Hi,
Vault newbie here!
A few months ago I deployed a very simple vault setup to host three secrets for (currently) 15 hosts as part of a automation project using salt.
Vault and Salt are working together as expected, however I recently noticed Vault is consuming an awful lot of disk space (just short of 2GB) for just a simple setup and it’s been slowly incrementing every 15 minutes since it was deployed (based on my internal monitoring).
The salt minion on all the endpoints is scheduled to run salt-call state.apply every 15 minutes to keep their state in sync, hence I think that it highly likely to be relevant.
The salt minion does require a password stored in vault in certain circumstances, but I suspect the salt minion is request it on every execution to populate it’s pillar data.
The salt master is configured to use Vault using an approle as per the documentation at Basic Configuration - Salt Extension for interacting with Vault. A redacted copy of the config is available at the bottom of this message.
I suspect Salt is generating a refresh token for each minion for the requests made every fifteen minutes. I presumed the old tokens would expire and be removed from the database, but now I’m not so sure.
I also wondered if they are being expired but the “database” needs defragging or similar. As I’m using the simple file storage backend it seems all the tokens are placed on the disk as flat files, not entered into a single database file so that seems unlikely.
I’ve been unable to find a ridiculous amount of tokens “hanging around” when I navigate Vault using the Web GUI.
Does anyone have any advice as to how I might find and (if there’s loads of them) expire old tokens issued for the salt-minions?
Failing that, does anyone have any advice on how to troubleshoot this further?
Lastly, is using the file storage backend okay for this very simple setup or is there a better storage backup which would allow me to deal with this type of issue better?
Many thanks
Steve
Disk space incrementing every 15 minutes.
Output from vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.17.5
Build Date 2024-08-30T15:54:57Z
Storage Type file
Cluster Name vault-cluster-XXXXXXXX
Cluster ID XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
HA Enabled false
Vault’s config file
cat /etc/vault.d/vault.hcl
ui = true
storage "file" {
path = "/opt/vault/data"
}
# HTTPS listener
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/etc/nginx/ssl/XXXX.cer"
tls_key_file = "/etc/nginx/ssl/XXXX.key"
}
Disk usage highlighting the disk space is being consumed in /opt/vault/data/sys/token
sudo du -d 1 -h /opt/vault/data/
196K /opt/vault/data/auth
20K /opt/vault/data/audit
240K /opt/vault/data/logical
100K /opt/vault/data/core
1.8G /opt/vault/data/sys
1.8G /opt/vault/data/
sudo du -d 1 -h /opt/vault/data/sys/
124K /opt/vault/data/sys/counters
1.2G /opt/vault/data/sys/token
586M /opt/vault/data/sys/expire
40K /opt/vault/data/sys/policy
1.8G /opt/vault/data/sys/
sudo du -d 1 -h /opt/vault/data/sys/token
581M /opt/vault/data/sys/token/accessor
587M /opt/vault/data/sys/token/id
1.2G /opt/vault/data/sys/token
cat /etc/salt/master.d/vault.conf
vault:
auth:
method: approle
approle_mount: salt-master-approle
role_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
secret_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
issue:
type: approle
approle:
mount: salt-minions-approle
server:
url: https://FQDN:8200
verify: /etc/ssl/certs/ca-certificates.crt
metadata:
entity:
minion-id: '{minion}'
secret:
saltstack-jid: '{jid}'
saltstack-minion: '{minion}'
saltstack-user: '{user}'
policies:
assign:
- saltstack/minions
- saltstack/{minion}
cache_time: 60
refresh_pillar: null
