I have three nodes cluster (ubuntu 18 VMs running 1.8.2) , raft database file is growing by 0.5GB daily over past 10 days… Not sure what started this…
My question : how to find out what’s causing this growth ? Is there a way to compress/defrag the database ? . The secrets we have are very small.
What do your metrics show for tokens, KVs, etc being used?
What does the audit log show is happening with the usage?
metrics are not enabled, audit logs show lots of activities with same pattern.
expiration: revoked lease: lease_id=auth/xxxx/login/xxxxxxxxxxxxxxx
How to get the makeup of raft database now ?
The tutorial starts by putting vault in recovery mode … Do I have to put vault in recovery mode while inspecting the storage ?
What I do is save a snapshot, and load the snapshot into a docker instance on my machine and restore the data. Now you can use recovery mode to inspect the structure.
Note: You have to -force the restore and use your server’s keys after the restore.
If you are relying on Vault for anything production-like, you really need to enable and monitor it… You could see if there’s errant/unexpected growth, which is what I suspect here.
Although I agree with @mikegreen as far as monitoring the metrics, I’ve not gained the same level of detail from them as he has.
Metrics are very good for performance monitoring and looking for odd patterns in long term data, but they don’t usually tell you why your database size is growing.
Between these two, you should be able to see what’s going on.
vault.secret.kv.count
vault.token.count
And they can be broken down by path/policy/ttl/etc…
and *edit - this should show secret leases vault.secret.lease.creation