How to free up disk space for deleted/destroyed secrets using integrated storage?

Using integrated storage for Vault HA setup, there is one single raft.db file in the raft storage backend e.g. /opt/vault/data/raft/raft.db. I noticed that this file keeps growing regardless of creation or deletion operations.

I can understand that changing any data or metadata will increase the DB size. But what if I delete a K/V v1 secret or destroy a K/V v2 secret version/metadata? Which means that secret is permanently deleted and I will never need to access it anymore. Is there a way to shrink or compact the vault.db to free up the space occupied by permanently removed data?

In addition, worth to mention that even if I overwrite an existing k/v v1 secret which does not have multiple versions, or an existing k/v v2 secret already exceeding it max_versions, the DB size keeps growing all the same.

With this, if we open a vault for company use, the vault.db will grow sky high. It can easily max out the disk capacity by one bad operation/automation, and we have no way to clean things up.

It’s very unlikely that your KV engine is going to eat up a major portion of your space.

This isn’t for the faint of heart but you can inspect the storage to see what’s using what. This was simpler in the past with Consul as your backend where you could inspect the snapshots offline but it’s a bit more complicated with IS.

Thanks for the information.

I did some extensive reading and this seems not an easy or straight answer. I believe it is due to GitHub - etcd-io/bbolt: An embedded key/value database for Go. that the space for free pages will not be reclaimed to free up the disk space, but can be reused by later transactions.

I made further testing on this. When the DB size reaches to my VM RAM size, it stopped increasing. System resource usage. This makes sense now.

top - 22:09:25 up 43 days, 21:23,  4 users,  load average: 0.50, 0.40, 0.31
Tasks: 191 total,   1 running, 129 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.9 us,  2.1 sy,  0.0 ni, 90.0 id,  1.7 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :  8124312 total,   142184 free,  1076168 used,  6905960 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  6741324 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 30826 vault     20   0  0.197t 6.088g 5.517g S  31.7 78.6  20:14.23 vault

Probably this is just a no brainer conclusion. But it makes me feel more comfortable to host a vault service until I tried this out.

Remember that IS (raft) is a memory based database, so the memory profile is going to significantly change from disk i/o to memory i/o and usage.