I’m new to vault/consul admin work. We have a vault cluster with PKI engine and kv-2 engine enabled on different paths. Recently we find that vault cluster backup takes longer and longer time. Your help will be much more than appreciated.
One area that we are looking at is secret engine paths, more specifically, the accumulated generated private/public key pairs, whether they are stored inside the vault cluster.
According to document PKI - Secrets Engines | Vault by HashiCorp, the generated private/public key pairs won’t be stored on vault cluster, do I read the meaning correctly? Thanks,
if PKI secrets doesn’t cost much storage, any way we can find the storage consumption of kv/kv-2 engine paths, Thanks,
You can use the consul snapshot inspect <backup-snap> . That’ll break down what is using up the space in the backup …which tells you what’s using it in Vault.
Most likely and most common issue is leases and tokens that have been created and are continuously created by teams without realizing it and left around until they TTL out.
PKI - the certificate is stored, but the private keys are not. You need to track your private keys. PKIs storage footprint is tiny … In my previous company I had well over 5,000 certs and it barely hit a few MB.
But the output only has the header part, ID/Size/Index/Term/Version. The body part is empty, no list of service/count/size rows for backend storages.
I also try to consul inspect command with options kv*, but it complains that the flags not defined.
# consul snapshot inspect -kvdetails -kvdepth 3 -kvfilter /var/tmp/a.sanpshot
flag provided but not defined: -kvdetails
The consul binary version in service is 1.8.0, vault is 1.4.3:
consul --version
Consul v1.8.0
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
vault --version
Vault v1.4.3
Do I need to upgrade consul & vault cluster with some newer version? Thanks,
Not needed – Vault 1.4 is VERY old, and you should upgrade that to 1.7 at a minimum, 1.9 if you want to be on the latest version. The further back you fall the more chance of not being able to upgrade and having massive headache when you do need to upgrade.
With Consul yes, absolutely … this is with a simple test vault 1.9.1 instance using Consul 1.10.5:
$ consul snapshot inspect foo.snap
ID 14-224163-1639560747634
Size 101924
Index 224163
Term 14
Version 1
Type Count Size
---- ---- ----
KVS 101 82.2KB
Register 15 11.7KB
ConnectCA 1 1.2KB
ConnectCAProviderState 1 1.2KB
ACLToken 2 753B
Index 22 670B
ACLPolicy 1 565B
CoordinateBatchUpdate 3 519B
Session 1 212B
Autopilot 1 199B
ConnectCAConfig 1 180B
FederationState 1 154B
SystemMetadata 1 72B
ChunkingState 1 12B
---- ---- ----
Total 99.5KB