Hi I was hoping someone could guide me in the right direction, We recently built a 1.7.2 consul server servicing vault. We have noticed that this server is constantly crashing due to memory swap with most memory being used by the consul process.
From the analysis I’ve done I can see in the state file is storing alot of kvs
Record Type Count Total Size
KVS 3643061 2.6GB
Tombstone 2171 270.5KB
Register 3 1.7KB
Index 10 288B
Autopilot 1 199B
CoordinateBatchUpdate 1 162B
ChunkingStateType 1 12B
TOTAL: 2.6GB
The server is currently sized at 12GB
Is there a way to see what is stored inside these kvs and is it possible to remove them?
Hi garoadx,
Maybe try to set linux swappiness to a small number.
Step 1. vi /etc/sysctl.conf
Step 2. vm.swappiness=10
Step 3. sysctl -p
Hi Son76040408, tried the suggestion and it didnt help.
It appears the memory usage is related to whats store in the KVS of the state.bin. I suspect if I can clear them out the memory usage would decrease.
Does anyone know how I may do this?
@garoadx Please upgrade to v1.7.4. I am going to guess that you’re using consul-dns a lot, there was a memory leak discovered for consul-dns. Please update to v1.7.4 and the memory issue should be resolved.
Hi billykwooten,
Thank you for the suggestion. I’ll give it a shot.
Unfortunately upgrading to 1.7.4 has not helped.
On further investigation I can see there is a large amount of vault/sys/expire/id/auth/approle/login records in state.bin.
Around 4.6Million in fact.
Is there a way I can batch remove these entries?
Resolved by setting auth max-lease to 2mins then force removing leases
For future reference, you can see what datais in the KV store with this guide https://learn.hashicorp.com/vault/monitoring/inspecting-data-consul