High memory usage in consul 1.7.2

Hi I was hoping someone could guide me in the right direction, We recently built a 1.7.2 consul server servicing vault. We have noticed that this server is constantly crashing due to memory swap with most memory being used by the consul process.

From the analysis I’ve done I can see in the state file is storing alot of kvs

       Record Type    Count       Total Size

       KVS                   3643061   2.6GB
       Tombstone     2171         270.5KB
       Register           3               1.7KB
       Index               10             288B
       Autopilot        1               199B
       CoordinateBatchUpdate        1         162B
       ChunkingStateType        1          12B

       TOTAL:        2.6GB

The server is currently sized at 12GB

Is there a way to see what is stored inside these kvs and is it possible to remove them?

Hi garoadx,

Maybe try to set linux swappiness to a small number.
Step 1. vi /etc/sysctl.conf
Step 2. vm.swappiness=10
Step 3. sysctl -p

Hi Son76040408, tried the suggestion and it didnt help.

It appears the memory usage is related to whats store in the KVS of the state.bin. I suspect if I can clear them out the memory usage would decrease.

Does anyone know how I may do this?

@garoadx Please upgrade to v1.7.4. I am going to guess that you’re using consul-dns a lot, there was a memory leak discovered for consul-dns. Please update to v1.7.4 and the memory issue should be resolved.

Hi billykwooten,

Thank you for the suggestion. I’ll give it a shot.

Unfortunately upgrading to 1.7.4 has not helped.

On further investigation I can see there is a large amount of vault/sys/expire/id/auth/approle/login records in state.bin.

Around 4.6Million in fact.

Is there a way I can batch remove these entries?

Resolved by setting auth max-lease to 2mins then force removing leases

For future reference, you can see what datais in the KV store with this guide https://learn.hashicorp.com/vault/monitoring/inspecting-data-consul