Hi,
I am using vault-k8s deployed in aws eks cluster.
Current configuration is as follows:
Setup - HA
Storage - Raft
Number of Pods - 3 (1 leader, 2 standby).
We started using vault transit engine a couple of weeks ago with the traffic on the endpoints /transit/encrypt being close to 15-20k rpm.
Currently we have around 3M transit keys.
As soon as we started using transit engine, vault pod memory keeps on increasing, currently it is about 12 GB.
We even tried disabling vault cache using “disable_cache” = true, but no effect is observed.
What could be the possible reasons behind this continuous surge in memory consumption by vault pods (even standby nodes are taking too much memory)?
We are using “container_memory_working_set_bytes” metric to monitor the memory consumption by pods.
Is it possible that vault is still caching any data?
Any help would be greatly appreciated.
Thanks