How to set the Vault Heap Objects threshold?

Vault
1

Hi everyone

We’ve been running Vault servers in Kubernetes for a number of years, for a specific purpose of generating certificates for a specific application.

When we originally deployed the clusters, we went through the best practice guide to work out what metrics we should monitor. There was one metric that we added, but we’ve never really worked out what we should set it at, and that’s Vault Heap Objects ( vault.runtime.heap_objects). The documentation states “Number of objects on the heap. This is a good general memory pressure indicator worth establishing a baseline and thresholds for alerting.” but doesn’t give us much more, so I guess my question is, how do we accurately set an alert for this metric? What is the correct method of setting this value?

Thanks in advance.

John

2

I hate bumping posts, but I’d appreciate any response, even if it’s RTFM :slight_smile:

3

Hi @jeg1972 ,

This metric is very much a “it depends” value (I wish I had a more specific answer), as the number of available heap objects per cluster/application/etc varies based on your underlying infra, specific Vault config, etc.

In the past, with similar heap metrics (not this one specifically) what I tend to do is capture the metric over low and high usage periods and start there. If I observe a heap size of N during known peak usage times, you can set that + some buffer.

If you have the opportunity to load test, you could throw vault-benchmark at it

and see if you can get your vault cluster to perform poorly and record the metrics during that period.