Vault expire_num_leases very high on a specific timeframe

Dear HashiCorp Team,

We are experiencing a strange situation with our Vault Cluster deployed in an Redhat Openshift Cluster, where we face a high number of leases (to be expired) at a specific timeframe (see attachment).
This particular load appears every day at the same time window. I don’t know if this is some kind of internal process within the Vault Cluster, but I didn’t found any clear explanation.

Vault Cluster properties :

  • License OSS
  • Version 1.10.0
  • Mode HA Cluster (3 nodes)
  • Openshift Cluster 4.9

Any kind of hint would be helpful.

Thanks

You should look to identify which kind of leases these are.

But before you start, make sure you’ve understood exactly what that metric means - it’s not leases due for expiration, it’s the total number of leases being tracked by the expiration manager - i.e. all leases.

Given you have such an impressive step-change, perhaps the Vault server log or audit log has useful clues?

If not, some other interesting metrics to look at could be:

  • vault_token_creation - i.e. rate of lease creation by authentications, which is broken down by several useful labels
  • vault_secret_lease_creation - i.e. rate of lease creation by access to leased secrets - also with useful labels

It wouldn’t be internal process, there is a process or team that’s doing something they shouldn’t be. I’d suggest turning on your audit device and tracking the auth that is generating the high number of leases.