We are experiencing a strange situation with our Vault Cluster deployed in an Redhat Openshift Cluster, where we face a high number of leases (to be expired) at a specific timeframe (see attachment).
This particular load appears every day at the same time window. I don’t know if this is some kind of internal process within the Vault Cluster, but I didn’t found any clear explanation.
You should look to identify which kind of leases these are.
But before you start, make sure you’ve understood exactly what that metric means - it’s not leases due for expiration, it’s the total number of leases being tracked by the expiration manager - i.e. all leases.
Given you have such an impressive step-change, perhaps the Vault server log or audit log has useful clues?
If not, some other interesting metrics to look at could be:
vault_token_creation - i.e. rate of lease creation by authentications, which is broken down by several useful labels
vault_secret_lease_creation - i.e. rate of lease creation by access to leased secrets - also with useful labels
It wouldn’t be internal process, there is a process or team that’s doing something they shouldn’t be. I’d suggest turning on your audit device and tracking the auth that is generating the high number of leases.
