We are using dynamic secrets for database credentials, which the vault agent should and seems to fully manage correctly.
However, the vault agent logs the following warning every once in a while:
2022-03-17T06:52:00.645Z [WARN] vault.read(database/creds/profileservice): TTL of "24h" exceeded the effective max_ttl of "3h49m49s"; TTL value is capped accordingly
2022-03-17T06:52:00.645Z [WARN] vault.read(database/creds/profileservice): renewer done (maybe the lease expired)
2022-03-17T06:52:10.802Z [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/mysql.conf"
As far as I understand, the vault agent should renew its lease on the database secret every once in a while (based on default TTL), but at some point the default TTL of 24h would exceed the max TTL of the issued credentials and then this warning occurs. The vault agent seems to handle this fine and request a new set of credentials. However, since this is a warning I was wondering if this is something I can fix, and how to fix it. I can’t seem to find anything about this warning in the docs.
The default and max TTL seem to be fine.
From the database secret role in the vault UI:
We also do not set the global setting, so that should default to 768h I believe?
However, if I navigate to the configuration of the database secret engine I do see the following TTL:
My understanding is that if the Default/Max lease TTL is set to 0 it’ll use the next higher level’s TTL. So if a role doesn’t have TTLs defined it’ll use the secret engine config configured TTLs. If the secret engine config doesn’t have TTLs specified it’ll use the global.
With that said, secret leases are also, generally, dependent on your auth token’s TTL - when your auth TTL expires your secret lease will be revoked. What are your auth role TTL settings?
The configuration of the Kubernetes auth method which we use has both Default Lease TTL and Max Lease TTL set to 0.
Under a specific role within that auth method, the Generated Token’s Explicit Maximum TTL, Maximum Uses of Generated Tokens, Generated Token’s Period and Generated Token’s Initial TTL are all set to 0. Only the Generated Token’s Maximum TTL is set to 1800000.
Interesting. I’m not sure then, unless there is something in the agent config or its own code. Perhaps someone most familiar with the agent could offer some insight.
We are seeing the same thing, in our case, it is the “GCP” Secrets engine. We have seen problems where the application team says the token is expired, so we created an “excessively” low TTL/Max to test…
We are seeing that the Vault agent just seems to blindly refresh the lease, it doesn’t seem to pay much heed to “max_ttl”. This happens every hour (max_ttl), where it is like “OMG, the lease is gone!? Get a new one!?” This is not the expected behavior. It should be renewing the lease at like 80-90% of its life, and when it reaches max_ttl, it should get a new lease, right?
