I have set up Vault to generate ephemeral MongoDB credentials for an application running on Kubernetes. My current solution works well, but it does not have zero downtime. Once a request to MongoDB fails because the credentials expired, the application makes a request to Vault to generate a new set of credentials. While this work, all requests to the application from the moment the credentials expire to the moment Vault generates the new ones will fail.
I was considering setting a TTL of let’s say 2 weeks and then using consul-template with a grace period of 1 week, so that a new key would be generated well before the existing one expires. Then the application would start using the new key. I believe this approach should have zero downtime.
The problem is that the grace period in consul-template (and vault) was removed quite a while ago. See:
I can think of any solution as simple as using the grace period to avoid downtime. Does consul-template (or maybe vault-agent itself) provide now any way to generate new credentials before the currently existing ones expire?