Handling max_ttl in kubernetes pod

We are using vault-k8s to inject vault-agent in our pods and slowly moving to dynamic accesses on all our applications, after some fiddling I now have a system I am happy with but there is still one thing I am not sure how to handle correctly:

leased secrets all have a max_ttl of 32days by default, I know I can change it but not remove it and I understand the reasoning but how do you handle this in k8s ? Do you have something that does a rollout restart of the deployment when the pod reach a specified uptime ?
pods rarely reach 32 days uptime but it happens and handling the credentials change in app is not always easy as some library don’t let you do it (consul in that case).

I’m not sure I understand your question. If you’re asking how to remove the TTL, you can’t. Everything in vault has a TTL (excluding the root token).

Otherwise, the TTL in k8s applications are handled by the sidecar container. You can set the TTL of the auth much lower than the 32 days, as it’ll automatically renew it’s auth connection.

I will try to reformulate the question (which was not if max_ttl can be removed):
In the order of operation you will have:

  • vault-agent log into vault & fetch your secrets (in my cas dynamic access to consul)
  • my application starts and load the consul token from disk
  • vault-agent renew the token until it reaches max_ttl, after that it gets a new consul token

At this point my application is holding a consul client with the wrong token and no way of swapping it, every request made past this point will fail due to authentication.
The simple option would be to commit suicide and kill the pod but if I do that I might impact the service, the other option would be to do it like an erlang app would and internally restart the app, recreate the consul client and every dependency which might also impact the service.

So my question is: how do you handle this ? How are you handling this case in your applications ? Do you only use libraries which are able to reload the new token on the fly ? Does nobody do this ? Do you regularly restart your deployments ?
I am using go but I am not even talking specifically about a language, I am really wondering what is the best strategy.

What your describing is badly written code not a design issue.

There is no way of getting a consul token from Vault or Vault agent. If you are doing some trickery to get a consul token out of the agent, then it’s completely up to you to refresh it or get a new one by talking to Consul.

One more note, there is no such thing as a dynamic secrets engine for Consul (AFAIK) and even if there was then you would need to reauth to Vault to get a new Consul connection.

I am really not sure what you are talking about but I am talking about Consul - Secrets Engines | Vault by HashiCorp
and btw we also use this one that probably does not exists either: Google Cloud - Secrets Engines | Vault by HashiCorp

Thanks for letting me know that the documentation is false, my system does not work and on top of that I write bad code (even if I only talked about the official consul go client library and vault-agent which is quite funny), unless anyone else has real input on this I think I will just let this topic dies.

1 Like