Hi there.
In the pursuit of safer and more reliable deploys of our main API, we were looking for a way to prevent all allocations for restarting at the same time when in need of renewing a Vault dynamic secret, i.e. database credentials. We came across splay and vault_grace options in the documentation , who seemed like a good combo.
E.g.: the allocations would ask for new creds (vault_grace) 2 minutes before the TTL expiration and Nomad would restart them randomly in the course of the next 60 seconds (splay). That 1 minute + our shutdown_delay of 30 seconds would guarantee we would restart all allocations with fresh credentials with 30 seconds to spare before the old ones expired.
Well, the PR was raised and ready to be merged today when the engineer who was going to do it noticed the documentation has been update and that vault_grace is deprecated.
It’s my bad for not realizing that it was deprecated before because we trusted the nomad docs. But finally, here is my question: is there a way to achieve the scenario described above with other parameters? How would you prevent allocations from restarting all at once when renewing credentials via Vault?
Thanks in advance for any help.