My situation is that we run (12-factor setup) apps with Nomad, and with its Vault integration it provides dynamic database credentials for the app, which are loaded in the env. These credentials at some point expire (as they should, the earlier the better, right). However, Nomad won’t restart/refetch new credentials in time before they expire. So my app will attempt to use expired credentials for a while!
How can I make Nomad take care of this better? A larger splay doesn’t seem to help. It actually makes it worse, as Nomad will leave some apps running for longer with expired credentials.
Nomad won’t restart/refetch new credentials in time before they expire
Is it that the env variables aren’t be updated with the new database credentials at all , or the env variables are being updated just not before the token expiration?
It’d also be helpful to know the TTL of this problem token
That’s a good question. The template’s change_mode is set to restart, as the app won’t notice changes in env vars without it. We just see errors that the credentials are no longer valid, and soon after the application will restart.
TTL is now set to two weeks, as not to run into this (as we deploy more often than that).
