Hi,
we are using consul integration in Nomad as described in the documentation with workload identities. As stated in the documentation Nomad is not able to refresh this token on its own.
Because the Consultoken has only a limited lifetime (which is one of the main reasons for using this whole system), this leads to two problems in our production deployment
-
before token expiry we have to refresh the token, and have to restart the nomad process (reloading is not enough)
-
during (any) restart, nomad loose track of all running connect-proxies and their tokens.
This means after a few days the envoy-proxies stop working, because their consul_tokens are expired and nomad will no longer handle the proxy tokens
Our working hack is currently to drain all nomad-hosts explicitly before doing any restart of the nomad process. Even if the consul token lasts for 30days, because of our large number of nomad-hosts, we have nearly every day applications which has some rescheduling downtimes only because of this token-issue.
So I’m interested what should be the recommended way for rotating/updating the consul/nomad connection ? Or is the “best” way really to use 10000days tokens in consul ? oO