@idrennanvmware I was able to reproduce the issue again with explicit lower default ttl and max-lease-ttl.
The watcher is doing the job just fine. The problem is that nomad does not persist state (at least template related state, havent looked anyother places) across client restart.
So when nomad client restarts, the lease issued by previous client session leaks (no one revokes it, it will expire when it reaches its ttl).
And the new client session will follow the normal NEW job start procedure, mainly:
- run prehooks
- run templates and initialize watcher
- start containers (already running)
So it will template in a new VALID token into the container, but since the container was already running, the process running inside the container still held the old token. Then what happens is nomad template watcher will keep the new lease up-to-date, but when the old lease expires the process running inside the container start to fail.
I don’t think this should be an intended behavior… But solving this issue correctly seems require adding state persistent into nomad.
Want to hear from what nomad team says @tgross