I use force_pull = true (Drivers: Docker | Nomad by HashiCorp) for docker images with mutable tags, but I didn’t find a way to tell nomad to evaluate if there is a newer docker image and reschedule without modifiing the job (or quick and evil stop/start the job). Any hints?
Modifying the job is probably the most common way to do this historically. HCL2 makes this a bit easier, because you can use variables or the uuid functions. I have an example of that in this example job. (Fair warning, if you wander around in that repo, you will encounter jobs of very different quality and utility. )
Something I have considered in the past, but haven’t written up yet, is using a value stored in Consul KV for a CI/CD driven job. This would allow you to provide a minimally privileged token to your CI environment that could only write to specific KV paths. Your Nomad job would then have a template stanza that consumed that key.
The value itself doesn’t have to be meaningful, but—in your CI case—rather than watching a docker tag you could have the CI send the exact version id of the container it wrote to the repository.
Changing the value that a template depends on will trigger the change_mode behavior. This can be used to have Nomad restart the job. The downside here is that you will not get a new “deployment” of the job, the tasks are restarted using the splay value to try to spread load.
For the mutable tag case, as you mentioned When paired with force_pull you could get the behavior you want. You wouldn’t need to use force-pull if you were using an environment variable and interpolation to get the value from Consul to write into the image value, and you would be able to leverage local caching if the allocation needed to restart for some reason.
Hopefully some of these ideas might come in handy, but they are at least some food for thought.
Product Education Engineer, Nomad.