Force reschedule, pull latest mutable docker image

Hi

I use force_pull = true (Drivers: Docker | Nomad by HashiCorp) for docker images with mutable tags, but I didn’t find a way to tell nomad to evaluate if there is a newer docker image and reschedule without modifiing the job (or quick and evil stop/start the job). Any hints?

Just for my own clarification … do you want Nomad to periodically poll the Docker repo to see if the tag has changed (or not changed) and if it has indeed changed, then pull the newer image?

This would be useful as well but for now just a way to trigger it manually would be helpful, e.g. in a CI pipeline to deploy <image>:latest to dev and <image>:prod to prod.

Modifying the job is probably the most common way to do this historically. HCL2 makes this a bit easier, because you can use variables or the uuid functions. I have an example of that in this example job. (Fair warning, if you wander around in that repo, you will encounter jobs of very different quality and utility. :rofl: )

Something I have considered in the past, but haven’t written up yet, is using a value stored in Consul KV for a CI/CD driven job. This would allow you to provide a minimally privileged token to your CI environment that could only write to specific KV paths. Your Nomad job would then have a template stanza that consumed that key.

The value itself doesn’t have to be meaningful, but—in your CI case—rather than watching a docker tag you could have the CI send the exact version id of the container it wrote to the repository.

Changing the value that a template depends on will trigger the change_mode behavior. This can be used to have Nomad restart the job. The downside here is that you will not get a new “deployment” of the job, the tasks are restarted using the splay value to try to spread load.

For the mutable tag case, as you mentioned When paired with force_pull you could get the behavior you want. You wouldn’t need to use force-pull if you were using an environment variable and interpolation to get the value from Consul to write into the image value, and you would be able to leverage local caching if the allocation needed to restart for some reason.

Hopefully some of these ideas might come in handy, but they are at least some food for thought. :smiley:

Best,
-cv


Charlie Voiselle
Product Education Engineer, Nomad.

1 Like

:grinning_face_with_smiling_eyes: :grinning_face_with_smiling_eyes:

Thanks for the explanations. I think it should be easier.

I wish that this would be possible with nomad job eval job1, perhaps in connection with -force-reschedule.

My workaround for now is to

job "..." {
  meta {
    uuid = uuidv4()
  }
}

I’m using Levant so my workaround was this:

meta {
    deployed_at = "[[ timeNowUTC ]]"
}