Nomad - Persist Docker Container When Host Loses Connectivity With Server


I am new to Hashistack and I am doing a Proof of Concept for the company I work for.

The scenario is:

Hashistack → Consul + Nomad + Vault

  1. Nomad Server on AWS instance
  2. Few clients in different availability zones communicating with the server via WAN, not necessarily on AWS.

I need to make it work in a way that:

  1. I can run docker containers on the clients remotely through the nomad server. Therefore I can update tags/images/redeploy.
  2. When the client loses internet connectivity, the container must persist running on the client and when wan connectivity is available again, if that container is running, nothing should be changed.

I have the whole environment already set up and running, but the issue is: whenever the client node loses wan connectivity and comes back on again, nomad dispatch action to start a new container and kills the previously running container.

Also, is there a way to run containers with nomad without being a service? I mean, make the container run in the client node but if we stop the job from running on the server node, the container will not stop running on the client?

Thank you all!

Updating this topic…

I’ve already tested running docker with shell scripts using raw_exec driver, and it works fine. But if I could use the docker driver would be best because we would have the ability to manage the allocation/container from the nomad cluster.