Nomad - server & client upgrade strategies

Nomad
1

Hello,

I would like to hear from community members about their Nomad upgrade procedures, and how they relate to immutable infrastructure.

At my company, we are moving away from chef-client configuring base AMI/vmware templates to packer-generated AMIs and templates whose cloud init config is updated by terraform. Nomad is installed during the packer build, so when we upgrade Nomad, we rebuild the AMIs/templates and then update terraform modules to use these. Finally, we replace all the VMs. For servers, we add 3 new servers, and then remove the old servers one at a time. For clients, we create replacements with the new AMIs/templates, and then drain all the old ones and terminate them.

This is time consuming. For nomad version upgrades, is this typical? I have seen other threads which reference replacing the nomad binary and restarting nomad to do these upgrades. This does not sound immutable, but perhaps this is a good compromise to make these upgrades easier?

Thanks,
Dave G

2

This is basically the same process we run through when updating our cluster. It is somewhat time consuming(though we don’t have a very large environment) but the OldestInstance termination setting in our ASGs make clean up easy and Nomad is critical enough piece of infrastructure that it’s worth baby sitting once or twice a quarter.

3

Thanks for your response Andrew. We have been using aws cli to terminate and scale in, but setting this on ASGs and then resetting desired is better.

In vmware, we are stuck with using govc to shutdown the guest and destroy the vm.