I would like to hear from community members about their Nomad upgrade procedures, and how they relate to immutable infrastructure.
At my company, we are moving away from chef-client configuring base AMI/vmware templates to packer-generated AMIs and templates whose cloud init config is updated by terraform. Nomad is installed during the packer build, so when we upgrade Nomad, we rebuild the AMIs/templates and then update terraform modules to use these. Finally, we replace all the VMs. For servers, we add 3 new servers, and then remove the old servers one at a time. For clients, we create replacements with the new AMIs/templates, and then drain all the old ones and terminate them.
This is time consuming. For nomad version upgrades, is this typical? I have seen other threads which reference replacing the nomad binary and restarting nomad to do these upgrades. This does not sound immutable, but perhaps this is a good compromise to make these upgrades easier?