Nomad rolling upgrade - allocation ordering

Nomad
1

Hello!

I have a question about nomad task updates. We have a rolling deploy for a task group that we update by changing the artifact. We’d like to control the order that the allocations get updated in - or at least specify that one allocation is “last”. Is this possible somehow? I haven’t seen anything obvious in the docs.

Use case: we’re running a service that uses RAFT consensus, and we would like to restart the current leader last to prevent multiple leadership changes during the update.

Thanks!

  • Kevin
2

Hi @kevin.arthur,

This is not currently possible and likely something that would not natively be made available in Nomad. If we were to add this sort of functionality, it could be some form of update selector within the update block, however, there are many implementation problems I can think of straight away. We would not be able to add specific logic to handle just Raft applications unless we created some form of plugin system for job updates which I don’t believe is something we would consider adding nor have ever discussed.

A potential workaround that would be possible is to run each Raft service within its own task group. You could then alter the job specification artifacts one at a time, leaving the current leader until the end. I concede that this approach is very manual and not ideal but can’t think of a better solution currently.

Thanks,
jrasell and the Nomad team

3

Thanks @jrasell! Yea, we also came up with the same workaround approach. Definitely not ideal, but it will work.

I have a suggestion for a potential solution. I recall seeing something about promoting canary deployments via the API. If there was an API to “promote” individual allocations, there could potentially be a mode that starts the deploy (paused), and then we call the API to trigger each allocation to update. That way, we’d be able to choose which allocation to update and trigger it explicitly. We have a Java API tool that we’re using to trigger the artifact updates already.

That’s a pretty rough sketch of the process, I can elaborate further if it would be useful.

Thanks!

Kevin

4

Hi @kevin.arthur,

That is an interesting idea and probably worth additional discussion. Would you be able to open a feature request against the Nomad repository so engineers and community members can input additional thoughts?

Thanks,
jrasell and the Nomad team