Nomad update groups separately within a job

Nomad
1

Hi,

According to the update stanza, the parameter max_parallel applies for tasks within a group.
The doc also states that The task groups themselves are updated in parallel
I was wondering if there is a way to update groups within a single job separately, using stagger.
Meaning, a job with 3 groups will wait X between updating groups 1-3.

Thanks

2

Hi @Dgotlieb,

The update stanza can be used to control how the deployment progresses in exactly the manner you describe. The rolling update Learn guide has more information on this along with examples that could be useful.

Thanks,
jrasell and the Nomad team

3

Thanks @jrasell for your answer!
So, what I did was adding the update block at the job level.

  update {
    max_parallel = 1
    stagger      = "5m"
  }

My job has 3 identical groups.

I changed an env variable in all 3 groups and ran the updated job.
The result was that all 3 groups got updated at once.

Just to clarify… I’m not using a single group with count of 3, I’m using 3 different groups with count of 1

I will gladly add the job HCL, but before that I just want to make sure I got the logic correctly.
Thanks

4

This is obviously a very old thread, but the lack of this feature seems to still exist today.

One use case for sequential group deployments is when running workloads that are clustered applications with quorum implementations. In my case I came across this now when trying to deploy RustFS, it needs 4 unique nodes and listens to port 9000 and 9001 with each container. This means that you can’t put all the allocations into one group because the same port can’t be in the network namespace four times, and you have to instead spread them out by using a dynamic group block to create 4 unique groups with unique network namespaces.

This all however fails completely when deploying any change to the environment variables or other config for the allocations, as they will indeed get replaced all at the same time instead of a controlled rolling update due to the lack of options for group rollouts.

5

I don’t necessarily think you need four groups for this. Allocations in one group can all use the same static port. The only conflict is two of them landing on the same node, and distinct_hosts handles that:

hcl

  group "rustfs" {
    count = 4

    constraint {
      operator = "distinct_hosts"
      value    = "true"
    }

    update {
      max_parallel     = 1
      health_check     = "checks"
      min_healthy_time = "30s"
      healthy_deadline = "5m"
    }

    network {
      port "api" { static = 9000 }
      port "console" { static = 9001 }
    }

    service {
      name = "rustfs"
      port = "api"

      # Use whatever endpoint only passes once the node is back in the cluster
      check {
        type     = "http"
        path     = "/health"
        interval = "10s"
        timeout  = "2s"
      }
    }

    task "rustfs" {
      driver = "docker"

      config {
        image   = "rustfs/rustfs:<version>"
        volumes = ["/opt/rustfs/data-${NOMAD_ALLOC_INDEX}:/data"]
      }
    }
  }

With one group, update { max_parallel = 1 } gets you the rolling update back. It’s the four separate groups that make everything get replaced at once. If each member needs its own identity or data dir, use ${NOMAD_ALLOC_INDEX} (0-3).

One caveat: by default the rollout moves on once a task has been running for min_healthy_time, not once that member has rejoined the cluster. Use health_check = “checks” with a check that only passes when the node is back in quorum.

I run postgres with patroni more or less the same way if you want to see something similar that is working for me right now:

the redis-sentinel job in that repo has some similar concepts but I think the patroni one is probably closer to what you are describing