Rolling deploy across hosts with exec

How could nomad run a raw_exec across a group of hosts one at a time? Aborting if any of them fail.

ssh node1 /bin/foobar
ssh node2 /bin/foobar
ssh node3 /bin/foobar

I’ve tried using type=batch and the distinct_host operator however this attempts to run on all hosts at the same time.

job "example2" {
  datacenters = ["dc1"]
  constraint {
    operator  = "distinct_hosts"
    value     = "true"
  }
  type = "batch"
  task "example" {
    driver = "raw_exec"
    config {
      command = "/bin/sleep"
      args    = ["10"]
    }
  }
}

I understand that nomad is an orchestrator, and not really intended to be used for a procedural operations. However I want to get a non-dockerized legacy application adopted into nomad without rearchitecting it.

1 Like

As far as I know, the batch job may choose the same host again and again.

Also, once the batch job has exited with 0, it won’t re-run at all, unless you artificially change some data in the job (say an env variable, etc.)

Also, as I believe, the distinct_hosts etc. would be applicable for the “current” run and not consider previous jobs.
The use of distinct_hosts would be useful for a job of type service with count > 1.

An idea what comes to mind is that you could templatize the job with the constraint iterating over the “node name”. This would sort of achieve what you want, I think.
note: remember about the changing environment variable, else the batch job will not rerun.

If it is possible, could you elaborate a little more on the real use case?

Another open thought: I wonder if your use case can be also solved with a “parameterized” job, each time with a different constraint (or something) which makes it run on the required host.

To elaborate on the use case.

Salt does a rolling deploy across vms. If the rolling deploy fails, salt will abort. Servers are deployed to one at a time.

I could replace salt with nomad on this once service, if I could do a rolling exec / raw_exec

I am confused what Salt (I assume Saltstack) has to do with this? Are you using the salt-ssh module to iterate over the nodes.

If this is the use case, then the answer is pretty straightforward … just generate a small job file with a constraint which will target each of the machine, one by one.

You can use levant to use templates so that you can iterate over the name list.

That said, I would suggest the above method if-and-only-if a straightforward cleaner method is not available.

If you could explain all the steps like an algorithm, maybe a simpler, “Nomadic” way could be thought of! :smiley:

I am attempting to rip out salt, and replace it with nomad. I have provided an example of what I have attempted (see above). The problem is that it does not successfully target machines one by one, instead it runs on all machines simultaneously.

What would a job look like that runs a raw_exec on all servers one by one?

@spuder the example you provided in the first post is a batch job, but it sounds more like you’re looking to do a rolling deploy of a service type of job here? In that case, you might want to look into the update stanza. You can use max_parallel to define a maximum number of tasks to restart at a time.