Multiple Tasks in a Single Job with order

Nomad
1

Hi There, I have a nomad question.

Is it possible to have multiple tasks in a single Job where some of the groups are dependent on others and need to be completed first. It’s a chain of at least 3 links. I wasn’t sure if init has an order sequence or if it’s just binary.

2

Hi Ravi.

I believe you’re looking at the lifecycle stanza that allows this. Have a look at poststart.

Let me know how you get on.

3

Thanks Pandom, That info looked spot on.

I had a follow-up questions. Does the Lifecycle stanza work on groups too?
for example; if I have multiple groups is there a way that I wait for all groups to finish before running the final group?

like poststop for groups instead of tasks.

4

Hi @ravi :wave:

No, lifecycle can only be defined for tasks. And even for tasks they revolved around the idea of main tasks and support tasks, meaning that they are made for building deep dependency graphs for example.

If you need finer-grain coordination between tasks and groups you will need to control it externally.

Depending on how complex is your use case, I think you could use template blocks and Consul KV store to block and unblock tasks. You would have a prestart task that checks for a Consul key. This will block the main task until the key is set. And then a poststop tasks that updates the key in Consul to unblock the next group.

You could probably use templating as well to build a job file that suits your need. Here’s an experiment as a example: Hacking Nomad Job Dependencies

5

Thanks. I’ll take a look; My use cases is interesting I suppose.

I have 5 tasks that all require 1 task to run first. Once the 1 task is done; the 4 tasks can run in parallel. I would prefer to use different computers (clients) since 1 of the 4 tasks requires a GPU the other do not…

6

Hum…interesting. Then yeah, you would need them to be in separate groups. I think using Consul templates as a gate for your other 4 tasks could work.

The first task can write to Consul on a poststop task, and the other four would have a prestart that just blocks until the key is set.

7

My 0.02 …

  • using the trick of waiting for the service to show up in Consul also works (dig ...) as explained in one of the learn guides.

  • Q: @lgfa29 Consul Template can block for missing keys only, correct? can there be a CT expression to say “block until value fetched == X” ? ( the reason I ask is, unless using distinct keys for each run, subsequent runs will not “wait” as expected, correct?)

8

That’s true, you would need to do some clean-up. Or maybe some hacky bash?

while [ "{{ env "my_key"}} != "exptected_value"]; do
  sleep 10
done

If this is a dispatch job, you could use the NOMAD_JOB_ID environment variable as a unique key.

job "example" {
  datacenters = ["dc1"]
  type        = "batch"

  parameterized {}

  group "init" {
    task "work" {
      driver = "raw_exec"

      config {
        command = "sleep"
        args    = ["20"]
      }
    }

    task "unblock" {
      driver = "raw_exec"

      config {
        command = "local/script.sh"
      }

      lifecycle {
        hook = "poststop"
      }

      template {
        data        = <<EOF
#!/usr/bin/env bash
consul kv put ${NOMAD_JOB_ID} done
EOF
        destination = "local/script.sh"
      }
    }
  }

  group "main-1" {
    task "gate" {
      driver = "raw_exec"

      config {
        command = "local/script.sh"
      }

      lifecycle {
        hook = "prestart"
      }

      template {
        data        = <<EOF
#!/usr/bin/env bash
echo {{ key (env "NOMAD_JOB_ID") }}
EOF
        destination = "local/script.sh"
      }
    }

    task "main" {
      driver = "raw_exec"

      config {
        command = "sleep"
        args    = ["10"]
      }
    }
  }

  group "main-2" {
    task "gate" {
      driver = "raw_exec"

      config {
        command = "local/script.sh"
      }

      lifecycle {
        hook = "prestart"
      }

      template {
        data        = <<EOF
#!/usr/bin/env bash
echo {{ key (env "NOMAD_JOB_ID") }}
EOF
        destination = "local/script.sh"
      }
    }

    task "main" {
      driver = "raw_exec"

      config {
        command = "sleep"
        args    = ["10"]
      }
    }
  }
}

Kind of a hack as well, but it works :sweat_smile:

1 Like
9

This is great, this should be put in the learn guides. (I would use the command as a template output which has the date command in it, to show which group/task ran before and after), but this is good stuff!

1 Like
10

That’s true. My default throw away task is a raw_exec running sleep so I didn’t pay much attention on trying to make it actually do something useful :grimacing:

1 Like