Nomad job run removes Meta[index]

Nomad
1

Hello,
We recently upgraded from Nomad version 1.4.4 to 1.8.3, and we also updated consul to 1.19.2. We have a small script, that will perform a nomad job run on all jobs. In new versions, If we stop the job through the UI, start it back, it gets Meta[index]. When we perform another nomad job run using our template, it finds the Meta[index], and perform redeploy to remove it. How can we resolve this, that nomad job run does not redeploy the job depending on Meta[index]?

Thank you!

2

Hi There are “destructive” job specification updates and “in-place” job updates. You can see if the job update is in-place with nomad job plan.

Updating Meta is destructive, because meta values are exposed as NOMAD_META_… environment variables. Refreshing environment variables requires job restart.

There are not many fields that result in in-place job updates and are also arbitrary. One of them is for example tags or meta inside service block and you can use those to store your arbitrary data.

But! You can “hide” job meta values by overwriting them in group or task, that way the NOMAD_META_index environment variable will not change. I tested and this works for me for version 1.8.4.

job "test-upgrade2" {
  meta {
    index = uuidv4()
  }
  group "upgrade2" {
    task "upgrade2" {
      meta {
        index = "constant"
      }
      driver = "raw_exec"
      config {
        command = "bash"
        args = ["-xc", <<EOF
          echo "start $(date)"
          sleep infinity
          trap 'echo "stop  $(date)"' EXIT
          EOF
        ]
      }
    }
  }
}

The index = "constant" at task scope overwrites index = uuidv4(), so although index changes, the task is in-place updated:

$ NOMAD_NAMESPACE=test nomad job plan test-upgrade2.nomad.hcl
+/- Job: "test-upgrade2"
+/- Meta[index]: "f1e41c80-2e39-4531-9658-3b9a10fdfdaf" => "d2786506-f1d5-411b-baf6-f6606ce25a84"
    Task Group: "upgrade2" (1 in-place update)
      Task: "upgrade2"

Another idea with a “not-really-supported trick”, you can also hide everything inside a group with count=0 and store your arbitrary data there:

job "test-upgrade2" {
  group "upgrade2" {
    task "upgrade2" {
      driver = "raw_exec"
      config {
        command = "bash"
        args = ["-xc", <<EOF
          echo "start $(date)"
          sleep infinity
          trap 'echo "stop  $(date)"' EXIT
          EOF
        ]
      }
    }
  }

  group "disabled" {
    count = 0
    meta {
      index = uuidv4()
    }
    task "upgrade2" {
      driver = "invalid"
    }
  }
}

The group "disabled" will never run, because of count=0, and even if, the driver is non-existent.