Specify or modify a job constraint when invoking dispatch job API

Nomad
1

Hi! I love me some Nomad. We’re using the dispatch job capability to provide a compute backend for our ML efforts. I’m curious if there’s anyway to dynamically (at the time dispatch job API is invoked) set a constraint, or a piece of metadata evaluated by an existing constraint on the job? The use case here is I have multiple auto scaling groups in AWS. The nomad clients have metadata declaring which ASG they’re in. I would like, at dispatch job time, to specify which ASG the job should run in without having to have a job definition per ASG. Is this possible?

Thanks!

2

I guess this is quite similar to my question from earlier this year: Dynamic resource allocations for parameterized jobs - the answer to which was no so maybe that also applies in this case. I was hoping that via the dispatch metadata overrides, I could override job metadata that could be compared against client metadata

3

Hi @kylejmcintyre :wave:

There isn’t a way to do this with parameterized job meta params because these two things happen at different time.

Constraints are resolved in the servers during scheduling. It’s one of the ways Nomad scheduler filters out clients that are eligible to run a job.

Meta params are resolved in the clients running the allocation, so by the time the meta param is read, the scheduling must already have happened.

I was going to suggest HCL2 variables, but those won’t work either because they are evaluated when the job is registered, not dispatched.

So I’m afraid you found a hole in our scheduler where it can’t see job meta data when doing scheduling decision :sweat_smile:

Could file a feature request for this?

Thanks!