# Start job allocations atomically

**URL:** <https://discuss.hashicorp.com/t/start-job-allocations-atomically/53689>\
**Category:** Nomad\
**Created:** [May 11, 2023, 12:42pm UTC](https://discuss.hashicorp.com/t/start-job-allocations-atomically/53689 "2023-05-11T12:42:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![HadarGreinsmark](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/hadargreinsmark/32/35643_2.png) [@HadarGreinsmark](https://discuss.hashicorp.com/u/HadarGreinsmark)\
**Post date:** [May 11, 2023, 12:42pm UTC](https://discuss.hashicorp.com/t/start-job-allocations-atomically/53689/1 "2023-05-11T12:42:15Z")

</div>

I’m trying to start a Nomad batch job that runs two containers on two different hosts. My job consists of two task groups with the constraint `distinct_hosts = true` to force the containers to run on the different hosts.

Problem: I would like these containers to start simultaneously. If one of the host’s capacity is full, the entire job should stay in `pending` state until all resources are available to schedule both containers. When I try this, it seems like one of the task starts running and the other stays in the “queued” state if one of the host is full.

I also tried to set the job parameter `all_at_once = true`. This didn’t help and the documentation also states that it cannot be used for “atomic placement”. The outline of my job:

```auto
job "parallel-work" {
  type = "batch"
  all_at_once = true
  constraint {
      operator = "distinct_hosts"
      value = "true"
  }
  # Both work1 and work2 should start about the same time
  group "work1" {
    task "main" {
      # To be executed on host A
    }
  }
  group "work2" {
    task "main" {
      # To be executed on host B
    }
  }
}

```

Is there another way to achieve this or any plans on implementing such functionality?

---

<div class="post-metadata">

**Author:** ![shantanugadgil](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/shantanugadgil/32/923_2.png) [@shantanugadgil](https://discuss.hashicorp.com/u/shantanugadgil)\
**Post date:** [May 11, 2023, 7:30pm UTC](https://discuss.hashicorp.com/t/start-job-allocations-atomically/53689/2 "2023-05-11T19:30:50Z")

</div>

Interesting use case … could there be `poststart` tasks which could kill the main task if the other task is not healthy !?

Maybe the health of the other task can be determined using something like `dig` or `nslookup` ?

just-a-thought

---

<div class="post-metadata">

**Author:** ![HadarGreinsmark](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/hadargreinsmark/32/35643_2.png) [@HadarGreinsmark](https://discuss.hashicorp.com/u/HadarGreinsmark)\
**Post date:** [May 12, 2023, 5:58am UTC](https://discuss.hashicorp.com/t/start-job-allocations-atomically/53689/3 "2023-05-12T05:58:50Z")

</div>

Sound like an interesting approach. In my use case, there might be many `parallel-work` batch jobs in the queue waiting to be started. I guess one challenge could be a scenario where `parallel-work1` has a task running on host A and a `parallel-work2` has another task running on host B. Thus, no one can make progress if both needs two hosts. I guess, such cases could be dealt with by sleeping a random amount before checking the health of all expected tasks, so that one of the job would be killed and the other one could make progress. Though the question is how performant such an approach is if a batch job needs many nodes simultaneously and many other jobs are waiting the queue.

Best would be if it could be detected at the scheduling stage, but Nomad is maybe not primarily designed for such batch-heavy workloads.

---

<div class="post-metadata">

**Author:** ![HadarGreinsmark](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/hadargreinsmark/32/35643_2.png) [@HadarGreinsmark](https://discuss.hashicorp.com/u/HadarGreinsmark)\
**Post date:** [August 12, 2024, 1:10pm UTC](https://discuss.hashicorp.com/t/start-job-allocations-atomically/53689/4 "2024-08-12T13:10:44Z")

</div>

After investigating multiple schedulers, the algorithm I was looking for here was [Gang scheduling](https://en.wikipedia.org/wiki/Gang_scheduling).

Nomad doesn’t support this, neither does the default scheduler of Kubernetes. But Kubernetes has scheduler plugins in projects like [Volcano](https://volcano.sh/en/) and [Apache YuniKorn](https://yunikorn.apache.org/docs/get_started/core_features/#gang-scheduling) that make Gang scheduling possible. Also, HPC schedulers like [Slurm](https://en.wikipedia.org/wiki/Slurm_Workload_Manager) support it.