Reducing group count ignores node eligibility

Nomad
1

I’m trying to get my head around reducing a group count (from say 3 to 2) in a 3 node cluster such that i can drain a single node prior to OS patching / restart.

On a Vagrant 3-node test cluster, i have a simple job task with group count=3 and the job is healthy and stable.

My assumption would be that I could:

  1. Mark one node as ineligible
  2. Reduce the job group count from 3 to 2
  3. The scheduler would identify that it ‘should’ kill the allocation on the node that is ineiligible.
  4. I could then drain the node and patch / restart

The issue is the scheduler is killing a random allocation, which means when i go to drain the node, a migration will occur for the allocation running on that node, back to another node (where it was effectively just killed as part of the reduced count).

Is there a more efficient way to reduce group count and identify which client node the allocation should be killed on (ie examine ineligible nodes with running allocations).

1 Like
2

just curious … what is the main intent?
if its to patch/update a node, then how about just drain, which makes the node ineligible as well, and then immediately make count from 3 to 2 (if reducing the count is also what you want)
after the node update, when you mark the node eligible, the tasks should restart on the node (assuming they haven’t already started elsewhere)

3

I tried drain initially, but this just rescheduled the allocation on another node (such that one node now has two allocations), which is an issue because the task requires a dedicated network port, and so fails

Would a unique_host constraint prevent this?