Auto migrate when nomad client get low resources

Nomad
1

Hi there!

Brief explanation:
Our cluster is 3 Nomad servers, and 3 clients. We were doing some tests to see that Nomad can create new allocations on another client if the client with existing allocations accidently crashes. This worked fine, and worked as expected. Now we have 2 clients with all the allocations.
When we restarted the “crashed” client, it started successfully, but it didn’t start any new allocations. Makes sense, don’t really wanna close down any existing working allocations.
So my question is - what happens if one of the nomad clients get low on resources? I have currently 3 clients, 2 working a lot, while 1 is idle just waiting. do they ever spread if the client gets low on resources, or do an allocation need to crash to migrate to a new client?

I guess I can manually stop every job and start it again, which would spread it more evenly. Want to find something that handles it automatically.

we are using 0.10.2

thanks in advance!

2

Hi @Cronnay :wave:

do they ever spread if the client gets low on resources

No, Nomad would not reschedule jobs, unless you are using preemption.

By default Nomad uses binpack as the scheduler algorithm, which will try to fit as many allocations in a node before using other nodes. You can switch to the spread algorithm using the /v1/operator/scheduler/configuration API if you would like the opposite behaviour.

EDIT: sorry, I missed the part where you mentioned that you are using Nomad 0.10.2, so spread would not be an option for you…

1 Like