Our cluster is 3 Nomad servers, and 3 clients. We were doing some tests to see that Nomad can create new allocations on another client if the client with existing allocations accidently crashes. This worked fine, and worked as expected. Now we have 2 clients with all the allocations.
When we restarted the “crashed” client, it started successfully, but it didn’t start any new allocations. Makes sense, don’t really wanna close down any existing working allocations.
So my question is - what happens if one of the nomad clients get low on resources? I have currently 3 clients, 2 working a lot, while 1 is idle just waiting. do they ever spread if the client gets low on resources, or do an allocation need to crash to migrate to a new client?
I guess I can manually stop every job and start it again, which would spread it more evenly. Want to find something that handles it automatically.
we are using 0.10.2
thanks in advance!