Hi everyone. I have three nomad clients+servers, running nomad 0.12.3 and 0.11.2.
Everything is running fine, however Nomad does not seem to balance the allocations across the clients. I am seeing that two clients have their RAM usage at about 70% while the other always sits at 23%. And the jobs are not all system jobs or bound to a client, they can be shifted freely.
I now added another job of the system type. Nomad said that one allocation can not be placed due to:
- Resources exhausted on 1 node
- Dimension memory exhausted on 1 node
The node in question is one of the 70% ram usage clients. However first of all I don’t understand what exactly is meant by “resources exhausted” and “dimension memory exhausted”, which resource specifically? RAM? Disk space? Because unique.storage.bytesfree says I have 50GB free and my job needs about 300MB, which should not cause problems.
The other thing is, why does Nomad not reschedule existing jobs so the job can be started on this client? There are plenty of possibilities.
This is the reschedule policy on all jobs:
“ReschedulePolicy”: {
“Attempts”: 0,
“Interval”: 0,
“Delay”: 30000000000,
“DelayFunction”: “exponential”,
“MaxDelay”: 3600000000000,
“Unlimited”: true
},