[Nomad] Always stuck at +- 125K Allocation and wont go further

Hi, we do load test to our nomad environment,
and currently we faced that its always stuck at around 125K then won’t go further. although the nomad clients node still have many unallocated resource. we try change the nomad clients instance type but no luck. still stuck at 125K as always

this some screenshot about it :




I have a nomad debug profile, how can I send it to you guys as it sensitive ?

Hi @petrukngantuk1,

For additional details it’s possible to use the following address:

Logs and other artifacts may also be sent to: nomad-oss-debug@hashicorp.com

Please link to your Github issue in the email and reference it in the subject
line:

To: nomad-oss-debug@hashicorp.com

Subject: GH-1234: Errors garbage collecting allocs

Emails sent to that address are readable by all HashiCorp employees but are
not publicly visible.

Do you have any further information regarding the allocations that are stuck? What is their current status and such?

Thanks,
jrasell and the Nomad team

Nomad UI, didn’t show anything,

but found multiple logs in nomad server like this :

    2021-08-26T07:26:32.880Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: evaluation is not outstanding"
    2021-08-26T07:26:32.880Z [ERROR] worker: failed to nack evaluation: eval_id=7a91967b-cc1f-661c-d4cf-c5d7df26481e error="Evaluation ID not found"
    2021-08-26T07:26:32.934Z [ERROR] worker: failed to submit plan for evaluation: eval_id=2c00510b-2526-2abb-37a8-2afbb178268b error="evaluation is not outstanding"

will send the nomad prof shortly

Hi @jrasell , sended to you,
let me know if you have solutions or comment,

Thanks,
Kholis RA Gumelar

another similar logs :

    2021-08-27T08:15:53.625Z [ERROR] worker: failed to submit plan for evaluation: eval_id=a23f405c-bbc8-1774-9698-eb569869b587 error="rpc error: evaluation is not outstanding"
    2021-08-27T08:15:53.626Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
    2021-08-27T08:15:53.711Z [ERROR] worker: failed to nack evaluation: eval_id=a23f405c-bbc8-1774-9698-eb569869b587 error="rpc error: Evaluation ID not found"
    2021-08-27T08:16:46.100Z [ERROR] worker: failed to submit plan for evaluation: eval_id=f0d1e007-3a39-19c7-1b63-a33d87102f7f error="rpc error: evaluation is not outstanding"
    2021-08-27T08:16:46.104Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
    2021-08-27T08:16:46.140Z [ERROR] worker: failed to nack evaluation: eval_id=f0d1e007-3a39-19c7-1b63-a33d87102f7f error="rpc error: Evaluation ID not found"
    2021-08-27T08:17:06.193Z [ERROR] worker: failed to submit plan for evaluation: eval_id=f815a9f3-e3d5-1c40-cd23-09774e011275 error="rpc error: evaluation is not outstanding"
    2021-08-27T08:17:06.294Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
    2021-08-27T08:17:06.297Z [ERROR] worker: failed to nack evaluation: eval_id=f815a9f3-e3d5-1c40-cd23-09774e011275 error="rpc error: Evaluation ID not found"
    2021-08-27T08:17:10.854Z [ERROR] worker: failed to submit plan for evaluation: eval_id=9b258528-f4ad-5fc0-39ca-3b2861ebf4a7 error="rpc error: evaluation is not outstanding"
    2021-08-27T08:17:10.855Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
    2021-08-27T08:17:10.879Z [ERROR] worker: failed to nack evaluation: eval_id=9b258528-f4ad-5fc0-39ca-3b2861ebf4a7 error="rpc error: Evaluation ID not found"
    2021-08-27T08:18:24.964Z [ERROR] worker: failed to submit plan for evaluation: eval_id=0a7ac52e-4127-a122-93d0-b478ca9329f7 error="rpc error: evaluation is not outstanding"
    2021-08-27T08:18:24.966Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
    2021-08-27T08:18:25.688Z [ERROR] worker: failed to nack evaluation: eval_id=0a7ac52e-4127-a122-93d0-b478ca9329f7 error="rpc error: Evaluation ID not found"
    2021-08-27T08:19:06.220Z [ERROR] worker: failed to submit plan for evaluation: eval_id=b4b41143-5a3e-5a37-431c-cf205b690f40 error="rpc error: evaluation is not outstanding"
    2021-08-27T08:19:06.226Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
    2021-08-27T08:19:06.284Z [ERROR] worker: failed to nack evaluation: eval_id=b4b41143-5a3e-5a37-431c-cf205b690f40 error="rpc error: Evaluation ID not found"
    2021-08-27T08:19:09.430Z [ERROR] worker: failed to submit plan for evaluation: eval_id=a4d76295-5f5f-80ae-b7e6-c8031fb1e749 error="rpc error: evaluation is not outstanding"
    2021-08-27T08:19:09.431Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
    2021-08-27T08:19:09.493Z [ERROR] worker: failed to nack evaluation: eval_id=a4d76295-5f5f-80ae-b7e6-c8031fb1e749 error="rpc error: Evaluation ID not found"
    2021-08-27T08:19:40.239Z [ERROR] worker: failed to submit plan for evaluation: eval_id=0a7ac52e-4127-a122-93d0-b478ca9329f7 error="rpc error: evaluation is not outstanding"
    2021-08-27T08:19:40.239Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
    2021-08-27T08:19:40.280Z [ERROR] worker: failed to nack evaluation: eval_id=0a7ac52e-4127-a122-93d0-b478ca9329f7 error="rpc error: Evaluation ID not found"
    2021-08-27T08:19:42.594Z [ERROR] worker: failed to submit plan for evaluation: eval_id=cb202235-df51-0a82-6c92-c364e4834c50 error="rpc error: evaluation is not outstanding"
    2021-08-27T08:19:42.595Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
    2021-08-27T08:19:42.626Z [ERROR] worker: failed to nack evaluation: eval_id=cb202235-df51-0a82-6c92-c364e4834c50 error="rpc error: Evaluation ID not found"
    2021-08-27T08:19:45.398Z [ERROR] worker: failed to submit plan for evaluation: eval_id=5bbfebff-63f2-1a27-c2d3-57458f48d19d error="rpc error: evaluation is not outstanding"
    2021-08-27T08:19:45.398Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
    2021-08-27T08:19:45.477Z [ERROR] worker: failed to nack evaluation: eval_id=5bbfebff-63f2-1a27-c2d3-57458f48d19d error="rpc error: Evaluation ID not found"

Hi, just want to update,
after I implement the nomad raft log ../server/raft to use instance storage,
still have same problems, stuck at ± 125K

Warm Regards,
Kholis RA Gumelar