Hi, we do load test to our nomad environment,
and currently we faced that its always stuck at around 125K then won’t go further. although the nomad clients node still have many unallocated resource. we try change the nomad clients instance type but no luck. still stuck at 125K as always
this some screenshot about it :
I have a nomad debug profile, how can I send it to you guys as it sensitive ?
Hi @petrukngantuk1,
For additional details it’s possible to use the following address:
Logs and other artifacts may also be sent to: nomad-oss-debug@hashicorp.com
Please link to your Github issue in the email and reference it in the subject
line:
To: nomad-oss-debug@hashicorp.com
Subject: GH-1234: Errors garbage collecting allocs
Emails sent to that address are readable by all HashiCorp employees but are
not publicly visible.
Do you have any further information regarding the allocations that are stuck? What is their current status and such?
Thanks,
jrasell and the Nomad team
Nomad UI, didn’t show anything,
but found multiple logs in nomad server like this :
2021-08-26T07:26:32.880Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: evaluation is not outstanding"
2021-08-26T07:26:32.880Z [ERROR] worker: failed to nack evaluation: eval_id=7a91967b-cc1f-661c-d4cf-c5d7df26481e error="Evaluation ID not found"
2021-08-26T07:26:32.934Z [ERROR] worker: failed to submit plan for evaluation: eval_id=2c00510b-2526-2abb-37a8-2afbb178268b error="evaluation is not outstanding"
will send the nomad prof shortly
Hi @jrasell , sended to you,
let me know if you have solutions or comment,
Thanks,
Kholis RA Gumelar
another similar logs :
2021-08-27T08:15:53.625Z [ERROR] worker: failed to submit plan for evaluation: eval_id=a23f405c-bbc8-1774-9698-eb569869b587 error="rpc error: evaluation is not outstanding"
2021-08-27T08:15:53.626Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
2021-08-27T08:15:53.711Z [ERROR] worker: failed to nack evaluation: eval_id=a23f405c-bbc8-1774-9698-eb569869b587 error="rpc error: Evaluation ID not found"
2021-08-27T08:16:46.100Z [ERROR] worker: failed to submit plan for evaluation: eval_id=f0d1e007-3a39-19c7-1b63-a33d87102f7f error="rpc error: evaluation is not outstanding"
2021-08-27T08:16:46.104Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
2021-08-27T08:16:46.140Z [ERROR] worker: failed to nack evaluation: eval_id=f0d1e007-3a39-19c7-1b63-a33d87102f7f error="rpc error: Evaluation ID not found"
2021-08-27T08:17:06.193Z [ERROR] worker: failed to submit plan for evaluation: eval_id=f815a9f3-e3d5-1c40-cd23-09774e011275 error="rpc error: evaluation is not outstanding"
2021-08-27T08:17:06.294Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
2021-08-27T08:17:06.297Z [ERROR] worker: failed to nack evaluation: eval_id=f815a9f3-e3d5-1c40-cd23-09774e011275 error="rpc error: Evaluation ID not found"
2021-08-27T08:17:10.854Z [ERROR] worker: failed to submit plan for evaluation: eval_id=9b258528-f4ad-5fc0-39ca-3b2861ebf4a7 error="rpc error: evaluation is not outstanding"
2021-08-27T08:17:10.855Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
2021-08-27T08:17:10.879Z [ERROR] worker: failed to nack evaluation: eval_id=9b258528-f4ad-5fc0-39ca-3b2861ebf4a7 error="rpc error: Evaluation ID not found"
2021-08-27T08:18:24.964Z [ERROR] worker: failed to submit plan for evaluation: eval_id=0a7ac52e-4127-a122-93d0-b478ca9329f7 error="rpc error: evaluation is not outstanding"
2021-08-27T08:18:24.966Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
2021-08-27T08:18:25.688Z [ERROR] worker: failed to nack evaluation: eval_id=0a7ac52e-4127-a122-93d0-b478ca9329f7 error="rpc error: Evaluation ID not found"
2021-08-27T08:19:06.220Z [ERROR] worker: failed to submit plan for evaluation: eval_id=b4b41143-5a3e-5a37-431c-cf205b690f40 error="rpc error: evaluation is not outstanding"
2021-08-27T08:19:06.226Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
2021-08-27T08:19:06.284Z [ERROR] worker: failed to nack evaluation: eval_id=b4b41143-5a3e-5a37-431c-cf205b690f40 error="rpc error: Evaluation ID not found"
2021-08-27T08:19:09.430Z [ERROR] worker: failed to submit plan for evaluation: eval_id=a4d76295-5f5f-80ae-b7e6-c8031fb1e749 error="rpc error: evaluation is not outstanding"
2021-08-27T08:19:09.431Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
2021-08-27T08:19:09.493Z [ERROR] worker: failed to nack evaluation: eval_id=a4d76295-5f5f-80ae-b7e6-c8031fb1e749 error="rpc error: Evaluation ID not found"
2021-08-27T08:19:40.239Z [ERROR] worker: failed to submit plan for evaluation: eval_id=0a7ac52e-4127-a122-93d0-b478ca9329f7 error="rpc error: evaluation is not outstanding"
2021-08-27T08:19:40.239Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
2021-08-27T08:19:40.280Z [ERROR] worker: failed to nack evaluation: eval_id=0a7ac52e-4127-a122-93d0-b478ca9329f7 error="rpc error: Evaluation ID not found"
2021-08-27T08:19:42.594Z [ERROR] worker: failed to submit plan for evaluation: eval_id=cb202235-df51-0a82-6c92-c364e4834c50 error="rpc error: evaluation is not outstanding"
2021-08-27T08:19:42.595Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
2021-08-27T08:19:42.626Z [ERROR] worker: failed to nack evaluation: eval_id=cb202235-df51-0a82-6c92-c364e4834c50 error="rpc error: Evaluation ID not found"
2021-08-27T08:19:45.398Z [ERROR] worker: failed to submit plan for evaluation: eval_id=5bbfebff-63f2-1a27-c2d3-57458f48d19d error="rpc error: evaluation is not outstanding"
2021-08-27T08:19:45.398Z [ERROR] worker: error invoking scheduler: error="failed to process evaluation: rpc error: evaluation is not outstanding"
2021-08-27T08:19:45.477Z [ERROR] worker: failed to nack evaluation: eval_id=5bbfebff-63f2-1a27-c2d3-57458f48d19d error="rpc error: Evaluation ID not found"
Hi, just want to update,
after I implement the nomad raft log ../server/raft
to use instance storage,
still have same problems, stuck at ± 125K
Warm Regards,
Kholis RA Gumelar