Nomad Version: v1.7.3
Consul Version: v1.17.3
3 server cluster setup with 160+ clients .
we are facing an intermittent issue with Nomad ,Consul .
Nomad is unable to register/de-register services in consul because of “ACL not found” error.
Initially permission issue occurred for reading secrets from vault which caused the allocation to restart after 12 attempts and after which we observed “ACL not found” Error.
Services were not getting registered to consul due to ACL error.
we are using the JWT mechanism outlined in Consul ACL with Nomad Workload Identities | Nomad | HashiCorp Developer to authenticate my Nomad workloads against Consul.
went through ACL trouble shooting page. All Tokens used are valid.
Logs:
a-trading-kv/test/legacy_secrets\nCode: 403. Errors:\n\n* permission denied (retry attempt 12 after \"1m0s\")","@module":"agent","@timestamp":"2024-06-18T13:44:13.776081+01:00"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-18T13:45:09.425982+01:00","alloc_id":"ecd95bd7-caf0-76f2-f129-f21853470dbb","failed":false,"msg":"Task not running by healthy_deadline of 5m0s","task":"app-mecs-service-task","type":"Alloc Unhealthy"}
{"@level":"warn","@message":"unable to fingerprint consul","@module":"client.fingerprint_mgr.consul","@timestamp":"2024-06-18T13:45:10.825105+01:00","attribute":"consul.partition","cluster":"default"}
{"@level":"error","@message":"(view) vault.read(app-kv/test/legacy_secrets): vault.read(app-kv/test/legacy_secrets): Error making API request.\n\nURL: GET https://vault.dt.sec.inaut.io/v1/d
ma-trading-kv/test/legacy_secrets\nCode: 403. Errors:\n\n* permission denied (exceeded maximum retries)","@module":"agent","@timestamp":"2024-06-18T13:45:13.836598+01:00"}
{"@level":"error","@message":"(runner) watcher reported error: vault.read(app-kv/test/legacy_secrets): vault.read(app-kv/test/legacy_secrets): Error making API request.\n\nURL: GET https://vault.dt.sec.inaut.io/v1/app-kv/test/legacy_secrets\nCode: 403. Errors:\n\n* permission denied","@module":"agent","@timestamp":"2024-06-18T13:45:13.836671+01:00"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-18T13:45:13.836694+01:00","alloc_id":"ecd95bd7-caf0-76f2-f129-f21853470dbb","failed":true,"msg":"Template failed: vault.read(app-kv/test/legacy_secrets): vault.read(app-kv/test/legacy_secrets): Error making API request.\n\nURL: GET https://vault.dt.sec.inaut.io/v1/app-kv/test/legacy_secrets\nCode: 403. Errors:\n\n* permission denied","task":"app-mecs-service-task","type":"Killing"}
{"@level":"warn","@message":"timed out waiting for read-side of process output pipe to close","@module":"logmon","@timestamp":"2024-06-18T13:45:17.846759+01:00","alloc_id":"ecd95bd7-caf0-76f2-f129-f21853470dbb","task":"app-mecs-service-task","timestamp":"2024-06-18T13:45:17.846+0100"}
{"@level":"warn","@message":"timed out waiting for read-side of process output pipe to close","@module":"logmon","@timestamp":"2024-06-18T13:45:17.846859+01:00","alloc_id":"ecd95bd7-caf0-76f2-f129-f21853470dbb","task":"app-mecs-service-task","timestamp":"2024-06-18T13:45:17.846+0100"}
{"@level":"info","@message":"plugin process exited","@module":"client.alloc_runner.task_runner.task_hook.logmon","@timestamp":"2024-06-18T13:45:17.850260+01:00","alloc_id":"ecd95bd7-caf0-76f2-f129-f21853470dbb","id":"81661","plugin":"/usr/bin/nomad","task":"app-mecs-service-task"}
{"@level":"info","@message":"(runner) stopping","@module":"agent","@timestamp":"2024-06-18T13:45:17.850911+01:00"}
{"@level":"info","@message":"marking allocation for GC","@module":"client.gc","@timestamp":"2024-06-18T13:45:17.851427+01:00","alloc_id":"ecd95bd7-caf0-76f2-f129-f21853470dbb"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-18T13:45:43.884139+01:00","alloc_id":"33df797c-4a0e-3e6c-2151-5b139849e129","failed":false,"msg":"Task received by client","task":"app-mecs-service-task","type":"Received"}
{"@level":"info","@message":"Task event","@module":"client.alloc_runner.task_runner","@timestamp":"2024-06-18T13:45:43.921208+01:00","alloc_id":"33df797c-4a0e-3e6c-2151-5b139849e129","failed":false,"msg":"Building Task Directory","task":"app-mecs-service-task","type":"Task Setup"}
{"@level":"info","@message":"(runner) creating new runner (dry: false, once: false)","@module":"agent","@timestamp":"2024-06-18T13:45:43.980812+01:00"}
{"@level":"info","@message":"(runner) creating watcher","@module":"agent","@timestamp":"2024-06-18T13:45:43.980943+01:00"}
{"@level":"info","@message":"(runner) starting","@module":"agent","@timestamp":"2024-06-18T13:45:43.981065+01:00"}
{"@level":"warn","@message":"(view) health.service(tcp.app-host-agent|passing): Unexpected response code: 403 (rpc error making call: ACL not found) (retry attempt 1 after \"250ms\")","@module":"agent","@timestamp":"2024-06-18T13:45:43.982636+01:00"}