Agent.rpcclient.health ACL not found error

Consul
,
1

Hi folks,

I’m having a bit of an issue with Nomad + Consul in that everything appears to be working fine, but my Consul logs on the (single) client are filling up with the following errors:

[ERROR] agent.rpcclient.health: subscribe call failed: err="rpc error: code = Unknown desc = ACL not found" failure_count=16 key=psm-redis topic=ServiceHealth

and sometimes

Feb 14 03:31:35 vmi1633713 consul[16957]: 2024-02-14T03:31:35.308+0100 [ERROR] agent.http: Request error: method=GET url="/v1/health/service/postgres?index=67798&passing=1&stale=&wait=60000ms" from=127.0.0.1:46172 error="rpc error: code = Unknown desc = ACL not found"

The server logs are totally clean.

I ran through the steps from Troubleshoot Consul ACL issues – HashiCorp Help Center and checked client’s token /opt/consul/acl-tokens.json is valid - I set it to my own CONSUL_HTTP_TOKEN environment variable and I can do consul catalog services and get my services back, which shows the Consul server recognises it as a valid token.

What’s interesting is that if I restart consul via systemctl restart consul, the errors go away. Until I run a Nomad job. At which point, these errors appear with a topic= corresponding to each service I used discovery with in the Nomad job. I’m using the JWT mechanism outlined in Consul ACL with Nomad Workload Identities | Nomad | HashiCorp Developer to authenticate my Nomad workloads against Consul.

At this point, I’m pretty stuck - it doesn’t seem to be causing any issues, and all my services report healthy in Consul and can be discovered from other jobs. But I don’t really want to start relying on the cluster while it is spamming errors :confused:

Does anyone know how I might be able to get the token that is supposedly not found and work out where it is coming from? My only clue as to what is sending these requests is the from=127.0.0.1:46172 in one of the log lines, but that port number doesn’t correspond to any running service.

2

I think the error obviously complains about incorrect ACL token on the agent.
Instead of setting env environment, you can configure the agent to present the token by specifying the token in the agent configure file

acl = {
  enabled = true
  tokens = {
    agent = "<token>"
  }
}

or using command

consul acl set-agent-token agent <token>
2 Likes
3

Thanks for the reply - sorry I’m 5 months late responding! I didn’t seem to get a notification, or perhaps I missed the email.

I recently changed my config file to have

acl {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
  tokens {
    default = "<token>"
  }
}

(maybe having persistence and a token enabled is redundant, but I have the same value as in <token> in /opt/consul/acl-tokens.json :person_shrugging:)

I think this should be equivalent to setting the agent token per the docs, as the agent token will fall back on this: Agents - Configuration File Reference | Consul | HashiCorp Developer

Despite all that, I am still getting these errors in the logs whenever I deploy a service. It’s really confusing as I have personally validated that <token> is a valid Consul token and can be used to operate the CLI, so ‘ACL not found’ makes no sense unless the agent is somehow using a different token.

4

The same issue. It does not work even if {token} is management:
tokens {
default = “{token}”
}

It’s reproduced after deploying. Errors disappear after restarting Consul.

5

Hey guys, I have the same issue. These ACL errors happen only when I run a Nomad job (Traefik with Consul Catalog integration). My setup was working fine until suddenly it stopped. All my services are reported as healthy in Consul, but since these errors started, Traefik now returns a bad gateway for those services from one specific client. The issue is also inconsistent. After many restarts and redeploying the same job, the errors disappear and everything works for a while. Then the cycle repeats. What is going on?

@rincler do you have any updates, I know it’s been long time.

6

I don’t know if it helps, but at least in our case I think the issue was related to the Traefik Consul Catalog implementation and continuously abruptly stopping it when redeploying its job. The Traefik Consul Catalog provider has a watch option that we enabled:

consulCatalog:
cache: true
refreshInterval: 5s
connectAware: true
watch: true

It seems this watch feature somehow breaks Traefik and how it uses Consul and workflow identity:

Sep 26 19:02:05 flowing-panda consul[1237859]: 2025-09-26T19:02:05.052+0300 [ERROR] agent.rpcclient.health: subscribe call failed: err="rpc error: code = Unknown desc = ACL not found" failure_count=21 key=search-manager--production-sidecar-proxy topic=ServiceHealthConnect error="rpc error: code = Unknown desc = ACL not found"
Sep 26 19:02:07 flowing-panda consul[1237859]: 2025-09-26T19:02:07.155+0300 [ERROR] agent.rpcclient.health: subscribe call failed: err="rpc error: code = Unknown desc = ACL not found" failure_count=20 key=api--production topic=ServiceHealthConnect error="rpc error: code = Unknown desc = ACL not found"
Sep 26 19:02:10 flowing-panda consul[1237859]: 2025-09-26T19:02:10.443+0300 [ERROR] agent.rpcclient.health: subscribe call failed: err="rpc error: code = Unknown desc = ACL not found" failure_count=20 key=api--production-sidecar-proxy topic=ServiceHealthConnect error="rpc error: code = Unknown desc = ACL not found"

So we had to disable it and everything started working fine again.

For reference we use:

  • Traefik v3.3.5
  • Nomad v1.10.5
  • Consul v1.21.4
7

Update after long time, it seems that turning watch: false just solves it in a way that it breaks less frequently, but problem still persists.