Stuck troubleshooting ACLs

Consul
1

Consul is repeatedly spamming my logs with messages of the form:

[ERROR] agent: Coordinate update error: error="ACL not found"

and

[ERROR] agent.anti_entropy: failed to sync remote state: error="ACL not found"

At this point I’ve pared down my policy that is assigned to the agent token to the following:

  rules = jsonencode({
    node_prefix = {
      "" = {
        policy = "write"
      }
    }
    agent_prefix = {
      "" = {
	policy = "write"
      } 
    } 
    service_prefix = {
      "" = {
        policy = "read"
      }
    }
  })

I would be using node-identity, but since the vault consul backend still doesn’t work with that, this is what I’m trying for the moment until I figure out how to get the templating to work better. This is similar to this unanswered thread from 2021.

Is there any way to get a useful error message out of consul for these cases? Something with even a fraction more context might help debug what is going on. I’m running Consul 1.16.1 from Hashicorp’s own repo on debian, so I have to assume the build I have is fine or other people would have noticed a problem.

2

kind of sounds like it’s saying the ACL token sent isn’t valid at all, rather than it having the wrong policy.

It’s something else to investigate, at least.

3

The agent has a token specified for acl.tokens.agent which suggests that if it isn’t being sent this is an obvious bug. One thing I have yet to try because I couldn’t figure out how to get from a secret-id to an accessor-id was to look up the token itself and see if it was valid at all. Do you know of a way to do that conversion?

4

It appears that the token specified in your Consul configuration under acl.tokens.agent cannot be located. To verify its existence, you can use the command consul acl token list -token ROOT_TOKEN.

As a temporary solution, you can acquire a valid token from another Consul agent that is functioning correctly. Subsequently, apply this token to the agent facing the issue.

$ consul acl set-agent-token agent TOKEN_ID