[ERR] agent: failed to sync remote state: rpc error making call: ACL not found

Consul
1

I bootstrapped ACL and created an agent policy with token for my consul server.
Still I’m puzzled by this remaining error message in the logs:

[ERR] agent: failed to sync remote state: rpc error making call: ACL not found

Without the policy I had lots of thes error message in the logs:

[WARN] agent: Coordinate update blocked by ACLs

Is there a permission missing in this policy? What is needed to sync remote state??

node "myserver" {
  policy = "write"
}
node_prefix "" {
  policy = "read"
}
service "" {
  policy = "write"
}
service_prefix "secure-" {
  policy = "read"
}
event "" {
  policy = "write"
}
query "" {
  policy = "read"
}
key "" {
  policy = "read"
}
key_prefix "config/properties/" {
  policy = "read"
}
2

Did you ever figure this one out? I’m experiencing something similar. On my Consul servers I’m getting these two errors:

[ERROR] agent: Coordinate update error: error=“rpc error making call: ACL not found”
[ERROR] agent.anti_entropy: failed to sync remote state: error=“ACL not found”

On the Consul client I’m getting those two, in addition to this one:

agent.client: RPC failed to server: method=Coordinate.Update server=10.0.2.6:8300 error=“rpc error making call: ACL not found”

In the ACL troubleshooting guide, this is stated regarding the catalog command:

If “TaggedAddresses” is null for any of the agents, that agent’s ACLs are not configured correctly.

My output of consul catalog nodes -detailed is:

Node      ID Address   DC   TaggedAddresses                                                   Meta
client-0  a  10.0.2.4  dc1                                                                    
server-0  b  10.0.2.5  dc1  lan=10.0.2.5, lan_ipv4=10.0.2.5, wan=10.0.2.5, wan_ipv4=10.0.2.5  consul-network-segment=
server-1  c  10.0.2.6  dc1  lan=10.0.2.6, lan_ipv4=10.0.2.6, wan=10.0.2.6, wan_ipv4=10.0.2.6  consul-network-segment=
server-2  d  10.0.2.7  dc1  lan=10.0.2.7, lan_ipv4=10.0.2.7, wan=10.0.2.7, wan_ipv4=10.0.2.7  consul-network-segment=

So it looks like it has be an issue with my client.
The common error guide says that:

This indicates that you have ACL enabled in your cluster, but you aren’t passing a valid token. Make sure that when creating your tokens that they have the correct permissions set. In addition, you would want to make sure that an agent token is provided on each call.

I have the same permissions set in the same way, but with a token generated from a policy with a different name for both the Consul clients and servers. So what is the difference between Consul agents for clients and for servers that’s making the client’s TaggedAddresses being wrong? Is it something in the Consul configuration that is making it?

3

Hi @kollstrom,

Is the token with node:write privileges correctly specified in the config file under acl.tokens.agent?

4

No, I’ve set the token with consul acl set-agent-token agent "<agent token here>", and restarted the service with systemctl restart consul.service. Doesn’t work if I hardcode it into the config file either.

My config looks like this:

    {
      "ui": true,
      "log_level": "INFO",
      "data_dir": "/opt/consul/data",
      "bind_addr": "0.0.0.0",
      "client_addr": "0.0.0.0",
      "advertise_addr": "10.0.2.4",
      "retry_join": ["provider=azure tag_name=ConsulAutoJoin tag_value=auto-join subscription_id=zzz tenant_id=yyy secret_access_key=xxx"],
      "acl": {
        "enabled": true,
        "default_policy": "allow",
        "enable_token_persistence": true
      }
    }

And the systemd definition looks like this:

[Unit]
Description=Consul Agent
Requires=network-online.target
After=network-online.target

[Service]
Restart=on-failure
ExecStart=/usr/local/bin/consul agent -config-dir="/etc/consul.d" -recursor="168.63.129.16"
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM
User=root
Group=root
Type=notify

[Install]
WantedBy=multi-user.target
5

Which token are you using when running this command?

I ask because that token will need agent:write permissions for the node in order to update and persist the token across restarts (see permission requirements under Agent HTTP API - Update ACL Tokens), and I didn’t see this permission specified in the policy you previously shared.

6

I used the bootstrap token generated with: consul acl bootstrap:

AccessorID:   4d123dff-f460-73c3-02c4-8dd64d136e01
SecretID:     86cddfb9-2760-d947-358d-a2811156bf31
Description:  Bootstrap Token (Global Management)
Local:        false
Create Time:  2018-10-22 11:27:04.479026 -0400 EDT
Policies:
   00000000-0000-0000-0000-000000000001 - global-management

(Output is copied from the documentation, I’m not sharing my global-management token :sweat_smile:)
Before I run the commands I do export CONSUL_HTTP_TOKEN=86cddfb9-2760-d947-358d-a2811156bf31, which is the SecretID in the command above.