RPC failed to server: method=Coordinate.Update server=192.168.1.1:8300 error="rpc error making call: Permission denied"

Consul
1

Hello,

I have Consul v1.11.1 and it seems all parts are working … but I get every few seconds on servers and agents the message:

agent.client: RPC failed to server: method=Coordinate.Update server=192.168.1.1:8300 error="rpc error making call: Permission denied"

The IP is the consul server (one of three with 1.2 / 1.3) and I have noe clue, which ACL is missing. The config uses a Agent Token which contains:

node_prefix "" {
   policy = "write"
}
service_prefix "" {
   policy = "read"
}

agent_prefix "" {
   policy = "read"
}

query_prefix "" {
   policy = "read"
}

Anonymous allows only list nodes.

Any suggestions ?

cu denny

2

Hi @linuxmail,

Do you have acl.enable_token_persistence set to true, and did you ever apply an agent token with the consul acl set-agent-token agent <token> command? If yes, Consul will create a copy of the tokens inside the data directory, and those tokens will take precedence compared to the ones you set in the config file.

If this is the case, you have two options based on how you intend to manage tokens for an agent:

  1. If you are planning to use the configuration file to set the tokens:
    a. delete (backup in case you need it) the <data-dir>/acl-tokens.json
    b. set the acl.enable_token_persistence to false (default)
    c. set the token in config file
    d. reload the consul agent.

  2. If you prefer to use the set-agent-token command or the API, remove the token from the config file (to avoid confusion), and set the right agent token using the CLI/API.

I hope this helps.

3

hi @Ranjandas

thanks for the reply :slight_smile:

I have for the agents:

...
      'acl'                         => {
        enabled        => true,
        default_policy => 'deny',
        down_policy    => 'extend-cache',
        policy_ttl     => '180s',
        token_ttl      => '180s',
        'tokens'       => {
          agent              => "$consul_agent_token",
        },
      },
....

and for the servers:

...
      'acl'                         => {
        enabled        => true,
        default_policy => 'deny',
        down_policy    => 'extend-cache',
        policy_ttl     => '180s',
        token_ttl      => '180s',
        'tokens'       => {
          agent              => "$consul_agent_token",
          initial_management => "$acl_api_token",
          agent_recovery     => "$consul_agent_token",
        },
      },
...

for the servers… I’ve set consul_agent_token / acl_api_token to the same … as someone wrote, it helped him.

cu denny