Consul Agent access to KV denied by ACL

I am likely making a glaring error in my configuration, but so far am failing to spot it. I am attempting to develop an ACL strategy for an existing Consul deployment, but am currently working with a small test configuration.

  • I am using Consul 1.6.3 on Centos 7 hosts
  • I have 5 agents running in the server role
  • a handful of others just running as basic agents
  • I have a couple of simple keys in the KV store
  • ACLs have been enabled and at a basic node level look to be working fine
  • I have created a couple of policies and tokens for the UI, which also appear to be working fine

I am now attempting to grant an agent on one node, access to the KV store (accessing a key containing a simple string value):

[user@computer ~]# consul kv get docker/test
Error querying Consul agent: Unexpected response code: 403

In the logs:

Feb 12 10:45:57 computer consul: 2020/02/12 10:45:57 [ERR] consul: "KVS.Get" RPC failed to server 192.168.10.22:8300: rpc error making call: rpc error making call: Permission denied
Feb 12 10:45:57 computer consul: 2020/02/12 10:45:57 [ERR] http: Request GET /v1/kv/docker/test, error: rpc error making call: rpc error making call: Permission denied from=127.0.0.1:32902

from the same logs:

Feb 12 09:54:19 computer consul: 2020/02/12 09:54:19 [INFO] serf: EventMemberUpdate: computer
Feb 12 09:54:20 computer consul: 2020/02/12 09:54:20 [INFO] agent: Synced node info

which I am taking to confirm that basic node ACLs are working OK otherwise the sync would fail?.

The agent is configured to use a token that has the following policy attached:

{
  "node": {
    "computer": {
      "policy": "write"
    }
  },
  "key": {
    "docker/test": {
      "policy": "read"
    }
  }
}

Attempting the same consul kv get ... command from the host with the Global Management token set in CONSUL_HTTP_TOKEN works fine.

I have tried playing around with ‘key’ and ‘key_prefix’, have added the token to the acl.tokens.agent section of the config and restarted consul, tried using HCL versus JSON, tried splitting the key policy out into its own policy file and added to the token. So far, no change.

(I have changed the actual host name in the example above, but the actual value is correct, based on the agents name in the cluster)

What obvious mistake am I missing?

Hi @stewartm,

I don’t see any issues with your policy.

Can you verify the token you’re using (either via CONSUL_HTTP_TOKEN or -token) at the CLI has that policy assigned?

consul acl token read -id <accessor ID>

Hi @blake,

Thanks for getting back to me and the suggestion.

Using CONSUL_HTTP_TOKEN with the relevant token worked fine, which then lead me to review the “acl” section of my config file, specifically the “tokens” section. I had misunderstood the difference between “agent” and “default”, and thought it would be sufficient to set the token in the “agent” field. Re-reading the docs I now see I should have placed it in “default”.

I have made this change and all is working fine now. Thanks very much for the pointer.

2 Likes