Consul Agent access to KV denied by ACL

stewartm · February 12, 2020, 11:15am

I am likely making a glaring error in my configuration, but so far am failing to spot it. I am attempting to develop an ACL strategy for an existing Consul deployment, but am currently working with a small test configuration.

I am using Consul 1.6.3 on Centos 7 hosts
I have 5 agents running in the server role
a handful of others just running as basic agents
I have a couple of simple keys in the KV store
ACLs have been enabled and at a basic node level look to be working fine
I have created a couple of policies and tokens for the UI, which also appear to be working fine

I am now attempting to grant an agent on one node, access to the KV store (accessing a key containing a simple string value):

[user@computer ~]# consul kv get docker/test
Error querying Consul agent: Unexpected response code: 403

In the logs:

Feb 12 10:45:57 computer consul: 2020/02/12 10:45:57 [ERR] consul: "KVS.Get" RPC failed to server 192.168.10.22:8300: rpc error making call: rpc error making call: Permission denied
Feb 12 10:45:57 computer consul: 2020/02/12 10:45:57 [ERR] http: Request GET /v1/kv/docker/test, error: rpc error making call: rpc error making call: Permission denied from=127.0.0.1:32902

from the same logs:

Feb 12 09:54:19 computer consul: 2020/02/12 09:54:19 [INFO] serf: EventMemberUpdate: computer
Feb 12 09:54:20 computer consul: 2020/02/12 09:54:20 [INFO] agent: Synced node info

which I am taking to confirm that basic node ACLs are working OK otherwise the sync would fail?.

The agent is configured to use a token that has the following policy attached:

{
  "node": {
    "computer": {
      "policy": "write"
    }
  },
  "key": {
    "docker/test": {
      "policy": "read"
    }
  }
}

Attempting the same consul kv get ... command from the host with the Global Management token set in CONSUL_HTTP_TOKEN works fine.

I have tried playing around with ‘key’ and ‘key_prefix’, have added the token to the acl.tokens.agent section of the config and restarted consul, tried using HCL versus JSON, tried splitting the key policy out into its own policy file and added to the token. So far, no change.

(I have changed the actual host name in the example above, but the actual value is correct, based on the agents name in the cluster)

What obvious mistake am I missing?

blake · February 12, 2020, 4:31pm

Hi @stewartm,

I don’t see any issues with your policy.

Can you verify the token you’re using (either via CONSUL_HTTP_TOKEN or -token) at the CLI has that policy assigned?

consul acl token read -id <accessor ID>

stewartm · February 13, 2020, 9:40am

Hi @blake,

Thanks for getting back to me and the suggestion.

Using CONSUL_HTTP_TOKEN with the relevant token worked fine, which then lead me to review the “acl” section of my config file, specifically the “tokens” section. I had misunderstood the difference between “agent” and “default”, and thought it would be sufficient to set the token in the “agent” field. Re-reading the docs I now see I should have placed it in “default”.

I have made this change and all is working fine now. Thanks very much for the pointer.

rkibbe · August 10, 2020, 8:55pm

I am having similiar issue: here is my config.json:
{
“server”: false,
“datacenter”: “dev”,
“data_dir”: “/home/jenkins/Consul/Consul/data”,
“encrypt”: “7n6FFC8HlaqGnRZE5t1NJg==”,
“log_level”: “INFO”,
“bind_addr”: “10.169.xx”,
“start_join”: [“10.169.xx”],
“ca_file”: “/home/jenkins/Consul/Consul/consul.d/ssl/ca.cert”,
“cert_file”: “/home/jenkins/Consul/Consul/consul.d/ssl/consul.cert”,
“key_file”: “/home/jenkins/Consul/Consul/consul.d/ssl/consul.key”,
“acl_default_policy”: “allow”,
“verify_incoming”: true,
“verify_outgoing”: true
}

I set acl_default to allow.
==> Starting Consul agent…
==> Joining cluster…
Join completed. Synced with 1 initial agents
==> Consul agent running!
Version: ‘v1.2.1’
Node ID: ‘b3154c18-f17b-b3e0-de72-283b5e831dc1’
Node name: ‘f5baca29e625’
Datacenter: ‘dev’ (Segment: ‘’)
Server: false (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.169.44.5 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: true

==> Log data will now stream in as it occurs:

2020/08/10 19:59:56 [INFO] serf: EventMemberJoin: f5baca29e625 10.169.44.xx
2020/08/10 19:59:56 [INFO] serf: Attempting re-join to previously known node: mdlmsvrxx2: 10.169.xx.xx:8301
2020/08/10 19:59:56 [WARN] agent/proxy: running as root, will not start managed proxies
2020/08/10 19:59:56 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2020/08/10 19:59:56 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2020/08/10 19:59:56 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
2020/08/10 19:59:56 [INFO] agent: (LAN) joining: [10.169.44.5]
2020/08/10 19:59:56 [INFO] serf: EventMemberJoin: mdldjenxx1 10.169.xx.114
2020/08/10 19:59:56 [INFO] serf: EventMemberJoin: mdlmsvrxx1 10.169.xx.58
2020/08/10 19:59:56 [INFO] serf: EventMemberJoin: mdlsvrrxx1 10.169.xx.57
2020/08/10 19:59:56 [INFO] serf: EventMemberJoin: mdlmsvrxx2 10.169.xx.59
2020/08/10 19:59:56 [INFO] serf: EventMemberJoin: mdljenkxx1 10.169.xx.54
2020/08/10 19:59:56 [INFO] serf: Re-joined to previously known node: mdlmsvrxx2: 10.169.xx.59:8301
2020/08/10 19:59:56 [INFO] consul: adding server mdlsvrrxx1 (Addr: tcp/10.169.x1.xx:8300) (DC: dev)
2020/08/10 19:59:56 [INFO] agent: (LAN) joined: 1 Err: <nil>
2020/08/10 19:59:56 [INFO] agent: started state syncer
2020/08/10 19:59:56 [ERR] consul: "Catalog.Register" RPC failed to server 10.169.xx.xx:8300: rpc error making call: Permission denied
2020/08/10 19:59:56 [WARN] agent: Service "name-address-normalizer-api-57000" registration blocked by ACLs
2020/08/10 19:59:56 [ERR] consul: "Catalog.Register" RPC failed to server 10.169.xx.xx:8300: rpc error making call: Permission denied
2020/08/10 19:59:56 [WARN] agent: Service "membership-verification-api-57010-management" registration blocked by ACLs
2020/08/10 19:59:56 [ERR] consul: "Catalog.Register" RPC failed to server 10.169.xx.xx:8300: rpc error making call: Permission denied
2020/08/10 19:59:56 [WARN] agent: Service "name-address-normalizer-api-57000-management" registration blocked by ACLs
2020/08/10 19:59:56 [ERR] consul: "Catalog.Register" RPC failed to server 10.169.xx.xx:8300: rpc error making call: Permission denied
2020/08/10 19:59:56 [WARN] agent: Service "membership-verification-api-57010" registration blocked by ACLs
2020/08/10 19:59:56 [INFO] agent: Synced node info

2020/08/10 19:59:59 [INFO] agent: Synced service "name-address-normalizer-api-57000"
2020/08/10 19:59:59 [INFO] agent: Synced service "membership-verification-api-57010-management"
2020/08/10 19:59:59 [INFO] agent: Synced service "name-address-normalizer-api-57000-management"
2020/08/10 19:59:59 [INFO] agent: Synced service "membership-verification-api-57010"
2020/08/10 19:59:59 [WARN] agent: Check "service:membership-verification-api-57010-management" HTTP request failed: Get https://mdlmsvrxx1.cgi.int:57011/membership-verification-api/manage/health: dial tcp 10.169.xx.xx:57011: connect: connection refused

==> Newer Consul version available: 1.8.1 (currently running: 1.2.1)

2020/08/10 20:00:12 [WARN] agent: Check "service:membership-verification-api-57010" HTTP request failed: Get https://mdlmsvrxx1.cgi.int:57011/membership-verification-api/manage/health: dial tcp 10.169.xx.xx:57011: connect: connection refused

2020/08/10 20:00:13 [WARN] agent: Check "service:name-address-normalizer-api-57000" HTTP request failed: Get https://mdlmsvrxx1.cgi.int:57001/name-address-normalizer-api/manage/health: dial tcp 10.169.xx.xx:57001: connect: connection refused

jsosulska · August 11, 2020, 2:29pm

Hi @rkibbe,

Welcome to the forums!
Have you tried looking through our Bootstrapping the ACL System guide? Simply setting acl_default_policy does not turn on ACLs. The configuration you posted is missing the acls.enabled set to true.

Give the guide a try and let us know how it goes!

Hope this helps!

Wolfsrudel · August 11, 2020, 3:27pm

Duplicate: Rpc error making call: Permission denied

Also it is a client config. The acl.enabled set to true should be configured on the server-side, shouldn’t it?

rkibbe · August 11, 2020, 4:16pm

I have looked thru the acl docs. The issue is I cant just recreate a policy and tokens since it works for our other two jenkins servers. I am using the same token, certs, keys, etc… as those servers. The only guy who knew anything about this is no longer with the company so i dont want to break what is working.

Is there an “easy” way to disable the acl so I can test if this is actually the issue or if it is maybe network\firewall related ? In my config.json I have “acl_default_policy”: “allow”, but does it have to be changed on the servers as opposed to the agents ?

jsosulska · August 11, 2020, 5:01pm

@rkibbe, Hmmm, did you add the acls.token.default to the client config? There needs to be some token definition for default, iirc. That’s what the OP ran into.

Edit: As @Wolfsrudel pointed out in the other thread, is there a particular reason you are using 1.2.1? The current ACL system is for > 1.4.0+ Here is the legacy documentation.

@Wolfsrudel - Thanks for finding the dupe, I will go and close that thread in favor of this one.

rkibbe · August 11, 2020, 5:59pm

I appreciate your help. I have read the legacy docs since i have an older version. I still dont really understand what you are asking me add the acls.token.default to the client config ?

I have 5 servers, 2 of which are jenkins servers running the agents on them. When i start mine it elects a leader. If i go on the leader server and look at client.token.json it matches to what i have and also to what the other two jenki sservers are running as well. In fact what I installed on new jenkins server is just a zip of one of the other jenkins servers that work.
Is there an easy way to temporarily disable acls or check what acls may be turned on with the “leader” server ?