Rpc error making call: Permission denied

Consul
1

i am running a Consul agent in a Docker container, connecting to Consul servers running on VMs.

I set acl_policy to allow .

I checked the tokens on the servers that i know can connect and it is the same.

cat /home/jenkins/Consul/Consul/consul.d/client/config.json
{
“server”: false,
“datacenter”: “dev”,
“data_dir”: “/home/jenkins/Consul/Consul/data”,
“encrypt”: “7n6FFC8HlaqGnRZE5t1NJg==”,
“log_level”: “INFO”,
“bind_addr”: “10.169.xx”,
“start_join”: [“10.169.xx”],
“ca_file”: “/home/jenkins/Consul/Consul/consul.d/ssl/ca.cert”,
“cert_file”: “/home/jenkins/Consul/Consul/consul.d/ssl/consul.cert”,
“key_file”: “/home/jenkins/Consul/Consul/consul.d/ssl/consul.key”,
“acl_default_policy”: “allow”,
“verify_incoming”: true,
“verify_outgoing”: true
}

==> Starting Consul agent…
==> Joining cluster…
Join completed. Synced with 1 initial agents
==> Consul agent running!
Version: ‘v1.2.1’
Node ID: ‘b3154c18-f17b-b3e0-de72-283b5e831dc1’
Node name: ‘f5baca29e625’
Datacenter: ‘dev’ (Segment: ‘’)
Server: false (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.169.44.5 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: true

==> Log data will now stream in as it occurs:

2020/08/10 19:59:56 [INFO] serf: EventMemberJoin: f5baca29e625 10.169.44.xx
2020/08/10 19:59:56 [INFO] serf: Attempting re-join to previously known node: mdlmsvrxx2: 10.169.xx.xx:8301
2020/08/10 19:59:56 [WARN] agent/proxy: running as root, will not start managed proxies
2020/08/10 19:59:56 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2020/08/10 19:59:56 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2020/08/10 19:59:56 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
2020/08/10 19:59:56 [INFO] agent: (LAN) joining: [10.169.44.5]
2020/08/10 19:59:56 [INFO] serf: EventMemberJoin: mdldjenxx1 10.169.xx.114
2020/08/10 19:59:56 [INFO] serf: EventMemberJoin: mdlmsvrxx1 10.169.xx.58
2020/08/10 19:59:56 [INFO] serf: EventMemberJoin: mdlsvrrxx1 10.169.xx.57
2020/08/10 19:59:56 [INFO] serf: EventMemberJoin: mdlmsvrxx2 10.169.xx.59
2020/08/10 19:59:56 [INFO] serf: EventMemberJoin: mdljenkxx1 10.169.xx.54
2020/08/10 19:59:56 [INFO] serf: Re-joined to previously known node: mdlmsvrxx2: 10.169.xx.59:8301
2020/08/10 19:59:56 [INFO] consul: adding server mdlsvrrxx1 (Addr: tcp/10.169.x1.xx:8300) (DC: dev)
2020/08/10 19:59:56 [INFO] agent: (LAN) joined: 1 Err: <nil>
2020/08/10 19:59:56 [INFO] agent: started state syncer
2020/08/10 19:59:56 [ERR] consul: "Catalog.Register" RPC failed to server 10.169.xx.xx:8300: rpc error making call: Permission denied
2020/08/10 19:59:56 [WARN] agent: Service "name-address-normalizer-api-57000" registration blocked by ACLs
2020/08/10 19:59:56 [ERR] consul: "Catalog.Register" RPC failed to server 10.169.xx.xx:8300: rpc error making call: Permission denied
2020/08/10 19:59:56 [WARN] agent: Service "membership-verification-api-57010-management" registration blocked by ACLs
2020/08/10 19:59:56 [ERR] consul: "Catalog.Register" RPC failed to server 10.169.xx.xx:8300: rpc error making call: Permission denied
2020/08/10 19:59:56 [WARN] agent: Service "name-address-normalizer-api-57000-management" registration blocked by ACLs
2020/08/10 19:59:56 [ERR] consul: "Catalog.Register" RPC failed to server 10.169.xx.xx:8300: rpc error making call: Permission denied
2020/08/10 19:59:56 [WARN] agent: Service "membership-verification-api-57010" registration blocked by ACLs
2020/08/10 19:59:56 [INFO] agent: Synced node info

2020/08/10 19:59:59 [INFO] agent: Synced service "name-address-normalizer-api-57000"
2020/08/10 19:59:59 [INFO] agent: Synced service "membership-verification-api-57010-management"
2020/08/10 19:59:59 [INFO] agent: Synced service "name-address-normalizer-api-57000-management"
2020/08/10 19:59:59 [INFO] agent: Synced service "membership-verification-api-57010"
2020/08/10 19:59:59 [WARN] agent: Check "service:membership-verification-api-57010-management" HTTP request failed: Get https://mdlmsvrxx1.cgi.int:57011/membership-verification-api/manage/health: dial tcp 10.169.xx.xx:57011: connect: connection refused

==> Newer Consul version available: 1.8.1 (currently running: 1.2.1)

2020/08/10 20:00:12 [WARN] agent: Check "service:membership-verification-api-57010" HTTP request failed: Get https://mdlmsvrxx1.cgi.int:57011/membership-verification-api/manage/health: dial tcp 10.169.xx.xx:57011: connect: connection refused
2

Are you sure that acl default policy is allow at the server side?

registration blocked by ACLs

Beside your agent is very, very old using version 1.2.1. :wink:

3

how do i check if acl default policy is allow at the server side?

4

@rkibbe,

You can check the value of acl.default_policy in the server’s configuration files, or use the /v1/agent/self HTTP API endpoint to determine the default ACL policy.

$ curl --silent \
       --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
       $CONSUL_HTTP_ADDR/v1/agent/self | jq .DebugConfig.ACLDefaultPolicy
"deny"