Acls errors, k8s-consul servers outside kubernetes

Hello,
I am trying to run consul-k8s with an existing external consul servers with acls enabled. values.yaml for the helm chart as follows:

global:
  enabled: false
  name: consul
  domain: "external.dc1.com"
  datacenter: "dc1"
  gossipEncryption:
    secretName: consul-gossip-encryption-key
    secretKey: key
  acls:
    manageSystemACLs: false
    bootstrapToken:
      secretName: bootstrap-token
      secretKey: token
client:
  enabled: true
  exposeGossipPorts: true
  join:
    - 'xxx.xx.xx.xx'
    - 'xxx.xx.xx.xx'
    - 'xxx.xx.xx.xx'
externalServers:
  enabled: true
  hosts: ["xxx.xx.xx.xx", "xxx.xx.xx.xx", "xxx.xx.xx.xx"]

Checking the logs of one of the clients, I am getting a lot of error=“rpc error making call: rpc error making call: Permission denied”


    ==> Starting Consul agent...
               Version: '1.9.4'
               Node ID: 'bxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
             Node name: 'node1'
            Datacenter: 'dc1' (Segment: '')
                Server: false (Bootstrap: false)
           Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
          Cluster Addr: xxx.xx.xx.xx (LAN: 8301, WAN: 8302)
               Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false

    ==> Log data will now stream in as it occurs:

        2021-05-19T09:47:01.605Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: node1 xxx.xx.xx.xx
        2021-05-19T09:47:01.605Z [INFO]  agent.router: Initializing LAN area manager
        2021-05-19T09:47:01.605Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=udp
        2021-05-19T09:47:01.605Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=tcp
        2021-05-19T09:47:01.605Z [INFO]  agent: Starting server: address=[::]:8500 network=tcp protocol=http
        2021-05-19T09:47:01.606Z [WARN]  agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
        2021-05-19T09:47:01.606Z [INFO]  agent: Started gRPC server: address=[::]:8502 network=tcp
        2021-05-19T09:47:01.606Z [INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
        2021-05-19T09:47:01.606Z [INFO]  agent: Joining cluster...: cluster=LAN
        2021-05-19T09:47:01.606Z [INFO]  agent: (LAN) joining: lan_addresses=[xxx.xx.xx.xx, xxx.xx.xx.xx, xxx.xx.xx.xx]
        2021-05-19T09:47:01.607Z [INFO]  agent: started state syncer
    ==> Consul agent running!
        2021-05-19T09:47:01.607Z [WARN]  agent.router.manager: No servers available
        2021-05-19T09:47:01.607Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
        2021-05-19T09:47:01.610Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: enode1 xxx.xx.xx.xx
        2021-05-19T09:47:01.610Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: enode3 xxx.xx.xx.xx
        2021-05-19T09:47:01.610Z [WARN]  agent.client.memberlist.lan: memberlist: Refuting a dead message (from: node1)
        2021-05-19T09:47:01.610Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: enode2 xxx.xx.xx.xx
        2021-05-19T09:47:01.610Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: inode1 xxx.xx.xx.xx
        2021-05-19T09:47:01.610Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: nnode1 xxx.xx.xx.xx
        2021-05-19T09:47:01.610Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: inode3 xxx.xx.xx.xx
        2021-05-19T09:47:01.610Z [INFO]  agent.client: adding server: server="inode1 (Addr: tcp/xxx.xx.xx.xx:8300) (DC: dc1)"
        2021-05-19T09:47:01.610Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: node2 xxx.xx.xx.xx
        2021-05-19T09:47:01.610Z [INFO]  agent.client: adding server: server="inode3 (Addr: tcp/xxx.xx.xx.xx:8300) (DC: dc1)"
        2021-05-19T09:47:01.610Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: dnode1 xxx.xx.xx.xx
        2021-05-19T09:47:01.610Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: inode2 xxx.xx.xx.xx
        2021-05-19T09:47:01.610Z [INFO]  agent.client: adding server: server="inode2 (Addr: tcp/xxx.xx.xx.xx:8300) (DC: dc1)"
        2021-05-19T09:47:01.610Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: mnode1 xxx.xx.xx.xx
        2021-05-19T09:47:01.610Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: knode1 xxx.xx.xx.xx
        2021-05-19T09:47:01.610Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: anode1 xxx.xx.xx.xx
        2021-05-19T09:47:01.615Z [INFO]  agent: (LAN) joined: number_of_nodes=3
        2021-05-19T09:47:01.615Z [INFO]  agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3
        2021-05-19T09:47:04.331Z [ERROR] agent.client: RPC failed to server: method=Catalog.Register server=xxx.xx.xx.xx:8300 error="rpc error making call: Permission denied"
        2021-05-19T09:47:04.332Z [WARN]  agent: Node info update blocked by ACLs: node=bxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx accessorID=
        2021-05-19T09:47:06.466Z [ERROR] agent.client: RPC failed to server: method=Catalog.Register server=xxx.xx.xx.xx:8300 error="rpc error making call: rpc error making call: Permission denied"
        2021-05-19T09:47:06.466Z [WARN]  agent: Node info update blocked by ACLs: node=bxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx accessorID=
        2021-05-19T09:47:19.056Z [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=xxx.xx.xx.xx:8300 error="rpc error making call: rpc error making call: Permission denied"
        2021-05-19T09:47:19.057Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID=

I am using the main bootstrapToken(global management).

Checking the docs for consul-K8s with external servers (Consul Servers Outside of Kubernetes - Kubernetes | Consul by HashiCorp). The method shown here is with manageSystemACLs: true, is it possible to run it without it? creating the needed tokens manually.

consul agent: 1.9.4
consul-k8s: 0.31.1

Thank you :slight_smile:

Hey @gmaytham

Yeah, it’s possible. You’d have to create a token for consul clients manually, create a secret with the token as config for clients and provide it via client.extraVolumes Helm value. Note that in that case you don’t need to provide the bootstrapToken or set externalServers in your Helm values.

Hope this helps!

1 Like

thank you. it helped.