Consul ACL on openshift with existing bootstrapToken: not working

mattlegall · May 28, 2021, 1:27pm

I am trying to install a single consul node in openshift with a customized values.yaml for helm. I want a single bootstrapToken for all interactions with consul (no complex policies needed).

So I am creating the secret beforehand:
kubectl create secret generic master-token -n consul-ns --from-literal=‘token=supersecret’

My values.yml looks like:
global:
acls:
manageSystemACLs: true
bootstrapToken:
secretName: master-token
secretKey: token
enabled: false
datacenter: dev-itaag108-corporatehmb01
openshift:
enabled: true
server:
enabled: true
replicas: 1
bootstrap_expect: 1
storage: 1Gi
client:
enabled: false
dns:
enabled: false
ui:
enabled: true
tests:
enabled: false

When I install with helm, consul ui is available but at login time I am getting “Invalid token, The token entered does not exist. Please enter a valid token to log in.”

The pod consul-server-acl-init-gjbn7 keeps logging:
Failure: calling /agent/self to get datacenter: err=“Unexpected response code: 403 (ACL not found)”

The pod consul-server-0 keeps logging:
oc logs pod/dev-itaag108-corporatehmb01-consul-server-0 -n consul-ns
==> Starting Consul agent…
Version: ‘1.9.4’
Node ID: ‘9a46b178-823f-74b2-afe5-0136f533db1d’
Node name: ‘dev-itaag108-corporatehmb01-consul-server-0’
Datacenter: ‘dev-itaag108-corporatehmb01’ (Segment: ‘’)
Server: true (Bootstrap: true)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
Cluster Addr: 10.128.2.40 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false

==> Log data will now stream in as it occurs:

2021-05-28T13:20:48.608Z [WARN]  agent: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
2021-05-28T13:20:48.608Z [WARN]  agent: bootstrap = true: do not enable unless necessary
2021-05-28T13:20:48.805Z [WARN]  agent.auto_config: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
2021-05-28T13:20:48.805Z [WARN]  agent.auto_config: bootstrap = true: do not enable unless necessary
2021-05-28T13:20:48.827Z [INFO]  agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9a46b178-823f-74b2-afe5-0136f533db1d Address:10.12

8.2.40:8300}]"
2021-05-28T13:20:48.827Z [INFO] agent.server.raft: entering follower state: follower=“Node at 10.128.2.40:8300 [Follower]” leader=
2021-05-28T13:20:48.828Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: dev-itaag108-corporatehmb01-consul-server-0.dev-itaag108-corporatehmb01 10.128.2.40
2021-05-28T13:20:48.904Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: dev-itaag108-corporatehmb01-consul-server-0 10.128.2.40
2021-05-28T13:20:48.904Z [INFO] agent.router: Initializing LAN area manager
2021-05-28T13:20:48.904Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp
2021-05-28T13:20:48.904Z [INFO] agent.server: Handled event for server in area: event=member-join server=dev-itaag108-corporatehmb01-consul-server-0.dev-itaag108
-corporatehmb01 area=wan
2021-05-28T13:20:48.905Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2021-05-28T13:20:48.907Z [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http
2021-05-28T13:20:48.907Z [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Cons
ul. Set telemetry { disable_compat_1.9 = true } to disable them.
2021-05-28T13:20:49.003Z [INFO] agent.server: Adding LAN server: server=“dev-itaag108-corporatehmb01-consul-server-0 (Addr: tcp/10.128.2.40:8300) (DC: dev-itaag1
08-corporatehmb01)”
2021-05-28T13:20:49.003Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods=“aliyun aws azure digitalocean
gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere”
2021-05-28T13:20:49.003Z [INFO] agent: Joining cluster…: cluster=LAN
2021-05-28T13:20:49.003Z [INFO] agent: (LAN) joining: lan_addresses=[dev-itaag108-corporatehmb01-consul-server-0.dev-itaag108-corporatehmb01-consul-server.consul
-ns.svc:8301]
2021-05-28T13:20:49.003Z [INFO] agent: started state syncer
==> Consul agent running!
2021-05-28T13:20:49.305Z [WARN] agent.server.memberlist.lan: memberlist: Failed to resolve dev-itaag108-corporatehmb01-consul-server-0.dev-itaag108-corporatehmb0
1-consul-server.consul-ns.svc:8301: lookup dev-itaag108-corporatehmb01-consul-server-0.dev-itaag108-corporatehmb01-consul-server.consul-ns.svc on 172.30.0.10:53: no s
uch host
2021-05-28T13:20:49.305Z [WARN] agent: (LAN) couldn’t join: number_of_nodes=0 error="1 error occurred:

Failed to resolve dev-itaag108-corporatehmb01-consul-server-0.dev-itaag108-corporatehmb01-consul-server.consul-ns.svc:8301: lookup dev-itaag108-corporatehmb
01-consul-server-0.dev-itaag108-corporatehmb01-consul-server.consul-ns.svc on 172.30.0.10:53: no such host

"
2021-05-28T13:20:49.305Z [WARN] agent: Join cluster failed, will retry: cluster=LAN retry_interval=30s error=
2021-05-28T13:20:51.040Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”
2021-05-28T13:20:52.041Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”
2021-05-28T13:20:53.041Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”
2021-05-28T13:20:54.044Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”
2021-05-28T13:20:55.046Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”
2021-05-28T13:20:56.046Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”
2021-05-28T13:20:56.254Z [ERROR] agent.anti_entropy: failed to sync remote state: error=“No cluster leader”
2021-05-28T13:20:57.047Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”
2021-05-28T13:20:58.048Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”
2021-05-28T13:20:58.387Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=
2021-05-28T13:20:58.387Z [INFO] agent.server.raft: entering candidate state: node=“Node at 10.128.2.40:8300 [Candidate]” term=2
2021-05-28T13:20:58.395Z [INFO] agent.server.raft: election won: tally=1
2021-05-28T13:20:58.395Z [INFO] agent.server.raft: entering leader state: leader=“Node at 10.128.2.40:8300 [Leader]”
2021-05-28T13:20:58.395Z [INFO] agent.server: cluster leadership acquired
2021-05-28T13:20:58.395Z [INFO] agent.server: New leader elected: payload=dev-itaag108-corporatehmb01-consul-server-0
2021-05-28T13:20:58.399Z [INFO] agent.server: initializing acls
2021-05-28T13:20:58.402Z [INFO] agent.server: Created ACL ‘global-management’ policy
2021-05-28T13:20:58.408Z [INFO] agent.server: Created ACL anonymous token from configuration
2021-05-28T13:20:58.408Z [INFO] agent.leader: started routine: routine=“legacy ACL token upgrade”
2021-05-28T13:20:58.408Z [INFO] agent.leader: started routine: routine=“acl token reaping”
2021-05-28T13:20:58.408Z [INFO] agent.server.serf.lan: serf: EventMemberUpdate: dev-itaag108-corporatehmb01-consul-server-0
2021-05-28T13:20:58.408Z [INFO] agent.server.serf.wan: serf: EventMemberUpdate: dev-itaag108-corporatehmb01-consul-server-0.dev-itaag108-corporatehmb01
2021-05-28T13:20:58.408Z [INFO] agent.server: Updating LAN server: server=“dev-itaag108-corporatehmb01-consul-server-0 (Addr: tcp/10.128.2.40:8300) (DC: dev-itaa
g108-corporatehmb01)”
2021-05-28T13:20:58.408Z [INFO] agent.server: Handled event for server in area: event=member-update server=dev-itaag108-corporatehmb01-consul-server-0.dev-itaag1
08-corporatehmb01 area=wan
2021-05-28T13:20:58.417Z [INFO] agent.leader: started routine: routine=“federation state anti-entropy”
2021-05-28T13:20:58.418Z [INFO] agent.leader: started routine: routine=“federation state pruning”
2021-05-28T13:20:58.522Z [INFO] agent.server.connect: initialized primary datacenter CA with provider: provider=consul
2021-05-28T13:20:58.522Z [INFO] agent.leader: started routine: routine=“intermediate cert renew watch”
2021-05-28T13:20:58.522Z [INFO] agent.leader: started routine: routine=“CA root pruning”
2021-05-28T13:20:58.528Z [INFO] agent.server: member joined, marking health alive: member=dev-itaag108-corporatehmb01-consul-server-0
2021-05-28T13:20:58.556Z [INFO] agent.server: federation state anti-entropy synced
2021-05-28T13:20:59.049Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”
2021-05-28T13:20:59.777Z [WARN] agent: Node info update blocked by ACLs: node=9a46b178-823f-74b2-afe5-0136f533db1d accessorID=00000000-0000-0000-0000-00000000000
2
2021-05-28T13:21:00.051Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”
2021-05-28T13:21:01.051Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”
2021-05-28T13:21:02.053Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”
2021-05-28T13:21:03.054Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.38:52188 error=“ACL not found”

Any idea?

Ranjandas · May 28, 2021, 3:32pm

Hi @mattlegall,

Welcome to the forums!

Looks like you have to pass the acl.tokens.master configuration using extraConfig to the Consul server when you are using a custom bootstrap token.

So according to the values.yaml you shared, your server section will become:

server:
  enabled: true
  replicas: 1
  bootstrapExpect: 1
  extraConfig: |
    {
      "acl": {
        "tokens": {
          "master": "supersecret",
          "agent": "supersecret"
        }
      }
    }

Ref: How to bootstrap ACL using a well known master token? · Issue #569 · hashicorp/consul-helm · GitHub

mattlegall · May 31, 2021, 10:54am

Hi Ranjandas,

It worked, thanks. The only thing is that there are some acl related pods ERROR. Do you happen to know how to get rid of them?

$ oc get all -n consul-ns
NAME READY STATUS RESTARTS AGE
pod/dev-itaag108-corporatehmb01-consul-server-0 1/1 Running 0 60m
pod/dev-itaag108-corporatehmb01-consul-server-acl-init-255pc 0/1 Error 0 59m
pod/dev-itaag108-corporatehmb01-consul-server-acl-init-bc52m 0/1 Error 0 60m
pod/dev-itaag108-corporatehmb01-consul-server-acl-init-pdr2p 0/1 Error 0 59m
pod/dev-itaag108-corporatehmb01-consul-server-acl-init-vsmk5 0/1 Error 0 57m
pod/dev-itaag108-corporatehmb01-consul-server-acl-init-w4md9 0/1 Error 0 59m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
service/dev-itaag108-corporatehmb01-consul-server ClusterIP None 8500/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP
service/dev-itaag108-corporatehmb01-consul-ui ClusterIP 172.30.15.144 80/TCP
NAME READY AGE
statefulset.apps/dev-itaag108-corporatehmb01-consul-server 1/1 60m
NAME COMPLETIONS DURATION AGE
job.batch/dev-itaag108-corporatehmb01-consul-server-acl-init 0/1 60m 60m

These pods is ERROR are logging:
$ oc logs pod/dev-itaag108-corporatehmb01-consul-server-acl-init-bc52m -n consul-ns
2021-05-31T09:47:16.918Z [INFO] No bootstrap token from previous installation found, continuing on to bootstrapping
2021-05-31T09:47:17.032Z [ERROR] Failure: bootstrapping ACLs - PUT /v1/acl/bootstrap: err=“Put “http://dev-itaag108-corporatehmb01-consul-server-0.dev-itaag108-corpor
atehmb01-consul-server.consul-ns.svc:8500/v1/acl/bootstrap”: dial tcp: lookup dev-itaag108-corporatehmb01-consul-server-0.dev-itaag108-corporatehmb01-consul-server.co
nsul-ns.svc on 172.30.0.10:53: no such host”
2021-05-31T09:47:17.032Z [INFO] Retrying in 1s
…
2021-05-31T09:47:56.765Z [INFO] Retrying in 1s
2021-05-31T09:47:57.769Z [ERROR] Failure: bootstrapping ACLs - PUT /v1/acl/bootstrap: err=“no leader elected: Unexpected response code: 500 (The ACL system is current
ly in legacy mode.)”
2021-05-31T09:47:57.769Z [INFO] Retrying in 1s
…
2021-05-31T09:48:01.774Z [INFO] Success: bootstrapping ACLs - PUT /v1/acl/bootstrap
2021-05-31T09:48:01.774Z [ERROR] ACLs already bootstrapped but the ACL token was not written to a Kubernetes secret. We can’t proceed because the bootstrap token is l
ost. You must reset ACLs.

Ranjandas · May 31, 2021, 12:49pm

Sorry, I am not sure about this error. I haven’t seen this in my CRC environment when I tested the answer I shared in my previous response. Did you try a clean install after uninstalling and removing the PVC associated with the previous deployment?

mattlegall · June 2, 2021, 8:47am

Hi Ranjandas,

Yes, I just did try a clean install after uninstalling and removing the PVC and PV associated with the previous deployment but I am still getting these 6 acl init pods in error (with the same log output as in my previous note). The acl init cleanup pod is completing correctly on the other hand. Could it be openshift specific?

[eesbadmin@eesb-devops ~]$ oc get all -n consul-ns
NAME READY STATUS RESTARTS AGE
pod/dev-itaag108-corporatehmb01-consul-server-0 1/1 Running 0 6m5s
pod/dev-itaag108-corporatehmb01-consul-server-acl-init-4vczk 0/1 Error 0 4m52s
pod/dev-itaag108-corporatehmb01-consul-server-acl-init-5pvzk 0/1 Error 0 4m32s
pod/dev-itaag108-corporatehmb01-consul-server-acl-init-crsk9 0/1 Error 0 6m5s
pod/dev-itaag108-corporatehmb01-consul-server-acl-init-dcjrw 0/1 Error 0 2m32s
pod/dev-itaag108-corporatehmb01-consul-server-acl-init-fh6kf 0/1 Error 0 3m52s
pod/dev-itaag108-corporatehmb01-consul-server-acl-init-nmx27 0/1 Error 0 5m14s

Matthieu

Ranjandas · June 2, 2021, 12:19pm

@mattlegall, could you please check whether your ACL related options in values.yaml file is indented properly. I got the same error as yours when bootstrapToken wasn’t placed under acls.

The below acls section fixed it for me.

 global:
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: master-token
      secretKey: token
  enabled: false

mattlegall · June 3, 2021, 1:06pm

Hi Ranjandas,

I actually had NOT defined ANY section with bootstrapToken/secretName and secretKey.

Now I have created the secret prior running the helm chart using:
kubectl create secret generic consul-secret --from-literal=token=supersecret -n consul-ns

Then I added the section:
global:
acls:
manageSystemACLs: true
bootstrapToken:
secretName: consul-secret
secretKey: token
datacenter: {{env|lower}}-{{zone|lower}}-{{site|lower}}
enabled: false
openshift:
enabled: true

server:
enabled: true
replicas: 1
bootstrap_expect: 1
storage: 1Gi
extraConfig: |
{
“acl”: {
“tokens”: {
“master”: “supersecret”,
“agent”: “supersecret”
}
}
}

And it worked: acl working and no acl-init pods left behind in error.
Thanks.

Matthieu

Topic		Replies	Views
Enable ACL's and Generate Token - consul-bootstrap-acl-token Consul	5	1467	September 17, 2021
Found no secret Consul consul-k8s	0	276	August 29, 2023
Consul HCS Helm Deployment - Pod error Consul k8s , helm , azure , hcs	5	656	June 10, 2021
Consul-Kubernetes Consul	6	1862	March 30, 2020
Enable ACL in consul helm chart Consul k8s , acl	0	191	October 23, 2023

Consul ACL on openshift with existing bootstrapToken: not working

Related topics