Hi everyone, I have issues in helm chart kubernetes setup
consul client pods error:
[ERROR] agent.auto_config: AutoEncrypt.Sign RPC failed: addr=192.168.115.25:8300 error=“rpcinsecure error making call: rpcinsecure error making call: ACL not found” ERROR] agent.auto_config: AutoEncrypt.Sign RPC failed: addr=192.168.10.87:8300 error=“rpcinsecure error making call: rpcinsecure error making call: ACL not found” [ERROR] agent.auto_config: AutoEncrypt.Sign RPC failed: addr=192.168.112.206:8300 error="rpcinsecure error making call: ACL not found [ERROR] agent.auto_config: No servers successfully responded to the auto-encrypt request
consul server pods error:
[ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=192.168.10.82:34406 error=“ACL not found”
consul-server-acl-init-cleanup:
[INFO] job “consul-server-acl-init” has not yet succeeded, waiting 1s
consul-server-acl-init:
[ERROR] Failure: calling /agent/self to get datacenter: err=“Unexpected response code: 403 (ACL not found)”
After 6 tries consul-server-acl-init no longer create any jobs. All previous jobs will be in a error state
[ERROR] Error getting datacenter name: err=“reached command timeout”
Also, UI loads as below
I used below command to get the token and pasted in UI in secretID which results in an error
kubectl get secrets/consul-bootstrap-acl-token --template={{.data.token}} | base64 --decode
My config.yaml
global:
name: consul
datacenter: ***DC1
# kubectl create secret generic consul-gossip-encryption-key --from-literal=key=$(consul keygen)
# Run above command for gossip key
gossipEncryption:
secretName: 'consul-gossip-encryption-key'
secretKey: 'key'
tls:
enabled: true
# This configuration sets `verify_outgoing`, `verify_server_hostname`,
# and `verify_incoming` to `false` on servers and clients,
# which allows TLS-disabled nodes to join the cluster.
enableAutoEncrypt: true
verify: true
acls:
manageSystemACLs: true
# createReplicationToken: true
server:
replicas: 3
bootstrapExpect: 3
disruptionBudget:
enabled: true
maxUnavailable: 0
updatePartition:
securityContext:
runAsNonRoot: false
runAsUser: 0
ui:
# Add service Loadbalancer for consul ui to be on a random port. Check in kubernetes services
service:
type: "LoadBalancer"
enabled: true
connectInject:
enabled: true
controller:
enabled: true
prometheus:
enabled: true
grafana:
enabled: true
Pods Information
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
consul-6pr65 0/1 Running 0 19h 192.168.115.24 tmp-k8c1w2 <none> <none>
consul-connect-injector-webhook-deployment-69788f9b9f-mh4c5 1/1 Running 0 19h 192.168.115.28 tmp-k8c1w2 <none> <none>
consul-connect-injector-webhook-deployment-69788f9b9f-rdckj 1/1 Running 0 19h 192.168.10.83 tmp-k8c1w3 <none> <none>
consul-controller-5fcd5cd9-5pns9 1/1 Running 0 19h 192.168.10.85 tmp-k8c1w3 <none> <none>
consul-dxrjq 0/1 Running 0 19h 192.168.10.84 tmp-k8c1w3 <none> <none>
consul-gqhh6 0/1 Running 0 19h 192.168.112.203 tmp-k8c1w1 <none> <none>
consul-server-0 1/1 Running 0 19h 192.168.115.25 tmp-k8c1w2 <none> <none>
consul-server-1 1/1 Running 0 19h 192.168.10.87 tmp-k8c1w3 <none> <none>
consul-server-2 1/1 Running 0 19h 192.168.112.206 tmp-k8c1w1 <none> <none>
consul-server-acl-init-fm8xr 0/1 Error 0 19h 192.168.10.86 tmp-k8c1w3 <none> <none>
consul-server-acl-init-gvjpd 0/1 Error 0 18h 192.168.115.36 tmp-k8c1w2 <none> <none>
consul-server-acl-init-h8ks9 0/1 Error 0 19h 192.168.10.82 tmp-k8c1w3 <none> <none>
consul-server-acl-init-jjxfx 0/1 Error 0 19h 192.168.115.30 tmp-k8c1w2 <none> <none>
consul-server-acl-init-nzgtf 0/1 Error 0 19h 192.168.115.31 tmp-k8c1w2 <none> <none>
consul-server-acl-init-s9tqf 0/1 Error 0 19h 192.168.115.32 tmp-k8c1w2 <none> <none>
consul-webhook-cert-manager-674f8979c8-4qk7g 1/1 Running 0 19h 192.168.115.26 tmp-k8c1w2 <none> <none>
prometheus-server-666569649c-rc785 2/2 Running 0 19h 192.168.112.200 tmp-k8c1w1 <none> <none>
Please share your thoughts on this.
Thank you
