Hi all,
I’m trying to deploy consul in the k8 cluster with 17 worker nodes.
Following is my config file:
global:
name: consul
datacenter: AcuityDC1
# kubectl create secret generic consul-gossip-encryption-key --from-literal=key=$(consul keygen)
# Run above command for gossip key
gossipEncryption:
secretName: 'consul-gossip-encryption-key'
secretKey: 'key'
tls:
enabled: true
# This configuration sets `verify_outgoing`, `verify_server_hostname`,
# and `verify_incoming` to `false` on servers and clients,
# which allows TLS-disabled nodes to join the cluster.
enableAutoEncrypt: true
verify: true
acls:
manageSystemACLs: true
# createReplicationToken: true
server:
replicas: 3
bootstrapExpect: 3
# disruptionBudget:
# enabled: true
# maxUnavailable: 0
# updatePartition:
securityContext:
runAsNonRoot: false
runAsUser: 0
nodeSelector: |
consul.client: "true"
client:
nodeSelector: |
consul.client: "true"
ui:
# Add service Loadbalancer for consul ui to be on a random port. Check in kubernetes services
service:
type: "LoadBalancer"
enabled: true
connectInject:
enabled: true
controller:
enabled: true
prometheus:
enabled: true
grafana:
enabled: true
Consul client and Consul-connect-injector-webhook-deployment:
Failed to load logs: container "consul" in pod "consul-l5rlm" is waiting to start: PodInitializing
Reason: BadRequest (400)
consul-server-acl-init
2021-09-12T15:51:01.207Z [INFO] No bootstrap token from previous installation found, continuing on to bootstrapping
2021-09-12T15:51:31.208Z [ERROR] Failure: bootstrapping ACLs - PUT /v1/acl/bootstrap: err="Put "https://consul-server-0.consul-server.default.svc:8501/v1/acl/bootstrap": dial tcp: i/o timeout"
2021-09-12T15:51:31.208Z [INFO] Retrying in 1s
2021-09-12T16:01:20.243Z [ERROR] reached command timeout
consul-webhook-cert-manager
2021-09-12T15:50:59.293Z [INFO] Updated certificate bundle received for consul-controller-mutating-webhook-configuration; Updating webhook certs.
2021-09-12T15:50:59.496Z [INFO] Creating Kubernetes secret with certificate: mutatingwebhookconfig=consul-controller-mutating-webhook-configuration secret=consul-controller-webhook-cert secretNS=default
2021-09-12T15:50:59.585Z [INFO] Updating webhook configuration: mutatingwebhookconfig=consul-controller-mutating-webhook-configuration secret=consul-controller-webhook-cert secretNS=default
2021-09-12T15:50:59.605Z [INFO] Updated certificate bundle received for consul-connect-injector-cfg; Updating webhook certs.
2021-09-12T15:50:59.786Z [INFO] Creating Kubernetes secret with certificate: mutatingwebhookconfig=consul-connect-injector-cfg secret=consul-connect-inject-webhook-cert secretNS=default
2021-09-12T15:50:59.794Z [INFO] Updating webhook configuration: mutatingwebhookconfig=consul-connect-injector-cfg secret=consul-connect-inject-webhook-cert secretNS=default
Kubernetes nodes:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
va-k8s-apps-m1 Ready control-plane,master 57d v1.21.0 10.64.116.23 <none> CentOS Linux 7 (Core) 5.4.108-1.el7.elrepo.x86_64 docker://20.10.5
va-k8s-apps-m2 Ready control-plane,master 57d v1.21.0 10.64.116.17 <none> CentOS Linux 7 (Core) 5.4.108-1.el7.elrepo.x86_64 docker://20.10.5
va-k8s-apps-m3 Ready control-plane,master 57d v1.21.0 10.64.116.9 <none> CentOS Linux 7 (Core) 5.4.108-1.el7.elrepo.x86_64 docker://20.10.5
va-k8s-apps-w1 Ready frontend,worker 57d v1.21.0 10.64.116.100 <none> Ubuntu 20.04.3 LTS 5.4.0-81-lowlatency docker://20.10.8
va-k8s-apps-w10 Ready backend,worker 45d v1.21.0 10.64.116.165 <none> Ubuntu 20.04.2 LTS 5.4.0-80-generic docker://20.10.7
va-k8s-apps-w11 Ready backend,worker 45d v1.21.0 10.64.116.183 <none> Ubuntu 20.04.2 LTS 5.4.0-80-generic docker://20.10.7
va-k8s-apps-w12 Ready backend,worker 45d v1.21.0 10.64.116.184 <none> Ubuntu 20.04.2 LTS 5.4.0-80-generic docker://20.10.7
va-k8s-apps-w13 Ready backend,worker 45d v1.21.0 10.64.116.185 <none> Ubuntu 20.04.2 LTS 5.4.0-80-generic docker://20.10.7
va-k8s-apps-w14 Ready backend,worker 45d v1.21.0 10.64.116.176 <none> Ubuntu 20.04.2 LTS 5.4.0-80-generic docker://20.10.7
va-k8s-apps-w15 Ready backend,worker 45d v1.21.0 10.64.116.187 <none> Ubuntu 20.04.2 LTS 5.4.0-80-generic docker://20.10.7
va-k8s-apps-w16 Ready <none> 37d v1.21.0 10.64.116.160 <none> Ubuntu 20.04.2 LTS 5.4.0-80-generic docker://20.10.8
va-k8s-apps-w17 Ready <none> 37d v1.21.0 10.64.116.158 <none> Ubuntu 20.04.2 LTS 5.4.0-80-generic docker://20.10.8
va-k8s-apps-w2 Ready frontend,worker 57d v1.21.0 10.64.116.156 <none> Ubuntu 20.04.3 LTS 5.4.0-81-lowlatency docker://20.10.8
va-k8s-apps-w3 Ready frontend,worker 57d v1.21.0 10.64.116.161 <none> Ubuntu 20.04.3 LTS 5.4.0-81-lowlatency docker://20.10.8
va-k8s-apps-w4 Ready ingress,worker 48d v1.21.0 10.64.116.159 <none> Ubuntu 20.04.2 LTS 5.8.0-63-lowlatency docker://20.10.7
va-k8s-apps-w5 Ready backend,worker 46d v1.21.0 10.64.116.175 <none> Ubuntu 20.04.2 LTS 5.4.0-80-generic docker://20.10.7
va-k8s-apps-w6 Ready backend,worker 46d v1.21.0 10.64.116.172 <none> Ubuntu 20.04.2 LTS 5.4.0-80-lowlatency docker://20.10.7
va-k8s-apps-w7 Ready backend,worker 46d v1.21.0 10.64.116.168 <none> Ubuntu 20.04.2 LTS 5.4.0-80-lowlatency docker://20.10.7
va-k8s-apps-w8 Ready backend,worker 46d v1.21.0 10.64.116.162 <none> Ubuntu 20.04.2 LTS 5.4.0-81-lowlatency docker://20.10.7
va-k8s-apps-w9 Ready backend,worker 46d v1.21.0 10.64.116.174 <none> Ubuntu 20.04.2 LTS 5.4.0-81-lowlatency docker://20.10.7
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-default-consul-server-0 Bound data-consul-pv0 10Gi RWO 18m
data-default-consul-server-1 Bound data-consul-pv1 10Gi RWO 18m
data-default-consul-server-2 Bound data-consul-pv2 10Gi RWO 18m
Consul-server
2021-09-12T15:51:49.303Z [INFO] agent.server.serf.lan: serf: EventMemberUpdate: consul-server-1
2021-09-12T15:51:49.303Z [INFO] agent.server: Updating LAN server: server="consul-server-1 (Addr: tcp/172.29.153.17:8300) (DC: acuitydc1)"
2021-09-12T15:51:49.702Z [INFO] agent.server.serf.wan: serf: EventMemberUpdate: consul-server-1.acuitydc1
2021-09-12T15:51:49.702Z [INFO] agent.server: Handled event for server in area: event=member-update server=consul-server-1.acuitydc1 area=wan
2021-09-12T15:51:50.583Z [INFO] agent.server.serf.lan: serf: EventMemberUpdate: consul-server-0
2021-09-12T15:51:50.583Z [INFO] agent.server: Updating LAN server: server="consul-server-0 (Addr: tcp/172.29.2.207:8300) (DC: acuitydc1)"
2021-09-12T15:51:50.888Z [WARN] agent: Node info update blocked by ACLs: node=11cc3111-73ed-79d4-6d4c-459b84939c5f accessorID=00000000-0000-0000-0000-000000000002
2021-09-12T15:51:51.182Z [INFO] agent.server.serf.wan: serf: EventMemberUpdate: consul-server-0.acuitydc1
2021-09-12T15:51:51.182Z [INFO] agent.server: Handled event for server in area: event=member-update server=consul-server-0.acuitydc1 area=wan
2021-09-12T15:51:56.247Z [INFO] agent.server.serf.lan: serf: EventMemberUpdate: consul-server-2
2021-09-12T15:51:56.247Z [INFO] agent.server: Updating LAN server: server="consul-server-2 (Addr: tcp/172.29.192.39:8300) (DC: acuitydc1)"
2021-09-12T15:51:56.643Z [INFO] agent.server.serf.wan: serf: EventMemberUpdate: consul-server-2.acuitydc1
2021-09-12T15:51:56.643Z [INFO] agent.server: Handled event for server in area: event=member-update server=consul-server-2.acuitydc1 area=wan
2021-09-12T15:52:03.676Z [WARN] agent: Coordinate update blocked by ACLs: accessorID=00000000-0000-0000-0000-000000000002
2021-09-12T15:52:29.073Z [WARN] agent: Coordinate update blocked by ACLs: accessorID=00000000-0000-0000-0000-000000000002
2021-09-12T15:52:58.351Z [WARN] agent: Coordinate update blocked by ACLs: accessorID=00000000-0000-0000-0000-000000000002
2021-09-12T15:53:05.044Z [WARN] agent: grpc: addrConn.createTransport failed to connect to {172.29.153.17:8300 0 consul-server-1.acuitydc1 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 172.29.153.17:8300: operation was canceled". Reconnecting...
What am I missing in my config that this deployment is not successful?