Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Try the new cross-platform PowerShell https://aka.ms/pscore6
PS C:\Users\kiran> kubectl get deploy -n consul
NAME READY UP-TO-DATE AVAILABLE AGE
consul-connect-injector-webhook-deployment 0/2 2 0 155m
consul-controller 0/1 1 0 155m
consul-webhook-cert-manager 0/1 1 0 155m
PS C:\Users\kiran> kubectl get po -n consul
NAME READY STATUS RESTARTS AGE
consul-2jfmc 1/1 Running 0 155m
consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s 0/1 ContainerCreating 0 155m
consul-connect-injector-webhook-deployment-5d6b98587c-prq7c 0/1 ContainerCreating 0 155m
consul-controller-dff49c9f4-99tmj 0/1 ContainerCreating 0 155m
consul-server-0 1/1 Running 0 155m
consul-sync-catalog-78998c5f4-vvdp5 1/1 Running 0 155m
consul-webhook-cert-manager-56cdbb7648-7j654 0/1 CrashLoopBackOff 35 (93s ago) 155m
PS C:\Users\kiran> kubectl get po -n consul
NAME READY STATUS RESTARTS AGE
consul-2jfmc 1/1 Running 0 156m
consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s 0/1 ContainerCreating 0 156m
consul-connect-injector-webhook-deployment-5d6b98587c-prq7c 0/1 ContainerCreating 0 156m
consul-controller-dff49c9f4-99tmj 0/1 ContainerCreating 0 156m
consul-server-0 1/1 Running 0 156m
consul-sync-catalog-78998c5f4-vvdp5 1/1 Running 0 156m
consul-webhook-cert-manager-56cdbb7648-7j654 0/1 CrashLoopBackOff 35 (116s ago) 156m
PS C:\Users\kiran> kubectl get po -n consul
NAME READY STATUS RESTARTS AGE
consul-2jfmc 1/1 Running 0 173m
consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s 0/1 ContainerCreating 0 173m
consul-connect-injector-webhook-deployment-5d6b98587c-prq7c 0/1 ContainerCreating 0 173m
consul-controller-dff49c9f4-99tmj 0/1 ContainerCreating 0 173m
consul-server-0 1/1 Running 0 173m
consul-sync-catalog-78998c5f4-vvdp5 1/1 Running 0 173m
consul-webhook-cert-manager-56cdbb7648-7j654 0/1 CrashLoopBackOff 38 (4m16s ago) 173m
PS C:\Users\kiran> kubectl describe po consul-webhook-cert-manager-56cdbb7648-7j654 -n consul
Name: consul-webhook-cert-manager-56cdbb7648-7j654
Namespace: consul
Priority: 0
Node: aks-systempool-27136238-vmss000000/10.240.0.4
Start Time: Fri, 14 Jan 2022 12:08:48 +0000
Labels: app=consul
chart=consul-helm
component=webhook-cert-manager
heritage=Helm
pod-template-hash=56cdbb7648
release=consul
Annotations: consul.hashicorp.com/config-checksum: 44f20d3c49318074ca5a4aef932fc051358ba926a51ae01fd5b2fc9ea9cd5769
consul.hashicorp.com/connect-inject: false
Status: Running
IP: 10.240.0.15
IPs:
IP: 10.240.0.15
Controlled By: ReplicaSet/consul-webhook-cert-manager-56cdbb7648
Containers:
webhook-cert-manager:
Container ID: containerd://44cf5ff7d98d31326cea9cc17b40f1a51b44ab1238bb38fce46089917e849f2c
Image: hashicorp/consul-k8s:0.26.0
Image ID: docker.io/hashicorp/consul-k8s@sha256:16c8066aeb1d85b1b3e72e7a3a2c19f3f9b2c2742201d97a668ffb2657efd32f
Port: <none>
Host Port: <none>
Command:
/bin/sh
-ec
consul-k8s webhook-cert-manager \
-config-file=/bootstrap/config/webhook-config.json \
-deployment-name=consul-webhook-cert-manager \
-deployment-namespace=consul
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 14 Jan 2022 14:58:26 +0000
Finished: Fri, 14 Jan 2022 14:58:27 +0000
Ready: False
Restart Count: 38
Limits:
cpu: 100m
memory: 50Mi
Requests:
cpu: 100m
memory: 50Mi
Environment: <none>
Mounts:
/bootstrap/config from config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mf7qh (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: consul-webhook-cert-manager-config
Optional: false
kube-api-access-mf7qh:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 4m12s (x781 over 174m) kubelet Back-off restarting failed container
PS C:\Users\kiran> kubectl logs consul-webhook-cert-manager-56cdbb7648-7j654 -n consul
Error parsing config at index 0: MutatingWebhookConfiguration with name "consul-connect-injector-cfg" must exist in cluster
PS C:\Users\kiran> kubectl get po -n consul
NAME READY STATUS RESTARTS AGE
consul-2jfmc 1/1 Running 0 175m
consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s 0/1 ContainerCreating 0 175m
consul-connect-injector-webhook-deployment-5d6b98587c-prq7c 0/1 ContainerCreating 0 175m
consul-controller-dff49c9f4-99tmj 0/1 ContainerCreating 0 175m
consul-server-0 1/1 Running 0 175m
consul-sync-catalog-78998c5f4-vvdp5 1/1 Running 0 175m
consul-webhook-cert-manager-56cdbb7648-7j654 0/1 CrashLoopBackOff 39 (20s ago) 175m
PS C:\Users\kiran> kubectl describe po consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s -n consul
Name: consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s
Namespace: consul
Priority: 0
Node: aks-systempool-27136238-vmss000000/10.240.0.4
Start Time: Fri, 14 Jan 2022 12:08:48 +0000
Labels: app=consul
chart=consul-helm
component=connect-injector
pod-template-hash=5d6b98587c
release=consul
Annotations: consul.hashicorp.com/connect-inject: false
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/consul-connect-injector-webhook-deployment-5d6b98587c
Containers:
sidecar-injector:
Container ID:
Image: hashicorp/consul-k8s:0.26.0
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
Command:
/bin/sh
-ec
CONSUL_FULLNAME="consul"
consul-k8s inject-connect \
-default-inject=true \
-consul-image="hashicorp/consul:1.10.0" \
-envoy-image="envoyproxy/envoy-alpine:v1.18.3" \
-consul-k8s-image="hashicorp/consul-k8s:0.26.0" \
-release-name="consul" \
-release-namespace="consul" \
-listen=:8080 \
-default-enable-transparent-proxy=true \
-transparent-proxy-default-overwrite-probes=true \
-log-level=info \
-default-enable-metrics=false \
-default-enable-metrics-merging=false \
-default-merged-metrics-port=20100 \
-default-prometheus-scrape-port=20200 \
-default-prometheus-scrape-path="/metrics" \
-allow-k8s-namespace="*" \
-tls-cert-dir=/etc/connect-injector/certs \
-init-container-memory-limit=150Mi \
-init-container-memory-request=25Mi \
-init-container-cpu-limit=50m \
-consul-sidecar-memory-limit=50Mi \
-consul-sidecar-cpu-limit=20m \
-consul-sidecar-cpu-request=20m \
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
memory: 50Mi
cpu: 50m
memory: 50Mi
Environment:
NAMESPACE: consul (v1:metadata.namespace)
HOST_IP: (v1:status.hostIP)
CONSUL_HTTP_ADDR: http://$(HOST_IP):8500
Mounts:
/etc/connect-injector/certs from certs (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n6k98 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
certs:
Type: Secret (a volume populated by a Secret)
SecretName: consul-connect-inject-webhook-cert
Optional: false
kube-api-access-n6k98:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 19m (x49 over 173m) kubelet Unable to attach or mount volumes: unmounted volumes=[certs], unattached volumes=[certs kube-api-access-n6k98]: timed out waiting for the condition
Warning FailedMount 10m (x22 over 166m) kubelet Unable to attach or mount volumes: unmounted volumes=[certs], unattached volumes=[kube-api-access-n6k98 certs]: timed out waiting for the condition
Warning FailedMount 32s (x94 over 175m) kubelet MountVolume.SetUp failed for volume "certs" : secret "consul-connect-inject-webhook-cert" not found
PS C:\Users\kiran> kubectl logs consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s -n consul
Error from server (BadRequest): container "sidecar-injector" in pod "consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s" is waiting to start: ContainerCreating
PS C:\Users\kiran> kubectl get po -n consul
NAME READY STATUS RESTARTS AGE
consul-2jfmc 1/1 Running 0 176m
consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s 0/1 ContainerCreating 0 176m
consul-connect-injector-webhook-deployment-5d6b98587c-prq7c 0/1 ContainerCreating 0 176m
consul-controller-dff49c9f4-99tmj 0/1 ContainerCreating 0 176m
consul-server-0 1/1 Running 0 176m
consul-sync-catalog-78998c5f4-vvdp5 1/1 Running 0 176m
consul-webhook-cert-manager-56cdbb7648-7j654 0/1 CrashLoopBackOff 39 (117s ago) 176m
PS C:\Users\kiran> kubectl logs consul-connect-injector-webhook-deployment-5d6b98587c-prq7c -n consul
Error from server (BadRequest): container "sidecar-injector" in pod "consul-connect-injector-webhook-deployment-5d6b98587c-prq7c" is waiting to start: ContainerCreating
PS C:\Users\kiran> kubectl describe po consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s -n consul
Name: consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s
Namespace: consul
Priority: 0
Node: aks-systempool-27136238-vmss000000/10.240.0.4
Start Time: Fri, 14 Jan 2022 12:08:48 +0000
Labels: app=consul
chart=consul-helm
component=connect-injector
pod-template-hash=5d6b98587c
release=consul
Annotations: consul.hashicorp.com/connect-inject: false
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/consul-connect-injector-webhook-deployment-5d6b98587c
Containers:
sidecar-injector:
Container ID:
Image: hashicorp/consul-k8s:0.26.0
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
Command:
/bin/sh
-ec
CONSUL_FULLNAME="consul"
consul-k8s inject-connect \
-default-inject=true \
-consul-image="hashicorp/consul:1.10.0" \
-envoy-image="envoyproxy/envoy-alpine:v1.18.3" \
-consul-k8s-image="hashicorp/consul-k8s:0.26.0" \
-release-name="consul" \
-release-namespace="consul" \
-listen=:8080 \
-default-enable-transparent-proxy=true \
-transparent-proxy-default-overwrite-probes=true \
-log-level=info \
-default-enable-metrics=false \
-default-enable-metrics-merging=false \
-default-merged-metrics-port=20100 \
-default-prometheus-scrape-port=20200 \
-default-prometheus-scrape-path="/metrics" \
-allow-k8s-namespace="*" \
-tls-cert-dir=/etc/connect-injector/certs \
-init-container-memory-limit=150Mi \
-init-container-memory-request=25Mi \
-init-container-cpu-limit=50m \
-consul-sidecar-memory-limit=50Mi \
-consul-sidecar-memory-request=25Mi \
-consul-sidecar-cpu-limit=20m \
-consul-sidecar-cpu-request=20m \
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
cpu: 50m
memory: 50Mi
Requests:
cpu: 50m
memory: 50Mi
Environment:
NAMESPACE: consul (v1:metadata.namespace)
HOST_IP: (v1:status.hostIP)
CONSUL_HTTP_ADDR: http://$(HOST_IP):8500
Mounts:
/etc/connect-injector/certs from certs (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n6k98 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
certs:
Type: Secret (a volume populated by a Secret)
SecretName: consul-connect-inject-webhook-cert
Optional: false
kube-api-access-n6k98:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 20m (x49 over 175m) kubelet Unable to attach or mount volumes: unmounted volumes=[certs], unattached volumes=[certs kube-api-access-n6k98]: timed out waiting for the condition
Warning FailedMount 11m (x22 over 168m) kubelet Unable to attach or mount volumes: unmounted volumes=[certs], unattached volumes=[kube-api-access-n6k98 certs]: timed out waiting for the condition
Warning FailedMount 2m12s (x94 over 177m) kubelet MountVolume.SetUp failed for volume "certs" : secret "consul-connect-inject-webhook-cert" not found
PS C:\Users\kiran> kubectl get po -n consul
NAME READY STATUS RESTARTS AGE
consul-2jfmc 1/1 Running 0 177m
consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s 0/1 ContainerCreating 0 177m
consul-connect-injector-webhook-deployment-5d6b98587c-prq7c 0/1 ContainerCreating 0 177m
consul-controller-dff49c9f4-99tmj 0/1 ContainerCreating 0 177m
consul-server-0 1/1 Running 0 177m
consul-sync-catalog-78998c5f4-vvdp5 1/1 Running 0 177m
consul-webhook-cert-manager-56cdbb7648-7j654 0/1 CrashLoopBackOff 39 (2m53s ago) 177m
PS C:\Users\kiran>
Hey @ukreddy-erwin
It looks like the root of the error is that there’s no consul-connect-injector-cfg mutating webhook configuration in your cluster. It should be created when you helm install, so I’m not sure how/why it got lost (if it doesn’t exist). You can check if it exists by running kubectl get mutatingwebhookconfiguration.
Perhaps trying to reinstall fresh could help. I think running a helm upgrade should also recreate the webhook configuration if it doesn’t exist.
A bit late, but I’ll write it here anyway:
I’ve been struggling with setting up consul using helm the last few days (Rancher Desktop k3s), and this was one of the issues I had (or atleast very similar).
Most/all of my problems went away as soon as I changed my context to target namespace, before installing chart:
# Set default namespace
kubectl config set-context --current --namespace=consul
# Apply helm-chart (using kustomize)
kubectl kustomize --enable-helm v1.24.x/ | kubectl apply --dry-run=none -f -
I have no idea why this helped, but maybe som resources isn’t created in the correct namespace?
It fixed the deployment-issues I had with with the following:
- global.acls.manageSystemACLs
- globals.gossipEncryption.autoGenerate
- controller
- connectInject