Consul pods are not running, how to fix those?

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Try the new cross-platform PowerShell https://aka.ms/pscore6
PS C:\Users\kiran> kubectl get deploy -n consul
NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
consul-connect-injector-webhook-deployment   0/2     2            0           155m
consul-controller                            0/1     1            0           155m
consul-webhook-cert-manager                  0/1     1            0           155m
PS C:\Users\kiran> kubectl get po -n consul
NAME                                                          READY   STATUS              RESTARTS       AGE
consul-2jfmc                                                  1/1     Running             0              155m
consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s   0/1     ContainerCreating   0              155m
consul-connect-injector-webhook-deployment-5d6b98587c-prq7c   0/1     ContainerCreating   0              155m
consul-controller-dff49c9f4-99tmj                             0/1     ContainerCreating   0              155m
consul-server-0                                               1/1     Running             0              155m
consul-sync-catalog-78998c5f4-vvdp5                           1/1     Running             0              155m
consul-webhook-cert-manager-56cdbb7648-7j654                  0/1     CrashLoopBackOff    35 (93s ago)   155m
PS C:\Users\kiran> kubectl get po -n consul
NAME                                                          READY   STATUS              RESTARTS        AGE
consul-2jfmc                                                  1/1     Running             0               156m
consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s   0/1     ContainerCreating   0               156m
consul-connect-injector-webhook-deployment-5d6b98587c-prq7c   0/1     ContainerCreating   0               156m
consul-controller-dff49c9f4-99tmj                             0/1     ContainerCreating   0               156m
consul-server-0                                               1/1     Running             0               156m
consul-sync-catalog-78998c5f4-vvdp5                           1/1     Running             0               156m
consul-webhook-cert-manager-56cdbb7648-7j654                  0/1     CrashLoopBackOff    35 (116s ago)   156m
PS C:\Users\kiran> kubectl get po -n consul
NAME                                                          READY   STATUS              RESTARTS         AGE
consul-2jfmc                                                  1/1     Running             0                173m
consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s   0/1     ContainerCreating   0                173m
consul-connect-injector-webhook-deployment-5d6b98587c-prq7c   0/1     ContainerCreating   0                173m
consul-controller-dff49c9f4-99tmj                             0/1     ContainerCreating   0                173m
consul-server-0                                               1/1     Running             0                173m
consul-sync-catalog-78998c5f4-vvdp5                           1/1     Running             0                173m
consul-webhook-cert-manager-56cdbb7648-7j654                  0/1     CrashLoopBackOff    38 (4m16s ago)   173m
PS C:\Users\kiran> kubectl describe po consul-webhook-cert-manager-56cdbb7648-7j654 -n consul
Name:         consul-webhook-cert-manager-56cdbb7648-7j654
Namespace:    consul
Priority:     0
Node:         aks-systempool-27136238-vmss000000/10.240.0.4
Start Time:   Fri, 14 Jan 2022 12:08:48 +0000
Labels:       app=consul
              chart=consul-helm
              component=webhook-cert-manager
              heritage=Helm
              pod-template-hash=56cdbb7648
              release=consul
Annotations:  consul.hashicorp.com/config-checksum: 44f20d3c49318074ca5a4aef932fc051358ba926a51ae01fd5b2fc9ea9cd5769
              consul.hashicorp.com/connect-inject: false
Status:       Running
IP:           10.240.0.15
IPs:
  IP:           10.240.0.15
Controlled By:  ReplicaSet/consul-webhook-cert-manager-56cdbb7648
Containers:
  webhook-cert-manager:
    Container ID:  containerd://44cf5ff7d98d31326cea9cc17b40f1a51b44ab1238bb38fce46089917e849f2c
    Image:         hashicorp/consul-k8s:0.26.0
    Image ID:      docker.io/hashicorp/consul-k8s@sha256:16c8066aeb1d85b1b3e72e7a3a2c19f3f9b2c2742201d97a668ffb2657efd32f
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
      consul-k8s webhook-cert-manager \
        -config-file=/bootstrap/config/webhook-config.json \
        -deployment-name=consul-webhook-cert-manager \
        -deployment-namespace=consul

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 14 Jan 2022 14:58:26 +0000
      Finished:     Fri, 14 Jan 2022 14:58:27 +0000
    Ready:          False
    Restart Count:  38
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:        100m
      memory:     50Mi
    Environment:  <none>
    Mounts:
      /bootstrap/config from config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mf7qh (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      consul-webhook-cert-manager-config
    Optional:  false
  kube-api-access-mf7qh:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Warning  BackOff  4m12s (x781 over 174m)  kubelet  Back-off restarting failed container
PS C:\Users\kiran> kubectl logs consul-webhook-cert-manager-56cdbb7648-7j654 -n consul
Error parsing config at index 0: MutatingWebhookConfiguration with name "consul-connect-injector-cfg" must exist in cluster
PS C:\Users\kiran> kubectl get po -n consul
NAME                                                          READY   STATUS              RESTARTS       AGE
consul-2jfmc                                                  1/1     Running             0              175m
consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s   0/1     ContainerCreating   0              175m
consul-connect-injector-webhook-deployment-5d6b98587c-prq7c   0/1     ContainerCreating   0              175m
consul-controller-dff49c9f4-99tmj                             0/1     ContainerCreating   0              175m
consul-server-0                                               1/1     Running             0              175m
consul-sync-catalog-78998c5f4-vvdp5                           1/1     Running             0              175m
consul-webhook-cert-manager-56cdbb7648-7j654                  0/1     CrashLoopBackOff    39 (20s ago)   175m
PS C:\Users\kiran> kubectl describe po consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s -n consul
Name:           consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s
Namespace:      consul
Priority:       0
Node:           aks-systempool-27136238-vmss000000/10.240.0.4
Start Time:     Fri, 14 Jan 2022 12:08:48 +0000
Labels:         app=consul
                chart=consul-helm
                component=connect-injector
                pod-template-hash=5d6b98587c
                release=consul
Annotations:    consul.hashicorp.com/connect-inject: false
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/consul-connect-injector-webhook-deployment-5d6b98587c
Containers:
  sidecar-injector:
    Container ID:
    Image:         hashicorp/consul-k8s:0.26.0
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/sh
      -ec
      CONSUL_FULLNAME="consul"

      consul-k8s inject-connect \
        -default-inject=true \
        -consul-image="hashicorp/consul:1.10.0" \
        -envoy-image="envoyproxy/envoy-alpine:v1.18.3" \
        -consul-k8s-image="hashicorp/consul-k8s:0.26.0" \
        -release-name="consul" \
        -release-namespace="consul" \
        -listen=:8080 \
        -default-enable-transparent-proxy=true \
        -transparent-proxy-default-overwrite-probes=true \
        -log-level=info \
        -default-enable-metrics=false \
        -default-enable-metrics-merging=false  \
        -default-merged-metrics-port=20100 \
        -default-prometheus-scrape-port=20200 \
        -default-prometheus-scrape-path="/metrics" \
        -allow-k8s-namespace="*" \
        -tls-cert-dir=/etc/connect-injector/certs \
        -init-container-memory-limit=150Mi \
        -init-container-memory-request=25Mi \
        -init-container-cpu-limit=50m \
        -consul-sidecar-memory-limit=50Mi \
        -consul-sidecar-cpu-limit=20m \
        -consul-sidecar-cpu-request=20m \

    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  50Mi
      cpu:     50m
      memory:  50Mi
    Environment:
      NAMESPACE:         consul (v1:metadata.namespace)
      HOST_IP:            (v1:status.hostIP)
      CONSUL_HTTP_ADDR:  http://$(HOST_IP):8500
    Mounts:
      /etc/connect-injector/certs from certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n6k98 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  consul-connect-inject-webhook-cert
    Optional:    false
  kube-api-access-n6k98:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From     Message
  ----     ------       ----                 ----     -------
  Warning  FailedMount  19m (x49 over 173m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[certs], unattached volumes=[certs kube-api-access-n6k98]: timed out waiting for the condition
  Warning  FailedMount  10m (x22 over 166m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[certs], unattached volumes=[kube-api-access-n6k98 certs]: timed out waiting for the condition
  Warning  FailedMount  32s (x94 over 175m)  kubelet  MountVolume.SetUp failed for volume "certs" : secret "consul-connect-inject-webhook-cert" not found
PS C:\Users\kiran> kubectl logs consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s -n consul
Error from server (BadRequest): container "sidecar-injector" in pod "consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s" is waiting to start: ContainerCreating
PS C:\Users\kiran> kubectl get po -n consul
NAME                                                          READY   STATUS              RESTARTS        AGE
consul-2jfmc                                                  1/1     Running             0               176m
consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s   0/1     ContainerCreating   0               176m
consul-connect-injector-webhook-deployment-5d6b98587c-prq7c   0/1     ContainerCreating   0               176m
consul-controller-dff49c9f4-99tmj                             0/1     ContainerCreating   0               176m
consul-server-0                                               1/1     Running             0               176m
consul-sync-catalog-78998c5f4-vvdp5                           1/1     Running             0               176m
consul-webhook-cert-manager-56cdbb7648-7j654                  0/1     CrashLoopBackOff    39 (117s ago)   176m
PS C:\Users\kiran> kubectl logs consul-connect-injector-webhook-deployment-5d6b98587c-prq7c -n consul
Error from server (BadRequest): container "sidecar-injector" in pod "consul-connect-injector-webhook-deployment-5d6b98587c-prq7c" is waiting to start: ContainerCreating
PS C:\Users\kiran> kubectl describe po consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s -n consul
Name:           consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s
Namespace:      consul
Priority:       0
Node:           aks-systempool-27136238-vmss000000/10.240.0.4
Start Time:     Fri, 14 Jan 2022 12:08:48 +0000
Labels:         app=consul
                chart=consul-helm
                component=connect-injector
                pod-template-hash=5d6b98587c
                release=consul
Annotations:    consul.hashicorp.com/connect-inject: false
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/consul-connect-injector-webhook-deployment-5d6b98587c
Containers:
  sidecar-injector:
    Container ID:
    Image:         hashicorp/consul-k8s:0.26.0
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/sh
      -ec
      CONSUL_FULLNAME="consul"

      consul-k8s inject-connect \
        -default-inject=true \
        -consul-image="hashicorp/consul:1.10.0" \
        -envoy-image="envoyproxy/envoy-alpine:v1.18.3" \
        -consul-k8s-image="hashicorp/consul-k8s:0.26.0" \
        -release-name="consul" \
        -release-namespace="consul" \
        -listen=:8080 \
        -default-enable-transparent-proxy=true \
        -transparent-proxy-default-overwrite-probes=true \
        -log-level=info \
        -default-enable-metrics=false \
        -default-enable-metrics-merging=false  \
        -default-merged-metrics-port=20100 \
        -default-prometheus-scrape-port=20200 \
        -default-prometheus-scrape-path="/metrics" \
        -allow-k8s-namespace="*" \
        -tls-cert-dir=/etc/connect-injector/certs \
        -init-container-memory-limit=150Mi \
        -init-container-memory-request=25Mi \
        -init-container-cpu-limit=50m \
        -consul-sidecar-memory-limit=50Mi \
        -consul-sidecar-memory-request=25Mi \
        -consul-sidecar-cpu-limit=20m \
        -consul-sidecar-cpu-request=20m \

    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     50m
      memory:  50Mi
    Requests:
      cpu:     50m
      memory:  50Mi
    Environment:
      NAMESPACE:         consul (v1:metadata.namespace)
      HOST_IP:            (v1:status.hostIP)
      CONSUL_HTTP_ADDR:  http://$(HOST_IP):8500
    Mounts:
      /etc/connect-injector/certs from certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n6k98 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  consul-connect-inject-webhook-cert
    Optional:    false
  kube-api-access-n6k98:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                    From     Message
  ----     ------       ----                   ----     -------
  Warning  FailedMount  20m (x49 over 175m)    kubelet  Unable to attach or mount volumes: unmounted volumes=[certs], unattached volumes=[certs kube-api-access-n6k98]: timed out waiting for the condition
  Warning  FailedMount  11m (x22 over 168m)    kubelet  Unable to attach or mount volumes: unmounted volumes=[certs], unattached volumes=[kube-api-access-n6k98 certs]: timed out waiting for the condition
  Warning  FailedMount  2m12s (x94 over 177m)  kubelet  MountVolume.SetUp failed for volume "certs" : secret "consul-connect-inject-webhook-cert" not found
PS C:\Users\kiran> kubectl get po -n consul
NAME                                                          READY   STATUS              RESTARTS         AGE
consul-2jfmc                                                  1/1     Running             0                177m
consul-connect-injector-webhook-deployment-5d6b98587c-8sq2s   0/1     ContainerCreating   0                177m
consul-connect-injector-webhook-deployment-5d6b98587c-prq7c   0/1     ContainerCreating   0                177m
consul-controller-dff49c9f4-99tmj                             0/1     ContainerCreating   0                177m
consul-server-0                                               1/1     Running             0                177m
consul-sync-catalog-78998c5f4-vvdp5                           1/1     Running             0                177m
consul-webhook-cert-manager-56cdbb7648-7j654                  0/1     CrashLoopBackOff    39 (2m53s ago)   177m
PS C:\Users\kiran>

Hey @ukreddy-erwin

It looks like the root of the error is that there’s no consul-connect-injector-cfg mutating webhook configuration in your cluster. It should be created when you helm install, so I’m not sure how/why it got lost (if it doesn’t exist). You can check if it exists by running kubectl get mutatingwebhookconfiguration.

Perhaps trying to reinstall fresh could help. I think running a helm upgrade should also recreate the webhook configuration if it doesn’t exist.