Deployment not able to spin the vault containers

I have followed the steps mentioned in the link

Came till the deployment part, last before step.

After deployment, the pods status is stuck at creation.

kubectl get pods --watch
NAME                                READY   STATUS     RESTARTS   AGE
devwebapp                           0/2     Init:0/1   0          2m11s
nginx-6799fc88d8-9xnqv              1/1     Running    1          98m
vault-0                             1/1     Running    0          25m
vault-agent-injector-c5f9f8-zcv6q   1/1     Running    0          25m

So I ran the describe command and found nothing.

osboxes@osboxes:~$ kubectl describe pod devwebapp
Name:         devwebapp
Namespace:    default
Priority:     0
Node:         uday1-control-plane/172.19.0.2
Start Time:   Tue, 27 Jul 2021 15:50:56 -0400
Labels:       app=devwebapp
Annotations:  vault.hashicorp.com/agent-inject: true
              vault.hashicorp.com/agent-inject-secret-credentials.txt: secret/data/martwebapp/config
              vault.hashicorp.com/agent-inject-status: injected
              vault.hashicorp.com/role: martweb-app
Status:       Pending
IP:           10.244.0.10
IPs:
  IP:  10.244.0.10
Init Containers:
  vault-agent-init:
    Container ID:  containerd://a125495c63dc63e605146b9dd67d1e0e731c43c28e4130156d261efca2aaf54c
    Image:         vault:1.7.3
    Image ID:      docker.io/library/vault@sha256:6085e96fa42c2524eef7bf9af0cf5199da0b16964003b3f88e2b8195b6acb52b
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Running
      Started:      Tue, 27 Jul 2021 15:50:57 -0400
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      VAULT_LOG_LEVEL:   info
      VAULT_LOG_FORMAT:  standard
      VAULT_CONFIG:      eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoibWFydHdlYi1hcHAifX0sInNpbmsiOlt7InR5cGUiOiJmaWxlIiwiY29uZmlnIjp7InBhdGgiOiIvaG9tZS92YXVsdC8udmF1bHQtdG9rZW4ifX1dfSwiZXhpdF9hZnRlcl9hdXRoIjp0cnVlLCJwaWRfZmlsZSI6Ii9ob21lL3ZhdWx0Ly5waWQiLCJ2YXVsdCI6eyJhZGRyZXNzIjoiaHR0cDovL3ZhdWx0LmRlZmF1bHQuc3ZjOjgyMDAifSwidGVtcGxhdGUiOlt7ImRlc3RpbmF0aW9uIjoiL3ZhdWx0L3NlY3JldHMvY3JlZGVudGlhbHMudHh0IiwiY29udGVudHMiOiJ7eyB3aXRoIHNlY3JldCBcInNlY3JldC9kYXRhL21hcnR3ZWJhcHAvY29uZmlnXCIgfX17eyByYW5nZSAkaywgJHYgOj0gLkRhdGEgfX17eyAkayB9fToge3sgJHYgfX1cbnt7IGVuZCB9fXt7IGVuZCB9fSIsImxlZnRfZGVsaW1pdGVyIjoie3siLCJyaWdodF9kZWxpbWl0ZXIiOiJ9fSJ9XX0=
    Mounts:
      /home/vault from home-init (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9k5qp (ro)
      /vault/secrets from vault-secrets (rw)
Containers:
  devwebapp:
    Container ID:
    Image:          jweissig/app:0.0.1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9k5qp (ro)
      /vault/secrets from vault-secrets (rw)
  vault-agent:
    Container ID:
    Image:         vault:1.7.3
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      VAULT_LOG_LEVEL:   info
      VAULT_LOG_FORMAT:  standard
      VAULT_CONFIG:      eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoibWFydHdlYi1hcHAifX0sInNpbmsiOlt7InR5cGUiOiJmaWxlIiwiY29uZmlnIjp7InBhdGgiOiIvaG9tZS92YXVsdC8udmF1bHQtdG9rZW4ifX1dfSwiZXhpdF9hZnRlcl9hdXRoIjpmYWxzZSwicGlkX2ZpbGUiOiIvaG9tZS92YXVsdC8ucGlkIiwidmF1bHQiOnsiYWRkcmVzcyI6Imh0dHA6Ly92YXVsdC5kZWZhdWx0LnN2Yzo4MjAwIn0sInRlbXBsYXRlIjpbeyJkZXN0aW5hdGlvbiI6Ii92YXVsdC9zZWNyZXRzL2NyZWRlbnRpYWxzLnR4dCIsImNvbnRlbnRzIjoie3sgd2l0aCBzZWNyZXQgXCJzZWNyZXQvZGF0YS9tYXJ0d2ViYXBwL2NvbmZpZ1wiIH19e3sgcmFuZ2UgJGssICR2IDo9IC5EYXRhIH19e3sgJGsgfX06IHt7ICR2IH19XG57eyBlbmQgfX17eyBlbmQgfX0iLCJsZWZ0X2RlbGltaXRlciI6Int7IiwicmlnaHRfZGVsaW1pdGVyIjoifX0ifV19
    Mounts:
      /home/vault from home-sidecar (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9k5qp (ro)
      /vault/secrets from vault-secrets (rw)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-9k5qp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  home-init:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  home-sidecar:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  vault-secrets:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      Memory
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  7m24s  default-scheduler  Successfully assigned default/devwebapp to uday1-control-plane
  Normal  Pulled     7m24s  kubelet            Container image "vault:1.7.3" already present on machine
  Normal  Created    7m24s  kubelet            Created container vault-agent-init
  Normal  Started    7m23s  kubelet            Started container vault-agent-init

osboxes@osboxes:~$ kubectl logs devwebapp -c vault-agent-init
==> Vault agent started! Log data will stream in below:
2021-07-27T19:50:57.835Z [INFO]  sink.file: creating file sink
2021-07-27T19:50:57.836Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2021-07-27T19:50:57.837Z [INFO]  template.server: starting template server
[INFO] (runner) creating new runner (dry: false, once: false)

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.7.3
             Version Sha: 5d517c864c8f10385bf65627891bc7ef55f5e827

[INFO] (runner) creating watcher
2021-07-27T19:50:57.844Z [INFO]  sink.server: starting sink server
2021-07-27T19:50:57.844Z [INFO]  auth.handler: starting auth handler
2021-07-27T19:50:57.845Z [INFO]  auth.handler: authenticating
2021-07-27T19:51:57.847Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s
2021-07-27T19:51:58.847Z [INFO]  auth.handler: authenticating
2021-07-27T19:52:58.851Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1.91s
2021-07-27T19:53:00.774Z [INFO]  auth.handler: authenticating
2021-07-27T19:54:00.789Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=2.93s
2021-07-27T19:54:03.723Z [INFO]  auth.handler: authenticating
2021-07-27T19:55:03.724Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=4.87s
2021-07-27T19:55:08.595Z [INFO]  auth.handler: authenticating
2021-07-27T19:56:09.043Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=8.89s
2021-07-27T19:56:17.940Z [INFO]  auth.handler: authenticating
2021-07-27T19:57:17.942Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=13.86s
2021-07-27T19:57:31.811Z [INFO]  auth.handler: authenticating
2021-07-27T19:58:31.813Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=21.7s
2021-07-27T19:58:53.516Z [INFO]  auth.handler: authenticating
2021-07-27T19:59:53.521Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=39.73s
2021-07-27T20:00:33.254Z [INFO]  auth.handler: authenticating
2021-07-27T20:01:33.255Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1m7.23s
2021-07-27T20:02:40.492Z [INFO]  auth.handler: authenticating
2021-07-27T20:03:40.493Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=2m7.25s
2021-07-27T20:05:47.752Z [INFO]  auth.handler: authenticating
2021-07-27T20:06:47.756Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=3m42.25

Any suggestion on how to resolve this?

Your injector can’t authenticate to Vault’s kube auth.

Context deadline usually means “timeout”, so the injector can’t find/reach/talk to the Vault server. Fix the connection and setup the auth.

@ukreddy-erwin I know it’s over a year since this was posted but, I struggled with something similar to this same issue yesterday and today.
Here is a link that helped me: Kubernetes auth method : Permission Denied error – HashiCorp Help Center