0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims

Hello,

I have configured the k8s cluster on AWS EC2 with one master node and two worker nodes and have installed the HashiCorp vault on those using the Helm chart.

But it’s not working, the one pod stucks in the pending state. The following is the described result of the particular Pod.

This is the documentation that I have followed.

ubuntu@k8smaster:~$ kubectl describe pod vault-0 -n vault
Name:             vault-0
Namespace:        vault
Priority:         0
Service Account:  vault
Node:             <none>
Labels:           app.kubernetes.io/instance=vault
                  app.kubernetes.io/name=vault
                  component=server
                  controller-revision-hash=vault-5d9d5864f7
                  helm.sh/chart=vault-0.23.0
                  statefulset.kubernetes.io/pod-name=vault-0
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Controlled By:    StatefulSet/vault
Containers:
  vault:
    Image:       hashicorp/vault:1.12.1
    Ports:       8200/TCP, 8201/TCP, 8202/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Command:
      /bin/sh
      -ec
    Args:
      cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
      [ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
      [ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
      [ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
      [ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
      [ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
      [ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
      /usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl

    Readiness:  exec [/bin/sh -ec vault status -tls-skip-verify] delay=5s timeout=3s period=5s #success=1 #failure=2
    Environment:
      HOST_IP:               (v1:status.hostIP)
      POD_IP:                (v1:status.podIP)
      VAULT_K8S_POD_NAME:   vault-0 (v1:metadata.name)
      VAULT_K8S_NAMESPACE:  vault (v1:metadata.namespace)
      VAULT_ADDR:           http://127.0.0.1:8200
      VAULT_API_ADDR:       http://$(POD_IP):8200
      SKIP_CHOWN:           true
      SKIP_SETCAP:          true
      HOSTNAME:             vault-0 (v1:metadata.name)
      VAULT_CLUSTER_ADDR:   https://$(HOSTNAME).vault-internal:8201
      HOME:                 /home/vault
    Mounts:
      /home/vault from home (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kk2r6 (ro)
      /vault/config from config (rw)
      /vault/data from data (rw)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-vault-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      vault-config
    Optional:  false
  home:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-kk2r6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  2m41s  default-scheduler  0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod

The main error is “0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod…

What steps should I follow to resolve this issue?

This isn’t specifically related to Vault.

Two things are happening to your cluster that make the scheduling of the pods not succeed:

  • pod has unbound immediate PersistentVolumeClaims
  • No preemption victims found for incoming pod

The first is related to your cluster not being able to bind the pod to a persistent volume (PV) that fulfills the request created by the PVC. You may need to manually provision a PV or install the correct CSI driver for your cloud provider.

The second probably means your cluster is out of capacity or the worker nodes are unschedulable. You may need to scale up your resources.

In either case, you’ll probably find better answers in Kubernetes-related forums or Stackoverflow