0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims

Hello,

I have configured the k8s cluster on AWS EC2 with one master node and two worker nodes and have installed the HashiCorp vault on those using the Helm chart.

But it’s not working, the one pod stucks in the pending state. The following is the described result of the particular Pod.

This is the documentation that I have followed.

ubuntu@k8smaster:~$ kubectl describe pod vault-0 -n vault
Name:             vault-0
Namespace:        vault
Priority:         0
Service Account:  vault
Node:             <none>
Labels:           app.kubernetes.io/instance=vault
                  app.kubernetes.io/name=vault
                  component=server
                  controller-revision-hash=vault-5d9d5864f7
                  helm.sh/chart=vault-0.23.0
                  statefulset.kubernetes.io/pod-name=vault-0
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Controlled By:    StatefulSet/vault
Containers:
  vault:
    Image:       hashicorp/vault:1.12.1
    Ports:       8200/TCP, 8201/TCP, 8202/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Command:
      /bin/sh
      -ec
    Args:
      cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
      [ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
      [ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
      [ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
      [ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
      [ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
      [ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
      /usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl

    Readiness:  exec [/bin/sh -ec vault status -tls-skip-verify] delay=5s timeout=3s period=5s #success=1 #failure=2
    Environment:
      HOST_IP:               (v1:status.hostIP)
      POD_IP:                (v1:status.podIP)
      VAULT_K8S_POD_NAME:   vault-0 (v1:metadata.name)
      VAULT_K8S_NAMESPACE:  vault (v1:metadata.namespace)
      VAULT_ADDR:           http://127.0.0.1:8200
      VAULT_API_ADDR:       http://$(POD_IP):8200
      SKIP_CHOWN:           true
      SKIP_SETCAP:          true
      HOSTNAME:             vault-0 (v1:metadata.name)
      VAULT_CLUSTER_ADDR:   https://$(HOSTNAME).vault-internal:8201
      HOME:                 /home/vault
    Mounts:
      /home/vault from home (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kk2r6 (ro)
      /vault/config from config (rw)
      /vault/data from data (rw)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-vault-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      vault-config
    Optional:  false
  home:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-kk2r6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  2m41s  default-scheduler  0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod

The main error is “0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod…

What steps should I follow to resolve this issue?

This isn’t specifically related to Vault.

Two things are happening to your cluster that make the scheduling of the pods not succeed:

  • pod has unbound immediate PersistentVolumeClaims
  • No preemption victims found for incoming pod

The first is related to your cluster not being able to bind the pod to a persistent volume (PV) that fulfills the request created by the PVC. You may need to manually provision a PV or install the correct CSI driver for your cloud provider.

The second probably means your cluster is out of capacity or the worker nodes are unschedulable. You may need to scale up your resources.

In either case, you’ll probably find better answers in Kubernetes-related forums or Stackoverflow

unable to fix this issue…
pod state still in pending state…i have checked with different kubernetes versions but both having same issue…pv is not created hence pvc is in pending state.

root@ntt-master1:/opt/vault# kubectl describe pvc data-vault-0 -n vault
Name: data-vault-0
Namespace: vault
StorageClass:
Status: Pending
Volume:
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/name=vault
component=server
Annotations:
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: vault-0
Events:
Type Reason Age From Message


Normal FailedBinding 5s (x19 over 4m29s) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
Normal FailedBinding 4s (x19 over 4m29s) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
root@ntt-master1:/opt/vault#

i have created pv manullay but still same issue…

data 5Gi RWO Retain Available vault/data-vault-0 local-storage 2m32s

audit 5Gi RWO Retain Available vault/audit-vault-0 local-storage 115s

Did you install the EBS CSI driver in your EKS cluster?

eksctl create addon --name aws-ebs-csi-driver --cluster <CLUSTER_NAME> --service-account-role-arn arn:aws:iam::<AWS_ACCOUNT_NUMBER>:role/AmazonEKS_EBS_CSI_DriverRole --force