Warning : Readiness probe failed: Key Value Seal Type shamir Initialized true Sealed true Total Shares 5

Vault
, ,
1

I have 3 nodes of vault running on k8s , everything was fine, and suddenly today
i have event warning that says :

Readiness probe failed: Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 0/3 Unseal Nonce n/a Version 1.6.1 Storage Type raft HA Enabled true

when i look at the node-1 and node-2 logs i can see that the server is up and running

==> Vault server configuration:

Api Address: https://10.xxx.0.xxx:8200

Cgo: disabled

Cluster Address: https://vault-1.vault-internal:8201

Go Version: go1.15.4

Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")

Log Level: info

Mlock: supported: true, enabled: false

Recovery Mode: false

Storage: raft (HA available)

Version: Vault v1.6.1

Version Sha: 6d2db3f033e02e70xxxx360062b88b03

==> Vault server started! Log data will stream in below:

2021-01-26T10:11:14.437Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=

also here is the pod describe:

$ kubectl describe pod vault-1 -n vault-foo
Name:         vault-1
Namespace:    vault-foo
Priority:     0
Node:         ip-10-101-0-98.ec2.internal/xxx.xxx.0.98
Start Time:   Tue, 26 Jan 2021 12:11:05 +0200
Labels:       app.kubernetes.io/instance=vault
              app.kubernetes.io/name=vault
              component=server
              controller-revision-hash=vault-7694f4b78c
              helm.sh/chart=vault-0.9.0
              statefulset.kubernetes.io/pod-name=vault-1
              vault-active=false
              vault-initialized=false
              vault-perf-standby=false
              vault-sealed=true
              vault-version=1.6.1
Annotations:  kubernetes.io/psp: eks.privileged
Status:       Running
IP:           xxx.xxx.0.191
IPs:
  IP:           xxx.xxx.0.191
Controlled By:  StatefulSet/vault
Containers:
  vault:
    Container ID:  docker://077b501aef3eaeb5f9e75dc144f288d51dbff96edb093c157401e89e5738a447
    Image:         vault:1.6.1
    Image ID:      docker-pullable://vault@sha256:efe6036315aafbab771939cf518943ef704f5e02a96a0e1b2643666a4aab1ad4
    Ports:         8200/TCP, 8201/TCP, 8202/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      /bin/sh
      -ec
    Args:
      cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
      [ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
      [ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
      [ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
      [ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
      [ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
      [ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
      /usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl

    State:          Running
      Started:      Tue, 26 Jan 2021 12:11:14 +0200
    Ready:          False
    Restart Count:  0
    Readiness:      exec [/bin/sh -ec vault status -tls-skip-verify] delay=5s timeout=3s period=5s #success=1 #failure=2
    Environment:
      HOST_IP:               (v1:status.hostIP)
      POD_IP:                (v1:status.podIP)
      VAULT_K8S_POD_NAME:   vault-1 (v1:metadata.name)
      VAULT_K8S_NAMESPACE:  vault-foo (v1:metadata.namespace)
      VAULT_ADDR:           https://127.0.0.1:8200
      VAULT_API_ADDR:       https://$(POD_IP):8200
      SKIP_CHOWN:           true
      SKIP_SETCAP:          true
      HOSTNAME:             vault-1 (v1:metadata.name)
      VAULT_CLUSTER_ADDR:   https://$(HOSTNAME).vault-internal:8201
      HOME:                 /home/vault
    Mounts:
      /home/vault from home (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from vault-token-pb4vc (ro)
      /vault/config from config (rw)
      /vault/data from data (rw)
      /vault/userconfig/vault-tls from userconfig-vault-tls (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-vault-1
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      vault-config
    Optional:  false
  userconfig-vault-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  vault-tls
    Optional:    false
  home:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  vault-token-pb4vc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  vault-token-pb4vc
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                      From     Message
  ----     ------     ----                     ----     -------
  Warning  Unhealthy  2m24s (x32639 over 45h)  kubelet  Readiness probe failed: Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.6.1
Storage Type       raft
HA Enabled         true

what I’m missing here ? what are those warnings ?