I have 3 nodes of vault running on k8s , everything was fine, and suddenly today
i have event warning that says :
Readiness probe failed: Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 0/3 Unseal Nonce n/a Version 1.6.1 Storage Type raft HA Enabled true
when i look at the node-1 and node-2 logs i can see that the server is up and running
==> Vault server configuration:
Api Address: https://10.xxx.0.xxx:8200
Cgo: disabled
Cluster Address: https://vault-1.vault-internal:8201
Go Version: go1.15.4
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
Log Level: info
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: raft (HA available)
Version: Vault v1.6.1
Version Sha: 6d2db3f033e02e70xxxx360062b88b03
==> Vault server started! Log data will stream in below:
2021-01-26T10:11:14.437Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
also here is the pod describe:
$ kubectl describe pod vault-1 -n vault-foo
Name: vault-1
Namespace: vault-foo
Priority: 0
Node: ip-10-101-0-98.ec2.internal/xxx.xxx.0.98
Start Time: Tue, 26 Jan 2021 12:11:05 +0200
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/name=vault
component=server
controller-revision-hash=vault-7694f4b78c
helm.sh/chart=vault-0.9.0
statefulset.kubernetes.io/pod-name=vault-1
vault-active=false
vault-initialized=false
vault-perf-standby=false
vault-sealed=true
vault-version=1.6.1
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: xxx.xxx.0.191
IPs:
IP: xxx.xxx.0.191
Controlled By: StatefulSet/vault
Containers:
vault:
Container ID: docker://077b501aef3eaeb5f9e75dc144f288d51dbff96edb093c157401e89e5738a447
Image: vault:1.6.1
Image ID: docker-pullable://vault@sha256:efe6036315aafbab771939cf518943ef704f5e02a96a0e1b2643666a4aab1ad4
Ports: 8200/TCP, 8201/TCP, 8202/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/sh
-ec
Args:
cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
[ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
[ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
/usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl
State: Running
Started: Tue, 26 Jan 2021 12:11:14 +0200
Ready: False
Restart Count: 0
Readiness: exec [/bin/sh -ec vault status -tls-skip-verify] delay=5s timeout=3s period=5s #success=1 #failure=2
Environment:
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
VAULT_K8S_POD_NAME: vault-1 (v1:metadata.name)
VAULT_K8S_NAMESPACE: vault-foo (v1:metadata.namespace)
VAULT_ADDR: https://127.0.0.1:8200
VAULT_API_ADDR: https://$(POD_IP):8200
SKIP_CHOWN: true
SKIP_SETCAP: true
HOSTNAME: vault-1 (v1:metadata.name)
VAULT_CLUSTER_ADDR: https://$(HOSTNAME).vault-internal:8201
HOME: /home/vault
Mounts:
/home/vault from home (rw)
/var/run/secrets/kubernetes.io/serviceaccount from vault-token-pb4vc (ro)
/vault/config from config (rw)
/vault/data from data (rw)
/vault/userconfig/vault-tls from userconfig-vault-tls (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-vault-1
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: vault-config
Optional: false
userconfig-vault-tls:
Type: Secret (a volume populated by a Secret)
SecretName: vault-tls
Optional: false
home:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
vault-token-pb4vc:
Type: Secret (a volume populated by a Secret)
SecretName: vault-token-pb4vc
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 2m24s (x32639 over 45h) kubelet Readiness probe failed: Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.6.1
Storage Type raft
HA Enabled true
what I’m missing here ? what are those warnings ?