My vault worked well for a few months. Last days I saw error message when trying to reach UI:
503 Service Temporarily Unavailable
So I’ve started the pods again, and found these errors:
==> Vault server configuration:
Api Address: http://10.42.197.59:8200
Cgo: disabled
Cluster Address: https://vault-server-0.vault-server-internal:8201
Environment Variables: GODEBUG, HOME, HOSTNAME, HOST_IP, KUBERNETES_PORT, KUBERNETES_PORT_443_TCP, KUBERNETES_PORT_443_TCP_ADDR, KUBERNETES_PORT_443_TCP_PORT, KUBERNETES_PORT_443_TCP_PROTO, KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT, KUBERNETES_SERVICE_PORT_HTTPS, NAME, PATH, POD_IP, PWD, SHLVL, SKIP_CHOWN, SKIP_SETCAP, VAULT_ADDR, VAULT_API_ADDR, VAULT_CLUSTER_ADDR, VAULT_K8S_NAMESPACE, VAULT_K8S_POD_NAME, VAULT_SERVER_ACTIVE_PORT, VAULT_SERVER_ACTIVE_PORT_8200_TCP, VAULT_SERVER_ACTIVE_PORT_8200_TCP_ADDR, VAULT_SERVER_ACTIVE_PORT_8200_TCP_PORT, VAULT_SERVER_ACTIVE_PORT_8200_TCP_PROTO, VAULT_SERVER_ACTIVE_PORT_8201_TCP, VAULT_SERVER_ACTIVE_PORT_8201_TCP_ADDR, VAULT_SERVER_ACTIVE_PORT_8201_TCP_PORT, VAULT_SERVER_ACTIVE_PORT_8201_TCP_PROTO, VAULT_SERVER_ACTIVE_SERVICE_HOST, VAULT_SERVER_ACTIVE_SERVICE_PORT, VAULT_SERVER_ACTIVE_SERVICE_PORT_HTTP, VAULT_SERVER_ACTIVE_SERVICE_PORT_HTTPS_INTERNAL, VAULT_SERVER_AGENT_INJECTOR_SVC_PORT, VAULT_SERVER_AGENT_INJECTOR_SVC_PORT_443_TCP, VAULT_SERVER_AGENT_INJECTOR_SVC_PORT_443_TCP_ADDR, VAULT_SERVER_AGENT_INJECTOR_SVC_PORT_443_TCP_PORT, VAULT_SERVER_AGENT_INJECTOR_SVC_PORT_443_TCP_PROTO, VAULT_SERVER_AGENT_INJECTOR_SVC_SERVICE_HOST, VAULT_SERVER_AGENT_INJECTOR_SVC_SERVICE_PORT, VAULT_SERVER_AGENT_INJECTOR_SVC_SERVICE_PORT_HTTPS, VAULT_SERVER_PORT, VAULT_SERVER_PORT_8200_TCP, VAULT_SERVER_PORT_8200_TCP_ADDR, VAULT_SERVER_PORT_8200_TCP_PORT, VAULT_SERVER_PORT_8200_TCP_PROTO, VAULT_SERVER_PORT_8201_TCP, VAULT_SERVER_PORT_8201_TCP_ADDR, VAULT_SERVER_PORT_8201_TCP_PORT, VAULT_SERVER_PORT_8201_TCP_PROTO, VAULT_SERVER_SERVICE_HOST, VAULT_SERVER_SERVICE_PORT, VAULT_SERVER_SERVICE_PORT_HTTP, VAULT_SERVER_SERVICE_PORT_HTTPS_INTERNAL, VAULT_SERVER_STANDBY_PORT, VAULT_SERVER_STANDBY_PORT_8200_TCP, VAULT_SERVER_STANDBY_PORT_8200_TCP_ADDR, VAULT_SERVER_STANDBY_PORT_8200_TCP_PORT, VAULT_SERVER_STANDBY_PORT_8200_TCP_PROTO, VAULT_SERVER_STANDBY_PORT_8201_TCP, VAULT_SERVER_STANDBY_PORT_8201_TCP_ADDR, VAULT_SERVER_STANDBY_PORT_8201_TCP_PORT, VAULT_SERVER_STANDBY_PORT_8201_TCP_PROTO, VAULT_SERVER_STANDBY_SERVICE_HOST, VAULT_SERVER_STANDBY_SERVICE_PORT, VAULT_SERVER_STANDBY_SERVICE_PORT_HTTP, VAULT_SERVER_STANDBY_SERVICE_PORT_HTTPS_INTERNAL, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_PORT, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_PORT_8443_TCP, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_PORT_8443_TCP_ADDR, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_PORT_8443_TCP_PORT, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_PORT_8443_TCP_PROTO, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_SERVICE_HOST, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_SERVICE_PORT, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_SERVICE_PORT_HTTPS, VERSION
Go Version: go1.20.5
Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level:
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: raft (HA available)
Version: Vault v1.14.0, built 2023-06-19T11:40:23Z
Version Sha: 13a649f860186dffe3f3a4459814d87191efc321
==> Vault server started! Log data will stream in below:
2023-12-17T08:34:24.811Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2023-12-17T08:34:24.933Z [INFO] core: Initializing version history cache for core
2023-12-17T08:35:59.661Z [INFO] core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2023-12-17T08:35:59.664Z [ERROR] core.cluster-listener.tcp: error starting listener: error="listen tcp [::]:8201: socket: address family not supported by protocol"
2023-12-17T08:35:59.665Z [INFO] storage.raft: creating Raft: config="&raft.Config{ProtocolVersion:3, HeartbeatTimeout:15000000000, ElectionTimeout:15000000000, CommitTimeout:50000000, MaxAppendEntries:64, BatchApplyCh:true, ShutdownOnRemove:true, TrailingLogs:0x2800, SnapshotInterval:120000000000, SnapshotThreshold:0x2000, LeaderLeaseTimeout:2500000000, LocalID:\"87f6b245-3c4f-6b3c-e9b3-1ea9401f5376\", NotifyCh:(chan<- bool)(0xc0013ec150), LogOutput:io.Writer(nil), LogLevel:\"DEBUG\", Logger:(*hclog.interceptLogger)(0xc000674090), NoSnapshotRestoreOnStart:true, skipStartup:false}"
2023-12-17T08:35:59.666Z [INFO] storage.raft: initial configuration: index=40 servers="[{Suffrage:Voter ID:87f6b245-3c4f-6b3c-e9b3-1ea9401f5376 Address:vault-server-0.vault-server-internal:8201} {Suffrage:Voter ID:1644af06-390d-777a-b41b-3eadd239b35c Address:vault-server-1.vault-server-internal:8201} {Suffrage:Voter ID:210e1e0e-1599-895d-4631-69c5e20e4228 Address:vault-server-2.vault-server-internal:8201}]"
2023-12-17T08:35:59.667Z [INFO] core: vault is unsealed
2023-12-17T08:35:59.667Z [INFO] storage.raft: entering follower state: follower="Node at vault-server-0.vault-server-internal:8201 [Follower]" leader-address= leader-id=
2023-12-17T08:35:59.667Z [INFO] core: entering standby mode
2023-12-17T08:36:18.602Z [WARN] storage.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2023-12-17T08:36:18.602Z [INFO] storage.raft: entering candidate state: node="Node at vault-server-0.vault-server-internal:8201 [Candidate]" term=127204
2023-12-17T08:36:18.606Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 210e1e0e-1599-895d-4631-69c5e20e4228 vault-server-2.vault-server-internal:8201}" error="dial tcp 10.42.65.181:8201: connect: connection refused" term=127204
2023-12-17T08:36:18.607Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 1644af06-390d-777a-b41b-3eadd239b35c vault-server-1.vault-server-internal:8201}" error="dial tcp 10.42.196.179:8201: connect: connection refused" term=127204
2023-12-17T08:36:21.174Z [ERROR] core: error during forwarded RPC request: error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.42.197.59:8201: connect: connection refused\""
2023-12-17T08:36:21.174Z [ERROR] core: forward request error: error="error during forwarding RPC request"
2023-12-17T08:36:25.312Z [WARN] storage.raft: Election timeout reached, restarting election
2023-12-17T08:36:25.312Z [INFO] storage.raft: entering candidate state: node="Node at vault-server-0.vault-server-internal:8201 [Candidate]" term=127205
2023-12-17T08:36:25.317Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 210e1e0e-1599-895d-4631-69c5e20e4228 vault-server-2.vault-server-internal:8201}" error="dial tcp 10.42.65.181:8201: connect: connection refused" term=127205
2023-12-17T08:36:25.317Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 1644af06-390d-777a-b41b-3eadd239b35c vault-server-1.vault-server-internal:8201}" error="dial tcp 10.42.196.179:8201: connect: connection refused" term=127205
2023-12-17T08:36:32.031Z [WARN] storage.raft: Election timeout reached, restarting election
2023-12-17T08:36:32.031Z [INFO] storage.raft: entering candidate state: node="Node at vault-server-0.vault-server-internal:8201 [Candidate]" term=127206
2023-12-17T08:36:32.036Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 210e1e0e-1599-895d-4631-69c5e20e4228 vault-server-2.vault-server-internal:8201}" error="dial tcp 10.42.65.181:8201: connect: connection refused" term=127206
this is from the leader.
and
==> Vault server configuration:
Api Address: http://10.42.65.181:8200
Cgo: disabled
Cluster Address: https://vault-server-2.vault-server-internal:8201
Environment Variables: GODEBUG, HOME, HOSTNAME, HOST_IP, KUBERNETES_PORT, KUBERNETES_PORT_443_TCP, KUBERNETES_PORT_443_TCP_ADDR, KUBERNETES_PORT_443_TCP_PORT, KUBERNETES_PORT_443_TCP_PROTO, KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT, KUBERNETES_SERVICE_PORT_HTTPS, NAME, PATH, POD_IP, PWD, SHLVL, SKIP_CHOWN, SKIP_SETCAP, VAULT_ADDR, VAULT_API_ADDR, VAULT_CLUSTER_ADDR, VAULT_K8S_NAMESPACE, VAULT_K8S_POD_NAME, VAULT_SERVER_ACTIVE_PORT, VAULT_SERVER_ACTIVE_PORT_8200_TCP, VAULT_SERVER_ACTIVE_PORT_8200_TCP_ADDR, VAULT_SERVER_ACTIVE_PORT_8200_TCP_PORT, VAULT_SERVER_ACTIVE_PORT_8200_TCP_PROTO, VAULT_SERVER_ACTIVE_PORT_8201_TCP, VAULT_SERVER_ACTIVE_PORT_8201_TCP_ADDR, VAULT_SERVER_ACTIVE_PORT_8201_TCP_PORT, VAULT_SERVER_ACTIVE_PORT_8201_TCP_PROTO, VAULT_SERVER_ACTIVE_SERVICE_HOST, VAULT_SERVER_ACTIVE_SERVICE_PORT, VAULT_SERVER_ACTIVE_SERVICE_PORT_HTTP, VAULT_SERVER_ACTIVE_SERVICE_PORT_HTTPS_INTERNAL, VAULT_SERVER_AGENT_INJECTOR_SVC_PORT, VAULT_SERVER_AGENT_INJECTOR_SVC_PORT_443_TCP, VAULT_SERVER_AGENT_INJECTOR_SVC_PORT_443_TCP_ADDR, VAULT_SERVER_AGENT_INJECTOR_SVC_PORT_443_TCP_PORT, VAULT_SERVER_AGENT_INJECTOR_SVC_PORT_443_TCP_PROTO, VAULT_SERVER_AGENT_INJECTOR_SVC_SERVICE_HOST, VAULT_SERVER_AGENT_INJECTOR_SVC_SERVICE_PORT, VAULT_SERVER_AGENT_INJECTOR_SVC_SERVICE_PORT_HTTPS, VAULT_SERVER_PORT, VAULT_SERVER_PORT_8200_TCP, VAULT_SERVER_PORT_8200_TCP_ADDR, VAULT_SERVER_PORT_8200_TCP_PORT, VAULT_SERVER_PORT_8200_TCP_PROTO, VAULT_SERVER_PORT_8201_TCP, VAULT_SERVER_PORT_8201_TCP_ADDR, VAULT_SERVER_PORT_8201_TCP_PORT, VAULT_SERVER_PORT_8201_TCP_PROTO, VAULT_SERVER_SERVICE_HOST, VAULT_SERVER_SERVICE_PORT, VAULT_SERVER_SERVICE_PORT_HTTP, VAULT_SERVER_SERVICE_PORT_HTTPS_INTERNAL, VAULT_SERVER_STANDBY_PORT, VAULT_SERVER_STANDBY_PORT_8200_TCP, VAULT_SERVER_STANDBY_PORT_8200_TCP_ADDR, VAULT_SERVER_STANDBY_PORT_8200_TCP_PORT, VAULT_SERVER_STANDBY_PORT_8200_TCP_PROTO, VAULT_SERVER_STANDBY_PORT_8201_TCP, VAULT_SERVER_STANDBY_PORT_8201_TCP_ADDR, VAULT_SERVER_STANDBY_PORT_8201_TCP_PORT, VAULT_SERVER_STANDBY_PORT_8201_TCP_PROTO, VAULT_SERVER_STANDBY_SERVICE_HOST, VAULT_SERVER_STANDBY_SERVICE_PORT, VAULT_SERVER_STANDBY_SERVICE_PORT_HTTP, VAULT_SERVER_STANDBY_SERVICE_PORT_HTTPS_INTERNAL, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_PORT, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_PORT_8443_TCP, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_PORT_8443_TCP_ADDR, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_PORT_8443_TCP_PORT, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_PORT_8443_TCP_PROTO, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_SERVICE_HOST, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_SERVICE_PORT, VAULT_SERVER_VAULT_SECRETS_OPERATOR_METRICS_SERVICE_SERVICE_PORT_HTTPS, VERSION
Go Version: go1.20.5
Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level:
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: raft (HA available)
Version: Vault v1.14.0, built 2023-06-19T11:40:23Z
Version Sha: 13a649f860186dffe3f3a4459814d87191efc321
==> Vault server started! Log data will stream in below:
2023-12-17T08:34:26.810Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2023-12-17T08:34:26.821Z [INFO] core: Initializing version history cache for core
2023-12-17T08:36:57.713Z [INFO] core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2023-12-17T08:36:57.715Z [ERROR] core.cluster-listener.tcp: error starting listener: error="listen tcp [::]:8201: socket: address family not supported by protocol"
2023-12-17T08:36:57.716Z [INFO] storage.raft: creating Raft: config="&raft.Config{ProtocolVersion:3, HeartbeatTimeout:15000000000, ElectionTimeout:15000000000, CommitTimeout:50000000, MaxAppendEntries:64, BatchApplyCh:true, ShutdownOnRemove:true, TrailingLogs:0x2800, SnapshotInterval:120000000000, SnapshotThreshold:0x2000, LeaderLeaseTimeout:2500000000, LocalID:\"210e1e0e-1599-895d-4631-69c5e20e4228\", NotifyCh:(chan<- bool)(0xc000c30540), LogOutput:io.Writer(nil), LogLevel:\"DEBUG\", Logger:(*hclog.interceptLogger)(0xc000c22900), NoSnapshotRestoreOnStart:true, skipStartup:false}"
2023-12-17T08:36:57.717Z [INFO] storage.raft: initial configuration: index=40 servers="[{Suffrage:Voter ID:87f6b245-3c4f-6b3c-e9b3-1ea9401f5376 Address:vault-server-0.vault-server-internal:8201} {Suffrage:Voter ID:1644af06-390d-777a-b41b-3eadd239b35c Address:vault-server-1.vault-server-internal:8201} {Suffrage:Voter ID:210e1e0e-1599-895d-4631-69c5e20e4228 Address:vault-server-2.vault-server-internal:8201}]"
2023-12-17T08:36:57.717Z [INFO] storage.raft: entering follower state: follower="Node at vault-server-2.vault-server-internal:8201 [Follower]" leader-address= leader-id=
2023-12-17T08:36:57.717Z [INFO] core: vault is unsealed
2023-12-17T08:36:57.717Z [INFO] core: entering standby mode
2023-12-17T08:37:12.817Z [WARN] storage.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2023-12-17T08:37:12.817Z [INFO] storage.raft: entering candidate state: node="Node at vault-server-2.vault-server-internal:8201 [Candidate]" term=127202
2023-12-17T08:37:12.819Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 87f6b245-3c4f-6b3c-e9b3-1ea9401f5376 vault-server-0.vault-server-internal:8201}" error="dial tcp 10.42.197.59:8201: connect: connection refused" term=127202
2023-12-17T08:37:12.819Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 1644af06-390d-777a-b41b-3eadd239b35c vault-server-1.vault-server-internal:8201}" error="dial tcp 10.42.196.179:8201: connect: connection refused" term=127202
2023-12-17T08:37:19.670Z [WARN] storage.raft: Election timeout reached, restarting election
2023-12-17T08:37:19.670Z [INFO] storage.raft: entering candidate state: node="Node at vault-server-2.vault-server-internal:8201 [Candidate]" term=127203
2023-12-17T08:37:19.673Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 1644af06-390d-777a-b41b-3eadd239b35c vault-server-1.vault-server-internal:8201}" error="dial tcp 10.42.196.179:8201: connect: connection refused" term=127203
2023-12-17T08:37:19.673Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 87f6b245-3c4f-6b3c-e9b3-1ea9401f5376 vault-server-0.vault-server-internal:8201}" error="dial tcp 10.42.197.59:8201: connect: connection refused" term=127203
2023-12-17T08:37:27.718Z [WARN] storage.raft: Election timeout reached, restarting election
2023-12-17T08:37:27.718Z [INFO] storage.raft: entering candidate state: node="Node at vault-server-2.vault-server-internal:8201 [Candidate]" term=127204
2023-12-17T08:37:27.721Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 87f6b245-3c4f-6b3c-e9b3-1ea9401f5376 vault-server-0.vault-server-internal:8201}" error="dial tcp 10.42.197.59:8201: connect: connection refused" term=127204
2023-12-17T08:37:27.721Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 1644af06-390d-777a-b41b-3eadd239b35c vault-server-1.vault-server-internal:8201}" error="dial tcp 10.42.196.179:8201: connect: connection refused" term=127204
2023-12-17T08:37:37.412Z [WARN] storage.raft: Election timeout reached, restarting election
2023-12-17T08:37:37.412Z [INFO] storage.raft: entering candidate state: node="Node at vault-server-2.vault-server-internal:8201 [Candidate]" term=127205
2023-12-17T08:37:37.414Z [ERROR] storage.raft: failed to make requestVote RPC: target="{Voter 1644af06-390d-777a-b41b-3eadd239b35c vault-server-1.vault-server-internal:8201}" error="dial tcp 10.42.196.179:8201: connect: connection refused" term=127205
from the follower.
Not sure what is going on here, but I have ping from pod to pod.
The only thing that was changed in last days is turn off IPV6 on the K8s nodes.