Hello,
I trust this message finds you well. Our team is currently encountering the described issue within our three-node Vault High Availability (HA) cluster deployed on Kubernetes. The Vault version in use is v1.15.2. Despite attempts to address the problem by restarting the Vault cluster and attempting to promote a standby node to a leader, the issue persists.
We are reaching out for suggestions and assistance in identifying the root cause of the problem. As this is occurring in our production environment, any insights or recommendations would be greatly appreciated. Thanks
Vault Leader Node Logs:
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.079Z [INFO] storage.raft: added peer, starting replication: peer=eecd350a-f2a7-403f-9702-619ba8cebe40
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.079Z [INFO] storage.raft: added peer, starting replication: peer=9b285315-469d-d3c1-be5f-9ba1a41f6760
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.081Z [INFO] storage.raft: pipelining replication: peer=“{Voter 9b285315-469d-d3c1-be5f-9ba1a41f6760 vault-secrets-server-1.vault-secrets-server-internal:8201}”
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.081Z [INFO] storage.raft: pipelining replication: peer=“{Voter eecd350a-f2a7-403f-9702-619ba8cebe40 vault-secrets-server-0.vault-secrets-server-internal:8201}”
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.095Z [INFO] core: acquired lock, enabling active operation
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.116Z [INFO] core: post-unseal setup starting
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.138Z [INFO] core: loaded wrapping token key
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.138Z [INFO] core: successfully setup plugin runtime catalog
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.138Z [INFO] core: successfully setup plugin catalog: plugin-directory=“”
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.171Z [INFO] core: successfully mounted: type=system version=“v1.15.2+builtin.vault” path=sys/ namespace="ID: root. Path: "
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.172Z [INFO] core: successfully mounted: type=identity version=“v1.15.2+builtin.vault” path=identity/ namespace="ID: root. Path: "
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.172Z [INFO] core: successfully mounted: type=kv version=“v0.16.1+builtin” path=secret/ namespace="ID: root. Path: "
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.172Z [INFO] core: successfully mounted: type=cubbyhole version=“v1.15.2+builtin.vault” path=cubbyhole/ namespace="ID: root. Path: "
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.294Z [INFO] core: successfully mounted: type=token version=“v1.15.2+builtin.vault” path=token/ namespace="ID: root. Path: "
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.295Z [INFO] core: successfully mounted: type=kubernetes version=“v0.17.1+builtin” path=kubernetes/ namespace=“ID: root. Path: "
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.295Z [INFO] core: successfully mounted: type=userpass version=“v1.15.2+builtin.vault” path=userpass/ namespace=“ID: root. Path: "
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.311Z [INFO] rollback: Starting the rollback manager with 256 workers
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.311Z [INFO] rollback: starting rollback manager
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.312Z [INFO] core: restoring leases
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.394Z [INFO] expiration: lease restore complete
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.395Z [INFO] identity: entities restored
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.418Z [INFO] identity: groups restored
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.445Z [INFO] core: starting raft active node
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.445Z [INFO] storage.raft: starting autopilot: config=”&{false 0 10s 24h0m0s 1000 0 10s false redundancy_zone upgrade_version}” reconcile_interval=0s
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.470Z [INFO] core: usage gauge collection is disabled
2024-01-19T22:01:48+05:00 2024-01-19T17:01:48.718Z [INFO] core: post-unseal setup complete
2024-01-19T22:04:23+05:00 2024-01-19T17:04:23.948Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=
2024-01-19T22:14:19+05:00 2024-01-19T17:14:19.065Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=
2024-01-19T22:29:23+05:00 2024-01-19T17:29:23.960Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=
2024-01-19T22:29:23+05:00 2024-01-19T17:29:23.977Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=