VAULT opens and data appears but constantly getting error when navigating UI

Vault
1

1.) I am new to vault 2.) we have production set and it works fine. This issue is with our sandbox. 3.) K8S environment running HA with 3 node cluster. Version is 1.4.2 deployed via helm.

It was working the last time I was in it but that was a few weeks ago. Today I spun up the cluster and nodes unsealed. I enter the GUI but immediately get error when clicking on anything.
I look in logs and get random error messages. I can add more info if necessary. Has anyone experienced an issue where nodes are active, unsealed can log in but then get and error when clicking on data point? Or can you point me to the forum post with solution? I looked and tried some solutions but problem persists. Which is why I am posing.

Thanks in advanced for your time.

BROWSER
Error in dev tools of browser:
Error while processing route: vault.cluster.secrets.backend.list-root Failed to fetch TypeError: Failed to fetch

Vault-0
==> Vault server configuration:
Api Address: http://$(POD_IP):8200
Cgo: disabled
Cluster Address: https://$(HOSTNAME).vault-internal:8201
Listener 1: tcp (addr: “[::]:8200”, cluster address: “[::]:8201”, max_request_duration: “1m30s”, max_request_size: “33554432”, tls: “disabled”)
Log Level: info
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: raft (HA available)
Version: Vault v1.4.2
2023-01-12T18:47:50.896Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
2023-01-12T18:47:50.920Z [INFO] core: raft retry join initiated
==> Vault server started! Log data will stream in below:
2023-01-12T18:47:51.012Z [INFO] core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2023-01-12T18:47:51.012Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2023-01-12T18:47:51.016Z [INFO] storage.raft: initial configuration: index=77 servers=“[{Suffrage:Voter ID:vault-0 Address:vault-0.vault-internal:8201} {Suffrage:Voter ID:vault-1 Address:vault-1.vault-internal:8201} {Suffrage:Voter ID:vault-2 Address:vault-2.vault-internal:8201}]”
2023-01-12T18:47:51.016Z [INFO] core: vault is unsealed
2023-01-12T18:47:51.016Z [INFO] storage.raft: entering follower state: follower=“Node at (HOSTNAME).vault-internal:8201 [Follower]" leader= 2023-01-12T18:47:51.016Z [INFO] core: entering standby mode 2023-01-12T18:48:06.786Z [ERROR] core: error during forwarded RPC request: error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup (HOSTNAME).vault-internal: no such host”"
2023-01-12T18:48:06.786Z [ERROR] core: forward request error: error=“error during forwarding RPC request”

VAULT-1
==> Vault server configuration:
Api Address: http://$(POD_IP):8200
Cgo: disabled
Cluster Address: https://$(HOSTNAME).vault-internal:8201
Listener 1: tcp (addr: “[::]:8200”, cluster address: “[::]:8201”, max_request_duration: “1m30s”, max_request_size: “33554432”, tls: “disabled”)
Log Level: info
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: raft (HA available)
Version: Vault v1.4.2
==> Vault server started! Log data will stream in below:
2023-01-12T18:46:41.361Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
2023-01-12T18:46:41.375Z [INFO] core: raft retry join initiated
2023-01-12T18:46:41.485Z [INFO] core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2023-01-12T18:46:41.485Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2023-01-12T18:46:41.488Z [INFO] storage.raft: initial configuration: index=77 servers=“[{Suffrage:Voter ID:vault-0 Address:vault-0.vault-internal:8201} {Suffrage:Voter ID:vault-1 Address:vault-1.vault-internal:8201} {Suffrage:Voter ID:vault-2 Address:vault-2.vault-internal:8201}]”
2023-01-12T18:46:41.488Z [INFO] core: vault is unsealed
2023-01-12T18:46:41.489Z [INFO] storage.raft: entering follower state: follower=“Node at (HOSTNAME).vault-internal:8201 [Follower]" leader= 2023-01-12T18:46:41.489Z [INFO] core: entering standby mode 2023-01-12T18:47:23.416Z [ERROR] core: error during forwarded RPC request: error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup (HOSTNAME).vault-internal: no such host”"
2023-01-12T18:47:23.416Z [ERROR] core: forward request error: error=“error during forwarding RPC request”

VAULT-2
==> Vault server configuration:
Api Address: http://$(POD_IP):8200
Cgo: disabled
Cluster Address: https://$(HOSTNAME).vault-internal:8201
Listener 1: tcp (addr: “[::]:8200”, cluster address: “[::]:8201”, max_request_duration: “1m30s”, max_request_size: “33554432”, tls: “disabled”)
Log Level: info
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: raft (HA available)
Version: Vault v1.4.2
==> Vault server started! Log data will stream in below:
2023-01-12T18:45:09.613Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
2023-01-12T18:45:09.629Z [INFO] core: raft retry join initiated
2023-01-12T18:45:09.729Z [INFO] core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2023-01-12T18:45:09.730Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2023-01-12T18:45:09.734Z [INFO] storage.raft: initial configuration: index=77 servers=“[{Suffrage:Voter ID:vault-0 Address:vault-0.vault-internal:8201} {Suffrage:Voter ID:vault-1 Address:vault-1.vault-internal:8201} {Suffrage:Voter ID:vault-2 Address:vault-2.vault-internal:8201}]”
2023-01-12T18:45:09.734Z [INFO] core: vault is unsealed
2023-01-12T18:45:09.734Z [INFO] storage.raft: entering follower state: follower=“Node at (HOSTNAME).vault-internal:8201 [Follower]" leader= 2023-01-12T18:45:09.734Z [INFO] core: entering standby mode 2023-01-12T18:45:16.327Z [WARN] storage.raft: heartbeat timeout reached, starting election: last-leader= 2023-01-12T18:45:16.327Z [INFO] storage.raft: entering candidate state: node="Node at (HOSTNAME).vault-internal:8201 [Candidate]” term=111828
2023-01-12T18:45:16.384Z [INFO] storage.raft: election won: tally=2
2023-01-12T18:45:16.384Z [INFO] storage.raft: entering leader state: leader=“Node at $(HOSTNAME).vault-internal:8201 [Leader]”
2023-01-12T18:45:16.384Z [INFO] storage.raft: added peer, starting replication: peer=vault-0
2023-01-12T18:45:16.385Z [INFO] storage.raft: added peer, starting replication: peer=vault-1
2023-01-12T18:45:16.386Z [INFO] storage.raft: pipelining replication: peer=“{Voter vault-0 vault-0.vault-internal:8201}”
2023-01-12T18:45:16.402Z [INFO] core: acquired lock, enabling active operation
2023-01-12T18:45:16.431Z [INFO] storage.raft: pipelining replication: peer=“{Voter vault-1 vault-1.vault-internal:8201}”
2023-01-12T18:45:16.449Z [INFO] core: post-unseal setup starting
2023-01-12T18:45:16.482Z [INFO] core: loaded wrapping token key
2023-01-12T18:45:16.482Z [INFO] core: successfully setup plugin catalog: plugin-directory=
2023-01-12T18:45:16.493Z [INFO] core: successfully mounted backend: type=system path=sys/
2023-01-12T18:45:16.493Z [INFO] core: successfully mounted backend: type=identity path=identity/
2023-01-12T18:45:16.493Z [INFO] core: successfully mounted backend: type=kv path=TEST/
2023-01-12T18:45:16.493Z [INFO] core: successfully mounted backend: type=kv path=test-qa/
2023-01-12T18:45:16.493Z [INFO] core: successfully mounted backend: type=kv path=test-qa-2/
2023-01-12T18:45:16.493Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2023-01-12T18:45:16.494Z [INFO] core: successfully enabled credential backend: type=token path=token/
2023-01-12T18:45:16.494Z [INFO] core: successfully enabled credential backend: type=userpass path=userpass/
2023-01-12T18:45:16.494Z [INFO] core: successfully enabled credential backend: type=okta path=okta/
2023-01-12T18:45:16.494Z [INFO] core: successfully enabled credential backend: type=ldap path=ldap/
2023-01-12T18:45:16.494Z [INFO] core: restoring leases
2023-01-12T18:45:16.494Z [INFO] rollback: starting rollback manager
2023-01-12T18:45:16.495Z [INFO] identity: entities restored
2023-01-12T18:45:16.495Z [INFO] identity: groups restored
2023-01-12T18:45:16.495Z [WARN] core.raft: skipping new raft TLS config creation, keys are pending
2023-01-12T18:45:16.496Z [INFO] expiration: lease restore complete
2023-01-12T18:45:16.506Z [INFO] core: post-unseal setup complete
2023-01-12T18:46:27.002Z [INFO] storage.raft: aborting pipeline replication: peer=“{Voter vault-1 vault-1.vault-internal:8201}”
2023-01-12T18:46:27.072Z [ERROR] storage.raft: failed to heartbeat to: peer=vault-1.vault-internal:8201 error=EOF
2023-01-12T18:46:27.082Z [ERROR] storage.raft: failed to appendEntries to: peer=“{Voter vault-1 vault-1.vault-internal:8201}” error=“dial tcp 172.30.8.149:8201: connect: connection refused”
2023-01-12T18:46:29.502Z [WARN] storage.raft: failed to contact: server-id=vault-1 time=2.500147509s
2023-01-12T18:46:32.002Z [WARN] storage.raft: failed to contact: server-id=vault-1 time=4.999404505s
2023-01-12T18:46:34.459Z [WARN] storage.raft: failed to contact: server-id=vault-1 time=7.45714846s
2023-01-12T18:46:37.147Z [ERROR] storage.raft: failed to appendEntries to: peer=“{Voter vault-1 vault-1.vault-internal:8201}” error=“dial tcp 172.30.8.149:8201: i/o timeout”
2023-01-12T18:46:37.623Z [ERROR] storage.raft: failed to heartbeat to: peer=vault-1.vault-internal:8201 error=“dial tcp 172.30.8.149:8201: i/o timeout”

[ERROR] storage.raft: failed to heartbeat to: peer=vault-1.vault-internal:8201 error=EOF
[ERROR] storage.raft: failed to appendEntries to: peer=“{Voter vault-1 vault-1.vault-internal:8201}” error=“dial tcp 172.30.8.149:8201: connect: connection refused”
[WARN] storage.raft: failed to contact: server-id=vault-1 time=2.500147509s
[WARN] storage.raft: failed to contact: server-id=vault-1 time=4.999404505s
[WARN] storage.raft: failed to contact: server-id=vault-1 time=7.45714846s
[ERROR] storage.raft: failed to appendEntries to: peer=“{Voter vault-1 vault-1.vault-internal:8201}” error=“dial tcp 172.30.8.149:8201: i/o timeout”
[ERROR] storage.raft: failed to heartbeat to: peer=vault-1.vault-internal:8201 error=“dial tcp 172.30.8.149:8201: i/o timeout”

And these…
storage.raft: entering follower state: follower=“Node at vault-0.vault-internal:8201 [Follower]” leader=
storage.raft: failed to get previous log: previous-index=295873 last-index=295872 error=“log not found”

Thanks for your time.

2

Can’t speak to K8S, but I have seen a somewhat similar issue where if you’re trying to use the web UI through a load balancer (so you’re hitting various vault servers with each request) or you’re trying to get to the UI via one of the inactive nodes in the cluster, it can wig out. IIRC it sometimes sort of worked, and then other times didn’t know what life was.

We solved this a while back by redirecting the browser to the active node. It’s been a while, so it could be that hitting standby nodes for the UI is now supported.

1 Like
3

THANK YOU @rjhornsby !! It was in fact the K8S internal ingress for that namespace which was incorrect. I had it set to /vault instead of /vault-active

This is what caused the ‘redirect loop’ issue.