# VAULT opens and data appears but constantly getting error when navigating UI **URL:** \ **Category:** Vault\ **Tags:** k8s\ **Created:** [January 12, 2023, 11:20pm UTC](https://discuss.hashicorp.com/t/vault-opens-and-data-appears-but-constantly-getting-error-when-navigating-ui/48891 "2023-01-12T23:20:22Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![arthurala](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/arthurala/32/33099_2.png) [@arthurala](https://discuss.hashicorp.com/u/arthurala)\ **Post date:** [January 12, 2023, 11:20pm UTC](https://discuss.hashicorp.com/t/vault-opens-and-data-appears-but-constantly-getting-error-when-navigating-ui/48891/1 "2023-01-12T23:20:22Z")
1.) I am new to vault 2.) we have production set and it works fine. This issue is with our sandbox. 3.) K8S environment running HA with 3 node cluster. Version is 1.4.2 deployed via helm. It was working the last time I was in it but that was a few weeks ago. Today I spun up the cluster and nodes unsealed. I enter the GUI but immediately get error when clicking on anything. I look in logs and get random error messages. I can add more info if necessary. **Has anyone experienced an issue where nodes are active, unsealed can log in but then get and error when clicking on data point?** Or can you point me to the forum post with solution? I looked and tried some solutions but problem persists. Which is why I am posing. Thanks in advanced for your time. **BROWSER** Error in dev tools of browser: Error while processing route: vault.cluster.secrets.backend.list-root Failed to fetch TypeError: Failed to fetch **Vault-0** ==\> Vault server configuration: Api Address: http://$(POD\_IP):8200 Cgo: disabled Cluster Address: https://$(HOSTNAME).vault-internal:8201 Listener 1: tcp (addr: “[::]:8200”, cluster address: “[::]:8201”, max\_request\_duration: “1m30s”, max\_request\_size: “33554432”, tls: “disabled”) Log Level: info Mlock: supported: true, enabled: false Recovery Mode: false Storage: raft (HA available) Version: Vault v1.4.2 2023-01-12T18:47:50.896Z [INFO] proxy environment: http\_proxy= https\_proxy= no\_proxy= 2023-01-12T18:47:50.920Z [INFO] core: raft retry join initiated ==\> Vault server started! Log data will stream in below: 2023-01-12T18:47:51.012Z [INFO] core.cluster-listener.tcp: starting listener: listener\_address=[::]:8201 2023-01-12T18:47:51.012Z [INFO] core.cluster-listener: serving cluster requests: cluster\_listen\_address=[::]:8201 2023-01-12T18:47:51.016Z [INFO] storage.raft: initial configuration: index=77 servers=“[{Suffrage:Voter ID:vault-0 Address:vault-0.vault-internal:8201} {Suffrage:Voter ID:vault-1 Address:vault-1.vault-internal:8201} {Suffrage:Voter ID:vault-2 Address:vault-2.vault-internal:8201}]” 2023-01-12T18:47:51.016Z [INFO] core: vault is unsealed 2023-01-12T18:47:51.016Z [INFO] storage.raft: entering follower state: follower=“Node at (HOSTNAME).vault-internal:8201 [Follower]" leader= 2023-01-12T18:47:51.016Z [INFO] core: entering standby mode 2023-01-12T18:48:06.786Z [ERROR] core: error during forwarded RPC request: error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup (HOSTNAME).vault-internal: no such host”" 2023-01-12T18:48:06.786Z [ERROR] core: forward request error: error=“error during forwarding RPC request” **VAULT-1** ==\> Vault server configuration: Api Address: http://$(POD\_IP):8200 Cgo: disabled Cluster Address: https://$(HOSTNAME).vault-internal:8201 Listener 1: tcp (addr: “[::]:8200”, cluster address: “[::]:8201”, max\_request\_duration: “1m30s”, max\_request\_size: “33554432”, tls: “disabled”) Log Level: info Mlock: supported: true, enabled: false Recovery Mode: false Storage: raft (HA available) Version: Vault v1.4.2 ==\> Vault server started! Log data will stream in below: 2023-01-12T18:46:41.361Z [INFO] proxy environment: http\_proxy= https\_proxy= no\_proxy= 2023-01-12T18:46:41.375Z [INFO] core: raft retry join initiated 2023-01-12T18:46:41.485Z [INFO] core.cluster-listener.tcp: starting listener: listener\_address=[::]:8201 2023-01-12T18:46:41.485Z [INFO] core.cluster-listener: serving cluster requests: cluster\_listen\_address=[::]:8201 2023-01-12T18:46:41.488Z [INFO] storage.raft: initial configuration: index=77 servers=“[{Suffrage:Voter ID:vault-0 Address:vault-0.vault-internal:8201} {Suffrage:Voter ID:vault-1 Address:vault-1.vault-internal:8201} {Suffrage:Voter ID:vault-2 Address:vault-2.vault-internal:8201}]” 2023-01-12T18:46:41.488Z [INFO] core: vault is unsealed 2023-01-12T18:46:41.489Z [INFO] storage.raft: entering follower state: follower=“Node at (HOSTNAME).vault-internal:8201 [Follower]" leader= 2023-01-12T18:46:41.489Z [INFO] core: entering standby mode 2023-01-12T18:47:23.416Z [ERROR] core: error during forwarded RPC request: error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup (HOSTNAME).vault-internal: no such host”" 2023-01-12T18:47:23.416Z [ERROR] core: forward request error: error=“error during forwarding RPC request” **VAULT-2** ==\> Vault server configuration: Api Address: http://$(POD\_IP):8200 Cgo: disabled Cluster Address: https://$(HOSTNAME).vault-internal:8201 Listener 1: tcp (addr: “[::]:8200”, cluster address: “[::]:8201”, max\_request\_duration: “1m30s”, max\_request\_size: “33554432”, tls: “disabled”) Log Level: info Mlock: supported: true, enabled: false Recovery Mode: false Storage: raft (HA available) Version: Vault v1.4.2 ==\> Vault server started! Log data will stream in below: 2023-01-12T18:45:09.613Z [INFO] proxy environment: http\_proxy= https\_proxy= no\_proxy= 2023-01-12T18:45:09.629Z [INFO] core: raft retry join initiated 2023-01-12T18:45:09.729Z [INFO] core.cluster-listener.tcp: starting listener: listener\_address=[::]:8201 2023-01-12T18:45:09.730Z [INFO] core.cluster-listener: serving cluster requests: cluster\_listen\_address=[::]:8201 2023-01-12T18:45:09.734Z [INFO] storage.raft: initial configuration: index=77 servers=“[{Suffrage:Voter ID:vault-0 Address:vault-0.vault-internal:8201} {Suffrage:Voter ID:vault-1 Address:vault-1.vault-internal:8201} {Suffrage:Voter ID:vault-2 Address:vault-2.vault-internal:8201}]” 2023-01-12T18:45:09.734Z [INFO] core: vault is unsealed 2023-01-12T18:45:09.734Z [INFO] storage.raft: entering follower state: follower=“Node at (HOSTNAME).vault-internal:8201 [Follower]" leader= 2023-01-12T18:45:09.734Z [INFO] core: entering standby mode 2023-01-12T18:45:16.327Z [WARN] storage.raft: heartbeat timeout reached, starting election: last-leader= 2023-01-12T18:45:16.327Z [INFO] storage.raft: entering candidate state: node="Node at (HOSTNAME).vault-internal:8201 [Candidate]” term=111828 2023-01-12T18:45:16.384Z [INFO] storage.raft: election won: tally=2 2023-01-12T18:45:16.384Z [INFO] storage.raft: entering leader state: leader=“Node at $(HOSTNAME).vault-internal:8201 [Leader]” 2023-01-12T18:45:16.384Z [INFO] storage.raft: added peer, starting replication: peer=vault-0 2023-01-12T18:45:16.385Z [INFO] storage.raft: added peer, starting replication: peer=vault-1 2023-01-12T18:45:16.386Z [INFO] storage.raft: pipelining replication: peer=“{Voter vault-0 vault-0.vault-internal:8201}” 2023-01-12T18:45:16.402Z [INFO] core: acquired lock, enabling active operation 2023-01-12T18:45:16.431Z [INFO] storage.raft: pipelining replication: peer=“{Voter vault-1 vault-1.vault-internal:8201}” 2023-01-12T18:45:16.449Z [INFO] core: post-unseal setup starting 2023-01-12T18:45:16.482Z [INFO] core: loaded wrapping token key 2023-01-12T18:45:16.482Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2023-01-12T18:45:16.493Z [INFO] core: successfully mounted backend: type=system path=sys/ 2023-01-12T18:45:16.493Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2023-01-12T18:45:16.493Z [INFO] core: successfully mounted backend: type=kv path=TEST/ 2023-01-12T18:45:16.493Z [INFO] core: successfully mounted backend: type=kv path=test-qa/ 2023-01-12T18:45:16.493Z [INFO] core: successfully mounted backend: type=kv path=test-qa-2/ 2023-01-12T18:45:16.493Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2023-01-12T18:45:16.494Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2023-01-12T18:45:16.494Z [INFO] core: successfully enabled credential backend: type=userpass path=userpass/ 2023-01-12T18:45:16.494Z [INFO] core: successfully enabled credential backend: type=okta path=okta/ 2023-01-12T18:45:16.494Z [INFO] core: successfully enabled credential backend: type=ldap path=ldap/ 2023-01-12T18:45:16.494Z [INFO] core: restoring leases 2023-01-12T18:45:16.494Z [INFO] rollback: starting rollback manager 2023-01-12T18:45:16.495Z [INFO] identity: entities restored 2023-01-12T18:45:16.495Z [INFO] identity: groups restored 2023-01-12T18:45:16.495Z [WARN] core.raft: skipping new raft TLS config creation, keys are pending 2023-01-12T18:45:16.496Z [INFO] expiration: lease restore complete 2023-01-12T18:45:16.506Z [INFO] core: post-unseal setup complete 2023-01-12T18:46:27.002Z [INFO] storage.raft: aborting pipeline replication: peer=“{Voter vault-1 vault-1.vault-internal:8201}” 2023-01-12T18:46:27.072Z [ERROR] storage.raft: failed to heartbeat to: peer=vault-1.vault-internal:8201 error=EOF 2023-01-12T18:46:27.082Z [ERROR] storage.raft: failed to appendEntries to: peer=“{Voter vault-1 vault-1.vault-internal:8201}” error=“dial tcp 172.30.8.149:8201: connect: connection refused” 2023-01-12T18:46:29.502Z [WARN] storage.raft: failed to contact: server-id=vault-1 time=2.500147509s 2023-01-12T18:46:32.002Z [WARN] storage.raft: failed to contact: server-id=vault-1 time=4.999404505s 2023-01-12T18:46:34.459Z [WARN] storage.raft: failed to contact: server-id=vault-1 time=7.45714846s 2023-01-12T18:46:37.147Z [ERROR] storage.raft: failed to appendEntries to: peer=“{Voter vault-1 vault-1.vault-internal:8201}” error=“dial tcp 172.30.8.149:8201: i/o timeout” 2023-01-12T18:46:37.623Z [ERROR] storage.raft: failed to heartbeat to: peer=vault-1.vault-internal:8201 error=“dial tcp 172.30.8.149:8201: i/o timeout” [ERROR] storage.raft: failed to heartbeat to: peer=vault-1.vault-internal:8201 error=EOF [ERROR] storage.raft: failed to appendEntries to: peer=“{Voter vault-1 vault-1.vault-internal:8201}” error=“dial tcp 172.30.8.149:8201: connect: connection refused” [WARN] storage.raft: failed to contact: server-id=vault-1 time=2.500147509s [WARN] storage.raft: failed to contact: server-id=vault-1 time=4.999404505s [WARN] storage.raft: failed to contact: server-id=vault-1 time=7.45714846s [ERROR] storage.raft: failed to appendEntries to: peer=“{Voter vault-1 vault-1.vault-internal:8201}” error=“dial tcp 172.30.8.149:8201: i/o timeout” [ERROR] storage.raft: failed to heartbeat to: peer=vault-1.vault-internal:8201 error=“dial tcp 172.30.8.149:8201: i/o timeout” And these… storage.raft: entering follower state: follower=“Node at vault-0.vault-internal:8201 [Follower]” leader= storage.raft: failed to get previous log: previous-index=295873 last-index=295872 error=“log not found” Thanks for your time. ---
**Author:** ![rjhornsby](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/rjhornsby/32/4886_2.png) [@rjhornsby](https://discuss.hashicorp.com/u/rjhornsby)\ **Post date:** [January 13, 2023, 6:08pm UTC](https://discuss.hashicorp.com/t/vault-opens-and-data-appears-but-constantly-getting-error-when-navigating-ui/48891/2 "2023-01-13T18:08:54Z")
Can’t speak to K8S, but I have seen a somewhat similar issue where if you’re trying to use the web UI through a load balancer (so you’re hitting various vault servers with each request) or you’re trying to get to the UI via one of the inactive nodes in the cluster, it can wig out. IIRC it sometimes sort of worked, and then other times didn’t know what life was. We solved this a while back by redirecting the browser to the active node. It’s been a while, so it could be that hitting standby nodes for the UI is now supported. ---
**Author:** ![arthurala](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/arthurala/32/33099_2.png) [@arthurala](https://discuss.hashicorp.com/u/arthurala)\ **Post date:** [January 13, 2023, 9:17pm UTC](https://discuss.hashicorp.com/t/vault-opens-and-data-appears-but-constantly-getting-error-when-navigating-ui/48891/3 "2023-01-13T21:17:45Z")
THANK YOU @rjhornsby !! It was in fact the K8S internal ingress for that namespace which was incorrect. I had it set to /vault instead of /vault-active This is what caused the ‘redirect loop’ issue.