I have a 5-node vault cluster at HA mode in a kubernetes cluster. Last night, one physical host node was shutdown, thus this kubernetes master vault has to shutdown, and starting up in another physical node. When it starting up, it turned itself as standby vault, I saw error about “error during forwarded RPC request:”, it lasted about 30 minutes, then things get fixed by itself.
This log shows probably at that time this standby node cannot forward requesting to master node. I am trying to figure out why it can not request forwarding.
For master vault, VAULT_CLUSTER_ADDR=https://readacted-name.redacted-name.com:8201
We are using an external hostname address for the cluster address. This name is publicly resolvable. When this origin master shutdown, the other one took master role. This standby node was aware of that.
The vault config file:
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 0
tls_cert_file = "/usr/files/server-cert.pem"
tls_key_file = "/usr/files/server-key.pem"
tls_require_and_verify_client_cert = 0
}
api_addr = "https://100.96.27.48:8200"
disable_mlock = "true"
The api_address is internal address.
Help: Any suggestion about what we got such a long time RPC error, and How to reduce this kind problem keep happening? It happened several times.
Logs:
2020-12-14T05:42:47.983Z [INFO] core.cluster-listener: starting listener: listener_address=0.0.0.0:8201
2020-12-14T05:42:47.983Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2020-12-14T05:42:47.983Z [INFO] core: vault is unsealed
2020-12-14T05:42:47.983Z [INFO] core: entering standby mode
2020-12-14T05:43:08.013Z [ERROR] core: error during forwarded RPC request: error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing tls: DialWithDialer timed out""
2020-12-14T05:43:08.013Z [ERROR] core: forward request error: error="error during forwarding RPC request"
....
2020-12-14T06:10:21.063Z [ERROR] core: error during forwarded RPC request: error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing EOF""
2020-12-14T06:10:21.063Z [ERROR] core: forward request error: error="error during forwarding RPC request"
2020-12-14T06:10:49.738Z [INFO] http: TLS handshake error from 100.96.75.1:36902: EOF
2020-12-14T06:11:19.737Z [INFO] http: TLS handshake error from 100.96.75.1:37506: EOF
2020-12-14T06:11:49.737Z [INFO] http: TLS handshake error from 100.96.75.1:38104: EOF
2020-12-14T06:12:19.737Z [INFO] http: TLS handshake error from 100.96.75.1:38698: EOF
2020-12-14T06:12:49.737Z [INFO] http: TLS handshake error from 100.96.75.1:39308: EOF
2020-12-14T06:13:19.738Z [INFO] http: TLS handshake error from 100.96.75.1:39928: EOF
2020-12-14T06:13:49.738Z [INFO] http: TLS handshake error from 100.96.75.1:40564: EOF