RPC error when forwarding request

Vault
,
1

Hi everyone,

I encountered the issue, that standby nodes are occasionally logging the following:

2021-08-24T08:55:18.410Z [ERROR] core: error during forwarded RPC request: error="rpc error: code = Canceled desc = context canceled" 2021-08-24T08:55:18.410Z [ERROR] core: forward request error: error="error during forwarding RPC request"

I am running vault in the Google Kubernetes Engine with 3 pods. I can’t reproduce the issue with manual requests to the standby nodes (e.g. reading a secret/writing a secret). Do you have any idea what could cause this?

My configuration:

    - name: HOSTNAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: VAULT_API_ADDR
      value: https://$(HOSTNAME).vault-internal:8200
    - name: VAULT_CLUSTER_ADDR
      value: https://$(HOSTNAME).vault-internal:8201

Backend: gcs

Vault config file:

listener "tcp" { tls_disable = 0 tls_cert_file = "/etc/tls/vault-tls/ca_bundle.pem" tls_key_file = "/etc/tls/vault-tls/vault-key.pem" tls_client_ca_file = "/etc/tls/vault-tls/ca.pem" tls_min_version = "tls12" address = "[::]:8200" cluster_address = "[::]:8201" }

I’ve already tried editing both VAULT_CLUSTER_ADDR and VAULT_API_ADDR to:

    - name: VAULT_API_ADDR
      value: https://$(POD_IP):8200
    - name: VAULT_CLUSTER_ADDR
      value: https://$(POD_IP):8201

without success.

Thanks in advance!

1 Like