Raft can't access the right TLS certificates

Vault
, ,
1

Hi, I am trying to setup a vault cluster with raft storage with HA mode enabled. I’m installing vault via helm chart(Attached the hel values file below). I have a created ingress which points to vault-ui k8s service of type loadbalancer and I have domain(vault.domain.io) which points to this IP and have created TLS certificates for this using let’s encrypt. vault-internal is another k8s service used by raft to communicate with other peers. The internal endpoints for raft is https://vault-0.vault-internal:8200 I have created self-signed certificates for the internal domains and provide the path of these certificates in retry_join block of storage. But when I try to initialize the vault in the logs I see raft is getting the certificates of the public domain and not from the path provided in the retry_join block. Am I doing anything wrong here? How can I fix this?

helm values file

  • vault-https-cert - TLS certs for public domain(vault.domain.io)
  • self-signed-tls - TLS certs for internal(vault-*.vault-internal)
global:
  enabled: true
  tlsDisable: false

injector:
  enabled: false

server:
  extraVolumes:
    - type: secret
      name: vault-serviceaccount
    - type: secret
      name: vault-https-cert
    - type: secret
      name: slef-signed-tls
  
  auditStorage:
    enabled: true

  standalone:
    enabled: false

  ha:
    enabled: true
    raft:
      enabled: true
      config: |
        ui = true
        listener "tcp" {
          address = "0.0.0.0:8200"
          cluster_address = "0.0.0.0:8201"
          tls_cert_file = "/vault/userconfig/vault-https-cert/tls.crt"
          tls_key_file  = "/vault/userconfig/vault-https-cert/tls.key"
        }
        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/self-signed-tls/ca.crt"
            leader_client_cert_file = "/vault/userconfig/self-signed-tls/tls.crt"
            leader_client_key_file = "/vault/userconfig/self-signed-tls/tls.key"
          }
          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/self-signed-tls/ca.crt"
            leader_client_cert_file = "/vault/userconfig/self-signed-tls/tls.crt"
            leader_client_key_file = "/vault/userconfig/self-signed-tls/tls.key"
          }
          retry_join {
            leader_api_addr = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/self-signed-tls/ca.crt"
            leader_client_cert_file = "/vault/userconfig/self-signed-tls/tls.crt"
            leader_client_key_file = "/vault/userconfig/self-signed-tls/tls.key"
          }

          autopilot {
            cleanup_dead_servers = "true"
            last_contact_threshold = "200ms"
            last_contact_failure_threshold = "10m"
            max_trailing_logs = 250000
            min_quorum = 2
            server_stabilization_time = "10s"
          }
        }

        seal "gcpckms" {
            credentials = "/vault/userconfig/vault-serviceaccount/vault-serviceaccount.json"
            project     = "my-project-id"
            region      = "global"
            key_ring    = "vault-auto-unseal"
            crypto_key  = "vault-auto-unseal"
          }
        service_registration "kubernetes" {}
    replicas: 3
ui:
  enabled: true
  serviceType: "LoadBalancer"
  serviceNodePort: null
  externalPort: 8200

vault
vault1035×776 37.7 KB

Error Logs

core: failed to get raft challenge: leader_addr=https://vault-1.vault-internal:8200 error="error during raft bootstrap init call: Put \"https://vault-1.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge\": x509: certificate is valid for vault.domain.io, not vault-1.vault-internal"
2022-07-06T11:04:01.398Z [ERROR] core: failed to get raft challenge: leader_addr=https://vault-2.vault-internal:8200 error="error during raft bootstrap init call: Put \"https://vault-2.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge\": x509: certificate is valid for vault.domain.io, not vault-2.vault-internal"
2022-07-06T11:04:01.398Z [ERROR] core: failed to retry join raft cluster: retry=2s
2022-07-06T11:04:01.404Z [INFO]  http: TLS handshake error from 10.42.2.18:49598: remote error: tls: bad certificate
2022-07-06T11:04:02.951Z [INFO]  http: TLS handshake error from 10.42.0.20:58780: remote error: tls: bad certificate
2

Were you ever able to solve this? I am running into similar issues with my implementation.