Raft Cluster Certificates

Vault
1

Hi!

I’m trying to run ha(3 replicas/pods) Vault with integrated raft cluster storage.
For signing names with certificates, internal communication between followers and the leader, I use an init container.
I run Vault itself through helm and ArgoCD on k8s.

First of all, am I right in assuming that the problem is with certificates?
And if so, how do I create them and apply them correctly?

The problem itself

$ vault operator raft join -leader-ca-cert=/vault/tls/ca.pem -leader-client-cert=/vault/tls/tls.crt -leader-client-key=/vault/tls/tls.key https://vault-0.vault-internal:8200
Error joining the node to the Raft cluster: Error making API request.
URL: POST https://127.0.0.1:8200/v1/sys/storage/raft/join
Code: 400. Errors:
tls: failed to find any PEM data in certificate input

/ $ vault operator raft join -leader-ca-cert=/vault/tls/ca.pem -leader-client-cert=/vault/tls/tls.crt -leader-client-key=/vault/tls/tls.key https://10.8.111.32:8200
Error joining the node to the Raft cluster: Error making API request.
URL: POST https://127.0.0.1:8200/v1/sys/storage/raft/join
Code: 400. Errors:
tls: failed to find any PEM data in certificate input

/ $ vault operator raft join -leader-ca-cert=@/vault/tls/ca.pem -leader-client-cert=@/vault/tls/tls.crt -leader-client-key=@/vault/tls/tls.key https://vault-0.vault-internal:8200
Error joining the node to the Raft cluster: Error making API request.
URL: POST https://127.0.0.1:8200/v1/sys/storage/raft/join
Code: 500. Errors:
failed to join raft cluster: failed to get raft challenge

/ $ vault operator raft join -leader-ca-cert=@/vault/tls/ca.pem -leader-client-cert=@/vault/tls/tls.crt -leader-client-key=@/vault/tls/tls.key https://10.8.111.32:8200
Error joining the node to the Raft cluster: Error making API request.
URL: POST https://127.0.0.1:8200/v1/sys/storage/raft/join
Code: 500. Errors:
failed to join raft cluster: failed to get raft challenge

/ $ vault operator raft join https://10.8.111.32:8200
Error joining the node to the Raft cluster: Error making API request.
URL: POST https://127.0.0.1:8200/v1/sys/storage/raft/join
Code: 500. Errors:
failed to join raft cluster: failed to get raft challenge

/ $ vault operator raft join https://vault-0.vault-internal:8200
Error joining the node to the Raft cluster: Error making API request.
URL: POST https://127.0.0.1:8200/v1/sys/storage/raft/join
Code: 500. Errors:
failed to join raft cluster: failed to get raft challenge

On vault-0 container(pod) you can see logs like this:
2024-11-05T09:11:30.851Z [INFO] http: TLS handshake error from 10.8.111.229:33030: remote error: tls: bad certificate
2024-11-05T09:11:31.264Z [INFO] http: TLS handshake error from 10.8.111.229:33046: remote error: tls: bad certificate
2024-11-05T09:11:31.862Z [INFO] http: TLS handshake error from 10.8.111.229:33048: remote error: tls: bad certificate

10.8.111.229 corresponds to the address of the container from which I attempted to join

InitContainer 
extraInitContainers:
  - name: generate-cert
    image: alpine/openssl
    command:
      - /bin/sh
      - -c
      - |
        mkdir -p /vault/tls

        # Generate the CA key
        openssl genrsa -out /vault/tls/ca.key 4096

        # Generate the CA certificate
        openssl req -x509 -new -nodes -key /vault/tls/ca.key -sha256 -days 10950 -out /vault/tls/ca.pem -subj "/C=LK/ST=Western Province/L=Colombo/O=Example (Private) Limited/OU=Development/CN=Example Development Certificate Authority"

        # Verify certificate data
        openssl x509 -in /vault/tls/ca.pem -text -noout

        # Generate tls.key and tls.csr for *.${POD_NAME}.local domain.
        openssl req -new -nodes -newkey rsa:4096 -keyout /vault/tls/tls.key -out /vault/tls/tls.csr \
          -subj "/C=LK/ST=Western Province/L=Colombo/O=Example (Private) Limited/OU=Development/CN=${POD_NAME}.vault-internal" \
          -config <(printf "[req]\ndistinguished_name=req\n[san]\nsubjectAltName=IP:127.0.0.1,DNS:localhost,DNS:${POD_NAME}.vault-internal,DNS:*.${POD_NAME}.vault-internal") \
          -reqexts san

        # Generate and sign tls.crt for the domain *.example.local
        openssl x509 -req -in /vault/tls/tls.csr -CA /vault/tls/ca.pem -CAkey /vault/tls/ca.key -CAcreateserial -out /vault/tls/tls.crt -days 3650 -sha256 -extfile <(printf "subjectAltName=IP:127.0.0.1,DNS:localhost,DNS:${POD_NAME}.vault-internal,DNS:*.${POD_NAME}.vault-internal")
        
        # Verify if the signed certificate is valid
        openssl verify -verbose -CAfile /vault/tls/ca.pem /vault/tls/tls.crt
    env:
      - name: POD_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
    volumeMounts:
      - name: tls-shared
        mountPath: /vault/tls
ha config 
ha:
  enabled: true
  replicas: 3
  apiAddr: null
  clusterAddr: null
  raft:
    enabled: true
    setNodeId: true
    config: |
      ui = true
      cluster_name = "vault-integrated-storage"
      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file = "/vault/tls/tls.crt"
        tls_key_file = "/vault/tls/tls.key"
        tls_ca_file = "/vault/tls/ca.pem"
      }

      storage "raft" {
        path = "/vault/data"
        retry_join {
          leader_api_addr = "https://vault-0.vault-internal:8200"
          leader_ca_cert_file = "/vault/tls/ca.pem"
          leader_client_cert_file = "/vault/tls/tls.crt"
          leader_client_key_file = "/vault/tls/tls.key"
        }
        retry_join {
          leader_api_addr = "https://vault-1.vault-internal:8200"
          leader_ca_cert_file = "/vault/tls/ca.pem"
          leader_client_cert_file = "/vault/tls/tls.crt"
          leader_client_key_file = "/vault/tls/tls.key"
        }
        retry_join {
          leader_api_addr = "https://vault-2.vault-internal:8200"
          leader_ca_cert_file = "/vault/tls/ca.pem"
          leader_client_cert_file = "/vault/tls/tls.crt"
          leader_client_key_file = "/vault/tls/tls.key"
        }
      }
      service_registration "kubernetes" {}