# Raft Cluster Certificates

**URL:** <https://discuss.hashicorp.com/t/raft-cluster-certificates/71135>\
**Category:** Vault\
**Tags:** raft\
**Created:** [November 5, 2024, 9:02am UTC](https://discuss.hashicorp.com/t/raft-cluster-certificates/71135 "2024-11-05T09:02:00Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![gornication](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/gornication/32/43572_2.png) [@gornication](https://discuss.hashicorp.com/u/gornication)\
**Post date:** [November 5, 2024, 9:02am UTC](https://discuss.hashicorp.com/t/raft-cluster-certificates/71135/1 "2024-11-05T09:02:00Z")

</div>

Hi!

I’m trying to run ha(3 replicas/pods) Vault with integrated raft cluster storage.  
For signing names with certificates, internal communication between followers and the leader, I use an init container.  
I run Vault itself through [helm](https://github.com/hashicorp/vault-helm/tree/main) and ArgoCD on k8s.

**First of all, am I right in assuming that the problem is with certificates?**  
**And if so, how do I create them and apply them correctly?**

> **The problem itself**
>
> **$ vault operator raft join -leader-ca-cert=/vault/tls/ca.pem -leader-client-cert=/vault/tls/tls.crt -leader-client-key=/vault/tls/tls.key [https://vault-0.vault-internal:8200](https://vault-0.vault-internal:8200)**  
> _Error joining the node to the Raft cluster: Error making API request._  
> _URL: POST [https://127.0.0.1:8200/v1/sys/storage/raft/join](https://127.0.0.1:8200/v1/sys/storage/raft/join)_  
> _Code: 400. Errors:_  
> _tls: failed to find any PEM data in certificate input_
> 
> **/ $ vault operator raft join -leader-ca-cert=/vault/tls/ca.pem -leader-client-cert=/vault/tls/tls.crt -leader-client-key=/vault/tls/tls.key [https://10.8.111.32:8200](https://10.8.111.32:8200)**  
> _Error joining the node to the Raft cluster: Error making API request._  
> _URL: POST [https://127.0.0.1:8200/v1/sys/storage/raft/join](https://127.0.0.1:8200/v1/sys/storage/raft/join)_  
> _Code: 400. Errors:_  
> _tls: failed to find any PEM data in certificate input_
> 
> **/ $ vault operator raft join -leader-ca-cert=@/vault/tls/ca.pem -leader-client-cert=@/vault/tls/tls.crt -leader-client-key=@/vault/tls/tls.key [https://vault-0.vault-internal:8200](https://vault-0.vault-internal:8200)**  
> _Error joining the node to the Raft cluster: Error making API request._  
> _URL: POST [https://127.0.0.1:8200/v1/sys/storage/raft/join](https://127.0.0.1:8200/v1/sys/storage/raft/join)_  
> _Code: 500. Errors:_  
> _failed to join raft cluster: failed to get raft challenge_
> 
> **/ $ vault operator raft join -leader-ca-cert=@/vault/tls/ca.pem -leader-client-cert=@/vault/tls/tls.crt -leader-client-key=@/vault/tls/tls.key [https://10.8.111.32:8200](https://10.8.111.32:8200)**  
> _Error joining the node to the Raft cluster: Error making API request._  
> _URL: POST [https://127.0.0.1:8200/v1/sys/storage/raft/join](https://127.0.0.1:8200/v1/sys/storage/raft/join)_  
> _Code: 500. Errors:_  
> _failed to join raft cluster: failed to get raft challenge_
> 
> **/ $ vault operator raft join [https://10.8.111.32:8200](https://10.8.111.32:8200)**  
> _Error joining the node to the Raft cluster: Error making API request._  
> _URL: POST [https://127.0.0.1:8200/v1/sys/storage/raft/join](https://127.0.0.1:8200/v1/sys/storage/raft/join)_  
> _Code: 500. Errors:_  
> _failed to join raft cluster: failed to get raft challenge_
> 
> **/ $ vault operator raft join [https://vault-0.vault-internal:8200](https://vault-0.vault-internal:8200)**  
> _Error joining the node to the Raft cluster: Error making API request._  
> _URL: POST [https://127.0.0.1:8200/v1/sys/storage/raft/join](https://127.0.0.1:8200/v1/sys/storage/raft/join)_  
> _Code: 500. Errors:_  
> _failed to join raft cluster: failed to get raft challenge_
> 
> **On vault-0 container(pod) you can see logs like this:**  
> 2024-11-05T09:11:30.851Z [INFO] http: TLS handshake error from 10.8.111.229:33030: remote error: tls: bad certificate  
> 2024-11-05T09:11:31.264Z [INFO] http: TLS handshake error from 10.8.111.229:33046: remote error: tls: bad certificate  
> 2024-11-05T09:11:31.862Z [INFO] http: TLS handshake error from 10.8.111.229:33048: remote error: tls: bad certificate
> 
> 10.8.111.229 corresponds to the address of the container from which I attempted to join

> **InitContainer**
>
> ```
> extraInitContainers:
> - name: generate-cert
> image: alpine/openssl
> command:
> - /bin/sh
> - -c
> - |
> mkdir -p /vault/tls
> 
> # Generate the CA key
> openssl genrsa -out /vault/tls/ca.key 4096
> 
> # Generate the CA certificate
> openssl req -x509 -new -nodes -key /vault/tls/ca.key -sha256 -days 10950 -out /vault/tls/ca.pem -subj "/C=LK/ST=Western Province/L=Colombo/O=Example (Private) Limited/OU=Development/CN=Example Development Certificate Authority"
> 
> # Verify certificate data
> openssl x509 -in /vault/tls/ca.pem -text -noout
> 
> # Generate tls.key and tls.csr for *.${POD_NAME}.local domain.
> openssl req -new -nodes -newkey rsa:4096 -keyout /vault/tls/tls.key -out /vault/tls/tls.csr \
> -subj "/C=LK/ST=Western Province/L=Colombo/O=Example (Private) Limited/OU=Development/CN=${POD_NAME}.vault-internal" \
> -config <(printf "[req]\ndistinguished_name=req\n[san]\nsubjectAltName=IP:127.0.0.1,DNS:localhost,DNS:${POD_NAME}.vault-internal,DNS:*.${POD_NAME}.vault-internal") \
> -reqexts san
> 
> # Generate and sign tls.crt for the domain *.example.local
> openssl x509 -req -in /vault/tls/tls.csr -CA /vault/tls/ca.pem -CAkey /vault/tls/ca.key -CAcreateserial -out /vault/tls/tls.crt -days 3650 -sha256 -extfile <(printf "subjectAltName=IP:127.0.0.1,DNS:localhost,DNS:${POD_NAME}.vault-internal,DNS:*.${POD_NAME}.vault-internal")
>         
> # Verify if the signed certificate is valid
> openssl verify -verbose -CAfile /vault/tls/ca.pem /vault/tls/tls.crt
> env:
> - name: POD_NAME
> valueFrom:
> fieldRef:
> fieldPath: metadata.name
> volumeMounts:
> - name: tls-shared
> mountPath: /vault/tls
> 
> ```

> **ha config**
>
> ```
> ha:
> enabled: true
> replicas: 3
> apiAddr: null
> clusterAddr: null
> raft:
> enabled: true
> setNodeId: true
> config: |
> ui = true
> cluster_name = "vault-integrated-storage"
> listener "tcp" {
> address = "[::]:8200"
> cluster_address = "[::]:8201"
> tls_cert_file = "/vault/tls/tls.crt"
> tls_key_file = "/vault/tls/tls.key"
> tls_ca_file = "/vault/tls/ca.pem"
> }
> 
> storage "raft" {
> path = "/vault/data"
> retry_join {
> leader_api_addr = "https://vault-0.vault-internal:8200"
> leader_ca_cert_file = "/vault/tls/ca.pem"
> leader_client_cert_file = "/vault/tls/tls.crt"
> leader_client_key_file = "/vault/tls/tls.key"
> }
> retry_join {
> leader_api_addr = "https://vault-1.vault-internal:8200"
> leader_ca_cert_file = "/vault/tls/ca.pem"
> leader_client_cert_file = "/vault/tls/tls.crt"
> leader_client_key_file = "/vault/tls/tls.key"
> }
> retry_join {
> leader_api_addr = "https://vault-2.vault-internal:8200"
> leader_ca_cert_file = "/vault/tls/ca.pem"
> leader_client_cert_file = "/vault/tls/tls.crt"
> leader_client_key_file = "/vault/tls/tls.key"
> }
> }
> service_registration "kubernetes" {}
> 
> ```