Vault operator raft join getting : [ERROR] core: failed to join raft cluster: error="failed to join any raft leader node"

Vault
, ,
1

i try to install the 3 nodes using raft on k8s, i successfully init and unsealed vault-0 pod
but when i try to from other pods (vault-1,vault-2) invoke:
vault operator raft join:

echo $CA_CERT
-----BEGIN CERTIFICATE----- MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl xxxxxxi9ThJsj4xMxEw= -----END CERTIFICATE----- 

vault operator raft join -leader-ca-cert="${CA_CERT}" https://vault-0.vault-internal:8200

I’m getting this error :

Error joining the node to the Raft cluster: Error making API request.

URL: POST https://127.0.0.1:8200/v1/sys/storage/raft/join
Code: 500. Errors:

* failed to join raft cluster: failed to join any raft leader node

error in logs of vault-1

2021-01-23T11:17:18.939Z [INFO]  core: security barrier not initialized
2021-01-23T11:17:18.939Z [INFO]  core: seal configuration missing, not initialized
2021-01-23T11:17:23.942Z [INFO]  core: security barrier not initialized
2021-01-23T11:17:23.942Z [INFO]  core: seal configuration missing, not initialized
2021-01-23T11:17:28.941Z [INFO]  core: security barrier not initialized
2021-01-23T11:17:28.941Z [INFO]  core: security barrier not initialized
2021-01-23T11:17:28.941Z [INFO]  core: attempting to join possible raft leader node: leader_addr=https://vault-0.vault-internal:8200
2021-01-23T11:17:28.950Z [INFO]  core: security barrier not initialized
2021-01-23T11:17:28.950Z [INFO]  core: seal configuration missing, not initialized
2021-01-23T11:17:28.955Z [WARN]  core: join attempt failed: error="error during raft bootstrap init call: Error making API request.

URL: PUT https://vault-0.vault-internal:8200/v1/sys/storage/raft/bootstrap/challenge
Code: 503. Errors:

* Vault is sealed"
2021-01-23T11:17:28.955Z [ERROR] core: failed to join raft cluster: error="failed to join any raft leader node"
2021-01-23T11:17:33.952Z [INFO]  core: security barrier not initialized
2021-01-23T11:17:33.952Z [INFO]  core: seal configuration missing, not initialized
2021-01-23T11:17:38.930Z [INFO]  core: security barrier not initialized
2021-01-23T11:17:38.930Z [INFO]  core: seal configuration missing, not initialized
2021-01-23T11:17:43.939Z [INFO]  core: security barrier not initialized

This is how my-values.yaml looks :

global:
  enabled: true
  tlsDisable: false
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-tls/vault.ca
server:
  extraVolumes:
  - type: secret
    name: vault-tls

  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: false
      config: |
        ui = true

        listener "tcp" {
          address = "0.0.0.0:8200"
          cluster_address = "0.0.0.0:8201"
          tls_cert_file = "/vault/userconfig/vault-tls/vault.crt"
          tls_key_file = "/vault/userconfig/vault-tls/vault.key"
          tls_client_ca_file = "/vault/userconfig/vault-tls/vault.ca"
        }

        storage "raft" {
          path = "/vault/data"
        }

        service_registration "kubernetes" {}

and the csr.conf :

[req]
default_bits = 4096
prompt = no
encrypt_key = yes
default_md = sha256
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
emailAddress = admin@admin.dev
CN = vault.vault-perso.svc
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = vault
DNS.2 = vault.vault-perso
DNS.3 = vault.vault-perso.svc
DNS.4 = vault.vault-perso.svc.cluster.local
DNS.5 = *.vault-internal
IP.1 = 127.0.0.1
2

Hi Umen,

When use Vault CLI the default Vault address is 127.0.0.1,
so the error log show “URL: POST https://127.0.0.1:8200/v1/sys/storage/raft/join”

There are two ways to solve it.

  1. add “-address=” option
-address=<string>
      Address of the Vault server. The default is https://127.0.0.1:8200. This
      can also be specified via the VAULT_ADDR environment variable.

$ vault operator raft join -leader-ca-cert="${CA_CERT}" https://vault-0.vault-internal:8200 -address=https://vault-0.vault-internal:8200

see more information:

$ vault operator raft join -h
  1. add OS ENV
    Linux:
$ vi ~/.bashrc

Add

export VAULT_ADDR="https://vault-0.vault-internal:8200"

after save then apply

$ source ~/.bashrc
3

Thanks for your answer
i removed the use of TLS in raft, i can’t do all you suggested as I’m automating the deployment k8s

4

Try add ENV in your yaml.

containers:
    - name: vault
        image: <hashicorp-vault:last>
        imagePullPolicy: Always
        env:
        - name: VAULT_ADDR
          value: "http://vault-0.vault-internal:8200"