Unable to join Raft server node to cluster

Vault
1

We are trying to establish a Vault cluster (Windows Server 2012).
Here are the steps and configurations used:

172.24.32.184 (vault_1):

Step 1
C:\Vault>vault.exe server -config vault.hcl

  storage "raft" {
    path    = "C:\\raft"
    node_id = "vault_1"
  }
  listener "tcp" {
    address = "0.0.0.0:8200"
    //cluster_address = "172.24.32.184:8201"
    tls_disable = 1
  }
  api_addr = "http://172.24.32.184:8200"
  disable_mlock = true
  cluster_addr = "http://172.24.32.184:8201"
  ui = true

Step 2
C:\Vault>vault operator init -format=json -key-shares 1 -key-threshold 1

{
  "unseal_keys_b64": [
    "FvSkX8AbKUaZl3whKRp5e5rneEzlrOdC/skIzkRyYcA="
  ],
  "unseal_keys_hex": [
    "16f4a45fc01b294699977c21291a797b9ae7784ce5ace742fec908ce447261c0"
  ],
  "unseal_shares": 1,
  "unseal_threshold": 1,
  "recovery_keys_b64": [],
  "recovery_keys_hex": [],
  "recovery_keys_shares": 5,
  "recovery_keys_threshold": 3,
  "root_token": "s.MOPuGPXBTnvWbhTzasHspurL"
}

Step 3
C:\Vault>vault operator unseal
-> Pasted from above “unseal_keys_b64”

Output:

Initialized             true
Sealed                  false

172.24.32.186 (vault_2):

Step 1
C:\Vault>vault.exe server -config vault.hcl

  storage "raft" {
    path    = "C:\\raft"
    node_id = "vault_2"
  }
  listener "tcp" {
    address = "172.24.32.186:8200"
    //cluster_address = "172.24.32.186:8201"
    tls_disable = true
  }
  api_addr = "http://172.24.32.186:8200"
  disable_mlock = true
  cluster_addr = "http://172.24.32.186:8201"
  ui = true

Step 2
C:\Vault>vault operator init -format=json -key-shares 1 -key-threshold 1

{
  "unseal_keys_b64": [
    "Quk9vAXTvM1JpVfKoko1AA9iqhxQjIfcVPjpAQVUgB8="
  ],
  "unseal_keys_hex": [
    "42e93dbc05d3bccd49a557caa24a35000f62aa1c508c87dc54f8e9010554801f"
  ],
  "unseal_shares": 1,
  "unseal_threshold": 1,
  "recovery_keys_b64": [],
  "recovery_keys_hex": [],
  "recovery_keys_shares": 5,
  "recovery_keys_threshold": 3,
  "root_token": "s.xMXO3kJO2wzg8sCjM6HIUxqJ"
}

Step 3
C:\Vault>vault operator unseal
-> Pasted from above “unseal_keys_b64”

Step 4
C:\Vault>vault operator raft join http://172.24.32.184:8200

Key       Value
---       -----
Joined    true

Step 5
C:\Vault>vault operator raft list-peers

Node       Address               State     Voter
----       -------               -----     -----
vault_2    172.24.32.186:8201    leader    true

Switch back to 172.24.32.184 (vault_1)

Step 1
C:\Vault>vault operator raft list-peers

Node       Address               State     Voter
----       -------               -----     -----
vault_1    172.24.32.184:8201    leader    true

This is the tutorial we are trying to follow: https://learn.hashicorp.com/tutorials/vault/raft-storage. One difference is that we are manually unsealing the vault instances.

We would like help understanding why when we try to join to another server cluster it seems to not join to the other server, and instead appears to create a second cluster.

Thank you!

2

@stevehorn-wt, is there a firewall blocking communication between the nodes? Also, do you have a minimum of 3 nodes configured for a quorum?

3

Thanks for the suggestion. I checked connectivity between the two servers by successfully executing a GET request using the /v1/sys/health endpoint from each server.

4

Thanks to some inspiration from another post, I think I figured out what I was doing wrong.

For the second node, it should not be initialized. Instead, the server should be started:
vault.exe server -config vault.hcl.

Then, join the second node to the first:
vault operator raft join http://172.24.32.184:8200.

And finally, unseal the second node:
vault operator unseal (Here be sure to unseal using the unseal keys from the first (leader) node.)

5

After Following this procedure, getting this error
Error joining the node to the Raft cluster: Error making API request.

URL: POST http://:8200/v1/sys/storage/raft/join
Code: 500. Errors:

  • failed to join raft cluster: failed to join any raft leader node
6

The above issue got resolved now, the issue was with the firewall.
Now after joining and unsealing node 2. I am getting the following output when listing the peers.
Node Address State Voter

node1 127.0.0.1:8201 leader true
node2 172.16.31.140:8201 follower false

Expected Output:

Node Address State Voter

node1 172.16.31.116:8201 leader true
node2 172.16.31.140:8201 follower true

7

I have removed the localhost IP from the config file but still, it is showing here in the Peer listing.

8

check your cluster status: vault operator raft autopilot state, then try enabling Cleanup Dead Servers with set-config to autopilot-subcommand.

9

Thanks for your reply, I have removed the raft storage directory and re-created and started the Clustering process and finally I got the expected output.