Vault Operator Raft Join not working on an existing cluster with raft storage and awskms seal config

Vault
1

Hello Team,

I have been struggling to add an additional node to a 3 nodes cluster which works well so far. I have checked on this https://discuss.hashicorp.com/t/how-we-can-add-node-in-vault-cluster/46843 but it seems it’s not exactly the same as my case.

Below are my configuration

listener "tcp" {
  address                    = "0.0.0.0:8200"
  cluster_address            = "10.50.10.10:8201"
  tls_disable                = 1
}

storage "raft" {
  node_id                   = "hashi-vault-eu-west-1a"
  path                       = "/opt/vault/data"


 retry_join {
  leader_api_addr    = "http://10.50.11.10:8200"
 }
 retry_join {
  leader_api_addr    = "http://10.50.10.10:8200"
 }
 retry_join {
  leader_api_addr    = "http://10.52.10.10:8200"
 }
 retry_join {
  leader_api_addr    = "http://10.52.11.10:8200" // <--- the new node
 }
}


seal "awskms" {
  region     = "eu-west-1"
  kms_key_id = "854048be-xxxxxx-394xxxe583c"
}

api_addr                     = "http://10.50.10.10:8200"
cluster_addr                 = "https://10.50.10.10:8201"


disable_mlock                = true

ui                           = "True"
cluster_name                 = "sg_hashi_vault"
log_format                   = "json"

The above configuration is the same for all the now 4 nodes. After restarting them all I was expecting the 4th node to be part of the cluster but that didn’t happen.

I run a vault status command on the 4th node

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.10.2
Storage Type             raft
Cluster Name             sg_hashi_vault
Cluster ID               6e2a43ef-xxxxxxxx9f2dcb63
HA Enabled               true
HA Cluster               n/a. //<--- no cluster url
HA Mode                  standby
Active Node Address      <none> // <-- no active node address
Raft Committed Index     3893
Raft Applied Index       3893

So I checked on raft list peers and it shows the following:

Node                                           Address             State       Voter
----                                           -------             -----       -----
hashi-vault-eu-west-1a         10.50.10.10:8201    follower    true
hashi-vault-eu-west-1b         10.50.11.10:8201    leader      true
hashi-vault-eu-central-1a    10.52.10.10:8201    follower    true

the member list too is as the following:

Host Name                                API Address                Cluster Address             ActiveNode    Last Echo
---------                                -----------                ---------------             ----------    ---------
hashi-vault-eu-west-1b         http://10.50.11.10:8200    https://10.50.11.10:8201    true          <nil>
hashi-vault-eu-central-1a      http://10.52.10.10:8200    https://10.52.10.10:8201    false         2024-01-04 16:33:19.018024257 +0000 UTC
hashi-vault-eu-west-1a         http://10.50.10.10:8200    https://10.50.10.10:8201    false         2024-01-04 16:33:19.709236564 +0000 UTC

So within the 4th I run the following:

vault operator raft join "http://10.50.11.10:8200"
Key       Value
---       -----
Joined    true

But all the raft list-peers and raft members commands remained unchanged. I have also attempted the same join command with the root token but got an error instead:

export VAULT_TOKEN="hvs.xxxxxxQayF" vault operator raft join http://10.50.11.10:8200
-bash: export: `http://10.50.11.10:8200': not a valid identifier

I have also spotted in the log a warning: “not part of stable configuration, aborting”, not sure how relevant is that. I have attached the log
hashicorp_vault_log.txt (4.8 KB)

Not too sure way forward from this . Any help is deeply appreciated