Vault Operator Raft Join not working on an existing cluster with raft storage and awskms seal config

Hello Team,

I have been struggling to add an additional node to a 3 nodes cluster which works well so far. I have checked on this but it seems it’s not exactly the same as my case.

Below are my configuration

listener "tcp" {
  address                    = ""
  cluster_address            = ""
  tls_disable                = 1

storage "raft" {
  node_id                   = "hashi-vault-eu-west-1a"
  path                       = "/opt/vault/data"

 retry_join {
  leader_api_addr    = ""
 retry_join {
  leader_api_addr    = ""
 retry_join {
  leader_api_addr    = ""
 retry_join {
  leader_api_addr    = "" // <--- the new node

seal "awskms" {
  region     = "eu-west-1"
  kms_key_id = "854048be-xxxxxx-394xxxe583c"

api_addr                     = ""
cluster_addr                 = ""

disable_mlock                = true

ui                           = "True"
cluster_name                 = "sg_hashi_vault"
log_format                   = "json"

The above configuration is the same for all the now 4 nodes. After restarting them all I was expecting the 4th node to be part of the cluster but that didn’t happen.

I run a vault status command on the 4th node

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.10.2
Storage Type             raft
Cluster Name             sg_hashi_vault
Cluster ID               6e2a43ef-xxxxxxxx9f2dcb63
HA Enabled               true
HA Cluster               n/a. //<--- no cluster url
HA Mode                  standby
Active Node Address      <none> // <-- no active node address
Raft Committed Index     3893
Raft Applied Index       3893

So I checked on raft list peers and it shows the following:

Node                                           Address             State       Voter
----                                           -------             -----       -----
hashi-vault-eu-west-1a    follower    true
hashi-vault-eu-west-1b    leader      true
hashi-vault-eu-central-1a    follower    true

the member list too is as the following:

Host Name                                API Address                Cluster Address             ActiveNode    Last Echo
---------                                -----------                ---------------             ----------    ---------
hashi-vault-eu-west-1b    true          <nil>
hashi-vault-eu-central-1a    false         2024-01-04 16:33:19.018024257 +0000 UTC
hashi-vault-eu-west-1a    false         2024-01-04 16:33:19.709236564 +0000 UTC

So within the 4th I run the following:

vault operator raft join ""
Key       Value
---       -----
Joined    true

But all the raft list-peers and raft members commands remained unchanged. I have also attempted the same join command with the root token but got an error instead:

export VAULT_TOKEN="hvs.xxxxxxQayF" vault operator raft join
-bash: export: `': not a valid identifier

I have also spotted in the log a warning: “not part of stable configuration, aborting”, not sure how relevant is that. I have attached the log
hashicorp_vault_log.txt (4.8 KB)

Not too sure way forward from this . Any help is deeply appreciated