How can I rebuild a `ha_storage` enabled cluster using existing data in MySQL?

Vault
1

I have a three-node cluster that uses internal storage for HA coordination and uses MySQL for data persistence. All three nodes were dead and I tried to rebuild them from scratch. I use the exact same server configuration to build servers. Vault servers were started and unsealed without any problem. But I just can’t re-form the cluster. The vault command returns either local node not active but active cluster node not found or unable to fetch leadership entry. My questions are:

  1. Is there any way to re-initialize the cluster without wiping out the data in a database?
  2. Is it possible to force promote a server to become the leader?
  3. Since I still can run the Vault server in standalone mode by removing the ha_storage section in the config file. Is it possible to convert the server from standalone mode to HA mode?

Here are the steps to reproduce the issue.

  1. Start three vault servers using the configuration below, listening at 8200, 8210, and 8220 respectfully.
ui = true
api_addr = "http://127.0.0.1:8200"
cluster_addr = "http://127.0.0.1:8201"

storage "mysql" {
  address = "127.0.0.1:3306"
  username = "****"
  password = "****"
  database = "vault"
  plaintext_connection_allowed = true
  max_parallel = "5"
}

ha_storage "raft" {
  path    = "/vault/node0"
  node_id = "vault_0"
}

listener "tcp" {
  address         = "127.0.0.1:8200"
  cluster_address = "127.0.0.1:8201"
  tls_disable     = "true"
}

  1. Init the first server and join rest two servers to the cluster. Now the cluster now becomes normal.

  2. Stop Vault servers and wipe out the data path defined in ha_storage (/vault/node0, /vault/node1, /vault/node2 in my example).

  3. Start Vault servers again and unseal them.

2

I found the solution. The /v1/sys/storage/raft/bootstrap API is exactly what I need. But it requires a manual step to remove the core/raft/tls key from storage before calling this API. Otherwise you will get a could not generate TLS keyring during bootstrap: TLS keyring already present error. Thanks for the information in the thread

3

Big brother, how to join the cluster, help me