I had a critical issue in my cluster and had to restore the vault.
At first, I created the vault and then used the restore file to restore the vault, pod number 0 (from 3 pods) is unseal as expected, but 2 others are not.
Pod 1 & 2 getting this error:
/ $ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce e68d14e5-e996-f00c-94fc-cd950ee188d0
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
/ $ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce e68d14e5-e996-f00c-94fc-cd950ee188d0
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
/ $ vault operator unseal
Unseal Key (will be hidden):
Error unsealing: Error making API request.
URL: PUT http://127.0.0.1:8200/v1/sys/unseal
Code: 500. Errors:
* raft is already initialized
You can see the third unseal I’m doing finish with an error raft is already initialized
Why is that? what I can do to resolve it? Currently, the vault working with only one pod unsealed, and the 2 others are sealed.
/ $ vault operator raft join http://vault-server-0.vault-server-internal:8200
Key Value
--- -----
Joined true
/ $ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 37711a37-0fbf-fa3d-9f29-14ea2eae73e4
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
/ $ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 37711a37-0fbf-fa3d-9f29-14ea2eae73e4
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
HA Enabled true
/ $ vault operator unseal
Unseal Key (will be hidden):
Error unsealing: Error making API request.
URL: PUT http://127.0.0.1:8200/v1/sys/unseal
Code: 500. Errors:
* error bootstrapping cluster: cluster already has state
Ah, the classic “raft is already initialized” message can certainly throw a wrench in your day, especially when you’re just trying to get your Vault cluster back on its feet! Let’s see if we can untangle this a bit.
So, you’re dealing with a 3-pod Vault setup using Raft as the storage backend, and after a restore, only one pod is playing nice. The other two are stuck with their seals on, giving you the cold shoulder with that “raft is already initialized” error on the third unseal attempt. It’s like they’re saying, “Hey, we’re already set up here, no more unsealing needed!” But obviously, that’s not the case since they’re not up and running like their buddy pod 0.
First off, it’s great that you’ve got one pod up; that means your data is intact and your cluster is somewhat operational. The issue with the other two pods likely stems from how Raft’s consensus mechanism got a bit confused during the restore process.
Here’s a plan to tackle this:
Check Cluster Peers: Start by checking the Raft peer set in the operational pod (pod 0). You can do this by running vault operator raft list-peers. This will give you an idea of what the cluster’s consensus state looks like from the perspective of the unsealed pod.
Remove Unresponsive Peers: If pods 1 and 2 are listed as peers but are in an unresponsive state, consider removing them from the peer set using vault operator raft remove-peer command. Be careful with this, as it directly manipulates the Raft consensus state.
Restart and Unseal Pods: After removing the problematic peers, restart pods 1 and 2. Upon restart, they should recognize that they’re not part of a Raft cluster and start in a fresh state. You’ll then attempt to unseal them again. If they unseal successfully, they should automatically join the Raft cluster as new nodes.
Verify Cluster Health: Once all pods are unsealed, verify the health and status of the Raft cluster again using vault operator raft list-peers from any of the pods. All three should now be listed as peers and in a healthy state.
It’s a bit like convincing the two stubborn pods that they’re not yet part of the club and need to be formally reintroduced. A touch finicky, but that’s Raft for you – it’s all about consensus and making sure every member is in agreement.
If you’re still hitting a wall after these steps, it might be worth looking into the specific restore process you used and ensuring there’s no residual state in pods 1 and 2 that’s causing them to think they’re already part of the cluster. Sometimes, a deeper clean (like purging Raft data files) is needed before they’ll play ball.
Remember, messing with the Raft state can have significant implications, so tread lightly and ensure you have backups or snapshots you can revert to if things go sideways. And hey, it’s always an adventure, right? Let’s get those pods back in line!
To ensure it will have no old data after the pod is started.
But here comes the plot, I’ve used IP to join pod 0 instead of DNS, which worked when DNS didn’t worked (By using DNS raft was joined but the unseal didn’t worked).
In this case my next step was:
vault operator raft join http://10.40.190.170:8200 #Pod-0 IP
vault operator unseal #3 times
Copy your Vault Raft Snapshots for the Primary and Secondary Performance Replica clusters onto restored members of the respective clusters and run the below following command, replacing the filename with that of your snapshot file.
This means using the snapshot on every pod, I’ve tried this, and it’s not working, the guide is not good.
When restoring, you need to restore it on one pod and join the others to follow him, otherwise, every pod thinks he is the leader.