Raft snapshot restore issue

Vault
1

Hello,

I am trying to restore a Raft snapshot onto a new Vault cluster. I can restore successfully, but none of the nodes ever join as a leader. The old cluster was 5 nodes, but now it is just 3.

I’ve been restoring with the command:
vault operator raft snapshot restore -force /path/to/snapshot

Not sure what I am doing wrong. Any insight appreciated, thanks.

2

What steps are you doing? Are you unsealing after you restore?

3

I initialize the vault cluster, then run the restore command. After that I restart the Vault service which unseals via auto-unseal.

4

We did have Audit enabled and I see error messages about it not being able to open the audit file because it doesn’t exist.

I know that Vault stops serving requests if it can’t run the audit log. Going to try and get the audit file created before the restore to see if that is the issue.

5

I was able to get it restored. Had a couple issues:

  1. The previous cluster was a quorum of 5 hosts, new 3
  2. The backup had audit logging configured, with the new hosts not having the file
  3. Vault will stop serving requests if audit is enabled and can’t log

So I was able to successfully join as a leader with 1 node by following this process:

Now just need to figure out bringing the other nodes online.

6

How did you manage to unseal after snapshot restore when you have transit engine for autounseal. I am getting error

  • failed to decrypt encrypted stored keys: Error making API request.

URL: PUT http://vtransit_transit:8200/v1/transit/decrypt/autounseal

7

I recommend you start a new topic, this does not appear to be strongly connected to the previous discussion.