Error restoring Vault backup

Vault
1

Hello, I am attempting to restore a vault backup snap file to a newly created vault instance to validate the backup files are intact and working as expected.

I am running into an issue where the manually create snap file will not restore over a fresh vault instance.

Details:
Vault Validation instance info:
Key Value

Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.6.2
Storage Type raft

Vault Backup source instance:
/ $ vault status
Key Value

Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
Version 1.6.2
Storage Type raft

All vault instances run on k8s 1.19.15 infrastructure. One note, we use auto unseal for our clusters that are generating the backups. I am not sure if this would cause an issue with a fresh basic cluster used to validate the snap file.

When I copy the snap file generated using the following command:

vault operator raft snapshot save vault-snapshot-date +%F.snap

to the fresh vault instance and attempt a restore with this command:

vault operator raft snapshot restore -force vault-snapshot-2022-06-08.snap

I get the following error in vault:

Error installing the snapshot: Error making API request.

URL: POST https://127.0.0.1:8200/v1/sys/storage/raft/snapshot-force
Code: 500. Errors:

  • 1 error occurred:
    • failed to read snapshot file: failed to read or write snapshot data: unexpected EOF

Of note, I am able to restore a much older snap file created prior to us moving to an auto unseal TSE model.

I am wondering is there is a way to test the snap file being created and what would likely cause the error seen above.

All k8s nodes run on CentOS7, kubelet 1.19.15, docker.

2

Are you restoring the snapshot to the active cluster node? I have reported a bug, that misleading errors happen if you attempt snapshot operations on a follower node: `vault operator raft snapshot save` and `restore` fail to handle redirection to the active node · Issue #15258 · hashicorp/vault · GitHub

You have mentioned that your source cluster uses auto-unseal. This means the cluster where you restore a backup must also be configured with access to the same auto-unseal key.

3

the node is the active node attempting to do the restore. HA mode is active for the node.

OK, so if I understand, using auto-unsealed backups would require the fresh instance to also access the TSE from the original cluster.

Is there a process for this type of restoration? Would I simply need to migrate the new fresh instance over to the TSE and then attempt a restore of the snap file? Any help on the process to restore and auto-unsealed backup would be very much appreciated. I think the basic backup/restore docs do not deal with this.

4

What’s TSE?

I’m not familiar with that acronym.

5

shoot, I am sorry transit secret engine, might be an inhouse term for the auto unseal instance.

6

Yes, you would need to migrate to (or just destroy the temporary cluster and start afresh using) the production auto unseal method, to make restoring the snapshot work.

7

Thank you, i will give that a try. Appreciate your time and responses.