Error restoring Vault backup

Hello, I am attempting to restore a vault backup snap file to a newly created vault instance to validate the backup files are intact and working as expected.

I am running into an issue where the manually create snap file will not restore over a fresh vault instance.

Vault Validation instance info:
Key Value

Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.6.2
Storage Type raft

Vault Backup source instance:
/ $ vault status
Key Value

Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
Version 1.6.2
Storage Type raft

All vault instances run on k8s 1.19.15 infrastructure. One note, we use auto unseal for our clusters that are generating the backups. I am not sure if this would cause an issue with a fresh basic cluster used to validate the snap file.

When I copy the snap file generated using the following command:

vault operator raft snapshot save vault-snapshot-date +%F.snap

to the fresh vault instance and attempt a restore with this command:

vault operator raft snapshot restore -force vault-snapshot-2022-06-08.snap

I get the following error in vault:

Error installing the snapshot: Error making API request.

Code: 500. Errors:

  • 1 error occurred:
    • failed to read snapshot file: failed to read or write snapshot data: unexpected EOF

Of note, I am able to restore a much older snap file created prior to us moving to an auto unseal TSE model.

I am wondering is there is a way to test the snap file being created and what would likely cause the error seen above.

All k8s nodes run on CentOS7, kubelet 1.19.15, docker.

Are you restoring the snapshot to the active cluster node? I have reported a bug, that misleading errors happen if you attempt snapshot operations on a follower node: `vault operator raft snapshot save` and `restore` fail to handle redirection to the active node · Issue #15258 · hashicorp/vault · GitHub

You have mentioned that your source cluster uses auto-unseal. This means the cluster where you restore a backup must also be configured with access to the same auto-unseal key.

the node is the active node attempting to do the restore. HA mode is active for the node.

OK, so if I understand, using auto-unsealed backups would require the fresh instance to also access the TSE from the original cluster.

Is there a process for this type of restoration? Would I simply need to migrate the new fresh instance over to the TSE and then attempt a restore of the snap file? Any help on the process to restore and auto-unsealed backup would be very much appreciated. I think the basic backup/restore docs do not deal with this.

What’s TSE?

I’m not familiar with that acronym.

shoot, I am sorry transit secret engine, might be an inhouse term for the auto unseal instance.

Yes, you would need to migrate to (or just destroy the temporary cluster and start afresh using) the production auto unseal method, to make restoring the snapshot work.

Thank you, i will give that a try. Appreciate your time and responses.