Backup/Restore K/V data

Vault
1

Hi there

We recently started using vault. We basically use vault as a password manager and therefore only use K/V v2 secret engines. The goal now is, to run regular backups/snapshots of all the secret engines for disaster recovery. Orinially we started with a file-storage. For testing purposes I switched to raft (integrated-storage) to make use of the snapshot function.

I created some secrets and then created a snapshop. I then deleted (permanantly) some secrets and restored said snapshot. I expected the deleted secrets to be restored but that was not the case.

We only have a single instance of vault running and don’t really need HA.

Can somebody explain, how I can create backups and restores of my k/v engines?

I would really appreciate the help.

Thanks in advance,
Regards

2

A snapshot will restore everything. Post your exact steps for snapshot and restore.
https://learn.hashicorp.com also a backup and restore guide you can search for.

3

I’ll try to provide some more information:

Our Vault config.json:

{
  "listener":  {
    "tcp":  {
      "address":  "0.0.0.0:8200",
      "tls_disable":  "false",    
	  "tls_cert_file": "/path/to/cert/file",
	  "tls_key_file": "/path/to/key/file"
	}
  },
  "backend": {
    "raft": {
      "path": "/vault/raft",
	  "node_id": "raft_node_1"
	  }
	  
  },
  "seal": {
    "azurekeyvault" : {
       //auto unseal information 
     }
  },
  "default_lease_ttl": "168h",
  "max_lease_ttl": "0h",
  "api_addr": "https://0.0.0.0:8200",
  "cluster_address": "https://127.0.0.1:8201",
  "cluster_addr": "https://127.0.0.1:8201",
  "ui": true
  }

With this config I can use Vault normally without errors.

Currently, I have several k/v v2 secret engines. For testing purposes, I created some secrets in one of them.
Then I use following CLI command to create a snapshot:

vault operator raft snapshot save raft.snap

This creates the file raft.snap

I then create a new version of a secret by removing one k/v-pair

Then I want to use the following CLI command to restore the previous state:

vault operator raft snapshot restore raft.snap

The command runs without error.

The vault then is sealed (auto-unseal doesn’t seem to work with a snapshot) so I just restart the vault server with

vault server -config=config.json

I would then expect that the one secret where I created the new version is restored to the previous version but that isn’t the case.

Same happens if I try to permanantly delete a secret. The restore command does not restore any data.

As I mentioned, we only have one instance of Vault server running. No HA or anything like that.

What am I doing wrong?

Thanks for the help

Regards

4

Just a small addition:

If I, during runtime, just copy the whole folder “raft” to some other location, I can use that to restore my data. I’m just not sure if this is save. I probably could get corrupted data this way.
This is just my “backup-plan”.

5

Try

vault operator raft snapshot restore -force raft.snap
6

I tried that as well. Didn’t work.

7

Are you following this?

You didn’t mention anything about shutting down vault, reinit’ing, etc.

8

I tried to follow that. I struggle to understand the first point

Bring your Vault cluster back online following the circumstances that required you to restore from backup. You will need to reinitialise your Vault cluster and log in with the new root token that was generated during its reinitialisation. Note that these will be temporary- the original unseal keys will be needed following restore.

So, I only have one single vault server running (is that in my case the same than a “vault cluster”?). And why “back online”? It’s still running. My scenario is, that I permanantly deleted data that I shouldn’t have. How do I “reinitialise” it? I tried to just delete the contents of the raft folder and then do a “vault operator init”. With this I get a new root token as well as unseal keys. I then unsealed vault with the new keys, logged in with the new token and tried the snapshot restore. But I still had to use the new unseal keys to unseal it afterwards and there was no data restored.

I’m sure I’m doing something wrong but I have no idea what.

IMO this first step here has to be a bit better explained.

9

I might have to add that I’m working on a Windows 2019 Server

10

Restore is intended to bring back your data from a dead server. As such, you need to start with a fresh build of Vault with no data or initialization history, outside of the steps outlined there.

It seems like the doc assumes you know how you init’d the cluster originally and some admin duties, if you’re doing backup and restore ops already.

You’ll want to stop the server, delete the integrated storage db files, then start it and init it.

11

Hi

Thanks for your response. I tried that again. I discovered an error and I have no idea why.
Can you help?

I ran the whole thing as administrator. That didn’t help either.

One thing I have seen is, that there is no 3-12802-1626067134301.tmp in raft/raft/snapshots but a folder named “3-12802-1626067134301” that includes a vault.db.

VaultSnapshotError
VaultSnapshotError1159Ă—340 50 KB

Regards

12

Are you using Windows? I’d guess to be permissions issues, but not familiar w/ Windows.

13

I gave everything full access for testing purposes but still get the error.

Here is exactly what I do.

  • Server is running
  • Stop the server
  • Delete everything unter vault/raft
  • Start server again aith same config
  • run “vault operator init” → get new root token and unseal keys
  • login with new token
  • run “vault operator raft snapshot restore -force raft.snap”
  • get the error: “failed to move snapshot into place: error sync \vault\raft\raft\snapshots: The handle is invalid.”

If you have another idea, I’ll behappy to try it out. Otherwise I give up

Regards.

14

@mitti2000 Were you able to resolve this issue ?
I am also facing this issue. I have one vault instance running on k8s with integrated raft storage and restoring snapshot doesn’t restore my secret.

15

@shubham-bansal96 Were you able to solve the problem?

16

@Gokul-C no, not yet

1 Like
17

Hi ,

Yesterday, I worked on resolving these errors, and I’m happy to report that I successfully fixed the issues. I can now perform backups and restorations without any problems. I tested two scenarios:

  1. Restoring secrets within the same cluster
  2. Restoring secrets to a different cluster

Both scenarios worked perfectly fine.

18

@Gokul-C
Did you perform any extra steps other than what mentioned on vault webpage ??

19

Yes @shubham-bansal96 I have modified step 3

  1. Exec into the vault leader pod
    if using kubernetes:
    $ kubectl exec -it vault-0 sh

    if using openshift:
    $ oc rsh vault-0

  2. Login in to Vault
    $ vault login <root-token>

  3. Take a backup to tmp folder
    $ vault operator raft snapshot save /tmp/vaultsnapshot.snap

The user, when executing into the Vault pod, doesn’t have permission to create files in the entire filesystem except the /tmp directory. So, saving the snapshot to the /tmp directory works!