How to unseal vault server after raft snapshot restore?

Vault
1

Hello,

I am trying to restore a Raft snapshot onto a new Vault cluster.

I have Cluster A (Vault raft HA Cluster with transit auto-unseal) and new Cluster B (Vault raft HA Cluster with transit auto-unseal.) up and running.

steps:

  1. Take raft snapshot from cluster A.
  2. Restore in cluster B
    $ vault operator raft snapshot restore -force raft.snap
  3. Now vault server is sealed because Autounseal keys are different in the snapshot restored.
  4. Tried to unseal using new and old Recovery Keys but no luck. Restored database has different key and tranist engine is not able to decrypt.
    Error:
  • failed to decrypt encrypted stored keys: Error making API request.

URL: PUT http://vtransit_transit:8200/v1/transit/decrypt/autounseal
Code: 400. Errors:

Not sure what I am doing wrong. Any insight appreciated, thanks.

2

Recovery keys cannot unseal Vault.

IMO, that makes their very name a false promise, and I can’t imagine why Vault was designed that way.

But, all recovery keys are useful for, is for a quorum of Vault administrators to prove to the Vault software that they have a quorum, to authenticate operations which require that.

In your situation, you need to reconfigure cluster B to use cluster A’s transit autounseal configuration.

(And then, if desired, follow the seal migration process on cluster B.)

3

Since this keeps coming up, I’ve filed Auto-unseal Recovery Keys should be able to unseal Vault · Issue #15490 · hashicorp/vault · GitHub

4

Thanks for your inputs. I tried reconfiguring cluster B to using cluster A’s transit autounseal configuration but same error.

Steps:

  1. copied transit server ‘data’ directory from Cluster A to Cluster B and brought up the server in Cluster B.
    Transit server config used to bring up Cluster A:
    storage “file” {

path = “/vault/data”

node_id = “node1”

}

listener “tcp” {

address = “0.0.0.0:8200”

tls_disable = true

}

disable_mlock = true

------- Script to start and generate token-----------

vault login $VAULT_TOKEN > /dev/null

vault status

vault audit enable file file_path=/var/log/vault.d/vault_audit.log

vault secrets enable transit

vault write -f transit/keys/autounseal

tee autounseal.hcl <<EOF

        path "transit/encrypt/autounseal" {

        capabilities = [ "update" ]

        }

        path "transit/decrypt/autounseal" {

        capabilities = [ "update" ]

        }

EOF

vault policy write autounseal autounseal.hcl

vault token create -policy="autounseal" -wrap-ttl=120 >/vault/data/policy-token.txt

vault unwrap $(cat /vault/data/policy-token.txt | grep -w "wrapping_token:" | awk '{print $2}') >/vault/data/vault_token.txt

------Below executed on Cluster A and Cluster B in one vault server (initial leader) --------
vault operator init -recovery-shares=5 -recovery-threshold=5 > /vault/data/recovery-key.txt

  1. Verified all the vault server were up( auto-unseald).
  2. restored raft snapshot in Cluster B.
  3. Vault sealed and not able to unseal.

Please help !

5

I am confused by what you mean by this. Please explain more. Please also show the configuration file of the main Vault clusters.

6

Copied this data directory of transit server from cluster A to Cluster B so that it will not initialize again and can be brought up with old token and configuration.

– Vault config–

image
image785×837 52.2 KB

7

I think that to make progress here you need to look at the errors being returned from vtransit_transit - the ones which are cut off and not included in your original message:

What comes after this?

8

The simplest way around this is to migrate to shamir first, then do your backup and restore. When you have auto-unseal your keys are no longer “unseal keys” they are “recovery keys” for cluster recovery, not for the assembling the encryption key that Vault needs. Moving back to shamir, you’ll end up with “unseal keys” again.

There are more complicated steps you can go through to do this, but I think this is the fastest method. A safer method does exist but it does require an Enterprise license.