Manual raft snapshot creation fails

Vault
1

Hi everyone,
I have a Vault cluster with 5 nodes, and they are configured to auto-unseal from another Vault node (http://vault-node-1:8200) which is configures with transit secrets engine.

Auto Unseal works as expected when I restart Vault service on any of the 5 cluster nodes.
But when I want to create a raft snapshot manually it fails with this error in CLI

Error taking the snapshot: incomplete snapshot, unable to read SHA256SUMS.sealed file

When I look at logs it shows me this:

vault[2806038]: 2025-02-06T12:10:59.334+0200 [WARN]  core.autoseal: seal wrapper health check failed: seal_name=transit
vault[2806038]:   err=
vault[2806038]:   | failed to encrypt test value, seal wrapper may be unreachable: Error making API request.
vault[2806038]:   |
vault[2806038]:   | URL: PUT http://vault-node-1:8200/v1/transit/encrypt/unseal-key
vault[2806038]:   | Code: 403. Errors:
vault[2806038]:   |
vault[2806038]:   | * 2 errors occurred:
vault[2806038]:   | \t* permission denied
vault[2806038]:   | \t* invalid token
vault[2806038]:   |

I am doing this from the Leader node while logged in with root token.
Not sure why it complains on permission denied and invalid token ?
It should use the token from the configuration from seal transit stanza right ?

seal "transit" {
  address     = "http://vault-node-1:8200"
  token       = "hvs.XXXXXXXX"
  key_name    = "unseal-key"
  mount_path  = "transit"
  tls_skip_verify = "true"
}

When I check that token on vault-node-1 it’s still valid

Key                  Value
---                  -----
accessor             Od4jqaOpYPSZGFfgtibo6VyR
creation_time        1738360801
creation_ttl         768h
display_name         token
entity_id            n/a
expire_time          2025-03-05T00:00:01.346753128+02:00
explicit_max_ttl     0s
id                   hvs.XXXXXXXX
issue_time           2025-02-01T00:00:01.732063539+02:00
last_renewal         2025-02-04T17:26:10.346753258+02:00
last_renewal_time    1738682770
meta                 <nil>
num_uses             0
orphan               false
path                 auth/token/create
policies             [default unseal]
renewable            true
ttl                  635h58m15s
type                 service

Am I missing something ? Perhaps I need to add something in policy that looks like this ?

path "transit/encrypt/unseal-key" {
  capabilities = [ "update" ]
}

path "transit/decrypt/unseal-key" {
  capabilities = ["update"]
}

Thanks !

2

Ok after lot of checks I just restarted vault services on all nodes and it solved the issue

1 Like