Backup for integrated storage(raft) with autounseal without restart

Vault
1

We have set up automatic unsealing for the vault using the Transit Engine and Vault Agent, which generates tokens for accessing the external vault and stores them in a file that is used as the VAULT_TOKEN variable when starting the service through the EnvironmentFile in systemd. However, we encountered an issue when using integrated storage (Raft) while trying to create a backup. We received the error
Error taking the snapshot: incomplete snapshot, unable to read SHA256SUMS.sealed file because the token becomes invalid after a certain period of time. The only way to read it from the file again is to restart the vault, after which the backup will function again until the token expires. This is not a convenient solution, especially for a production environment. Perhaps there are other, more elegant ways to solve this problem?

2

Possible you are running into this, and not starting the backup on the primary node?

3

unfortunately, no, I double-checked that the node is the master before launching

4

If you have tracked this to token expiration, have you tried extending the TTL (either dedicated role for backups, or one time as a test) with a long enough window to complete?

5

I have enough time to create a backup, but I don’t want to have to restart Vault every time the token expires. Also, the Vault team doesn’t recommend using long-lived tokens, if I remember correctly

6

Is Vault Agent generating a new Vault token and replacing it every time the old one expires? If so, you may be able to send a SIGHUP to tell Vault to reload its configuration file. You should be able to do that by configuring Vault Agent to run systemctl reload vault.service after creating the new token.

However, this may or may not work with EnvironmentFile. I believe systemd only uses that on startup/launch, not on reload but its worth a try. You may need to use indirect value references within the seal stanza to point directly to the file rather than using EnvironmentFile.

seal "transit" {
  address         = "https://vault.example.com"
  token           = "file://....." # value/path of response-wrapped token
  disable_renewal = "true" # if using Vault Agent

  # Key configuration
  mount_path         = "transit/"
  key_name           = "vault"
}

Be sure to checkout the best practices when it comes to the Vault transit engine for auto-unseal.

8

Thank you for the suggestion, @michaelkosir . I have tested it in my test environment and the vault does read the token from the file. However, unfortunately, it does not reread the token when the service is reloaded, and the issue with the expired token persists until I restart the service

9

Yeah, it seems to be that Vault does not reload the token file on SIGHUP, which makes it difficult to manage externally (using Vault Agent/etc). Possibly a feature enhancement if you would like to submit a request.

The simplest option would be to use a periodic token instead, and have Vault manage the token internally, rather than using Vault Agent. A periodic token can live “forever” as long as it is continuously renewed. In this case, as long as Vault is online, it will renew the token so it never expires. Be sure to make it an orphan token, so that it is not attached to a parent token.

vault token create \
  -orphan \
  -policy="autounseal" \
  -period=24h