Transit autounseal tokens aren't renewing themselves

Vault
1

Hi all

We have a few Vault servers running in Kubernetes clusters which unseal with keys from a master Vault. Since transitioning over to autounseal on all subordinate vaults, we’ve seen that the unseal tokens aren’t being renewed which results in failure to unseal should Vault be restarted for any reason.

Configuration is pretty vanilla, we’re running Vault 1.2.0 all round with pretty much the same settings as used here: https://learn.hashicorp.com/vault/operations/autounseal-transit

Path          Plugin       Accessor              Default TTL    Max TTL     Force No Cache    Replication    Seal Wrap    Options    Description
----          ------       --------              -----------    -------     --------------    -----------    ---------    -------    -----------
transit/      transit      transit_0a1d21b7      15768000       31536000    false             replicated     false        map[]      n/a

Subordinate Vaults have the following seal config:

seal "transit" {
  address = "https://vault.domain.com"
  disable_renewal = "false"
  key_name = "autounseal"
  mount_path = "transit/"
}

The auth/token path on the master Vault was configured with -default-lease-ttl=4380h -max-lease-ttl=8760h after it was stood up, but otherwise config is the same as the link above, and initial subordinate tokens were created with vault token create -policy="autounseal" -period=4320h. When a Vault gets restarted, the unseal request is met with a 403/permission denied, and a manual lookup on the unseal token returns bad token.

In order to debug this, I created some shorter-lived unseal tokens (between 5 minutes and several hours) and these all renewed fine - the only difference between them and the initial tokens is the TTL they were created with. Examining the audit logs on the master for the initial expired tokens shows no attempt was made to renew them.

I’m not sure where to proceed from here; any suggestions are gratefully received!

2

Just a thought - is there any way to monitor if the token has expired? Or potentially if the subordinate Vault has tried to renew it?

3

Here are a couple of things to check:

  1. Vault with try to renew the token when it first starts up. What do your server logs say during startup? If you enable trace logging, you will also see all the renewals in the server logs.
  2. Check out the information on token hierarchies https://www.vaultproject.io/docs/concepts/tokens.html#token-hierarchies-and-orphan-tokens. It could be possible that you are issuing a token that is a child of another token. When the parent expires, so does the child.
  3. If you enable audit logging, you will see each renewal that happens on the Vault that hosts the transit mount.
    Hope this helps!
4

Thanks Chris, that’s been really helpful.

Trace logging does indeed show that the unseal token is renewed on startup, and based off of your second point I think I can see the issue. We use Github auth for manual interaction with the master Vault and obviously those tokens have a limited lifespan. Given that the unseal tokens were created manually, I think they’re expiring because the parent token from the Github auth endpoint has a shorter ttl than the unseal token which is a child of it. Does that seem sensible? It would also explain why the short-lived tokens I generated for testing were renewed successfully.

If so, the easiest way I can see of resolving this is to shorten the ttl of the unseal tokens to be less than the parent auth token. Does that make sense?

5

Even if you set the TTL shorter than your Github token, once the auth token expires, so will your other token. The simplest thing would be to set the token you generate to be an orphan token which will have no linkage to the parent token.

6

Ok, that’s an easy solution. Thanks Chris!

7

Hi.

I’m investigating a similar issue with Transit auto-unseal. Did creating the autounseal token as an orphan token resolve your issue, or did you find a different solution?

Thanks.

8

Hi Peter - yes it was the solution in the end. Maybe not a perfect solution as the tokens then have no parent and therefore no hierarchy to ensure they can be automatically expired, but as the subordinate vault will handle renewal you can set a short TTL which mitigates the lack of parent to some degree.