This is my first post in here, and I’m relatively new to Vault. Where I work we use it for a number of applications within various automations.
Historically we used to have a problem whereby if an automation didn’t run for 96 hours then the token would expire and we’d need to renew it. This is actually quite painful for us as we then need to go and update all the automation software and scripts with the new token code.
I wrote some powershell that checks the token ttl (using vault token lookup) and if it’s under 26hrs it then triggers a renew (using vault token renew). This all works absolutely fine.
I also wrote some powershell that on a weekly basis logs in as the user that created the token above so that our user access token doesn’t expire when it comes to running the lookup & renew commands above.
What we’re noticing though, is that every 32 days without fail the token is still expiring even though the user access token has 700+ hours remaining, and the ttl of the vault token is well over 90hrs. Is there some other token somewhere that I need to or can refresh as I don’t see anything when I do the lookup on either the login token or the actual vault token.
In short, if it’s not a special ‘periodic’ token, or a root token with infinite TTL, there is a maximum lifetime enforced beyond which renewal can no longer extend it.
The global maximum defaults to 32 days
The global maximum can be adjusted in the Vault server configuration file (max_lease_ttl)
The global maximum can be overridden on a per-auth-method basis (e.g. see the vault auth tune -max-lease-ttl=... CLI command)
The maximum can be further limited (but not extended) by configuration profiles (roles) within some auth-methods.
Thanks for the information, hopefully there’s something there I can work with. If we can do something with that auth method override command. I’ll take a look next week.
Thanks again for your help, I’ll report back with if and how we solve this.
If you truly need your tokens to live forever, if continuously renewed, look into making periodic tokens.
Alternatively, you can give your automation credentials which are not tokens - e.g. AppRole and let them perform a login request to get a new token as needed.
In general it isn’t good having very long lived tokens isn’t a great idea. Instead of trying to make things last longer as @maxb mentions you are probably better off at having short lived tokens created via AppRoles when they are needed.
Well, maybe. Although, if you just trade your long-lived token for a long-lived AppRole SecretID, you’ve really just moved the long-lived credential that you need to be sure to properly secure into one deeper layer of abstraction…
Thanks for the replies. Apologies for the delay in my response but I was out of the office for a couple of days. I’m thinking that going down the periodic token route is the way we need to go, with a ttl of something like 96 hours as the automation we run typically runs every day but we’ll leave some ‘slack’ as it doesn’t get run over a weekend or bank holiday.
Anyway, I’ve read the documentation that references periodic tokens but I can’t quite get my head around how I make one. At the moment to create a token I’d login off the CLI with the vault.exe and then use the ‘vault token create’ command and define the policy that’s used. I’d have to admit that someone else (much cleverer than me) set this up initially and I just follow instruction to recreate the token when it fails! If I have a look at the policy (vault policy read command) I see:
All tokens (apart from a few very special administrative ones) are created through auth methods.
Most auth methods implement a “login” operation - a user or system presents some kind of credential to prove its identity, and Vault returns a token to be used in further API calls. AppRole, as referred to in your quoted policy snippet, is one such auth method.
The token auth method is technically still an auth method in the Vault model, but an unusual one. It is the only built in auth method. It does not have a login operation, and you can only ever use it if you are already authenticated, and have relevant permissions. It’s a way to manage tokens that already exist, and create more, subject to policy, without going through a “login” step.
Periodic tokens can be created through either route.
In the case of an auth method with a login operation, the auth method’s roles (a.k.a. configuration profiles) will generally include a token_period setting, which, if non-zero specifies to create a periodic token with that period.
In the case of the token auth method the above still applies, if you are using the auth/token/create/:role_name API (or in CLI terms, vault token create -role=...) - but in addition, there is also a period setting that can be passed to the APIs that create a token without using a role.
At this point, it would be helpful if you could post the full complete vault token create ... command line that you are using, so we can be clear as to what options you currently make use of.
From what you’ve shown, I can see that you are essentially an administrator of your Vault, having broad permissions to make use of any policy you feel like. This is fine, if it fits your requirements. I’m just used to working in an environment where corporate security rules prohibit any human from having this level of access !
A couple of things I feel I should call to your attention - this token you have created still retains a personal association with you (see the entity_id field) and inherits any policies assigned to you via the Vault identity system. Apparently there aren’t any right now (no identity_policies in the above output) but if any were later added, this token would acquire those permissions too. Additionally, since this token is a child of your own login token, which is not periodic, it will be revoked when your own login expires.
To address this, you’ll want to use the -orphan flag to create an unrelated token, not bound to your own identity.
Thanks for the information, that’s good to know. Next time I’ll create the token as an orphan.
I’m a bit confused though as to why we keep hitting the 32 day limit regardless of how many times the token gets accessed or refreshed.
That token was created by a service user that I have on a windows box that runs some poweshell. Literally I logged on as that user to create the token, and we then have 2 scripts that run as follows:
Every 4hrs one script runs as and does a vault token lookup on the token, and if it’s less than 26hrs remaining in ttl then we do a vault token renew
Every week I run another script that logs in to vault as the service user to generate a new login/accessor token otherwise we were find that would expire at 768hours/32 days.
Are we still getting tripped up at 32days because the user token that was used to create the token will expire at 32days even though we’ve been generating new ones every week since then?
Yes. (Assuming I correctly understood your high level description of your scripts. Showing the scripts, or at least the relevant parts, is usually better than paraphrasing them in English.)
Cool, I’ll let you know how it works out when we approach the next 32 days. I won’t be able to generate the new orphan token until next week, so it will be the end of March when we find out. Really appreciate all your help with this, and I’ll be sure to report back with the outcome. Thanks again for your help.
This morning I’ve changed our token to be an ‘orphan’ and have updated all the automation. Typically in 32 days from now I’m on leave, but I’ve set myself a calendar reminder when I’m back to check how it’s going. I’ll report back with the outcome in early April.
Just to let you know that changing the token to orphan has worked We didn’t get tripped up at the 32 day point and everything still appears to be working. Thanks so much for all your help.
