Vault Cluster token expired, Need to recreate

Hello, I am new to vault and am inheriting a vault 4 node environment. I have 3 servers in a cluster that talk to a master vault server. The token that the clients used to authenticate expired. I am trying to create a new token for the clients on the master vault server but I am unable to log in. I tried using auth methods userpass and ldap to log in, but both methods say I am missing a token. What token is it referring to?

Password (will be hidden):
Error authenticating: Error making API request.

Code: 400. Errors:

  • missing client token

After doing more research it seems I would need to generate a new unseal token. My original one has expired. Since the unseal token is expired my GUI is down. I do not have the root token and only know my personal token when I’m able to log in via the GUI and view it. So I know I need to generate a new token but am unable to do so because I don’t have a token. I’ve tried to log in user auth method userpass but that still requires a token.

I created a new root token. Now when trying to renew token I get
$vault token create -role=auto-unseal -policy=auto-unseal -orphan
Error creating token: Error making API request.
URL: POST https://hostname:8200/v1/auth/token/create/auto-unseal
Code: 403. Errors:

  • permission denied

You’re confusing terms here.

An unseal key, is a shard of your master key, and do not expire. Make sure you have your set of keys available. A good idea is to re-key the instance (rotate the unseal keys) but let’s leave that until you’re more comfortable with the setup.

A token is a just the result of a valid authentication to the system. All user tokens have a TTL (time-to-live) and expire. Tokens are provided as part of a request to the instance to prove that you’re authenticated and allowed to request the information you want.

To create a new token, you need a couple of items:

  • A way to authenticate yourself
  • What policies the token should get.

As far how to authenticate, This can be userpass, LDAP, or OIDC by default (and most common). If you don’t have any of these, there is the “root” token which does not expire and has full access to the instance. Common practice is not to leave the root token laying around and revoke it.

If you don’t have a way to authenticate, nor do you have your root token you can generate a new root token using your unseal keys, the number of keys you need depends on how the environment was setup.

# vault status
Total Shares .......... 5 
Threshold ............. 3 

Tells you that there are 5 keys in the set, but you only need 3 out of 5 to be able to unseal or generate a new root token.

Hello Aram, thanks for the response. When I referred to unseal token I am referring to the token that was created for my vault cluster to authenticate with the master vault server. Since my cluster cannot communicate with the master vault server, the cluster vault services are failing to start. We also have Haproxy running as a load balancer in front of the vault cluster. This VIP is failing because all the cluster vaults are down. The master vault server cannot contact the API due to the vault cluster being down. Hopefully, that makes sense. When my environment is in such state, how do I bypass the API and create a new token that my vault cluster can utilize?

I’m not sure what a “master vault server is”, the closest thing I can think of is the relationship between a performance replicate instance and the primary vault cluster. But that doesn’t sound like what you have.

I imagine, you have one cluster, multiple instances, and they’re all down because none of the vault nodes are up. This can be many different issues, the simplest issue is that Vault is sealed (locked).

If that’s not it, then you need to open a case with support (if you’re running Enterprise and have support). If you’re on the open source version you can find partners that you can get in as a Consultant.

Thanks for the response again. I figured out the issue. I agree the environment we have is very weird and is not supported by much documentation. We have a cluster of 3 nodes that authenticates to a master vault server. Not sure why it was set up like this.

1 Like