Number of tokens inconsistent?

Hello,
I see some inconsistent number of tokens in vault. When I query the API endpoint /v1/sys/internal/counters/tokens I get a reply where “data”:{“counters”:{“service_tokens”:{“total”:19053}}}.
However when I list /v1/auth/token/accessors I get a list with about 50 keys.
What are the other 19000 remaining service tokens?

A bit of background story: I want to do a migration to a new cluster following the standard backup/restore procedure with a raft storage. That seems to work fine, but I also do not want any cruft in the new system. The current vault is v1.6.2, the new vault I tried to use directly 1.16.1, and it seems to be fine.
The massive number of 19053 tokens is already existent in the old instance, I would like to avoid to start without this cruft in the new instance.

Is there a way to revoke every lease and every token besides the root token?

Thank you for any help.

I can’t seem to replicate (using Vault in dev mode so not a totally comparable environment) getting a difference in accessors and service token count. On that front it may be worth opening an issue, or if you have support opening a support ticket.

On bulk revoking existing accessors, there is nothing (AFAIK) built in but using the vault token lookup command you could script this. An ugly but working (at least in my test environment) example:

Please use with caution, this is not official

# Get root token accessor so you don't revoke it
accessor_root=$(vault token lookup -format json | jq -r '.data | .accessor')

# Get all existing accessors
eval "accessor_list=(${$(vault list -format json auth/token/accessors)//[\"\'\[\]]/})"

# Loop through accessors that do not match the root token accessor and revoke
for accessor in "${accessor_list[@]}";
do
   if [[ "${accessor//,/}" != "$accessor_root" ]]; then
      vault token revoke -accessor ${accessor//,/}
   fi
done

This is exactly where the mismatch is. There are no token accessors.
What I have found out is that if I enable the sys/raw endpoint, then I can find the list of tokens at /sys/raw/sys/token/id.
There is the list of ~19k tokens, which have neither a lease nor a token accessor in the standard vault API.
I tried deleting them through the raw endpoint (ignoring the root token), and did not find any side-effect so far. This seemed to have worked, but I am not sure how these tokens even ended up there.
Randomly inspecting some of them, there validity was also already expired, and even if the validity was not expired, trying to use them was not successful. So they were in the database somehow, but useless from what I can say with my limited tests.

Sounds similar

Wonder if it could be related.

I think this is it.
And the suggestion there is to use the /sys/raw endpoint for manual cleanup.
Maybe in a future version vault is able to cleanup these stale tokens, for now my question is answered. Thanks for the linkt to the github issue, I will follow from there.

1 Like

Thanks for talking/writing through your experience. I have added a note to highlight the /sys/raw endpoint in an upcoming tutorial on troubleshooting.

Could the discrepancy be due to different types of tokens being counted in /v1/sys/internal/counters/tokens? For example, are some tokens inactive, expired, or otherwise not accessible?