Hello, 2 years ago I started to use vault with kubernetes auth.
I now have 50k+ identities created and the UI for identity is unresponsive, either it times out or returns 5xx because of the amount of them.
Is there a way to clean those not used identities (I have around 600 micro-services running at a given time, far from 50k, so I guess not all is useful)?
thanks
Sorry. Do you mean entities? I don’t see how using a kubernetes auth method (it’s a perm/renewable connection using a single entity) would create entities. If that was the case the Enterprise User count licenses would be useless as simply using the tool would drive up the count and that’s not how it works.
I am not sure to me it looks like those are bound. There is one identity which has one alias in the entity. I do not know why, but each new k8s app release (even if using same ServiceAccount) will create an identity with an entity and an alias.
Sorry I have been absent for a while but my issue still persist, in the attachment you will see we have like 500ish pages of entities.
Each of our k8s deployment is using the same auth endpoint. Each of the deployment have their own ServiceAccount.
This is weird, I would like to upgrade our cluster but I want to clean all of those before.
Is there a way to automatically delete entities after some TTL maybe?
One thing to consider is how the Kubernetes Auth method maps roles to Entities. By default the Kuberentes Auth Method Role uses
alias_name_source = serviceaccount_uid
which means that every time a new k8s Service Account is created, it will create a new entity. If you use Helm to deploy a lot of workloads with Service Accounts, and the CI/CD process does a Helm uninstall/install when deploying workloads to k8s, then a new Service Account is created each time.
Another configuration option is to use the namespace name and the service account name as the alias.
alias_name_source = serviceaccount_name
When you do it this way, as long as the namespace name and service account name are the same, even if the Service Account UID itself changes, Vault will see it as the same Vault entity. Since your Vault is old enough, I’m not sure all of these options are supported.
This github issue comment walks through the conceptual logic of deleting old entities that are past a TTL.
I recently dealt with a problem that was fairly similar at work.
The first thing you need to understand is when and why Vault creates entities. Every time there is any login to Vault, using any auth method, Vault looks up the pair of (auth method, “username”) in the identity store, and creates an entity if there isn’t one there already.
Exactly what “username” means, varies depending on the auth method. As @jbayer points out, the Kubernetes auth method is unusual in that it gives you a choice of two options for what is used as the “username” in this context. If the reason you have so many entities, is because your Kubernetes service accounts are frequently being deleted and recreated with the same name, this option should slow the growth (after an initial peak, as changing it would cause all active identities to create a new identity on next login).
However, you need a way of cleaning up entities that are no longer relevant, and unfortunately this is one of the things that Vault provides no out of the box support for, and requires a Vault admin to write their own script using the API.
If you are using Vault Enterprise: Bear in mind that “number of entities that have performed operations against the Vault API in the calendar month” is the licensed quantity, so effectively re-using entities where appropriate, and not deleting them unnecessarily, has financial implications.
When it comes to cleaning up Kubernetes entities, the approach I used was to query the Kubernetes API server for a list of all service accounts in all namespaces, extract all the UIDs from the response, and use this as a filter to determine what to delete from Vault. I think this is better than TTL based approaches, as Kubernetes service accounts do not have an expiry, and Vault entities don’t track their last use time.
This is clearly complex enough to require writing a script - for implementation language, Python is a good option if you have any previous experience with it, as it has both a library for accessing the Kubernetes API, abstracting away its rather complex authentication peculiarities, and the lovely requests HTTP library that makes talking to Vault easy.
The general approach is:
When it comes to getting all the entities from Vault, it is handy to know that the Vault GET /v1/identity/entity/id/?list=true endpoint returns more than a mere list - it additionally returns summary information for all entries in the list. Here’s an example response from my test Vault (irrelevant fields elided):
{
"data": {
"key_info": {
"c128cebf-97f6-9654-35b8-8181a1ab4885": {
"aliases": [
{
"id": "1247f01b-afb2-dd22-cace-fe3f57827baf",
"mount_accessor": "auth_userpass_10f3c534",
"mount_path": "auth/userpass/",
"mount_type": "userpass",
"name": "admin"
}
],
"name": "admin"
}
},
"keys": [
"c128cebf-97f6-9654-35b8-8181a1ab4885"
]
}
}
Do note that for your test Vault this response is going to be gigantic, and there is a risk of it timing out completely, which would be a problem, as Vault APIs don’t support pagination 
.
The above example is for the userpass auth method because I don’t have Kubernetes set up in this environment. In particular, pbserve that entity-alias information has been provided in-line with the entity information:
mount_type will let you easily filter kubernetes entities and ignore othersmount_path will let you distinguish between multiple Kubernetes auth methods, if relevantname (within aliases) will be the Kubernetes service account UIDTechnically, one entity can have multiple entity-aliases, but this is never the case for auto-created entities - only ones administratively configured via the Vault API.
Hopefully that sets you on the path to constructing a cleanup script for your environment.
Lastly, a correction to incorrect information earlier in this topic:
As detailed earlier in my response, all auth methods create entities.
A multitude of different Kubernetes service accounts using a Kubernetes auth method is very much a concern for Enterprise users’ licensing - as I know from monitoring usage and planning to ensure we remain within our license count at my place of work.
Hello thank you for the replies.
@jbayer indeed this is a sweet alternative that I will definitely use in the future, as well as in addition of a script to clean the excess if any.
@maxb
Thank for your time to go throu all the steps. Indeed I ended up creating a python script, but I am going with hvac module, to avoid all the requests ugly code 
Yeah the pagination is a nightmare, and still 50k is ‘kinda’ ok, took few seconds and overload the vault a liltle bit but its manageable - when it doesnt crashes the network. Fortunately I always store a success call locally to avoid redoing the call when trying things.
Very useful information, I can’t wait to finally upgrade this cluster, its so old, hopefully the migrate will be without issues !
Thanks again.