How to debug leases?

Vault
1

Hi,

In last few days my Vault cluster reporting a lot of new leases which cause increasing memory usage.

I’m using Vault 1.4. Around 90 new leases are created every 60s.

How can I found why this is happen? Any idea how to debug it?

thanks,
jacek

2

If you don’t already have one setup, configure an Audit Device and get the data sent over to an aggregation/presentation tool, such as Kibana or Splunk.

You can then run queries against the data to narrow down the culprit, which should help you identify the cause.

3

hi @jeffsanicola

I already set up audit logging. For which patterns should i look?

4

I would start with request.operation="create" (I’m leaving off request.operation="update" as it sounds like these are new leases, but might be worth having a look at that too) and either request.path="auth/*" or NOT request.path="auth/*" depending on which leases you’re referring to.

I’d then look at a summary of a count on request.path and auth.display_name to see what and, hopefully who, is causing the spike.

Once you have some results, look at some of the other fields in the records and filter more as necessary.

5

hi,

it’s look like i have no create operations :confused:

sorry for rusty bash:

# tail -n 100000 audit.log | awk -F '"operation":"' '{print $2}' | awk -F '"' '{print $1}' | sort | uniq -c
    308 list
  97679 read
   2013 update
6

Hrmm, perhaps I had a bad assumption.

Might want to check the API documentation and see which HTTP methods are associated with generating the type of credentials in question then look at filtering by the relevant action.

Looks like read and update may be both valid so you may need to explore both of those more in depth.