Auditing Vault policies

I would like to find a way to audit Vault policies for access to particularly sensitive paths. I have attempted to use the capabilities endpoint but I don’t think it is suitable. I should say that it is useful but on its own is not necessarily accurate.

For example, I want to audit all policies to see which ones are able to read a particular KVv2 secret, let’s say “secret/application/very-secret/foo”.

My initial approach was to iterate through the list of policies from. Acquire a short ttl token from each, and then call vault token capabilities "$token" secret/application/very-secret/foo.

This mostly works, however if there is a policy with a more explicit deny rule, the capabilities endpoint will incorrectly say this token has access to the key.

Here is my script for this:

set -eou pipefail

policies=$(vault policy list | grep -v root)

for i in $policies; do
  echo -n "$i ... "
  tok=$(vault token create -orphan -ttl=1m -type=batch -field=token -policy="$i" )
  vault token capabilities "$tok" "secret/application/very-secret/foo"

Given a policy like this:

path "secret/+/*" {
    capabilities = ["read", "list"]
path "secret/+/application/very-secret/*"
    capabilities = ["deny"]

the capabilities API will report read, list. But actually trying to read secret/application/very-secret/foo will result in permission denied.

Is this expected behavior of the capabilities endpoint?

If so, I think the only way to truly verify the result of a policy is to issue a token and use it? That could work for some cases but may have undesirable side-effects in others.

Are there any other tools for auditing vault policies like this that I should look into?

Ok I realized my mistake. I forgot to take into account that the KV v2 engine and its data/metadata sub paths, and the fact that the vault kv cli hides these from you. But capabilities endpoint would not know about that so I need to explicitly add data to my path query. I should probably do two capabilities queries - one for data and one for metadata, to be complete