Templated ACL policy, randomly returns permission denied

Vault
,
1

Hello,

I am using 3 x Vault v1.14.2 instances in cluster with raft storage.
The cluster is updated regularly and all seemed to work OK, until I faced the following problem:

I have this very simple ACL:

path “secrets_mount/data/level2_test/{{identity.entity.aliases.auth_approle_xxxxxxxx.metadata.server}}” {
capabilities = [“create”, “update”]
}

path “secrets_mount/metadata/level2_test/{{identity.entity.aliases.auth_approle_xxxxxxxx.metadata.server}}” {
capabilities = [“read”]
}

Due to some reason, the above ACL do not always work
If I start a cycle and start iterating multiple times to get the metadata (the second ACL), very soon at some point I start getting ‘Permission Denied’, for no obvious reason and before the token is expired.

If I replace the second ACL with wildcard, like this:

path “secrets_mount/metadata/level2_test/*” {
capabilities = [“read”]
}

All is good, I can always read the metadata.

The same behavior I get for the first ACL, when I try to update the vault object. It just do not work sometimes.
Since there are around 200 app using this store, some of them are getting permission denied immediately, i.e., the authenticate and on the first try to update their allowed path they get permission denied…

If they try second time to authenticate and try to write, it works in most of the cases.

Unfortunatelly I can not put a wildcard for the first ACL, and preferrably do not want to create separate ACL for each app/machine connecting to the vault, this was the reason to use templating.

I tried to reproduce the same behavior on test instance inmemory and with raft backend, but was unable to reproduce it elsewhere.

Does anyone have any suggestions how can I fix this?

Thank you.

2

You have not explained how this “server” metadata is being set.

I’m pretty sure the problem lies in that.

3

I do not set metadata manually anywhere.
This is ACL for password rotation.

The servers regularly rotate passwords and save them in the vault.
The metadata read is needed so they can compare the date of the last change and make a decision if rotation is necessary or not. So metadata is set by the vault itself, the apps are only reading it, and more precisely the time of modification/creation

4

If you mean this metadata:

auth_approle_xxxxxxxx.metadata.server

It is attached to the secret_id, and looks like this:

{
  "metadata": "{\"server\": \"aaa.bbb.ccc\", \"type\": \"server_type\", \"path\": \"secret_mount/data/level2_test/aaa.bbb.ccc\" }"
}
5

This is indeed the problem. The design of Vault’s metadata system does not support this usage pattern. Each time a different Secret ID is used to authenticate, it overwrites the metadata on the Identity, which is one-per-Role ID.

You are not the first person to run into this issue.

You will need to provide each server with an entirely separate AppRole, rather than Secret IDs within a role.

1 Like
6

Hello,
You are correct, creating separate role and secretid for each instance of the app fixed the issue.
Thank you.