We’ve run into similar challenges when deciding how to organize secrets in Vault. We ended up trialing a setup where we name secrets in Vault according to the secret’s source of truth, rather than the secret’s consumer.
It seems like many of HashiCorp’s own examples, and most other examples I see online, organize secrets by the application that’s consuming them. For instance, if I’m running an application named my-application that needs access to a Datadog API key in my organization named my-organization with the key ID f62808e8-92db-4a55-81e5-b13b4d2c2f6e, these examples would probably create a secret in the path secrets/data/my-application/datadog, or something similar. They set up policies that allow my application to read everything under secrets/data/my-application/*, and allow the team that manages the application to write the same.
This approach is easy to get started with, but can present some problems down the line. If multiple applications need to use the same secret, you now have to copy the secret into secrets/data/my-other-application/* in order for the new application to read it. This means there’s no longer a single source of truth, and if the secret is rotated, whoever rotates it needs to know that it must be updated in more than one place in Vault. It also means that the administrators that manage the secret’s source of truth may not have permission to update it in Vault at all, and if they rotate it, they have to find every team that’s using the secret and transmit the new secret value through some secure channel so each team can update their copy of the secret.
Our approach has been to instead organize secrets by their sources of truth. In the Datadog API key example, we would store it in a path similar to secrets/data/datadog/my-organization/api-key/f62808e8-92db-4a55-81e5-b13b4d2c2f6e. We create a Vault group named readers/datadog/my-organization/api-key/f62808e8-92db-4a55-81e5-b13b4d2c2f6e which grants read access to the API key, and the users who administer our Datadog organization are given write access to secrets/data/datadog/my-organization/* as well as permission to add and remove users from the readers group.
Now, if our Datadog administrators ever need to rotate that API key, they only need to update it in one easy to find location in Vault, and teams with applications that consume that Vault secret don’t need to take any action. Additionally, our Datadog administrators have full control and visibility into which applications can read any given API key. If more applications need to use an API key, the application team can request access to it directly from the Datadog administrators, and the Datadog administrators can add the application to the Vault group without going through the Vault administrators. Another advantage is that applications can easily find any given secret in Vault, because the secret path has a predictable structure and requires no additional information. For example, if an application wants to use an LDAP service account in the example.com domain named my-service-account, it knows that it will be in Vault under the path secrets/data/ldap/example.com/user/my-service-account.
The main drawback of this scheme is it requires Vault administrators to set up reader groups and owner policies on a per-secret basis. This is the policy that grants, for example, the Datadog administrators write access to the secret, and the ability to add/remove members from the secret’s readers group. We currently do this in IaC, but it’s not scaling very well so we may revisit that decision. It seems like there’s no easy way to allow administrators permission to create their own reader groups without also granting them the ability to create Vault policies, which would be a huge privilege escalation vector. Namespaces could alleviate this if we were using Vault Enterprise.