Best practice for organizing a service's vault kv secrets

Vault
1

Hello! How do folks typically organize a service/application’s secrets in Vault KV? e.g. if we have a service named app1, you could put all the secrets for that service under a single item in the path app1, such that vault kv get -format=json -mount=secret app1 returns this json blob (edited):

...
  "data": {
    "data": {
      "secret-key-1": "value1",
      "secret-key-2": "value2"
    },
...

Or we could put each secret as it’s own item under the path app1/<item_name>, so we could list all keys with vault kv list -format=json -mount=secret app1 and then get a specific key with vault kv get -format=json -mount=secret app1/secret-key-1.

I think the later option of one secret per item is probably the safer route, because if using a single item for all secrets, it’d be too easy for an engineer to accidentally wipe out the whole json blob by using the put command instead of the patch command. We could protect against that via a policy, but when deploying a new service, the engineer needs access to the put command to create the initial vault item for that service.

A downside of using one item per secret is that the key becomes redundant, e.g. the path already has the key in it app1/secret-key-1 and so when writing to that path, what do you put for the key? Maybe just value e.g. vault kv write -format=json -mount=secret app1/secret-key-1 value=value1. This feels kind of clunky though.

What have you typically seen done? One vault item per service with all secrets as key/value pairs in that item, or many vault items per service, with the secret being the value of a single key/value pair per item.

2

This is a case of “figure out what works for you/your devs”, but there’s a couple of things you’re not taking into account that are important.

ACLs and metadata. You can set an ACL on a secret, not on a kv pair. Same with metadata. So, you want the kv pairs within a secret to be related enough that they can share security and any relevant metadata.

Also, with kv-v2, you have versioned secrets, so an accidental stomp shouldn’t be the end of the world, the data isn’t lost. But it does help make the point that kv pairs within a secret are within the blast radius of a whoopsie like that.

If you’re hung up on what string to put into the key if the secret name is already telling you exactly what’s in there, that’s not actually a problem. Call it whatever you (or your devs/customers) want to.

/edit- Hah, I think I answered this question similarly on Reddit, but having the answer here as well is useful for anyone else who finds this question on this forum.

1 Like
3

Thanks for the input Matt! Both here and on reddit :grin: That all makes sense. I think we will use the approach of one vault item per secret. So the json blob for each item will just have one secret (key/value pair). re: the key of the secret - you mentioned it doesn’t really matter (up to devs), which is true, but for us I think standardizing the key is helpful. We will be looping through all secrets from a service path and loading them in a nomad job file template stanza (via consul-template). Having a consistent key name to access each secret on will simplify the configuration and make it easier for our devs to copy/paste for onboarding services to vault and deploying new services. E.g. we loop through all secrets with {{ range secrets "secret/service/test-service" }} and then access the secret content with {{ .Data.data.value }}, assuming that value is the name of the sole key for each secret. I think you and others have already confirmed this but there isn’t a way with ACL policies to force a user to only be able to set the string value as the key for a secret, right? Or maybe ACL policy to prevent users from writing a secret with more than key/value pair? Thinking out loud here, thanks for any and all advice :pray: