Vault Secret Management

How does one add secrets to vault? The only option we see is the CLI commands (or via UI). Is it possible to have some sought of overview like PR reviews in GitHub before the secret is added or modified in vault?

Managing plaintext secret in private GitHub repo makes GitHub the weakest link. How does the Ops team manage version-controlled, audited secret management?

For static secrets, I don’t think there are many other options. Fundamentally a person or machine has to upload each static secret to Vault, and given that Vault is meant to be the only thing you trust with all the secrets, centralising the management of static secrets outside of Vault would defeat one of the main value propositions.

I think one of the main tools to help secret management scale is to use dynamic secrets. e.g.:

  • All database plugins support dynamic roles
  • Cloud secret engines like AWS support many dynamic short-lived credentials using a root credential
  • OpenLDAP supports dynamic credentials
  • And many more

Version control can be used for the terraform used to configure Vault if you use the Vault terraform provider, but otherwise the flow is going to be a little different. With regards to auditing, Vault does have auditing built-in, but again, the flow there is probably a little different to what you were imagining.

2 Likes

I have the same question. It’s 2022, so I would like to bump this up.

Is there a gitops way to add static secrets to Vault?

The GitOps way would be to commit your secrets to files in Git, and write a script that runs when Git changes, and uploads them to the Vault API.

But then your secrets would be stored in Git as well as Vault.

So that’s probably why people don’t generally use GitOps for this.

What kind of system or approach were you thinking of?

1 Like

Hi, I have thinking about it for a while but couldn’t figure out how to achieve this.

The idea is to have it reviewed, approved before any operation can happen. Of course, there should be audit logs for all operations.

Git can help achieve the above. But there are two drawbacks:

  1. If we store secrets in git and Vault, then there will be two sources of truth. And how to sync secrets from git to Vault is an unsolved problem
  2. The existing GitOps tool that uses a controller to unseal secrets on kubernetes cluster is less ideal, because it will create a kubernetes secret, which is less secure than directly attaching a secret into a memory volume.

There’s another aspect to the requirements, which will greatly shape the solution…

How much access are the operators adding these secrets allowed to have to them? For example is it necessary to split all secrets into an A and a B part for human handling, so no one human ever sees the entire secret?

How much access to the secret values does the change approver, and audit log reviewer get? For example, is the change approver merely approving the specified secret can be changed, or do the have to read the actual new secret value and confirm it is correct? Do the audit logs contain actual secret values, or not, or perhaps hashes thereof?

These requirements quite likely being different for different use cases, is another contributing factor to there not being a standard off-the-shelf solution to this class of problem.

To put secrets into Vault you can use Terraform. The Vault Terraform provider provides a nice structured way to interact with Vault, but you end up storing secrets in Git and the Terraform state.

A better solution is probably to write a Nomad or K8s job that retrieves the retrieves the secret and places it into Vault. Then you can separate permissions such that this job can write secrets but can’t read them, and your service can read the secret but can’t write it. You can also remove manual steps from the process so individuals don’t need to handle the creds.

This is effectively what the Vault secrets engines do for you, so you should try to use secret engine where ever possible. For static secrets, like API keys for services that don’t have APIs to generate creds, your best bet is probably setting them manually so they aren’t stored in source control.