Staging environment setup with multiple vault instances

Vault
1

I am utilising the helm chart for deploying a vault cluster for our staging environments. We have multiple apps, with multiple poles ad policies. We have seperate kubernetes namespaces for each of our Environments (Staging, QA, Review etc) and a “full stack” is deployed to each namespace - databases, applications, etc.

With this level of separation, it makes the likliness of secrets leaking between environments far less likely, and ensures that our development environment is as close as possible to our staging and production setup.

However, I’ve been fighting this for a couple of days, and cannot seem to get multiple vault instances running correctly across multiple namespaces. I deploy one vault instance with our stack (we are using devspace to automate most of this) and our secrets are correctly injected into our pods correctly, but the second vault instance that runs always fails and when the injector runs, I get a “auth.handler: error authenticating: error=“context deadline exceeded” backoff=31.09s” error.

Is this a normal pattern for vault?

What is the recommended way of running multiple environments in vault without leaking secrets?

NOTE: Just to add, this isn’t including our production vault instance - that’s on completely different kubernetes clusters, this is specifically for staging/QA which has ingresses locked behind VPNs, so security isn’t really a massive issue.

Any suggestions or help would be fantastic!

2

While I am not sure about the details of your setup, you mention “multiple Vault instances across multiple namespaces”. Do you mean by this that you have a Vault instance per environment on Kubernetes? If so, I would recommend having a standalone Vault for your environments.
Having asked this, the rest of my answer will be about how to utilise Vault with multiple environments rather than setting up Vault.

If you have Vault Enterprise, which is recommended, you can utilise the Vault namespaces for different environments (and even clusters).
If you do not have Vault Enterprise you can always make use of separate policies with each their own role or authentication method. Even going as far as having separate secret engines enabled. The Vault injector then authenticates to the authentication method and fetches the secrets you need, but this is a bit cumbersome.

Mentioning this since it’s not a normal pattern that you are not able to authenticate on the same cluster. Perhaps have a look at what service accounts are used to authenticate to Vault and look in your Vault logs, most of it can be caught there. Feel free to share any of this if you need help! :smile: