Architecture - Multiple environments

Vault
1

Hi,

I’m currently thinking how to deploy properly Vault in multiple environments: dev, staging, and production. Each environment has its own cluster. Here are multiple options:

  • One global vault (Vault Cloud)
  • One vault per environment (running in cluster)
  • One global vault (Vault Cloud) and one vault per environment (running in cluster)

Many people recommend having one vault per environment. However, what about external resources (S3, DB, …) specific to this environment ? If we destroy / recreate cluster with terraform, we will lose secrets whereas we didn’t destroy external resources so I’m not sure the right place to store credentials. Here are some use cases:

  • Stripe API credentials (secret shared by multiple env): If shared by staging and prod for eg, I guess it should be in global Vault ? Maybe it should be “synced” from global to local ?
  • MySQL as a Service user credentials (external secret): If we store credentials in environment vault, we lose credentials every time we rebuild cluster whereas MySQL as a service is “external” to cluster. But maybe it’s what we should expect ?
  • Redis operator K8S (local secret): As service only exist on environment cluster, I guess credentials should stick to environment vault
  • Dev S3 bucket (external secret): Same as MySQL database, if we store credentials in Dev cluster then we lose credentials everytime we deploy with terraform. Maybe we should expect this or store in global vault ?

I didn’t find documentation on this topic. What are your advices ?

Thanks

1 Like
2

Organisations that follow a strict policy of separation of environments will have a vault cluster for each environment. Environment can be dev, QA, prod, and maybe others.
They typically do that do prevent leakage of secrets from one environment into an other. An other motivation can be to prevent performance issues in one cluster to impact another.

Storing all secrets in one cluster (cloud or on prem) is possible but requires a strict hygiene regarding account management and tied policies to accounts.
You also need to come up with a working naming scheme.

Having multiple clusters let’s you have the same setup from Dev to prod without risk of the above.

If you are worried of losing your secrets when you want to rebuild your environment I suggest you have a look at taking backups through the vault snapshots command. And also doing a restore. :slight_smile:

Just have tested backups of all your environments.
(I would suggest having vault backups and terraform state backups that are in sync.)

1 Like