Gitlab review environments in Kubernetes + Vault?


Starting to use Vault in my Kubernetes cluster, and looking into the Kubernetes auth plugin, it seems pretty straightforward to retrieve secrets - as long as the namespace and service account is known.

I’m integrating my gitlab CI/CD chains with Kubernetes, using the review application feature. With this, every time I push a branch, a new installation of my Helm chart is created, in its own separate (and per-branch-unique) namespace, with the service accounts typically having unique names as well.

This makes it kind of hard to use the Kubernetes auth module, as you would typically want to lock down a role based on the namespace or serviceaccount name, or both.

Any ideas on how to resolve this problem to allow me to authenticate review applications towards vault?

doing a helm update shouldn’t be causing your kube-auth method to break. If you’re blowing away the whole pod every time, then that’s an excessive step.

Well, this is how gitlab review applications work. Basically, there are >=2 environments:

  • Staging, which is always in the same namespace (e.g. appname-staging)
  • Production, which is also always in the same namespace (e.g. appname-production)
  • A variable number of review applications, one for each feature or bugfix branch. They are created dynamically by gitlab creating a new namespace, and are removed when the branch is merged into main. Each such review application will live in its own namespace (e.g. appname-staging-feat-377

The staging and production environments are no problem to handle in vault configuration, as they have fixed namespaces and the service account for the various microservices within also have fixed names. The problem is how to handle the review environments, where I would like to authenticate against vault as some review-role and get credentials that allow me access only to review-databases. I can’t see a way to do this with the Kubernetes auth module, as you lock down the role on namespace and/or service account.

One idea is to authenticate against vault in the CI/CD step (something I already do with the jwt auth module), and retrieve a time-limited SSL certificate that is made available to the review applications, then authenticate against vault using this certificate, but only in review applications. This could work given that review applications usually live only for a couple of days. The staging and production environments would still use the Kubernetes auth method.

Better/simpler ideas are welcome!