Trying to wrap my head around using vault secrets in CI/CD

Just spun up an on-prem vault instance to play around with. Vault makes a lot of sense and I love the idea of it. But there are still some aspects I am wondering about.

I was reading gitlab docs for their process here:

But this seems backwards to me. For vault, it seems like the fewer places that actually use the secret the better. For a docker deployment, this would give gitlab access to the secret as well as the runner running somewhere. If a team member on a project wanted to echo out a secret to the pipeline, that seems trivial?

Wouldn’t it make more sense to do something like (random test I did):

export VAULT_ADDR=

 podman run -itd \
            -v ${PWD}/web:/var/www/html/:z \
            --name phptest \
            --network helpdesk-network \
            -p 8081:80 \
            -e "MYVAR=$(vault kv get -field testkey kv/mysecrettest)" \

The runner deploying the container (in this specific example) , would then get the secret value (assuming vault login was ran on that server on that same account as the runner) and the gitlab server would never have to the value.

Giving the actually gitlab server the token/keys to access the secret seems superfluous… But maybe there is something I’m not understanding?

Apologies if this is a dumb thing to ask, just trying to understand.

Ah yes, the eternal problem of doing anything secret in CI. :slight_smile:

In my experience, the only way forward is to just admit that the part of your CI that runs user-modifiable code needs to only have access to low-privilege mostly read-only secrets, that you’re willing for absolutely anyone who could trigger a CI job to be able to use.

Of course, that’s CI specifically. Presumably there has been review and approval of code before it gets to CD, making that different.

Yes… that’s what’s already happening, in the flow diagram in the article you linked to.

The GitLab server has the token, because it is literally the component that creates the token, because it posesses the necessary private key, and is the ultimate arbiter of which runner is running what.

An analogy: You apply to your country’s government passport office for a new passport. The passport office creates a new passport and gives it to you when they are satisfied with your identity. There is no point complaining that the passport office had posession of your new passport between creating it and giving it to you - because they are literally the authority in charge of creating passports. The GitLab server is the passport office here.

I guess I was thinking of was better for gitlab to have no keys/token, and authenticate the server the actual runner is running on… Which would in effect only allow the specific user account on the server to access the secret.

However, I see your point… Because if someone has access to the gitlab project they could still run a CI/CD using the same runner and output that value if they really wanted to.

Or if you really wanted to, you could have two non privileged accounts on the server the runner is running on. One with access to the secrets, and then give the non-privileged runner account sudo access to run a specific script,file as another user. I think this would, in effect prevent gitlab or the gitlab runner for executing anything specifically against vault?

Yet by doing that, you also give up all the fine-grained details offered by the GitLab JWT, such as being able to restrict Vault logins based on the individual repository being run, the branch/tag within the repository being run, and whether that branch/tag is protected… you replace all of that with “some kind of run that got dispatched to this particular runner”.

But what’s the point?

  • If the privileged script returns the secrets to the runner for it to use … what purpose has this extra level of indirection served?

  • If the privileged script runs arbitrary code from the GitLab repo … user-specified code from the repo is still in contact with the secrets.

  • If the privileged script doesn’t run arbitrary code from the GitLab repo … fine, you’re secure, but you’ve blocked most of the flexibility of the GitLab CI architecture and more-or-less turned it into an overly complicated webhook.