Read/Consume kv secret within GitHub workflow

Hello and good evening,

I’m facing with a problem using Vault within GitHub workflow, let me try to explain my scenario:

I’ve deployed on my k8s cluster (placed inside of an eks) a Vault instance using Helm and I’ve expose correctly this service to an ec2 instance (again in same eks) with a nodePort.
On this ec2 I’ve a docker container that run a self-hosted github runner, linked to a gh repo.
On this repo I’ve wrote a dummy workflow just to communicate with my vault instance to retrieve and consume secrets.

From vault-0 pod bash:

  • I successfully unseal server using the 3 vault keys
  • I’ve create kv/secret path (v2)
  • I wrote a policy just with capabilities = [“read”] for the kv/secret
  • Finally I’ve added a test secret like foo=bar

From vault-workflow in github action:

The problem is that when I run this workflow, job fail with Response code 500 (Internal Server Error).
I thought that was an ip or port issue but I tried to make a curl from inside of my gh runner container (in ec2) like:

curl -H "X-Vault-Token: <my-vault-root-token>" -X GET http://my.eks.cluster.ip:<exposed_service_nodeport>/v1/kv/data/secret?version=1

and vault service answered with response code 204 with the correct json (I can see my foo=bar secret), sign that the service is correctly exposed.

Now, idk what is the problem because it’s my first time with vault and I’m newby to this.
Maybe I’ve made a mistake or I miss something? My vault server is in standalone mode and I don’t want to run it in dev mode, neither in HA mode…can someone help me? It will be much appreciated, thanks!
Good evening at all :slight_smile:

If it works with cURL, it means the error is comming from the Github action.

Never saw that Github action before but I bet the url is Vault’s address. That’s everthing before /v1/kv/data/secret?......

The path to your secret goes in the secrets list. You will probably have to set version: 2 in there somewhere.

Hi ixe013,

yes the error comes from Github action.
I’ve tried to run pipe step adding cUrl command and it’s working as on the bash.
Now, I don’t want to use cUrl command instead of vault-action (showed in screen above) but this “not-working” code is came from official Github vault page example (Vault Secrets · Actions · GitHub Marketplace · GitHub) and I don’t know another way to retrieve secret from my vault server.

From curl:

If I use /v1/kv/data/secret?version=1 response show me foo=bar that is the first secret that I’ve created
If I use /v1/kv/data/secret?version=2 response show me foo2=bar2 that is the second secret that I’ve created so…I don’t think that is version issue or similar. It seems that the entire step is wrong.
I don’t even know how to activate debug mode to dig into logs…

Mind a thing: I’m using vault without TLS so I don’t need caCertificate as showd in official page, I’m just using token auth with correct vault root token.

(next time, do not post a screeshot, but the actual code. Saves other from retyping it to help you. +204 means no content, I don’t see how you could have gotten any JSON back. Post the full request response please)

There a multiple versions at stake here.

  1. The API always starts with /v1 don’t worry about it.
  2. The KV store has two implementation. Version 2 of the KV store adds secret versionning.
  3. Each secret’s versions.

We got the /v1 prefix out of the way.

Judging on your cURL request, you have enabled KV store version 2, the one with secret versionning. Great.

Let’s not get fancy and always query the latest version of the secret (not appending version=x at the end of the url)

I beleive you created your srecret with vault kv put kv/secret/data/secret foo=bar.

With those assumption, you should set:

url: http://my.eks.cluster.ip:1234/
secrets: kv/secret/data/secret | FOO

Keep the token parameter and remove tlsSkipVerify (but you should implement TLS and make it so that you can trust the certificate).

Hi @ixe013 and thanks for the answer.

Sorry, you’re right…I’ve posted a screen cause I thought it was better but I was wrong.
The response status code is 200 (sory, my bad).

I’ve tried to adjust my instructions and finally the pipeline is working.
The errors hiding in:

- url: http://my.eks.cluster.ip:<exposed_service_nodeport>/ ----> added last / missing
- token: ${{ secrets.VAULT_TOKEN }} ----> without initial \ (paste/copy error)
- #tlsSkipVerify: true ----> added a comment
- secrets: |
            kv/data/secret foo | FOO ----> fix the correct path 

Now I can read/consume my secrets vault.

Thanks a lot ixe, your explanation was useful and clear and you made me realize that I’ve had made mistakes.
Next step is to use a postgres db instance to save data instead of using in memory volume and risk to lose all if pod comes down or restart.

Because I’m inside an EC2 stored in EKS, is useful to add new security layer with tls or can I use this without it?

Using TLS in a controlled environment is always a tough question. It is a matter of risk. The threat is that your Vault credentials and the secrets will move around in plain text. To asses the risk, you must evaluate the likehood of a threat actor taking avantage of that. In short : risk=threat*likelyhood.

I would start by using stable storage (Integrated or DynamoDB) and AWS KMS auto-unseal. Good luck!

Hi @ixe013 …maybe you can answer to my new question.
I’ve created a postgres DB named “vault” and a table named “vault_kv_store” with key and value as column.
Now I want to read secrets from this db instance but I don’t understand how I can do it while staying in the workflow.
Following documentation I’ve activated db engine for postgresql, I’ve created vault_user (from pgAdmin) and used it as role, I’ve changed storage “file” to storage “postgresql” into config inside vault container with his correct url and table name but, when I try to reach database/ path from gh workflow (as before with kv engine but with different path) server answer with 404.
Where am I wrong? Thanks in advance!!

Ask a new question @plabate, this is not a personnal channel between us. Put the exact code (as text, not a screenshot) that you used to:

  1. Mount the KV store
  2. Create the secret
  3. Retreive the secret

+Clarify the role of MongoDB in your setup, maybe with a link to the documentation you used. Is i:

  • A storage backend?
  • Some service you want to authenticate to?
  • A database with dynamic password (that change automatically)