I am deploying Hashicorp Vault and want to inject Vault Secrets into our Kubernetes Pods via Vault Agent Containers. At the moment it doesn’t work and I am stuck when the Vault init container tries to connect to Vault with Kubernetes auth method:
$ kubectl logs mypod-d86fc79d8-hj5vv -c vault-agent-init -f
==> Note: Vault Agent version does not match Vault server version. Vault Agent version: 1.13.1, Vault server version: 1.13.2
==> Vault Agent started! Log data will stream in below:
==> Vault Agent configuration:
Api Address 1: http://bufconn
Cgo: disabled
Log Level: info
Version: Vault v1.13.1, built 2023-03-23T12:51:35Z
Version Sha: 4472e4a3fbcc984b7e3dc48f5a8283f3efe6f282
2023-06-22T15:01:43.754Z [INFO] agent.sink.file: creating file sink
2023-06-22T15:01:43.754Z [INFO] agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2023-06-22T15:01:43.754Z [INFO] agent.template.server: starting template server
2023-06-22T15:01:43.754Z [INFO] agent.auth.handler: starting auth handler
2023-06-22T15:01:43.754Z [INFO] agent.auth.handler: authenticating
2023-06-22T15:01:43.754Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-06-22T15:01:43.754Z [INFO] agent.sink.server: starting sink server
2023-06-22T15:01:43.755Z [INFO] (runner) creating watcher
2023-06-22T15:01:43.770Z [ERROR] agent.auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.REDACTED.net:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
I have followed two tutorials from Hashicorp (here and here).
My infrastructure
- Vault is installed on AWS EC2
- EKS (1.23) cluster which contains namespace for each env and a namespace for Vault-injector
What I did
- Installed Vault injector in EKS:
resource "helm_release" "vault_injector" {
repository = "https://helm.releases.hashicorp.com"
chart = "vault"
version = local.settings.vault-injector.version
name = "vault-injector"
create_namespace = true
namespace = "vault"
set {
name = "global.externalVaultAddr"
value = local.settings.vault-injector.address
}
}
Result:
$ k get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-injector-agent-injector-78b9c97bc9-tnjbj 1/1 Running 0 2d5h
$ k get sa -n vault
NAME SECRETS AGE
default 1 2d5h
vault-injector 1 2d5h
vault-injector-agent-injector 1 2d5h
- Following the tutorials, my next step was to configure Kubernetes authentication.
$ VAULT_HELM_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("vault-injector-token-")).name')
$ TOKEN_REVIEW_JWT=$(kubectl get secret $VAULT_HELM_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode)
$ KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
$ KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
$ KUBE_ISSUER=$(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer)
$ vault write auth/kubernetes/config \
token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
kubernetes_host="$KUBE_HOST" \
kubernetes_ca_cert="$KUBE_CA_CERT" \
issuer="$KUBE_ISSUER"
- I created a policy and a role for my deployment (service-copper):
$ vault policy write service-copper - <<EOF
path "apps/env/service-copper" {
capabilities = ["read"]
}
EOF
$ vault write auth/kubernetes/role/service-copper \
bound_service_account_names=copper \
bound_service_account_namespaces=env \
policies=service-copper \
ttl=24h
- At the point, I had to switch between tutorials to learn that I have to create a new Service Account (the one gave in
bound_service_account_names)
$ kubectl create sa copper
- I updated my deployment to add the service account
$ kubectl edit deployment service-copper
...
spec:
serviceAccountName: copper
containers:
...
At the restart of the pods, I got the following error:
2023-06-22T16:51:43.996+0200 WARN [,7494923433757325112,3984854980495805192] 7 --- [qtp833308761-45] o.s.c.k.f.Fabric8PodUtils : Failed to get pod with name:[service-copper-68c4444d8f-rj4ld]. You should look into this
if things aren't working as you expect. Are you missing serviceaccount permissions?
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.20.0.1/api/v1/namespaces/env/pods/service-copper-68c4444d8f-rj4ld. Message: Forbidden!Configured service account doesn't have access.
Service account may have been revoked. pods "service-copper-68c4444d8f-rj4ld" is forbidden: User "system:serviceaccount:env:copper" cannot get resource "pods" in API group "" in the namespace "env".
That’s the moment I learnt about ClusterRoleBindings. So I created one:
$ kubectl create clusterrolebinding copper-role --clusterrole=view --serviceaccount=env:copper --namespace=copper
- I created a secret in
apps/env/service-copperin Vault - I tried to inject secrets into the pod with an update of the deployment:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/role: 'service-copper'
vault.hashicorp.com/agent-inject-secret-copper.yaml: 'apps/env/service-copper'
Once the deployment is patched, the Init container starts and we arrive at the error:
[ERROR] agent.auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.REDACTED.net:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
Question
Since I mixed tutorial, I am pretty sure I got lost on the way. I am a bit confused about the different service accounts and I feel like it is where the issue is.
Can you help me see a bit clearer on the subject? Thank you
