Hello All,
I am facing a problem where I cannot connect to vault from pod or run curl command using service account token from different kubernetes cluster. its giving me “permission denied”
Below is the config I have:
I have deployed HA vault cluster(with 3 nodes) on one of kubernetes cluster(lets call it cluster-A). I have enabled endpoint via ingress UI which I can access via browser or using curl with root credentials.
Then I have deployed pod on same kubernetes cluster-A with new service account “app” and able to access secrets with no issues.
Then, I tried deploying same above pod with service account on different kubernetes cluster-B and facing “permission denied”
Below are the steps I have followed on Cluster-B:
- deployed vault agent injector using helm chart with “injector.externalVaultAddr=” as below:
helm install -f vault-custom-ha.yaml vault -n vault . --set injector.externalVaultAddr=http://<vault_ingress_url>
- create new service account name “vault-auth” under “vault” namespace and then cluster rolebindings for review token to access vault cluster which is on kubernetes cluster-A
- I have got the secret token of “vault-auth” as below:
SA_JWT_TOKEN=$(kubectl get secret/vault-auth-token-xmhbx -n vault -o jsonpath='{.data.token}' | base64 -d) --> exported this variable under vault pod
- Got K8s host and cert using below:
kubectl config view --raw -o jsonpath='{.clusters[*].cluster.certificate-authority-data}' | base64 -d > /tmp/ca-cert --> stored this ca file under vault pod
K8S_HOST=$(kubectl config view --raw -o jsonpath='{.clusters[*].cluster.server}')
- Enabled kubernetes auth and added config as below in vault pod:
vault auth enable -path=test-new kubernetes
vault write auth/test-new/config kubernetes_host="${K8S_HOST}" kubernetes_ca_cert=@/tmp/ca-cluster token_reviewer_jwt="${SA_JWT_TOKEN}"
vault write auth/test-new/role/k8s-app \
bound_service_account_names=vault-auth \
bound_service_account_namespaces=vault \
policies=k8s-secrets \
ttl=10d
using same secret policy which I used for pod when deployed on same kubernetes cluster-A(k8s-secrets)
- Created below secret:
vault kv put internal/config1 \
username='appuser' \
password='suP3rsec(et!' \
ttl='30h'
- Deployed application with new service account using below deployment yaml files and getting permission denied in agent-inector-init log file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
namespace: vault
labels:
app: vault-agent-demo
spec:
selector:
matchLabels:
app: vault-agent-demo
replicas: 1
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-secret-database: "internal/config1"
vault.hashicorp.com/role: "k8s-app"
vault.hashicorp.com/auth-path: "/auth/test-new"
vault.hashicorp.com/agent-inject-template-database: |
{{- with secret "internal/config1" -}}
postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
{{- end }}
labels:
app: vault-agent-demo
spec:
serviceAccount: vault-auth
serviceAccountName: vault-auth
containers:
- name: app
image: public/nginx:1.19.2
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: app
namespace: vault
labels:
app: vault-agent-demo
Below is the log from “vault-agent-init”:
2022-02-03T22:01:06.768Z [INFO] auth.handler: authenticating
2/3/2022 5:01:06 PM 2022-02-03T22:01:06.791Z [ERROR] auth.handler: error authenticating:
2/3/2022 5:01:06 PM error=
2/3/2022 5:01:06 PM | Error making API request.
2/3/2022 5:01:06 PM |
2/3/2022 5:01:06 PM | URL: PUT http://vault-ui.com/v1/auth/test-new/login
2/3/2022 5:01:06 PM | Code: 403. Errors:
2/3/2022 5:01:06 PM |
2/3/2022 5:01:06 PM | * permission denied
2/3/2022 5:01:06 PM backoff=1.8s
I see same error when executed via curl with same jwt token:
curl --insecure --request POST --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "k8s-app"}' http://<vault_ingress>/v1/auth/test-new/login
{"errors":["permission denied"]}
What am I missing here? Any help would be really appreciated
Below are versions I am using:
kubernetes deployed on rancher version: 1.18.20
vault version: 1.9.3