Kubernetes Authentication denied

Hi all,
This is my first post here so hello everyone.

I am trying to have a pod authenticate to Vault using Kubernetes. It of course fails which is why I hope the community at large might be able to help.

I have a 3 node cluster setup with mutual SSL. When I try and authenticate, I get the following error:

Logs

2020-05-28T14:03:32.188Z [ERROR] auth.kubernetes.auth_kubernetes_b273b73e: login unauthorized due to: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:vault:vault\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope","reason":"Forbidden","details":{"group":"authentication.k8s.io","kind":"tokenreviews"},"code":403}

Audit logs

{"time":"2020-05-28T14:03:32.088516658Z","type":"request","auth":{"token_type":"default"},"request":{"id":"6239f5e4-b013-ee0f-740d-e844e724475a","operation":"update","namespace":{"id":"root"},"path":"auth/kubernetes/login","data":{"jwt":"hmac-sha256:f7419a1c1caa08f62637fcd5242f69339751bfaef4f34f36040d5af244eb7d38","role":"hmac-sha256:0510100a0314572d65134a45f68e8f0acae7ce91275520e1cd35ce66f5058ac0"},"remote_address":"172.31.113.206"}}
{"time":"2020-05-28T14:03:32.188642092Z","type":"response","auth":{"token_type":"default"},"request":{"id":"6239f5e4-b013-ee0f-740d-e844e724475a","operation":"update","namespace":{"id":"root"},"path":"auth/kubernetes/login","data":{"jwt":"hmac-sha256:f7419a1c1caa08f62637fcd5242f69339751bfaef4f34f36040d5af244eb7d38","role":"hmac-sha256:0510100a0314572d65134a45f68e8f0acae7ce91275520e1cd35ce66f5058ac0"},"remote_address":"172.31.113.206"},"response":{},"error":"permission denied"}

The following are the steps I took to get there. I just don’t understand where “system:serviceaccount:vault:vault” comed from

kubectl create serviceaccount vault-auth

kubectl apply -f - << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: vault
EOF

in vault

vault auth enable kubernetes

vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

vault write auth/kubernetes/role/backup \
    bound_service_account_names=vault-auth \
    bound_service_account_namespaces='vault' \
    policies=backup \
    ttl=30m

On the pod trying to authenticate

export VAULT_ADDR='https://vault.vault.svc.cluster.local:8200'
export KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

curl -s --cert client.crt --key client.key --cacert /run/secrets/kubernetes.io/serviceaccount/ca.crt --request POST --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "backup"}' ${VAULT_ADDR}/v1/auth/kubernetes/login|jq
{
  "errors": [
    "permission denied"
  ]
}

Thanks for any help you can provide!

So, it turns out that by not creating a service account “vault-auth” and have the ClusterRoleBinding’s subject point to the “vault” service account" it works without issue.

Can someone explain to me if having a seperate service account is of any value?

Thanks. :smiley:

The seperate SA vault-auth is created only for TokenReview. I think the reason why it didnt worked out for you is that that the Pod you used to authenticate to vault wasnt launched with the identity/sa of vault-auth. But the role you created points to that specific service account.

Now you made the service account vault the token reviewer.

You can specify in your deployments which service account you want to use for your pod. If you dont specify a specific sa in a deployment the default service account in each namespace will be used, which in turn can also be granted permission to authenticate.

vault write auth/kubernetes/role/backup \
bound_service_account_names=default \
bound_service_account_namespaces=vault \
policies=backup

You can inspect which identity your pod is using if you check the jwt token with a website like jwt.io. Be careful with these websites!

I hope this helps in any way.

Regards,

Hi @rooftop90,

Thank you so much for your reply. What you describe is indeed how I understood it. I omitted to add the Deployment for clarity.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: backup
  namespace: vault
  labels:
    app: backup
spec:
  replicas: 1
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: backup
  template:
    metadata:
      labels:
        app: backup
    spec:
      serviceAccountName: vault-auth 
[...]

So The “sa” was defined in the pod but it still seemed to want to reference sa/vault rather than sa/vault-auth. Other than defining it it the pod, I can’t see what I am doing wrong. I have a workaround but would like to see the error of my ways :wink:

I think you can try to configure the RoleBinding back to the vault-auth sa. This SA will solely be responsible for TokenReview. After that you create a new service account in the vault namespace which u want to use for your app. E.g. “backup-app-sa”. After that you create a role for your new service account “backup-app-sa” like this:

vault write auth/kubernetes/role/backup \
bound_service_account_names=backup-app-sa \
bound_service_account_namespaces=vault \
policies=backup

Also, i just noticed you retrieve the Token for the TokenReview SA another way i am used to. Try this instead:

export VAULT_SA_NAME=$(kubectl -n default get sa vault-auth -o jsonpath="{.secrets[*]['name']}")
export SA_JWT_TOKEN=$(kubectl -n default get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)
export SA_CA_CRT=$(kubectl -n default get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)

Rewrite the Kubernetes Auth Method with these values + your K8s Host Address:

vault write auth/kubernetes/config \
    token_reviewer_jwt="$SA_JWT_TOKEN" \
    kubernetes_host="https://$K8S_HOST" \
    kubernetes_ca_cert="$SA_CA_CRT"

Exec into the Pod and verify your configuration.

To me it basically looks like you are using the Token of the system:serviceaccount:vault:vault for TokenReview instead of the Token from vault-auth.

Hope this helps.

Regards,

1 Like

Thanks again for your help @rooftop90, I’ll give it a go next week. If I give no feedback, it will have worked! :wink:

Hi @rooftop90,
I wanted to get back to give a bit more information on what my problem was and how you helped fix it.

The problem was due to how I created the auth/kubernetes/role/backup,

What I was doing was using the vault-0's sa token which ended up being sa/vault.

By doing the following, you specify the use of sa/vault-auth and therefore configure auth/kubernetes/config correctly.

Thanks again for your help!