Kubernetes Authentication denied

Vault
1

Hi all,
This is my first post here so hello everyone.

I am trying to have a pod authenticate to Vault using Kubernetes. It of course fails which is why I hope the community at large might be able to help.

I have a 3 node cluster setup with mutual SSL. When I try and authenticate, I get the following error:

Logs

2020-05-28T14:03:32.188Z [ERROR] auth.kubernetes.auth_kubernetes_b273b73e: login unauthorized due to: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:vault:vault\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope","reason":"Forbidden","details":{"group":"authentication.k8s.io","kind":"tokenreviews"},"code":403}

Audit logs

{"time":"2020-05-28T14:03:32.088516658Z","type":"request","auth":{"token_type":"default"},"request":{"id":"6239f5e4-b013-ee0f-740d-e844e724475a","operation":"update","namespace":{"id":"root"},"path":"auth/kubernetes/login","data":{"jwt":"hmac-sha256:f7419a1c1caa08f62637fcd5242f69339751bfaef4f34f36040d5af244eb7d38","role":"hmac-sha256:0510100a0314572d65134a45f68e8f0acae7ce91275520e1cd35ce66f5058ac0"},"remote_address":"172.31.113.206"}}
{"time":"2020-05-28T14:03:32.188642092Z","type":"response","auth":{"token_type":"default"},"request":{"id":"6239f5e4-b013-ee0f-740d-e844e724475a","operation":"update","namespace":{"id":"root"},"path":"auth/kubernetes/login","data":{"jwt":"hmac-sha256:f7419a1c1caa08f62637fcd5242f69339751bfaef4f34f36040d5af244eb7d38","role":"hmac-sha256:0510100a0314572d65134a45f68e8f0acae7ce91275520e1cd35ce66f5058ac0"},"remote_address":"172.31.113.206"},"response":{},"error":"permission denied"}

The following are the steps I took to get there. I just don’t understand where “system:serviceaccount:vault:vault” comed from

kubectl create serviceaccount vault-auth

kubectl apply -f - << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: vault
EOF

in vault

vault auth enable kubernetes

vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

vault write auth/kubernetes/role/backup \
    bound_service_account_names=vault-auth \
    bound_service_account_namespaces='vault' \
    policies=backup \
    ttl=30m

On the pod trying to authenticate

export VAULT_ADDR='https://vault.vault.svc.cluster.local:8200'
export KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

curl -s --cert client.crt --key client.key --cacert /run/secrets/kubernetes.io/serviceaccount/ca.crt --request POST --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "backup"}' ${VAULT_ADDR}/v1/auth/kubernetes/login|jq
{
  "errors": [
    "permission denied"
  ]
}

Thanks for any help you can provide!

2

So, it turns out that by not creating a service account “vault-auth” and have the ClusterRoleBinding’s subject point to the “vault” service account" it works without issue.

Can someone explain to me if having a seperate service account is of any value?

Thanks. :smiley:

3

The seperate SA vault-auth is created only for TokenReview. I think the reason why it didnt worked out for you is that that the Pod you used to authenticate to vault wasnt launched with the identity/sa of vault-auth. But the role you created points to that specific service account.

Now you made the service account vault the token reviewer.

You can specify in your deployments which service account you want to use for your pod. If you dont specify a specific sa in a deployment the default service account in each namespace will be used, which in turn can also be granted permission to authenticate.

vault write auth/kubernetes/role/backup \
bound_service_account_names=default \
bound_service_account_namespaces=vault \
policies=backup

You can inspect which identity your pod is using if you check the jwt token with a website like jwt.io. Be careful with these websites!

I hope this helps in any way.

Regards,

4

Hi @rooftop90,

Thank you so much for your reply. What you describe is indeed how I understood it. I omitted to add the Deployment for clarity.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: backup
  namespace: vault
  labels:
    app: backup
spec:
  replicas: 1
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: backup
  template:
    metadata:
      labels:
        app: backup
    spec:
      serviceAccountName: vault-auth 
[...]

So The “sa” was defined in the pod but it still seemed to want to reference sa/vault rather than sa/vault-auth. Other than defining it it the pod, I can’t see what I am doing wrong. I have a workaround but would like to see the error of my ways :wink:

5

I think you can try to configure the RoleBinding back to the vault-auth sa. This SA will solely be responsible for TokenReview. After that you create a new service account in the vault namespace which u want to use for your app. E.g. “backup-app-sa”. After that you create a role for your new service account “backup-app-sa” like this:

vault write auth/kubernetes/role/backup \
bound_service_account_names=backup-app-sa \
bound_service_account_namespaces=vault \
policies=backup

Also, i just noticed you retrieve the Token for the TokenReview SA another way i am used to. Try this instead:

export VAULT_SA_NAME=$(kubectl -n default get sa vault-auth -o jsonpath="{.secrets[*]['name']}")
export SA_JWT_TOKEN=$(kubectl -n default get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)
export SA_CA_CRT=$(kubectl -n default get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)

Rewrite the Kubernetes Auth Method with these values + your K8s Host Address:

vault write auth/kubernetes/config \
    token_reviewer_jwt="$SA_JWT_TOKEN" \
    kubernetes_host="https://$K8S_HOST" \
    kubernetes_ca_cert="$SA_CA_CRT"

Exec into the Pod and verify your configuration.

To me it basically looks like you are using the Token of the system:serviceaccount:vault:vault for TokenReview instead of the Token from vault-auth.

Hope this helps.

Regards,

1 Like
6

Thanks again for your help @rooftop90, I’ll give it a go next week. If I give no feedback, it will have worked! :wink:

7

Hi @rooftop90,
I wanted to get back to give a bit more information on what my problem was and how you helped fix it.

The problem was due to how I created the auth/kubernetes/role/backup,

What I was doing was using the vault-0’s sa token which ended up being sa/vault.

By doing the following, you specify the use of sa/vault-auth and therefore configure auth/kubernetes/config correctly.

Thanks again for your help!

8

Hi @edoardo-c ,

I have the same error message.
I would like to know is your Vault server install in Kubernetes or a standalone server?

9

Hi @sunny760408 ,
The vault cluster is running on a self-managed Kubernetes in AWS. We’ve come a long way since then.

Let me know if I can help and I’ll try my best but I’m still fairly new at it.

10

Hi @edoardo-c ,

Thank you for your response.
I follow this tutorials, below is my step.

Step 1. Create sa.

$ kubectl create serviceaccount issuer
serviceaccount/issuer created

Step 2. Get sa token and ca.crt

$ kubectl get serviceaccount issuer -o json | jq -r ".secrets[].name"
issuer-token-gfx4q

$ kubectl get secrets issuer-token-gfx4q -o jsonpath="{.data.token}" |base64 --decode
# save as file issuertoken

kubectl get secrets issuer-token-gfx4q -o jsonpath="{.data['ca\.crt']}" |base64 --decode
# save as file issuerca.crt

Step 3. Configure Kubernetes authentication

$ vault auth enable kubernetes

$ vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /path/issuertoken)" \
    kubernetes_host="https://url" \
    kubernetes_ca_cert=@/path/issuerca.crt

$ vault policy write pki - <<EOF
path "pki*"                        { capabilities = ["read", "list"] }
path "pki/roles/example-dot-com"   { capabilities = ["create", "update"] }
path "pki/sign/example-dot-com"    { capabilities = ["create", "update"] }
path "pki/issue/example-dot-com"   { capabilities = ["create"] }
EOF

$ vault write auth/kubernetes/role/issuer \
    bound_service_account_names=issuer \
    bound_service_account_namespaces=default \
    policies=pki \ 
    ttl=720h

Step 4. Create Issuer vault-issuer

$ kubectl apply -f token.yaml

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: issuer
  namespace: default
data:
  token: xxx.....

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: default
spec:
  vault:
    server: http://url
    path: pki/sign/example-dot-com
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: issuer
        secretRef:
          name: issuer-token-gfx4q
          key: token
EOF

Vault error message:

2021-03-18T08:54:02.693Z [ERROR] auth.kubernetes.auth_kubernetes_bf6de9c4: login unauthorized due to: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:default:issuer\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope","reason":"Forbidden","details":{"group":"authentication.k8s.io","kind":"tokenreviews"},"code":403}

Could you please help me point out which step is not correct?

11

Hi @sunny760408
I have exactly the same issue,
did you find how to solve it please ???
cc @rooftop90 @edoardo-c

12

Hi Hashi Team,

I am facing the same error, I copied the token and certificate inside the container, still facing the same same error.
Error writing data to auth/kubernetes/login: Error making API request.
URL: PUT https://vaultserver/v1/auth/kubernetes/login
Code: 403. Errors:

  • permission denied
13

@saigopi I also faced issue 403, it took me 2 hours to debug the issue but finally, I encountered the issue, its ClusterRoleBinding must be bind to the service account.
One More Important thing, Make sure Kubernetes CA Certificate is formatted.
If the command-line interface would not work then You can use Vault UI as well.
Go Access->Auth Methods → Kubernetes.
You can also debug the issue to make sure it’s returning 200.
curl --location --request POST ‘https://VAULT-APP-URL/v1/auth/kubernetes/login’
–header ‘Content-Type: application/json’
–data-raw ‘{“jwt”: “XXXXX”, “role”: “app-role”}’

14

Hi Anishmourya,

Thank you very much for your reply here.

That works for me.

I deployed the below config in my kubernetes namespace.

Ref: Kubernetes - Auth Methods | Vault by HashiCorp

kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth      #Give your service account name here.
    namespace: default  #Give your Kubernetes Namespace here.
15

Hi Guys, i am facing the same issue, permission denied, but the thing is if i tried it 2-3 times one of the request got passed and i got the token , but sometimes it is failing, the thing is keep trying, in 4-5 tries you will be get the token,

i have checked vault-logs as well got this:-

2021-07-23T13:39:11.336Z [ERROR] ****auth.kubernetes.auth_kubernetes_704e484a: login unauthorized due to: Post “https://load-balancer-dns.ap-south-1.elb.amazonaws.com/apis/authentication.k8s.io/v1/tokenreviews”: dial tcp: i/o timeout

Can anyone help me, even not sure how to troubleshoot, i checked everything

16

did you solve this problem? i have the same problem

17

I have the exact same issue, where it intermittently throws permission denied, but resolves automatically after 2-5 minutes. Does anyone know how to solve this. Would be a great help.