Kubernetes authentication

Vault
1

Hi everyone ! I try to configure the kubernetes authentication backend but I’m facing an issue when I try to authenticate a serviceaccount.

After some research I didn’t find any solution.

Vault log:

2020-08-18T15:39:20.102Z [ERROR] auth.kubernetes.auth_kubernetes_e36498cf: login unauthorized due to: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:vault:vault-auth\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope","reason":"Forbidden","details":{"group":"authentication.k8s.io","kind":"tokenreviews"},"code":403}

Kubernetes resources:

serviceaccounts:

$ kubectl get serviceaccounts -n vault vault-auth -o yaml
apiVersion: v1
automountServiceAccountToken: false
kind: ServiceAccount
metadata:
  creationTimestamp: "2020-08-18T15:18:05Z"
  name: vault-auth
  namespace: vault
  resourceVersion: "6276132075"
  selfLink: /api/v1/namespaces/vault/serviceaccounts/vault-auth
  uid: 3d563e0e-5f59-4806-916e-3731e4ac53da
secrets:
- name: vault-auth-token-gwcgj

clusterrolebinding:

$ kubectl get clusterrolebinding vault-auth -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2020-08-18T13:22:22Z"
  name: vault-auth
  resourceVersion: "6276132018"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/vault-auth
  uid: 66b1c102-4f49-4746-a459-4cd02e44ad97
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: vault

clusterrole:

$ kubectl get clusterrole system:auth-delegator -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2019-10-25T13:43:39Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:auth-delegator
  resourceVersion: "5746514639"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Aauth-delegator
  uid: ac0eee16-186b-4849-8933-6f33c6c982d7
rules:
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create

Concerning vault, it is configured by Terraform based on k8s datasource.

Thanks for your help !

1 Like
2

Hello,

I did some research on the issue you are having and i compared your yamls with mine and they seems to be matching.

The starting point for an investigation would be to verify that your pod is actually using the vault-auth SA. I think this thread will be useful for you : Kubernetes Authentication denied

Martin

3

Hello, first of all thanks for your response.

With your information I reconfigured the vault backend with the SA of the vault pod with these following information:

The SA attached to the pod:

$ kubectl get pods -n vault vault-0 -o yaml
...
  serviceAccount: vault
  serviceAccountName: vault
...

The clusterrolebiding:

$ kubectl get clusterrolebinding -n vault vault -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
  creationTimestamp: "2020-08-24T15:03:42Z"
  name: vault
  resourceVersion: "6488342403"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/vault
  uid: 54bb6548-224b-47c7-8d8c-8d63eda25a40
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault
  namespace: vault

Vault backend configuration (on vault-0 pod):

vault write auth/kubernetes/config \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Vault role (on vault-0 pod):

vault write auth/kubernetes/role/demo \
    bound_service_account_names=vault \
    bound_service_account_namespaces=vault \
    policies=default \
    ttl=30m

And the request:

$ curl \
    --request POST \
    --data '{"jwt": "<JWT>", "role": "demo"}' \
    https://<URL>/v1/auth/kubernetes/login

{"errors":["permission denied"]}

And unfortunately I have the same error.

2020-08-24T16:15:59.152Z [ERROR] auth.kubernetes.auth_kubernetes_e36498cf: login unauthorized due to: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:vault:vault\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope","reason":"Forbidden","details":{"group":"authentication.k8s.io","kind":"tokenreviews"},"code":403}

I guess, these configurations are correct, the SA on the pods is the same as the kubernetes authentication backend.

Do you know how can I resolve this issue?

4

Did you figure this out?

5

You can install kubectl and configure in your cluster and then communicate.

6

May be you can use the latest version and check controlling access on Kubernetes API .