Insane auth vault with kubernetes

Hi everybody!

I’ve been having this problem for a long time, let’s see if someone from the community can see the error.

I am integrating a test k8s cluster with an external vault, the k8s version is 1.21, the vault 1.9.
What I have done:

Deploy the vault injector with the following rbac configuration:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-agent-injector
  namespace: default
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
---
# Source: vault/templates/server-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault
  namespace: default
  labels:
    helm.sh/chart: vault-0.24.0
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
---
# Source: vault/templates/injector-clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vault-agent-injector-clusterrole
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
rules:
- apiGroups: ["admissionregistration.k8s.io"]
  resources: ["mutatingwebhookconfigurations"]
  verbs:
    - "get"
    - "list"
    - "watch"
    - "patch"
---
# Source: vault/templates/injector-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-agent-injector-binding
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vault-agent-injector-clusterrole
subjects:
- kind: ServiceAccount
  name: vault-agent-injector
  namespace: default
---
# Source: vault/templates/server-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-server-binding
  labels:
    helm.sh/chart: vault-0.24.0
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault
  namespace: default
---

One SA for the vault injector (vault-agent-injector) and another SA for the configuration of k8s with the external vault (vault).

I configure the authentication with k8s in the external vault as follows:

Get SA name
export VAULT_SA_NAME=$(kubectl get sa vault --output jsonpath="{.secrets[*]['name']}")

Get SA token
export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME --output 'go-template={{ .data.token }}'| base64 --decode)

Get SA CRT
export SA_CA_CRT=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)

Get k8s Hostname
export K8S_HOST=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.server}')

After that, I config in vault my k8s cluster, create a demo secret, config roles in k8s config and create a policy for my secret:

vault auth enable kubernetes
vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
vault policy write demo - <<EOF
path "demok8s/lab/appdata" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/demo bound_service_account_names=demo bound_service_account_namespaces=default policies=demo ttl=24h

I test that the SA (vault) can verify others SA in the k8s cluster, as follow:

echo SA_JWT_TOKEN

eyJhbGciOiJSUzI1NiIsImtpZCI6ImgxN1k4YVpiOGRkakExT0p3R2tJeGFHYUVQZnNic1RNekhoVkpDSjlMTlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LXRva2VuLTc4c2xiIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNjU4OWQ1ZWYtNDk1OC00ZTE3LTlkNGMtYmFhNGQ4MTEyNzUwIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQifQ.QsoIZmINknVey0kQXedmZmRgpFY8wGtFhhcaqm_0ALCAPYothb73NU415yjrx9RIfXrjC8LWoVjpRxx8pUTjjgJDspPkINHdfedLhcW-l404IQqximyEYZgE7xlCqVeFRUVOt3fhX75zr2AnOLebOi_Cg0tLFQ4jB3bMUAkmONxrgVaby92hHdPWVxq5z7JPtDe3B0T6NCWmxj51jZdqcsjbi936ftAYRW9xvvEuHW7dIZ7huwKqyL7gjMwwUJGgqumb_U1UFXRMZKE_U_Ie_Q7zHAW77NUZXymdgOdRD5mK9XtoWtPPtr45opgUoUvr8bYzL1apx72PGMHgdkHSWw

create another SA and get the token:

token="eyJhbGciOiJSUzI1NiIsImtpZCI6ImgxN1k4YVpiOGRkakExT0p3R2tJeGFHYUVQZnNic1RNekhoVkpDSjlMTlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlbW8tdG9rZW4tcnJyd2QiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVtbyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNhMDYyM2I0LTFlMmItNGYxYS1iODUzLTAxMjg1YzdkOGFkNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlbW8ifQ.eXjn6Igl7dAKuC6RLHG9lx4usPSyp1BeNT9qtPWi--7CgDdihCVnSeC7V8JUCc-2QuaDvs5FsHBFSlRqa1HnUWzqOXggp74ggzaWTNYBeYgoRkd19oGXuDafuqO0Bgfy5GAmfW3eB97ARMALqVYOu9VPmrNzUXBJkTfE_CMPjO2nIeZvDObfE-vVBnEsRJLmLtjxYcQtfMx2oTVXaKmubwbv68s4OjdvbHzNQaN2Vb710JZTwitEUl6Bj4on9sNoD2gqUy-1Htgjmc7fZMznqipu5kOtuW3gKh44BNF636mbw8SRLsK9r_GSzCZalbJq-H0jQpYzKgpCPZuEj8OgYQ"

ok, lets use curl to verify:

curl -k -X "POST" "https://x.x.x.x:6443/apis/authentication.k8s.io/v1/tokenreviews" -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImgxN1k4YVpiOGRkakExT0p3R2tJeGFHYUVQZnNic1RNekhoVkpDSjlMTlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LXRva2VuLTc4c2xiIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNjU4OWQ1ZWYtNDk1OC00ZTE3LTlkNGMtYmFhNGQ4MTEyNzUwIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQifQ.QsoIZmINknVey0kQXedmZmRgpFY8wGtFhhcaqm_0ALCAPYothb73NU415yjrx9RIfXrjC8LWoVjpRxx8pUTjjgJDspPkINHdfedLhcW-l404IQqximyEYZgE7xlCqVeFRUVOt3fhX75zr2AnOLebOi_Cg0tLFQ4jB3bMUAkmONxrgVaby92hHdPWVxq5z7JPtDe3B0T6NCWmxj51jZdqcsjbi936ftAYRW9xvvEuHW7dIZ7huwKqyL7gjMwwUJGgqumb_U1UFXRMZKE_U_Ie_Q7zHAW77NUZXymdgOdRD5mK9XtoWtPPtr45opgUoUvr8bYzL1apx72PGMHgdkHSWw' -H 'Content-Type: application/json; charset=utf-8' -d $'{"kind": "TokenReview","apiVersion": "authentication.k8s.io/v1","spec": {"token": "eyJhbGciOiJSUzI1NiIsImtpZCI6ImgxN1k4YVpiOGRkakExT0p3R2tJeGFHYUVQZnNic1RNekhoVkpDSjlMTlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlbW8tdG9rZW4tcnJyd2QiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVtbyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNhMDYyM2I0LTFlMmItNGYxYS1iODUzLTAxMjg1YzdkOGFkNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlbW8ifQ.eXjn6Igl7dAKuC6RLHG9lx4usPSyp1BeNT9qtPWi--7CgDdihCVnSeC7V8JUCc-2QuaDvs5FsHBFSlRqa1HnUWzqOXggp74ggzaWTNYBeYgoRkd19oGXuDafuqO0Bgfy5GAmfW3eB97ARMALqVYOu9VPmrNzUXBJkTfE_CMPjO2nIeZvDObfE-vVBnEsRJLmLtjxYcQtfMx2oTVXaKmubwbv68s4OjdvbHzNQaN2Vb710JZTwitEUl6Bj4on9sNoD2gqUy-1Htgjmc7fZMznqipu5kOtuW3gKh44BNF636mbw8SRLsK9r_GSzCZalbJq-H0jQpYzKgpCPZuEj8OgYQ"}}'

Curl response:

{
  "kind": "TokenReview",
  "apiVersion": "authentication.k8s.io/v1",
  "metadata": {
    "creationTimestamp": null,
    "managedFields": [
      {
        "manager": "curl",
        "operation": "Update",
        "apiVersion": "authentication.k8s.io/v1",
        "time": "2023-04-18T11:29:50Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:spec":{"f:token":{}}}
      }
    ]
  },
  "spec": {
    "token": "eyJhbGciOiJSUzI1NiIsImtpZCI6ImgxN1k4YVpiOGRkakExT0p3R2tJeGFHYUVQZnNic1RNekhoVkpDSjlMTlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlbW8tdG9rZW4tcnJyd2QiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVtbyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNhMDYyM2I0LTFlMmItNGYxYS1iODUzLTAxMjg1YzdkOGFkNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlbW8ifQ.eXjn6Igl7dAKuC6RLHG9lx4usPSyp1BeNT9qtPWi--7CgDdihCVnSeC7V8JUCc-2QuaDvs5FsHBFSlRqa1HnUWzqOXggp74ggzaWTNYBeYgoRkd19oGXuDafuqO0Bgfy5GAmfW3eB97ARMALqVYOu9VPmrNzUXBJkTfE_CMPjO2nIeZvDObfE-vVBnEsRJLmLtjxYcQtfMx2oTVXaKmubwbv68s4OjdvbHzNQaN2Vb710JZTwitEUl6Bj4on9sNoD2gqUy-1Htgjmc7fZMznqipu5kOtuW3gKh44BNF636mbw8SRLsK9r_GSzCZalbJq-H0jQpYzKgpCPZuEj8OgYQ"
  },
  "status": {
    "authenticated": true,
    "user": {
      "username": "system:serviceaccount:default:demo",
      "uid": "3a0623b4-1e2b-4f1a-b853-01285c7d8ad7",
      "groups": [
        "system:serviceaccounts",
        "system:serviceaccounts:default",
        "system:authenticated"
      ]
    },
    "audiences": [
      "https://kubernetes.default.svc.cluster.local"
    ]
  }
}

So, everything seems to be good, my SA (vault) can validate others SA (demo) but vault says other things.

From injector:

2023-04-18T11:35:56.594Z [ERROR] agent.auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT http://vault_ip:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied

From vault:

Apr 18 13:35:56 llabotpvl503 vault[50292]: 2023-04-18T13:35:56.592+0200 [ERROR] auth.kubernetes.auth_kubernetes_36c2d278: login unauthorized due to: Post "https://k8s_cluster_ip:6443/apis/authentication.k8s.io/v1/tokenreviews": **unexpected EOF**

Really, WTF is unexpected EOF to vault?

Im validated that my SA (vault) was able to verify others SA in k8s (demo)

Im assing to my demo pod the required annotations and the respective SA:

    vault.hashicorp.com/agent-inject: 'true'
    vault.hashicorp.com/role: 'demo'
    vault.hashicorp.com/agent-inject-secret-credentials.txt: 'demok8s/lab/appdata'
...

serviceAccountName: demo

I dont know what Im doing wrong, but this is insane!

Could anybody could help me to debug in deep what is happening?

Thanks in advance

Im was testing the same in 1.13.1 and work without problems. Appear that 1.9 has many problems with k8s auth.