Hi everybody!
I’ve been having this problem for a long time, let’s see if someone from the community can see the error.
I am integrating a test k8s cluster with an external vault, the k8s version is 1.21, the vault 1.9.
What I have done:
Deploy the vault injector with the following rbac configuration:
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-agent-injector
namespace: default
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
app.kubernetes.io/managed-by: Helm
---
# Source: vault/templates/server-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault
namespace: default
labels:
helm.sh/chart: vault-0.24.0
app.kubernetes.io/name: vault
app.kubernetes.io/instance: vault
app.kubernetes.io/managed-by: Helm
---
# Source: vault/templates/injector-clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: vault-agent-injector-clusterrole
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
app.kubernetes.io/managed-by: Helm
rules:
- apiGroups: ["admissionregistration.k8s.io"]
resources: ["mutatingwebhookconfigurations"]
verbs:
- "get"
- "list"
- "watch"
- "patch"
---
# Source: vault/templates/injector-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: vault-agent-injector-binding
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: vault
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: vault-agent-injector-clusterrole
subjects:
- kind: ServiceAccount
name: vault-agent-injector
namespace: default
---
# Source: vault/templates/server-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: vault-server-binding
labels:
helm.sh/chart: vault-0.24.0
app.kubernetes.io/name: vault
app.kubernetes.io/instance: vault
app.kubernetes.io/managed-by: Helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault
namespace: default
---
One SA for the vault injector (vault-agent-injector) and another SA for the configuration of k8s with the external vault (vault).
I configure the authentication with k8s in the external vault as follows:
Get SA name
export VAULT_SA_NAME=$(kubectl get sa vault --output jsonpath="{.secrets[*]['name']}")
Get SA token
export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME --output 'go-template={{ .data.token }}'| base64 --decode)
Get SA CRT
export SA_CA_CRT=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
Get k8s Hostname
export K8S_HOST=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.server}')
After that, I config in vault my k8s cluster, create a demo secret, config roles in k8s config and create a policy for my secret:
vault auth enable kubernetes
vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
vault policy write demo - <<EOF
path "demok8s/lab/appdata" {
capabilities = ["read"]
}
EOF
vault write auth/kubernetes/role/demo bound_service_account_names=demo bound_service_account_namespaces=default policies=demo ttl=24h
I test that the SA (vault) can verify others SA in the k8s cluster, as follow:
echo SA_JWT_TOKEN
eyJhbGciOiJSUzI1NiIsImtpZCI6ImgxN1k4YVpiOGRkakExT0p3R2tJeGFHYUVQZnNic1RNekhoVkpDSjlMTlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LXRva2VuLTc4c2xiIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNjU4OWQ1ZWYtNDk1OC00ZTE3LTlkNGMtYmFhNGQ4MTEyNzUwIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQifQ.QsoIZmINknVey0kQXedmZmRgpFY8wGtFhhcaqm_0ALCAPYothb73NU415yjrx9RIfXrjC8LWoVjpRxx8pUTjjgJDspPkINHdfedLhcW-l404IQqximyEYZgE7xlCqVeFRUVOt3fhX75zr2AnOLebOi_Cg0tLFQ4jB3bMUAkmONxrgVaby92hHdPWVxq5z7JPtDe3B0T6NCWmxj51jZdqcsjbi936ftAYRW9xvvEuHW7dIZ7huwKqyL7gjMwwUJGgqumb_U1UFXRMZKE_U_Ie_Q7zHAW77NUZXymdgOdRD5mK9XtoWtPPtr45opgUoUvr8bYzL1apx72PGMHgdkHSWw
create another SA and get the token:
token="eyJhbGciOiJSUzI1NiIsImtpZCI6ImgxN1k4YVpiOGRkakExT0p3R2tJeGFHYUVQZnNic1RNekhoVkpDSjlMTlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlbW8tdG9rZW4tcnJyd2QiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVtbyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNhMDYyM2I0LTFlMmItNGYxYS1iODUzLTAxMjg1YzdkOGFkNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlbW8ifQ.eXjn6Igl7dAKuC6RLHG9lx4usPSyp1BeNT9qtPWi--7CgDdihCVnSeC7V8JUCc-2QuaDvs5FsHBFSlRqa1HnUWzqOXggp74ggzaWTNYBeYgoRkd19oGXuDafuqO0Bgfy5GAmfW3eB97ARMALqVYOu9VPmrNzUXBJkTfE_CMPjO2nIeZvDObfE-vVBnEsRJLmLtjxYcQtfMx2oTVXaKmubwbv68s4OjdvbHzNQaN2Vb710JZTwitEUl6Bj4on9sNoD2gqUy-1Htgjmc7fZMznqipu5kOtuW3gKh44BNF636mbw8SRLsK9r_GSzCZalbJq-H0jQpYzKgpCPZuEj8OgYQ"
ok, lets use curl to verify:
curl -k -X "POST" "https://x.x.x.x:6443/apis/authentication.k8s.io/v1/tokenreviews" -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImgxN1k4YVpiOGRkakExT0p3R2tJeGFHYUVQZnNic1RNekhoVkpDSjlMTlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LXRva2VuLTc4c2xiIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNjU4OWQ1ZWYtNDk1OC00ZTE3LTlkNGMtYmFhNGQ4MTEyNzUwIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQifQ.QsoIZmINknVey0kQXedmZmRgpFY8wGtFhhcaqm_0ALCAPYothb73NU415yjrx9RIfXrjC8LWoVjpRxx8pUTjjgJDspPkINHdfedLhcW-l404IQqximyEYZgE7xlCqVeFRUVOt3fhX75zr2AnOLebOi_Cg0tLFQ4jB3bMUAkmONxrgVaby92hHdPWVxq5z7JPtDe3B0T6NCWmxj51jZdqcsjbi936ftAYRW9xvvEuHW7dIZ7huwKqyL7gjMwwUJGgqumb_U1UFXRMZKE_U_Ie_Q7zHAW77NUZXymdgOdRD5mK9XtoWtPPtr45opgUoUvr8bYzL1apx72PGMHgdkHSWw' -H 'Content-Type: application/json; charset=utf-8' -d $'{"kind": "TokenReview","apiVersion": "authentication.k8s.io/v1","spec": {"token": "eyJhbGciOiJSUzI1NiIsImtpZCI6ImgxN1k4YVpiOGRkakExT0p3R2tJeGFHYUVQZnNic1RNekhoVkpDSjlMTlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlbW8tdG9rZW4tcnJyd2QiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVtbyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNhMDYyM2I0LTFlMmItNGYxYS1iODUzLTAxMjg1YzdkOGFkNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlbW8ifQ.eXjn6Igl7dAKuC6RLHG9lx4usPSyp1BeNT9qtPWi--7CgDdihCVnSeC7V8JUCc-2QuaDvs5FsHBFSlRqa1HnUWzqOXggp74ggzaWTNYBeYgoRkd19oGXuDafuqO0Bgfy5GAmfW3eB97ARMALqVYOu9VPmrNzUXBJkTfE_CMPjO2nIeZvDObfE-vVBnEsRJLmLtjxYcQtfMx2oTVXaKmubwbv68s4OjdvbHzNQaN2Vb710JZTwitEUl6Bj4on9sNoD2gqUy-1Htgjmc7fZMznqipu5kOtuW3gKh44BNF636mbw8SRLsK9r_GSzCZalbJq-H0jQpYzKgpCPZuEj8OgYQ"}}'
Curl response:
{
"kind": "TokenReview",
"apiVersion": "authentication.k8s.io/v1",
"metadata": {
"creationTimestamp": null,
"managedFields": [
{
"manager": "curl",
"operation": "Update",
"apiVersion": "authentication.k8s.io/v1",
"time": "2023-04-18T11:29:50Z",
"fieldsType": "FieldsV1",
"fieldsV1": {"f:spec":{"f:token":{}}}
}
]
},
"spec": {
"token": "eyJhbGciOiJSUzI1NiIsImtpZCI6ImgxN1k4YVpiOGRkakExT0p3R2tJeGFHYUVQZnNic1RNekhoVkpDSjlMTlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlbW8tdG9rZW4tcnJyd2QiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVtbyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNhMDYyM2I0LTFlMmItNGYxYS1iODUzLTAxMjg1YzdkOGFkNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlbW8ifQ.eXjn6Igl7dAKuC6RLHG9lx4usPSyp1BeNT9qtPWi--7CgDdihCVnSeC7V8JUCc-2QuaDvs5FsHBFSlRqa1HnUWzqOXggp74ggzaWTNYBeYgoRkd19oGXuDafuqO0Bgfy5GAmfW3eB97ARMALqVYOu9VPmrNzUXBJkTfE_CMPjO2nIeZvDObfE-vVBnEsRJLmLtjxYcQtfMx2oTVXaKmubwbv68s4OjdvbHzNQaN2Vb710JZTwitEUl6Bj4on9sNoD2gqUy-1Htgjmc7fZMznqipu5kOtuW3gKh44BNF636mbw8SRLsK9r_GSzCZalbJq-H0jQpYzKgpCPZuEj8OgYQ"
},
"status": {
"authenticated": true,
"user": {
"username": "system:serviceaccount:default:demo",
"uid": "3a0623b4-1e2b-4f1a-b853-01285c7d8ad7",
"groups": [
"system:serviceaccounts",
"system:serviceaccounts:default",
"system:authenticated"
]
},
"audiences": [
"https://kubernetes.default.svc.cluster.local"
]
}
}
So, everything seems to be good, my SA (vault) can validate others SA (demo) but vault says other things.
From injector:
2023-04-18T11:35:56.594Z [ERROR] agent.auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT http://vault_ip:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
From vault:
Apr 18 13:35:56 llabotpvl503 vault[50292]: 2023-04-18T13:35:56.592+0200 [ERROR] auth.kubernetes.auth_kubernetes_36c2d278: login unauthorized due to: Post "https://k8s_cluster_ip:6443/apis/authentication.k8s.io/v1/tokenreviews": **unexpected EOF**
Really, WTF is unexpected EOF to vault?
Im validated that my SA (vault) was able to verify others SA in k8s (demo)
Im assing to my demo pod the required annotations and the respective SA:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/role: 'demo'
vault.hashicorp.com/agent-inject-secret-credentials.txt: 'demok8s/lab/appdata'
...
serviceAccountName: demo
I dont know what Im doing wrong, but this is insane!
Could anybody could help me to debug in deep what is happening?
Thanks in advance