I have HA vault/consul pods running with a local kubernetes cluster on minikube. I get “claim iss is invalid” when execing into another app pod and curling the k8s authentication endpoint with the JWT token. Not sure where to go from here.
deployment.yaml for my app code where I want to read the secret:
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-deployment
spec:
replicas: 2
selector:
matchLabels:
app: web-pod
template:
metadata:
labels:
app: web-pod
spec:
containers:
- name: web
image: kahunacohen/hello-k8s
imagePullPolicy: IfNotPresent
env:
- name: VAULT_ADDR
value: 'http://vault:8200'
- name: JWT_PATH
value: '/var/run/secrets/kubernetes.io/serviceaccount/token'
- name: SERVICE_PORT
value: '8080'
envFrom:
- configMapRef:
name: web-configmap
ports:
- containerPort: 3000
protocol: TCP
I created a ClusterRoleBinding:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-auth
namespace: default---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-auth
namespace: default
and I think I need this for the TokenReview API, whatever that is…
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-auth
namespace: default
All my pods are up and running. vault is initialized and unsealed.
- Create service account vault-auth:
$ kubectl create serviceaccount vault-auth - Get the vault service account name:
export VAULT_SA_NAME=$(kubectl get sa vault-auth --output jsonpath="{.secrets[*]['name']}") - Get the SA_JWT_TOKEN:
export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME \ --output 'go-template={{ .data.token }}' | base64 --decode) - Get the service account crt:
export SA_CA_CRT=$(kubectl config view --raw --minify --flatten \ --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode) - Get the k8s host:
export K8S_HOST=$(kubectl config view --raw --minify --flatten \ --output 'jsonpath={.clusters[].cluster.server}') - Get the root token:
root_token=$(cat cluster-keys.json | jq -r ".root_token") - I then pass these all to a script that I run on the vault pod.
- The script logs in with the root token and does the following steps, after each of them it sleeps for 20 seconds to “ensure” the previous step completed.
- Enable secrets:
$ vault secrets enable -path=secret kv-v2 - Set some dummy secrets:
vault kv put secret/webapp/config username="static-user" password="static-password" - Enable kubernets auth:
vault auth enable kubernetes - Configure the auth:
vault write auth/kubernetes/config \ token_reviewer_jwt="$SA_JWT_TOKEN" \ kubernetes_host="$K8S_HOST" \ kubernetes_ca_cert="$SA_CA_CERT" - Create a policy for accessing the secret:
vault policy write myapp-kv-ro - <<EOF path "secret/data/webapp/config" { capabilities = ["read"] } EOF - Create a role with the policy to access the secret:
vault write auth/kubernetes/role/webapp \ bound_service_account_names=vault-auth \ bound_service_account_namespaces=default \ policies=myapp-kv-ro \ ttl=24h
All the steps above are successful as far as I can see.
If I create this script on the app pod:
JWT=$(cat $JWT_PATH);
echo $JWT_PATH;
echo "$JWT";
curl -s http://vault:8200/v1/sys/health?standbyok=true
curl --header "Content-Type: application/json" -v --trace-ascii /dev/stdout --request POST --data "{\"jwt\": \"$JWT\", \"role\": \"webapp\"}" http://vault:8200/v1/auth/kubernetes/login
- The health check looks good.
- The JWT matches what curl sends
- When I execute the script the login fails with claim iss is invalid. I am truncating the actual JWT token:
/var/run/secrets/kubernetes.io/serviceaccount/token
eyJ...
{"initialized":true,"sealed":false,"standby":true,"performance_standby":false,"replication_performance_mode":"disabled","replication_dr_mode":"disabled","server_time_utc":1628146564,"version":"1.8.0","cluster_name":"vault-cluster-b323e711","cluster_id":"084375b5-b89e-6ef8-05d3-d850ac24bdc9"}
Warning: --trace-ascii overrides an earlier trace/verbose option
Note: Unnecessary use of -X or --request, POST is already inferred.
== Info: Trying 10.97.255.66...
== Info: TCP_NODELAY set
== Info: Connected to vault (10.97.255.66) port 8200 (#0)
=> Send header, 175 bytes (0xaf)
0000: POST /v1/auth/kubernetes/login HTTP/1.1
0029: Host: vault:8200
003b: User-Agent: curl/7.52.1
0054: Accept: */*
0061: Content-Type: application/json
0081: Content-Length: 1054
0097: Expect: 100-continue
00ad:
<= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 100 Continue
=> Send data, 1054 bytes (0x41e)
0000: {"jwt": "eyJ", "role": "webapp"}
== Info: We are completely uploaded and fine
<= Recv header, 36 bytes (0x24)
0000: HTTP/1.1 500 Internal Server Error
<= Recv header, 25 bytes (0x19)
0000: Cache-Control: no-store
<= Recv header, 32 bytes (0x20)
0000: Content-Type: application/json
<= Recv header, 37 bytes (0x25)
0000: Date: Thu, 05 Aug 2021 06:56:04 GMT
<= Recv header, 20 bytes (0x14)
0000: Content-Length: 40
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 40 bytes (0x28)
0000: {"errors":["claim \"iss\" is invalid"]}.
{"errors":["claim \"iss\" is invalid"]}
== Info: Curl_http_done: called premature == 0
== Info: Connection #0 to host vault left intact
I’m not sure what to try. Can someone tell me what I might be doing wrong in my setup? Is /var/run/secrets/kubernetes.io/serviceaccount/token the right path to this service account’s JWT?