Unable to fetch Vault Token for Pod Service Account

Vault
,
1

I’m trying to retrieve secrets from Vault for a pod running in a separate namespace (webapp) with its own service account (webapp-sa) following the steps in the blog.

So there are two things:

  • My secret stored in Vault that is going to be mounted as a Volume on the Pod.
  • The temporary Vault token that would be generated for my SA to allow the SA to be authorized to retrieved secrets from Vault.

If I understand correctly, My pod seems to fail to get this vault token with the msg - failed to create a service account token for requesting pod

Events:
  Type     Reason       Age                       From               Message
  ----     ------       ----                      ----               -------
  Normal   Scheduled    35m                       default-scheduler  Successfully assigned webapp/webapp to host-03

  Warning  FailedMount  4m38s (x23 over 35m)      kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod webapp/webapp, err: rpc error: code = Unknown desc = error making mount request: **failed to create a service account token for requesting pod** {webapp xxxx webapp webapp-sa}: the server could not find the requested resource

What I find weird is that when I try to get the vault token using cli in the pod namespace, I can get it,

$ vault write auth/kubernetes/login role=database jwt=$SA_JWT_TOKEN
Key                                       Value
---                                       -----
token                                     <snipped>

But when I use the API, I can’t get the vault token.

$ curl --request POST --data @payload.json https://127.0.0.1:8200/v1/auth/kubernetes/login
{"errors":["service account name not authorized"]}

Reference: Kubernetes - Auth Methods | Vault by HashiCorp

The Token Reviewer SA is configured as follows:

$ cat vault-auth-service-account.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-token-review-binding
  namespace: vault
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: vault

Kubernetes Auth Method is configured with JWT from the Token Reviewer SA as follows:

$ vault write auth/kubernetes/config \
    token_reviewer_jwt="< TOKEN Reviewer service account JWT>" \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" 
    kubernetes_ca_cert=@ca.crt

Vault Role is defined as follows:

$ vault write auth/kubernetes/role/database \
    bound_service_account_names=webapp-sa \
    bound_service_account_namespaces=webapp \
    policies=webapp-policy \
    ttl=72h
Success! Data written to: auth/kubernetes/role/database

Vault Policy is define as follows:

$ vault policy write webapp-policy - <<EOF
> path "secret/data/db-pass" {
>    capabilities = ["read"]
> }
> EOF
Success! Uploaded policy: webapp-policy

Am I missing something or doing incorrectly?

Would be grateful for any guidance.

Edit:

I tried to get the vault token by including the namespace name in the url, but that errors out too:

$ curl --request POST --data @payload.json https://127.0.0.1:8200/v1/auth/webapp/kubernetes/login {"errors":["missing client token"]}

2

Hi @sanakhanlibre, welcome to discuss! The error I’ve quoted means the Kubernetes TokenRequest API is unavailable. That’s the API that the CSI provider uses to generate a service account token, and then auth against Vault with. How that API gets enabled is documented here: Vault CSI Provider Installation | Vault | HashiCorp Developer. What version of Kubernetes are you using, and how is it administered (e.g. kind or GKE etc)?

With regards to vault vs curl, if you have a working vault command, then most of the time you should be able to add -output-curl-string to get a curl command equivalent. e.g. vault write -output-curl-string auth/kubernetes/login role=database jwt=$SA_JWT_TOKEN

3

Thanks @tomhjp for the reply.

Yup, I’ve been thinking the same that most likely the Token Review API is not enabled on the kube-apiserver.

I am using Charmed Kubernetes v1.19.

The kubernetes master charm version is #545.

On a quick look at options for the kube-apiserver process running on the Kubernetes Master node, I find the following:

--authentication-token-webhook-cache-ttl=1m0s 
--authentication-token-webhook-config-file=/root/charmed-k8s/auth-webhook/auth-webhook-conf.yaml 
--service-account-key-file=/root/charmed-k8s/serviceaccount.key

Although looking at a more recent version of the kubernetes master charm docs, it appears that it should be enabled in the newer version:

Thanks and Regards,
Sana

1 Like