Going through the guide on Kubernetes auth, is the Kubernetes auth method even applicable using an external vault (vault is not running Kubernetes). Kubernetes - Auth Methods | Vault by HashiCorp
Ultimately I want to issue certs to applications running in K8s from our external vault: Configure Vault as a Certificate Manager in Kubernetes with Helm | Vault - HashiCorp Learn
What auth method would work best in this case if Kubernetes auth isn’t applicable?
Yes it is, as long as your pointed to Kubernetes cluster’s API server is reachable by your Vault cluster.
If you are not a huge fan of the Kubernetes auth method, you can always use the OIDC auth replacement with Vault acting as an OIDC provider.
Yes, the Kubernetes cluster API is available to the Vault cluster. For the Kubernetes auth method, where do I find the value for this specified in the walk-through? It seems like it is assuming Vault is running kubernetes
token_reviewer_jwt="<your reviewer service account JWT>"
The token_reviewer_jwt is a token that is used to send a TokenReview request to your designated Kubernetes server. So this token needs to be obtained from your Kubernetes server from a service account with the right capabilities.
Would this method be applicable? It mentioned omitting the jwt token when configuring the auth method, but I assume this is only for when running vault in Kubernetes:
In your case, yes this would be applicable. In this case it will use the requesting service account’s JWT as the token reviewer. Do keep in mind that the requesting service account needs to have the capabilities to perform a token review.
Requesting service account meaning the account currently logged into vault when performing the vault write auth/kubernetes/config ?
Yes! At least if you are then using parameters derived from the Kubernetes cluster that is targeted in your auth/kubernetes config.
So the vault account you are using and the service account you’ve defined on the k8s cluster need to have the same name?
Yes, they need to be the exact same account. You will be using the service account token from the privileged service account you’ve defined in your k8s cluster to login to Vault. This service account in turn can also verify itself.
Gotcha. So in this order: define the service account in K8s, take the token there and define it in Vault and attach the appropriate policies, then setup the kubernetes auth in Vault
Exactly!
Just make sure that the service account in K8s has the right capabilities to make the TokenReview request.
define the service account in K8s, take the token there and define it in Vault and attach the appropriate policies, then setup the kubernetes auth in Vault
Just for a bit of a sum up. This approach is the will be the long-lived-auth approach.
If you want to go ahead and use the use-vault-client-jwt-as-the-reviewer approach, you can go ahead and emit the second step (take the token there and define it in Vault) and set disable_local_ca_jwt to true.
For the one other approach, please have a look here.
I hope this helps enough!
Hello here!
I’d like to add a question to that thread. When using Kubernetes auth with an external Vault, how do you handle the auth for more the one service (and thus multiple token_reviewer_jwt)?
With an external Vault, do we have to configure as many Kubernetes auth methods in Vault as service accounts ? How do we handle auth for multiple services ?
To detail a bit, I’m following the tutorial for setting up Vault CSI in Kubernetes but the part about the Kubernetes auth configuration seems to only cover Vault hosted on K8S with no token_reviewer_jwt specified since - if I understood correctly - it is accessible via a volume mounted into the pod in that case. The documentation lacks precision in the case of an external Vault cluster.
Authorizing multiple services does not imply multiple token_reviewer_jwt.
No, that woud be an antipattern. There should be one Kubernetes auth method per Kubernetes cluster.
Beware that some of the information in this thread appears to be incorrect.
This bit is not true:
Nor is this bit:
When using Vault external to the Kubernetes cluster(s), it comes down to this simple decision:
Either you allow all Kubernetes service accounts access to the TokenReview operation, and configure Vault to use the Kubernetes token submitted in each login attempt to talk to the Kubernetes API (token_reviewer_jwt unset, disable_local_ca_jwt=true)
Or you give Vault a dedicated Kubernetes service account token to use to invoke the TokenReview operation - meaning that you configure an explicit token_reviewer_jwt (manually or through your own automation).
Alright!
So to summarize, on my external Vault, I would have to configure my Kubernetes auth method this way :
$ vault auth enable kubernetes
$ vault write auth/kubernetes/config \
kubernetes_host="https://kubernetes-api.example.net:6443" \
kubernetes_ca_cert="$K8S_CA_CRT" \
disable_local_ca_jwt=true
Create the required service account on my K8S cluster :
$ cat myapp-sa.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: myapp-sa
namespace: myapp
$ kubectl apply -f myapp-sa.yaml
Then define a role attached to an existing policy this way :
$ vault write auth/kubernetes/role/myapp \
bound_service_account_names=myapp-sa \
bound_service_account_namespaces=myapp \
policies=myapp \
ttl=20m
And I would be all set?
So correct me if I’m wrong, but in that case, behind the curtains, when a Kubernetes service authenticates via Vault, it presents itself with a JWT, and since Vault doesn’t have any token_reviewer_jwt it will just check it against the tokenreview API via https://kubernetes-api.example.net:6443. Is that correct?
Nearly, but not quite.
Yes, but when Vault contacts the tokenreview API, it will re-use the presented Kubernetes JWT as Vault’s own authentication to call the tokenreview API.
For that reason, you need to create a K8s ClusterRoleBinding that, allows this to happen. It might look like this:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tokenreviews
subjects:
- kind: Group
name: system:serviceaccounts
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:auth-delegator
apiGroup: rbac.authorization.k8s.io
Hi!
Sorry for the late reply. I’ve just tested that, and it works like a charm 
Thanks so much for your help!
I would like to know more about this. Can we use the default SA created by Vault CSI provider for /v1/auth/kubernetes/config and use the same jwt for /v1/auth/kubernetes/login.
I am trying to set up the config from the pipeline using API which returns 204 but when I use the same jwt to login it returns 403. On the contrary, If I manually set this up via vault UI using same jwt it works but not via API.