Vault Secrets Operator with external Vault cluster: 403 Permission Denied

Vault
1

Hi there,

I have been struggling with this for 3 days and still cannot wrap my head around it:

I am trying to synchronize some kvv2’s static secrets using the Vault Secrets Operator in a client cluster but I am always getting a 403 response when trying to login as part of the synchronization process. Also, the vault server is not logging anything even if the ‘server.logLevel’ directive is set to “trace”.

I followed the VSO tutorial and used ‘*’ namespace instead of ‘app’ and also omitted the ‘audience’ field, as I don’t see anywhere that it is required.

Following the documentation I wanted to use the recommended way: short-lived tokens using the local SA(serviceaccount) as JWT reviewer, so I omitted every field in the k8s auth method configuration except for the host (which is set to my client cluster kube API address, port 6443).

I tried to manually use the token from the client SA on a curl request and it is in fact returning 403, the only way I can successfully login with curl is when I create a k8s secret engine for my client cluster and use it to generate a token for the same role I’m using with the Vault Secrets Operator, but I don’t think this is the expected workflow when using the VSO, right?

I find the documentation very confusing for this use case, can anybody shed some light on this? I can share whatever configuration is needed.

Thanks in advance.

2

Nevermind info about the secret engine. It only works for service accounts from inside the Vault cluster.

So in the end: what are the options for using the VSO and short-lived tokens from a client cluster?

3

When you want K8s service accounts in one cluster to authenticate to the Vault K8s auth method, running outside the cluster, you have this choice to make:

  1. Build your own cross-cluster automation to create a service account in the target cluster, take the service account JWT, and configure it in Vault (and refresh it as needed).

  2. Grant every service account access to the tokenreviews API. Then, configure Vault without a JWT (and disable_local_ca_jwt=true). Vault will then use the JWT that is presented in each K8s login attempt, as the credential to call the tokenreviews API.

1 Like
4

Thanks for your response,

Point 2 is exactly what I was trying to do, and I still don’t know what was missing but it ended up working after re-applying configurations and manifests all over the place.

5

hey teksuo, i am hitting into the exact same problem for like a week now, what configurations did you re-apply to get it working? if possible can you share ur configurations , I am also trying to achieve Point 2.

6

Basically you need to create an auth method for every cluster in which you want to sync secrets from Vault.

This auth method must define the base64-decoded version of your CA and the endpoint URI. You should also disable_local_ca_jwt=true as mentioned, and then create a Role for the service account you are using with the VSO with bound namespace=*

That’s all I do for every new cluster and it works fine.