Vault Secrets Operator with external Vault cluster: 403 Permission Denied

Vault
1

Hi there,

I have been struggling with this for 3 days and still cannot wrap my head around it:

I am trying to synchronize some kvv2’s static secrets using the Vault Secrets Operator in a client cluster but I am always getting a 403 response when trying to login as part of the synchronization process. Also, the vault server is not logging anything even if the ‘server.logLevel’ directive is set to “trace”.

I followed the VSO tutorial and used ‘*’ namespace instead of ‘app’ and also omitted the ‘audience’ field, as I don’t see anywhere that it is required.

Following the documentation I wanted to use the recommended way: short-lived tokens using the local SA(serviceaccount) as JWT reviewer, so I omitted every field in the k8s auth method configuration except for the host (which is set to my client cluster kube API address, port 6443).

I tried to manually use the token from the client SA on a curl request and it is in fact returning 403, the only way I can successfully login with curl is when I create a k8s secret engine for my client cluster and use it to generate a token for the same role I’m using with the Vault Secrets Operator, but I don’t think this is the expected workflow when using the VSO, right?

I find the documentation very confusing for this use case, can anybody shed some light on this? I can share whatever configuration is needed.

Thanks in advance.

2

Nevermind info about the secret engine. It only works for service accounts from inside the Vault cluster.

So in the end: what are the options for using the VSO and short-lived tokens from a client cluster?

3

When you want K8s service accounts in one cluster to authenticate to the Vault K8s auth method, running outside the cluster, you have this choice to make:

  1. Build your own cross-cluster automation to create a service account in the target cluster, take the service account JWT, and configure it in Vault (and refresh it as needed).

  2. Grant every service account access to the tokenreviews API. Then, configure Vault without a JWT (and disable_local_ca_jwt=true). Vault will then use the JWT that is presented in each K8s login attempt, as the credential to call the tokenreviews API.

1 Like
4

Thanks for your response,

Point 2 is exactly what I was trying to do, and I still don’t know what was missing but it ended up working after re-applying configurations and manifests all over the place.

5

hey teksuo, i am hitting into the exact same problem for like a week now, what configurations did you re-apply to get it working? if possible can you share ur configurations , I am also trying to achieve Point 2.

6

Basically you need to create an auth method for every cluster in which you want to sync secrets from Vault.

This auth method must define the base64-decoded version of your CA and the endpoint URI. You should also disable_local_ca_jwt=true as mentioned, and then create a Role for the service account you are using with the VSO with bound namespace=*

That’s all I do for every new cluster and it works fine.

7

Hi @teksuo,

I am running into the same error. I have a Vault installed on GCP Cluster and I want to test if I can sync secrets from a client cluster, in my case I am using “kind” for testing (PoC).

In the “kind” cluster I have VaultConnection and VaulthAuth configured correctly, since in the kubernetes object they both appear to me as valid and accepted.

When I try to create the VaultStaticSecret as the VaultDynamicSecret it returns the permission denied.

Can you share the yaml, the configurations or the step by step method to make it work?

Best regards

8

HI @teksuo
I am facing the same error and tried with all the options which you mentioned above.
It would be good if you share the config which you used.

10

Hi there,

I am not using this operator anymore, but make sure your auth method has the role correctly attached to it, with the right service account name and bound namespace (that was ‘*’ in my case). I also remember that in my case was necessary to have the “Generated token policies” field fulfilled with the corresponding policy name.

11

Hi,
This discussion was very helpful as the official documentation is not very helpful.
In my case, I have to check that my service account name was the same configured in Vault in bound_service_account_names.
Also, the namespace has to be included in bound_service_account_namespaces.
In the VaultAuth, the mount corresponds to the path in the Vault auth backend configuration.
In the VaultStaticSecret, the mount is the name of the secret engine and the path is the path to your secret.
Don’t forget to upgrade your CRDS