Vault-secrets-operator permission denied

Vault
1

I was trying to setup vault-secrets-operator with kubernetes authentication by deploying it via Helm chart (with default values) and following the configuration in the demo, but I get the following permission denied error:

* permission denied     {"type": "Warning", "object": {"kind":"VaultStaticSecret","namespace":"default","name":"test-secret","uid":"575ff59d-a02f-4751-887b-af8817005da2","apiVersion":"secrets.hashicorp.c
om/v1alpha1","resourceVersion":"13045869"}, "reason": "VaultClientConfigError"}                                                                                                                            
2023-05-17T18:03:16Z    ERROR   Reconciler error        {"controller": "vaultstaticsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultStaticSecret", "VaultStaticSecret": {"name":"test-secret","namespace":"default"}, "namespace": "default", "name": "test-secret", "reconcileID": "8b787497-6dd6-4f0c-8c8e-ef1912906232", "error": "Error making API request.\n\nURL: PUT https://XXX/v1/auth/kubernetes/login\nCode: 403. Errors:\n\n* permission denied"}

My Vault configuration looks like this:

resource "vault_auth_backend" "kubernetes" {
  type = "kubernetes"
}

resource "vault_kubernetes_auth_backend_config" "kubernetes" {
  backend                = vault_auth_backend.kubernetes.path
  kubernetes_host        = var.eks_api_server
  disable_iss_validation = true
}

resource "vault_kubernetes_auth_backend_role" "test" {
  backend                          = vault_kubernetes_auth_backend_config.kubernetes.backend
  role_name                        = "test-role"
  bound_service_account_names      = ["default"]
  bound_service_account_namespaces = ["default"]
  token_period                     = 120
  token_policies = [
    vault_policy.test.name,
  ]
  audience = "vault"
}

resource "vault_policy" "test" {
  name      = "vso-test"
  policy    = <<EOT
path "secret/data/kubernetes/test" {
  capabilities = ["read"]
}
EOT
}

These are my VaultConnection, VaultAuth & StaticSecret configurations:

---
apiVersion: secrets.hashicorp.com/v1alpha1
kind: VaultConnection
metadata:
  namespace: default
  name: default
spec:
  address: https://XXX
---
apiVersion: secrets.hashicorp.com/v1alpha1
kind: VaultAuth
metadata:
  namespace: default
  name: default
spec:
  vaultConnectionRef: default
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: test-role
    serviceAccount: default
    audiences: ["vault"]
---
apiVersion: secrets.hashicorp.com/v1alpha1
kind: VaultStaticSecret
metadata:
  namespace: default
  name: test-secret
spec:
  vaultAuthRef: default
  mount: secret
  type: kv-v2
  name: kubernetes/test
  refreshAfter: 60s
  destination:
    create: true
    name: test-secret

The namespace, connection reference, service account, role & audience are all correct, because if I change them to an incorrect value, I get a more specific error. I am not sure what the missing piece is.

2

This could mean various things. You should ensure the Vault server’s log level is set to DEBUG or TRACE, and check the logs for more clarity on why permission is denied.

Possible issues include the service account JWT being malformed, or more likely, it being rejected by the Kubernetes token review API, or problems calling that API.

3

hey Claude, did you get this working at last? im hitting into the same problem.

4

Yes, it was causes by the lack of access to TokenReview API, as mentioned above. In our case, the problem was we had multiple testing Vault instances in separate namespaces of the same EKS cluster. That was overwriting the ClusterRoleBinding that gives access to TokenReview API. So only the last deployed instance was working.

This was a useful resource: Getting the error cannot create resource "tokenreviews" in API group during Kubernetes Authentication – HashiCorp Help Center