User vault-csi-provider cannot create resource serviceaccounts token

Vault
, ,
1

I have created a vault (v1.15) with helm at a kubernetes cluster (openstack v1.25).
It is created as HA with 3 nodes, Raft, CSI and TLS (self signed). I am using CSI and SecretProviderClass to use the kv-v2-secret in my deployment. I have two namespaces (dev, prod) with my application deployed on both that currently should access the same secrets. (I tried to add a second vault but unfortunately this doesn’t worked.)

# Usage (namespace=dev):

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: redis-kv-pwd
  namespace: dev
spec:
  provider: vault
  secretObjects:
  - secretName: redis-secrets
    data:
    - key: redispwd
      objectName: REDIS-PWD
    type: Opaque
  parameters:
    vaultAddress: https://vault.vault:8200
    vaultCACertPath: "/vault/tls/vault.ca"
    roleName: "dev-app"
    objects:  |
      - objectName: REDIS-PWD
        secretPath: "secret/data/redis-secrets"
        secretKey: "redispwd"

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dev-redisdb-deployment
  namespace: dev
spec:
  [...]
    spec:
      serviceAccountName: dev-app-sa
      containers:
      - name: redisdb
        image: redis:6.2-alpine
        imagePullPolicy: IfNotPresent
        args: ["--requirepass", "$(REDIS_PWD)"]
        [...]
        env:
        - name: REDIS_PWD
          valueFrom:
            secretKeyRef:
              name: redis-secrets
              key: redispwd
        volumeMounts:
        - name: redis-kv-pwd
          mountPath: "/mnt/secrets-store-redis"
          readOnly: true
      volumes:
      - name: redis-kv-pwd
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "redis-kv-pwd"

# Error:
in the pod logs:
Warning FailedMount 52s (x21 over 27m) kubelet MountVolume.SetUp failed for volume “redis-kv-pwd” : rpc error: code = Unknown desc = failed to mount secrets store objects for pod dev/dev-redisdb-deployment-85969db85c-sfx2s, err: error connecting to provider “vault”: provider not found: provider “vault”

in the csi logs:
2023-11-07T15:16:56.744Z [INFO] server: Finished unary gRPC call: grpc.method=/v1alpha1.CSIDriverProvider/Mount grpc.time=6.672547ms grpc.code=Unknown err=“error making mount request: couldn’t read secret "REDIS-PWD": failed to create a service account token for requesting pod {dev-redisdb-deployment-85969db85c-sfx2s 43d8d7c2-e193-4058-872d-75334d35680f dev dev-app-sa }: serviceaccounts "dev-app-sa" is forbidden: User "system:serviceaccount:vault:vault-csi-provider" cannot create resource "serviceaccounts/token" in API group "" in the namespace "dev"”

# Configuration:
After helm installation, initialising, unsealing, raft joining, enabling kv-v2 with path=secret and putting secrets.

kubernetes auth
> vault auth enable kubernetes

this was how I did it previously with a standalone simple vault and older version but tried also here
> vault write auth/kubernetes/config token_reviewer_jwt=“$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)” kubernetes_host=“https://$KUBERNETES_PORT_443_TCP_ADDR:443” kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

I saw that asterisk can also be used for wildcard
> vault policy write internal-app - <<EOF
path "secret/data/" { capabilities = [“read”] }*
EOF

I have two serviceaccounts: app-sa for prod and dev-app-sa for dev namespace
> kubectl create serviceaccount -n prod app-sa
> kubectl create serviceaccount -n dev dev-app-sa

Roles
> vault write auth/kubernetes/role/dev-app bound_service_account_names=dev-app-sa bound_service_account_namespaces=dev policies=internal-app ttl=20m
> vault write auth/kubernetes/role/app bound_service_account_names=app-sa bound_service_account_namespaces=prod policies=internal-app ttl=20m

installing csi
> helm install csi secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system --set “syncSecret.enabled=true”

Previously I had a tls error. After adding vaultCACertPath into the SPC it has worked and I could read the secrets. But the next day I got this error. I tried several things but without success, for example I added a SA vault-auth:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth

---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: default
- kind: ServiceAccount
  name: dev-app-sa
  namespace: dev
- kind: ServiceAccount
  name: app-sa
  namespace: prod

And I tried changing the auth config without success:

> vault write auth/kubernetes/config token_reviewer_jwt=“$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)” kubernetes_host=“https://$KUBERNETES_PORT_443_TCP_ADDR:443” kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt disable_iss_validation=“true” disable_local_ca_jwt=“true”

and tried the same with the jwt-token of the vault-auth service-account.