I have created a vault (v1.15) with helm at a kubernetes cluster (openstack v1.25).
It is created as HA with 3 nodes, Raft, CSI and TLS (self signed). I am using CSI and SecretProviderClass to use the kv-v2-secret in my deployment. I have two namespaces (dev, prod) with my application deployed on both that currently should access the same secrets. (I tried to add a second vault but unfortunately this doesn’t worked.)
# Usage (namespace=dev):
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: redis-kv-pwd
namespace: dev
spec:
provider: vault
secretObjects:
- secretName: redis-secrets
data:
- key: redispwd
objectName: REDIS-PWD
type: Opaque
parameters:
vaultAddress: https://vault.vault:8200
vaultCACertPath: "/vault/tls/vault.ca"
roleName: "dev-app"
objects: |
- objectName: REDIS-PWD
secretPath: "secret/data/redis-secrets"
secretKey: "redispwd"
apiVersion: apps/v1
kind: Deployment
metadata:
name: dev-redisdb-deployment
namespace: dev
spec:
[...]
spec:
serviceAccountName: dev-app-sa
containers:
- name: redisdb
image: redis:6.2-alpine
imagePullPolicy: IfNotPresent
args: ["--requirepass", "$(REDIS_PWD)"]
[...]
env:
- name: REDIS_PWD
valueFrom:
secretKeyRef:
name: redis-secrets
key: redispwd
volumeMounts:
- name: redis-kv-pwd
mountPath: "/mnt/secrets-store-redis"
readOnly: true
volumes:
- name: redis-kv-pwd
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "redis-kv-pwd"
# Error:
in the pod logs:
Warning FailedMount 52s (x21 over 27m) kubelet MountVolume.SetUp failed for volume “redis-kv-pwd” : rpc error: code = Unknown desc = failed to mount secrets store objects for pod dev/dev-redisdb-deployment-85969db85c-sfx2s, err: error connecting to provider “vault”: provider not found: provider “vault”
in the csi logs:
2023-11-07T15:16:56.744Z [INFO] server: Finished unary gRPC call: grpc.method=/v1alpha1.CSIDriverProvider/Mount grpc.time=6.672547ms grpc.code=Unknown err=“error making mount request: couldn’t read secret "REDIS-PWD": failed to create a service account token for requesting pod {dev-redisdb-deployment-85969db85c-sfx2s 43d8d7c2-e193-4058-872d-75334d35680f dev dev-app-sa }: serviceaccounts "dev-app-sa" is forbidden: User "system:serviceaccount:vault:vault-csi-provider" cannot create resource "serviceaccounts/token" in API group "" in the namespace "dev"”
# Configuration:
After helm installation, initialising, unsealing, raft joining, enabling kv-v2 with path=secret and putting secrets.
kubernetes auth
> vault auth enable kubernetes
this was how I did it previously with a standalone simple vault and older version but tried also here
> vault write auth/kubernetes/config token_reviewer_jwt=“$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)” kubernetes_host=“https://$KUBERNETES_PORT_443_TCP_ADDR:443” kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
I saw that asterisk can also be used for wildcard
> vault policy write internal-app - <<EOF
path "secret/data/" { capabilities = [“read”] }*
EOF
I have two serviceaccounts: app-sa for prod and dev-app-sa for dev namespace
> kubectl create serviceaccount -n prod app-sa
> kubectl create serviceaccount -n dev dev-app-sa
Roles
> vault write auth/kubernetes/role/dev-app bound_service_account_names=dev-app-sa bound_service_account_namespaces=dev policies=internal-app ttl=20m
> vault write auth/kubernetes/role/app bound_service_account_names=app-sa bound_service_account_namespaces=prod policies=internal-app ttl=20m
installing csi
> helm install csi secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system --set “syncSecret.enabled=true”
Previously I had a tls error. After adding vaultCACertPath into the SPC it has worked and I could read the secrets. But the next day I got this error. I tried several things but without success, for example I added a SA vault-auth:
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
name: vault-auth
annotations:
kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-auth
namespace: default
- kind: ServiceAccount
name: dev-app-sa
namespace: dev
- kind: ServiceAccount
name: app-sa
namespace: prod
And I tried changing the auth config without success:
> vault write auth/kubernetes/config token_reviewer_jwt=“$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)” kubernetes_host=“https://$KUBERNETES_PORT_443_TCP_ADDR:443” kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt disable_iss_validation=“true” disable_local_ca_jwt=“true”
and tried the same with the jwt-token of the vault-auth service-account.