500 namespace not authorized with serviceaccount across namespaces

Vault
,
1

Hello, I was able to follow kubernetes-secret-store-driver tutorial without issue. In that tutorial, all actions are taking place within a single namespace. Now I am trying to actually configure this for our test environment.

I have two namespaces defined:
* vault - the namespace within which vault is deployed
* integration - the namespace we are testing within.

In the vault namespace:

  1. Installed vault
  2. populated secrets data
  3. Installed the secrets-store-csi-driver via helm
  4. configured the kubernetes auth role in vault:
vault write auth/kubernetes/role/integration bound_service_account_names=integration bound_service_account_namespaces=[integration] policies=integration ttl=20m

vault write auth/kubernetes/config \
> kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
  1. Defined an access policy:
vault policy write integration - << EOF
> path "webenv/*" {
>   capabilities = ["read"]
> }
> EOF

In the integration namespace

  1. Defined and applied the following SecretProviderClass:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: integration-fe
spec:
  provider: vault
  secretObjects:
  - data:
    - key: RAILS_ENV
      objectName: railsenv
    secretName: integration-fe-env
    type: Opaque

  parameters:
    vaultAddress: "http://vault.vault.svc.cluster.local:8200"
    roleName: "integration"
    objects: |
      - objectName: "railsenv"
        secretPath: "webenv/testing/integration/frontend"
        secretKey: "RAILS_ENV"
  1. Created the service account:
kubectl -n integration create sa integration
  1. Defined and created the following pod:
 kind: Pod
apiVersion: v1
metadata:
  name: intvaulttest
spec:
  serviceAccountName: integration
  containers:
  - image: jweissig/app:0.0.1
    name: intvaulttest
    env:
    - name: RAILS_ENV
      valueFrom:
        secretKeyRef:
          name: integration-fe-env
          key: RAILS_ENV
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "integration-fe"

The pod gets stuck with:

  Warning  FailedMount  44s (x20 over 26m)   kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod integration/intvaulttest, err: rpc error: code = Unknown desc = error making mount request: failed to login: Error making API request.

URL: POST http://vault.vault.svc.cluster.local:8200/v1/auth/kubernetes/login
Code: 500. Errors:

* namespace not authorized

Does anyone have any idea what I might be missing here? Any help would be appreciated.

Thank you,
-David

2

Each k-namespace needs to have a separate v-namespace authentication setup.

I’m not sure if this is supported in minikube but this article tells you how to setup the auth with Vault 1.12+/k8s 1.21+

3

It looks like you used square brackets in the Vault CLI command setting bound_service_account_namespaces.

I don’t think that’s the right syntax.

If you read the setting back, I think you’ll find its configured to look for a namespace with literal square brackets in its name.

4

This is not correct.

5

Max,

Good catch. That seems to have been the issue:

vault read auth/kubernetes/role/integration
Key                                 Value
---                                 -----
alias_name_source                   serviceaccount_uid
bound_service_account_names         [integration]
bound_service_account_namespaces    [[vault integration]]
policies                            [integration]
token_bound_cidrs                   []
token_explicit_max_ttl              0s
token_max_ttl                       0s
token_no_default_policy             false
token_num_uses                      0
token_period                        0s
token_policies                      [integration]
token_ttl                           20m
token_type                          default
ttl                                 20m

vault write auth/kubernetes/role/integration \
>      bound_service_account_names=integration \
>      bound_service_account_namespaces='integration' \
>      policies=integration \
>      ttl=20m

vault read auth/kubernetes/role/integration
Key                                 Value
---                                 -----
alias_name_source                   serviceaccount_uid
bound_service_account_names         [integration]
bound_service_account_namespaces    [integration]
policies                            [integration]
token_bound_cidrs                   []
token_explicit_max_ttl              0s
token_max_ttl                       0s
token_no_default_policy             false
token_num_uses                      0
token_period                        0s
token_policies                      [integration]
token_ttl                           20m
token_type                          default
ttl                                 20m

I now have a different error I need to track down, but it appears AUTH is occurring successfully:

URL: GET http://vault.vault.svc.cluster.local:8200/v1/webenv/testing/integration/frontend
Code: 404. Errors:
1 Like
6

An update,

Resolving the syntax error Max pointed out resolved the issue. This now works end-to-end:

kubectl -n integration exec -ti intvaulttest -- env | grep RAILS
RAILS_ENV=integration