Allow access from different k8s cluster

Vault
1

Hi all,
I have internal k8s cluster with Hashicorp Vault and Kubernetes Secrets Store CSI Driver installed by Helm 3 as below

helm -n vault-csi list
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
csi     vault-csi       1               2023-09-12 14:01:56.41588489 +0700 +07  deployed        secrets-store-csi-driver-1.3.4  1.3.4
vault   vault-csi       1               2023-09-12 14:06:28.294509781 +0700 +07 deployed        vault-0.25.0                    1.14.0

vault auth list
Path           Type          Accessor                    Description                Version
----           ----          --------                    -----------                -------
kubernetes/    kubernetes    auth_kubernetes_b5fc6707    n/a                        n/a
token/         token         auth_token_6676a736         token based credentials    n/a

vault read auth/kubernetes/config
Key                       Value
---                       -----
disable_iss_validation    true
disable_local_ca_jwt      false
issuer                    n/a
kubernetes_ca_cert        n/a
kubernetes_host           https://192.168.0.200:6443
pem_keys                  []

Everything works fine.
Now I have an external k8s cluster (vpn site to site is setup and 2 k8s cluster can see each other).
I proxied vault service port 8200 by ingress nginx, so k8s external can access to http://192.168.0.205:8200 of internal k8s
I created “external-k8s” kubernetes auth method in vault.

vault auth list
Path           Type          Accessor                    Description                Version
----           ----          --------                    -----------                -------
external-k8s/       kubernetes    auth_kubernetes_8928bee7    n/a                        n/a
kubernetes/    kubernetes    auth_kubernetes_b5fc6707    n/a                        n/a
token/         token         auth_token_6676a736         token based credentials    n/a

vault read auth/external-k8s/config
Key                       Value
---                       -----
disable_iss_validation    true
disable_local_ca_jwt      false
issuer                    n/a
kubernetes_ca_cert        n/a
kubernetes_host           https://external-k8s
pem_keys                  []

This is SecretProviderClass yaml

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: efms-spc
  namespace: efms
spec:
  provider: vault
  secretObjects:
  - data:
    - key: appsettings.Dev.json
      objectName: appsettings.Dev.json
    secretName: efms-secrets
    type: Opaque
  parameters:
    roleName: 'efms-role'
    vaultAddress: 'http://192.168.0.205:8200'
    vaultKubernetesMountPath: "external-k8s"
    objects: |
      - objectName: "appsettings.Dev.json"
        secretPath: "efms/data/identity"
        secretKey: "appsettings.Dev.json"

I get this error

Warning  FailedMount  7s (x5 over 15s)  kubelet            MountVolume.SetUp failed for volume "efms-secrets" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod efms/utilities-6dbf59b944-5cdd4, err: rpc error: code = Unknown desc = error making mount request: couldn't read secret "appsettings.Dev.json": failed to login: Error making API request.

URL: POST http://192.168.0.205:8200/v1/auth/external-k8s/login
Code: 403. Errors:

* permission denied

If you have experience on this issue please give me advice, thank you very much.

2

I found solution https://computingforgeeks.com/how-to-integrate-multiple-kubernetes-clusters-to-vault-server/

1 Like