Vault CSI : error making mount request: Error loading CA File

Vault
1

Hello,

I tried following the guide below for enabling TLS for vault.

And the guide below for CSI Vault

I set up everything but i get the error below on pods

Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    12s               default-scheduler  Successfully assigned fashionhousek8s/create-fashionhousek8s-db-job-4tl4l to ip-172-21-172-255.eu-central-1.compute.internal
  Warning  FailedMount  5s (x5 over 12s)  kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod fashionhousek8s/create-fashionhousek8s-db-job-4tl4l, err: rpc error: code = Unknown desc = error making mount request: Error loading CA File: open /home/admin/vault/certificates/vault.ca: no such file or directory

I have this config for my secretproviderclass

admin@skdeployer:~$ k get secretproviderclass -o yaml
apiVersion: v1
items:
- apiVersion: secrets-store.csi.x-k8s.io/v1
  kind: SecretProviderClass
  metadata:
    annotations:
      helm.sh/hook: pre-install
      helm.sh/hook-weight: "5"
    creationTimestamp: "2023-04-17T02:03:27Z"
    generation: 1
    name: vault-database
    namespace: fashionhousek8s
    resourceVersion: "1889291"
    uid: fd59201f-0942-45ce-9cf2-99cfa7cf713e
  spec:
    parameters:
      objects: |
        - objectName: "AWS_ACCESS_KEY_ID"
          secretPath: "secret/data/tokens"
          secretKey: "AWS_ACCESS_KEY_ID"
        - objectName: "AWS_SECRET_ACCESS_KEY"
          secretPath: "secret/data/tokens"
          secretKey: "AWS_SECRET_ACCESS_KEY"
      roleName: backoffice
      vaultAddress: https://vault.vault.svc.cluster.local:8200
      vaultCACertPath: /home/admin/vault/certificates/vault.ca
    provider: vault
    secretObjects:
    - data:
      - key: AWS_ACCESS_KEY_ID
        objectName: AWS_ACCESS_KEY_ID
      - key: AWS_SECRET_ACCESS_KEY
        objectName: AWS_SECRET_ACCESS_KEY
      secretName: account-token
      type: Opaque
kind: List
metadata:
  resourceVersion: ""
  selfLink: "
2

Hi @rommellll,

If you’ve followed the TLS tutorial along, you should have something similar to the following in your values.yaml:

   volumes:
      - name: userconfig-vault-ha-tls
        secret:
         defaultMode: 420
         secretName: vault-ha-tls
   volumeMounts:
      - mountPath: /vault/userconfig/vault-ha-tls
        name: userconfig-vault-ha-tls
        readOnly: true

Note that this is only defined for the Vault server but not for the CSI DaemonSet (because in that tutorial the csi is not enabled).

The error you’re getting is telling you the vault-csi-provider pods cannot find the CA certificate because it hasn’t been mounted to a volume.

In order to do that, just replicate the same configuration in the csi config section of your values.yaml as csi.volumes and csi.volumeMounts and pass the correct path to the mounted secret in vaultCACertPath

3

Thankyou @macmiranda for the reply.

I tried this override.yaml but still get the same error. Do you still have any idea?

global:
   enabled: true
   tlsDisable: false
injector:
   enabled: false
server:
   extraEnvironmentVars:
      VAULT_CACERT: /vault/userconfig/vault-ha-tls/vault.ca
      VAULT_TLSCERT: /vault/userconfig/vault-ha-tls/vault.crt
      VAULT_TLSKEY: /vault/userconfig/vault-ha-tls/vault.key
   volumes:
      - name: userconfig-vault-ha-tls
        secret:
         defaultMode: 420
         secretName: vault-ha-tls
   volumeMounts:
      - mountPath: /vault/userconfig/vault-ha-tls
        name: userconfig-vault-ha-tls
        readOnly: true
   standalone:
      enabled: false
   affinity: ""
   ha:
      enabled: true
      replicas: 3
      raft:
         enabled: true
         setNodeId: true
         config: |
            ui = true
            listener "tcp" {
               tls_disable = 0
               address = "[::]:8200"
               cluster_address = "[::]:8201"
               tls_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
               tls_key_file  = "/vault/userconfig/vault-ha-tls/vault.key"
               tls_client_ca_file = "/vault/userconfig/vault-ha-tls/vault.ca"
            }
            storage "raft" {
               path = "/vault/data"
            }
            disable_mlock = true
            service_registration "kubernetes" {}
csi:
   enabled: true
   volumes:
      - name: userconfig-vault-ha-tls
        secret:
         defaultMode: 420
         secretName: vault-ha-tls
   volumeMounts:
      - mountPath: /vault/userconfig/vault-ha-tls
        name: userconfig-vault-ha-tls
        readOnly: true

error

Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    80s                default-scheduler  Successfully assigned fashionhousek8s2/create-fashionhousek8s2-db-job-rdhm9 to ip-172-21-172-255.eu-central-1.compute.internal
  Warning  FailedMount  16s (x8 over 80s)  kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod fashionhousek8s2/create-fashionhousek8s2-db-job-rdhm9, err: rpc error: code = Unknown desc = error making mount request: Error loading CA File: open /home/admin/vault/certificates/vault.ca: no such file or directory
4

Still need to change this in the SecretProviderClass

5

Thank you for the quick response @macmiranda !

I have this in my secretproviderclass

  parameters:
    vaultAddress: "https://vault-internal.vault.svc.cluster.local:8200"
    vaultCACertPath: /home/admin/vault/certificates/vault.ca

is vaultCACertPath the local file path of the certificates or the location of the mounted certificates?

admin@skdeployer:~/vault/certificates$ pwd
/home/admin/vault/certificates
admin@skdeployer:~/vault/certificates$ ls -1
cluster-keys.json
csr.yaml
overrides.yaml
vault.ca
vault.crt
vault.csr
vault-csr.conf
vault.key

Thank you in advance.

6

The path of the mounted certificate in the Pod, in your case:
/vault/userconfig/vault-ha-tls/vault.ca

1 Like
7

That was it! Thank you very much @macmiranda

Something bothers me in the documentation, if you can confirm this that would be really helpful as well otherwise you’ve been a great help already.

On the documentation, when unsealing the vault-0, vault-1 and vault-2.
It was highlighted that the status of the vault-0 when unseased.
image

but for vault-1 and vault-2, the example output still says vault is sealed.

image
image741×256 15.1 KB

8

Did you follow the steps here?

9

Yes I did. Though, i did but did not use a single key.
On step 4,

kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator init \
    -format=json > ${WORKDIR}/cluster-keys.json

So i have to unseal 3 times. For vault-0, it did unsealed after 3 times but for vault-1 and vault-2 it stayed sealed like the one on the documentation with a single key.

10

I’m not very familiar with manual unsealing since I use auto-unseal but I believe if the node/pod has joined the cluster successfully, you should be able to run vault operator unseal without any arguments and it will ask you to input the 3 keys, one at a time.
It should report as unsealed afterwards.

11

Kind of … you have to re-run vault operator unseal the appropriate number of times, yourself, rather than it repeatedly prompting. This is because in a real deployment using split unseal keys, each key share would be provided by a different person.

12

This is a race condition.

When you provide an unseal key, that is going to be used to enable joining a Raft cluster, there is a lot more work for Vault to do, compared to a normal unseal.

As a result, when the automatic vault status display, that is automatically shown by vault operator unseal after submitting the unseal key, is fetched, Vault is still reading as sealed, because it is still processing the cluster join in the background.

If you repeatedly run vault status yourself after this, you will find the vault-1 and vault-2 pods do complete their transition to being unsealed after a little while.

1 Like