Couldn't read secret "xx-username": Error making API request

Vault
1

Hi All,

I’m trying to retrieve the passwords from the vault and seeing the below problem and the pod is unable to start.

JFYI, how to troubleshoot and fix this type of problem, please?

Warning  FailedMount  18s (x7 over 50s)  kubelet  
MountVolume.SetUp failed for volume "vault-es-secrets" : rpc 

error: code = Unknown desc = failed to mount secrets store objects 
for pod xxxx/xxxxx-xx97f45b86-6xwjf, err: rpc error: code = 
Unknown desc = error making mount request: couldn't read secret 
"es-username": Error making API request.

URL: GET http://vault:8200/v1/xxxx/data/configuration/xxx_config/es_config
Code: 403. Errors:

* 1 error occurred:
  * permission denied
2

As the error is saying you don’t have permission to the URL trying to be accessed. You need to check the policy and then add the missing permission.

3

Thank you, actually i am noticing the FailedMount issue, is this expected?

How to check the policy and add the permission, please.

Events:
  Type     Reason       Age                    From     Message
  ----     ------       ----                   ----     -------
  Warning  FailedMount  23m (x181 over 22h)    kubelet  Unable to attach or mount volumes: unmounted volumes=[vault-es-secrets], unattached volumes=[sial-config wildfly-standalone-log vault-cas-secrets vault-es-secrets kube-api-access-5bfxd]: timed out waiting for the condition
  Warning  FailedMount  11m (x99 over 22h)     kubelet  Unable to attach or mount volumes: unmounted volumes=[vault-es-secrets], unattached volumes=[vault-es-secrets kube-api-access-5bfxd sial-config wildfly-standalone-log vault-cas-secrets]: timed out waiting for the condition
  Warning  FailedMount  7m35s (x679 over 22h)  kubelet  MountVolume.SetUp failed for volume "vault-es-secrets" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxxxx/xxxxx-6c97f45b86-6xwjf, err: rpc error: code = Unknown desc = error making mount request: couldn't read secret "es-username": Error making API request.

URL: GET http://vault:8200/v1/searchcatalog/data/configuration/sial_config/es_config
Code: 403. Errors:

* 1 error occurred:
           * permission denied
  Warning  SecretRotationFailed  4m8s (x16893 over 21h)  csi-secrets-store-rotation  failed validation for secret object in spc xxxxx/xxxx-cas-secrets, err: secret type is empty
  Warning  FailedMount           2m48s (x100 over 22h)   kubelet                     Unable to attach or mount volumes: unmounted volumes=[vault-es-secrets], unattached volumes=[wildfly-standalone-log vault-cas-secrets vault-es-secrets kube-api-access-5bfxd sial-config]: timed out waiting for the condition
4

You need to check your Vault setup. Look at the policy being used with the Kubernetes auth you are doing.

5

I checked at the vault end, seeing the below output.
Able to read and view the secrets using the command line.

 $ vault --version
Vault v1.10.3 (af866591ee60485f05d6e32dd63dde93df686dfb)
/ $ vault policy list
default
es_config
prometheus-metrics
xxx_config
root
/ $ vault policy read xxx_config
path "xxxx/data/configuration/xxx_config" {
  capabilities = ["read"]
}
/

/ $ vault policy read es_config
path "xxxx/data/configuration/xxx_config/es_config" {
  capabilities = ["read"]
}
6 
$ vault kv get xxxxx/configuration/xxx_config/es_config
===================== Secret Path =====================
xxxxxx/data/configuration/xxx_config/es_config

======= Metadata =======
Key                Value
---                -----
created_time       2022-07-21T17:29:36.371468587Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    xxxxx
username    xxxxxx