Vault Enable/Configure REST Call

Vault
1

I am following the HashiCorp tutorial and it all looks fine until I try to launch the “webapp” pod - a simple pod whose only function is to demonstrate that it can start and mount a secret volume.

The error (permission denied on a REST call) is shown at the bottom of this command output:

kubectl describe pod webapp
Name:             webapp
Namespace:        default
Priority:         0
Service Account:  webapp-sa
Node:             docker-desktop/192.168.65.4
Start Time:       Tue, 14 Feb 2023 09:32:07 -0500
Labels:           <none>
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Containers:
  webapp:
    Container ID:
    Image:          jweissig/app:0.0.1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/secrets-store from secrets-store-inline (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5b76r (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  secrets-store-inline:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=vault-database
  kube-api-access-5b76r:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    42m                 default-scheduler  Successfully assigned default/webapp to docker-desktop
  Warning  FailedMount  20m (x8 over 40m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[secrets-store-inline kube-api-access-5b76r]: timed out waiting for the condition
  Warning  FailedMount  12m (x23 over 42m)  kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/webapp, err: rpc error: code = Unknown desc = error making mount request: couldn't read secret "db-password": Error making API request.

URL: GET http://vault.default:8200/v1/secret/data/db-pass
Code: 403. Errors:

* 1 error occurred:
           * permission denied
  Warning  FailedMount  2m19s (x4 over 38m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[kube-api-access-5b76r secrets-store-inline]: timed out waiting for the condition

The log of the vault-csi-provider shows this:

2023-02-14 09:25:19 2023-02-14T14:25:19.466Z [INFO]  server: Finished unary gRPC call: grpc.method=/v1alpha1.CSIDriverProvider/Mount grpc.time=22.703554ms grpc.code=Unknown
2023-02-14 09:25:19   err=
2023-02-14 09:25:19   | error making mount request: couldn't read secret "db-password": Error making API request.
2023-02-14 09:25:19   | 
2023-02-14 09:25:19   | URL: GET http://vault.default:8200/v1/secret/data/db-pass
2023-02-14 09:25:19   | Code: 403. Errors:
2023-02-14 09:25:19   | 
2023-02-14 09:25:19   | * 1 error occurred:
2023-02-14 09:25:19   | \t* permission denied

How can I fix this?

Actually, it seems that Vault is not responding to any query:

kubectl get pods
NAME                                    READY   STATUS    RESTARTS      AGE
vault-0                                 1/1     Running   1 (22m ago)   32m
vault-agent-injector-77fd4cb69f-mf66p   1/1     Running   1 (22m ago)   32m

but

vault status
Error checking seal status: Get "http://[::]:8200/v1/sys/seal-status": dial tcp [::]:8200: connect: connection refused

So if it does not respond to vault status, it probably will not accept any other command / connection.

How can I troubleshoot this?

2

The error you showed here means that Vault policy does not allow access to that secret. Re-visit the “Configure Kubernetes authentication” portion of the tutorial, as something seems to have not worked in that part.

This is a separate issue. Look at the host specified in the URL - it’s trying to talk to [::] which is localhost - not your Kubernetes Vault. You must set the VAULT_ADDR environment variable so the Vault CLI knows where to connect to.

3

The Vault logs (on Kubernetes on Docker on Win11) have the following:

Api Address: http://10.1.0.59:8200
Cluster Address: https://vault-0.vault-internal:8201
Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", tls: "disabled")
Version: Vault v1.12.1, built 2022-10-27T12:32:05Z
You may need to set the following environment variables:
$ export VAULT_ADDR='http://[::]:8200'

How should I set VAULT_ADDR?

UPDATE: I can run vault status from inside a terminal that is open on the vault pod. The pods involved in the tutorial are running in Kubernetes in Docker (under DockerDesktop) and they don’t seem to be exposing any ports to the outside. Maybe this is the solution to the vault status issue.

4

I ran the tutorial again and it all works as described.
Not sure what was wrong before.

Thank you!