Storage migration check error: error="open /vault/data/core/_migration: permission denied"

fnbk · October 5, 2021, 5:53pm

I am trying to setup vault in standalone mode and I get the following error:

WARNING! Unable to read storage migration status.
2021-10-05T17:44:09.157Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2021-10-05T17:44:09.157Z [WARN]  storage migration check error: error="open /vault/data/core/_migration: permission denied"
2021-10-05T17:44:11.158Z [WARN]  storage migration check error: error="open /vault/data/core/_migration: permission denied"
2021-10-05T17:44:13.159Z [WARN]  storage migration check error: error="open /vault/data/core/_migration: permission denied"
2021-10-05T17:44:15.159Z [WARN]  storage migration check error: error="open /vault/data/core/_migration: permission denied"
2021-10-05T17:44:17.160Z [WARN]  storage migration check error: error="open /vault/data/core/_migration: permission denied"
2021-10-05T17:44:19.161Z [WARN]  storage migration check error: error="open /vault/data/core/_migration: permission denied"
2021-10-05T17:44:21.161Z [WARN]  storage migration check error: error="open /vault/data/core/_migration: permission denied"
2021-10-05T17:44:23.162Z [WARN]  storage migration check error: error="open /vault/data/core/_migration: permission denied"

Here is the install command for kubernetes:

helm install vault hashicorp/vault --set injector.enabled=false --version 0.16.1 --namespace vault --create-namespace

side note: I am running the cluster in an ubuntu docker image (k3s). For the setup I use k3d.

k3d cluster create happy --k3s-server-arg '--no-deploy=traefik' --no-lb --api-port 127.0.0.1:6443 -p 443:443

aram · October 6, 2021, 9:28am

Looks like your PVC didn’t work or didn’t get assigned.
Each nodes needs a 10GB slice by default:

github.com

hashicorp/vault-helm/blob/main/values.yaml#L500

    
      
            # Target port to which the service should be mapped to
            targetPort: 8200
            # Extra annotations for the service definition. This can either be YAML or a
            # YAML-formatted multi-line templated string map of the annotations to apply
            # to the service.
            annotations: {}
          
          
# This configures the Vault Statefulset to create a PVC for data
          # storage when using the file or raft backend storage engines.
          # See https://www.vaultproject.io/docs/configuration/storage/index.html to know more
          dataStorage:
            enabled: true
            # Size of the PVC created
            size: 10Gi
            # Location where the PVC will be mounted.
            mountPath: "/vault/data"
            # Name of the storage class to use.  If null it will use the
            # configured default Storage Class.
            storageClass: null
            # Access Mode of the storage device being used for the PVC
            accessMode: ReadWriteOnce

fnbk · October 6, 2021, 10:47am

I see that the PV and PVC has the status Bound and the pod description looks fine to me too. So I guess the volume exists, maybe it is not mounted to the container, is there a way to validate that?

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS   REASON   AGE
pvc-cfa46aed-3681-4b56-8a9d-da25e1fd8ace   10Gi       RWO            Delete           Bound    vault/data-vault-0   local-path              20h

$ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-vault-0   Bound    pvc-cfa46aed-3681-4b56-8a9d-da25e1fd8ace   10Gi       RWO            local-path     20h

$ kubectl describe pod vault-0
Name:         vault-0
Namespace:    vault
Priority:     0
Node:         k3d-xxx-server-0/172.22.0.2
Start Time:   Wed, 06 Oct 2021 11:43:43 +0200
Labels:       app.kubernetes.io/instance=vault
              app.kubernetes.io/name=vault
              component=server
              controller-revision-hash=vault-9c984c4b7
              helm.sh/chart=vault-0.16.1
              statefulset.kubernetes.io/pod-name=vault-0
Annotations:  <none>
Status:       Running
IP:           10.42.0.53
IPs:
  IP:           10.42.0.53
Controlled By:  StatefulSet/vault
Containers:
  vault:
    Container ID:  containerd://da955e6743e41f86f038784048bca19359b00b36d69847a699fe77f7ef5e5aa5
    Image:         hashicorp/vault:1.8.3
    Image ID:      docker.io/hashicorp/vault@sha256:4db614d40d0ea5c02998b8ab01d0f67c47e3a5a76bae27e0bb9068523ab44482
    Ports:         8200/TCP, 8201/TCP, 8202/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      /bin/sh
      -ec
    Args:
      cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
      [ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
      [ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
      [ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
      [ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
      [ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
      [ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
      /usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl

    State:          Running
      Started:      Wed, 06 Oct 2021 11:43:43 +0200
    Ready:          False
    Restart Count:  0
    Readiness:      exec [/bin/sh -ec vault status -tls-skip-verify] delay=5s timeout=3s period=5s #success=1 #failure=2
    Environment:
      HOST_IP:               (v1:status.hostIP)
      POD_IP:                (v1:status.podIP)
      VAULT_K8S_POD_NAME:   vault-0 (v1:metadata.name)
      VAULT_K8S_NAMESPACE:  vault (v1:metadata.namespace)
      VAULT_ADDR:           http://127.0.0.1:8200
      VAULT_API_ADDR:       http://$(POD_IP):8200
      SKIP_CHOWN:           true
      SKIP_SETCAP:          true
      HOSTNAME:             vault-0 (v1:metadata.name)
      VAULT_CLUSTER_ADDR:   https://$(HOSTNAME).vault-internal:8201
      HOME:                 /home/vault
    Mounts:
      /home/vault from home (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vxsm5 (ro)
      /vault/config from config (rw)
      /vault/data from data (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-vault-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      vault-config
    Optional:  false
  home:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-vxsm5:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  10m                  default-scheduler  Successfully assigned vault/vault-0 to k3d-xxx-server-0
  Normal   Pulled     10m                  kubelet            Container image "hashicorp/vault:1.8.3" already present on machine
  Normal   Created    10m                  kubelet            Created container vault
  Normal   Started    10m                  kubelet            Started container vault
  Warning  Unhealthy  47s (x121 over 10m)  kubelet            Readiness probe failed: Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused

fnbk · October 6, 2021, 6:58pm

I now understand the problem and managed to come up with a working solution. Thanks @aram for pointing me into the right direction.

The persistant storage is not created successfully when running k3s with the k3d setup. (volumeClaimTemplates.status.phase is in “Pending” mode)

$ kubectl get statefulset vault -o yaml
...
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      volumeMode: Filesystem
    status:
      phase: Pending

The easiest way to work around this issue is to create your own volume.

vault-data-volume.yaml

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vault-data-pv
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/home/vagrant/vault"

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vault-data-pvc
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

Then executing the following commands will do the trick

# create pv and pvc
kubectl apply -f vault-data-volume.yaml

# install vault with custom pvc
helm install vault hashicorp/vault --version 0.16.1 --namespace vault --create-namespace --set injector.enabled=false --set server.volumes[0].name="data" --set server.volumes[0].persistentVolumeClaim.claimName="vault-data-pvc" --set server.volumeMounts[0].name="data" --set server.volumeMounts[0].mountPath="/vault/data" --set server.dataStorage.enabled=false