Failed to initialize barrier: failed to persist keyring: mkdir /vault/data/core: permission denied

Vault
1

Hello, I deployed vault using helm in OpenShift 4.12 and while performing the init received the following error.

/ $ vault operator init
Error initializing: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/init
Code: 400. Errors:

  • failed to initialize barrier: failed to persist keyring: mkdir /vault/data/core: permission denied
    / $

Note: hcl config is based on file to the path /vault/data and for the same a pvc has been created.

Apply the required rbacs and still the same. Any thoughts ?

2

The error:

reveals the exact problem - nothing to do with RBAC, but the file permissions on the Vault data directory prevent Vault writing there!

3

tried to add an init container, to solve the dir permission but still the same. Any thoughts to resolve this ?

4

Investigate and fix whatever is causing PVCs to be provisioned with incorrect file permissions, such that the services intended to run in them cannot write to them.

5

checked the pvc permissions and security context in pod and still the issue persists.

Is there any recommended security context for vault containers?

6

Perhaps you can post the values that used for your Helm install.

Seems to me the securityContext of the container is setting readOnlyRootFilesystem to true

7

here is the helm args - helm install vault . -n vault --set “server.image.repository=docker.io/hashicorp/vault” --set “injector.image.repository=docker.io/hashicorp/vault-k8s”

changed the following in values.yaml
global:
openshift: true
server:
route: true
ui:
enabled: true

Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-vault-0
ReadOnly: false

8

Related issue - Chart Is Not Compliant with the K8s Standard Hostpath Provisioner · Issue #85 · hashicorp/vault-helm · GitHub

Also, I have tried the option of init containers, scc, customrolebindings. Nothing works to resolve this.

Is there any other alternate approach for k8s or Openshift ???

9

You didn’t mention how you configured your persistent volume.

10

here is the storage conf from helm itself.

dataStorage:
enabled: true
# Size of the PVC created
size: 10Gi
# Location where the PVC will be mounted.
mountPath: “/vault/data”
# Name of the storage class to use. If null it will use the
# configured default Storage Class.
storageClass:
# Access Mode of the storage device being used for the PVC
accessMode: ReadWriteOnce
# Annotations to apply to the PVC
annotations: {}

11

And what’s your default storage class?

12

It’s my NFS, while sharing the above I removed it. But in my helm file, storage class name was specified.

13

Today I learned that: It seems that Kubernetes is inconsistent about whether it implements permissions initialization during volume setup, and hostPath and NFS are kinds of volumes that it does not do this for.

Unfortunately the Kubernetes documentation fails to make this clear - I had to dig around in the source code to confirm.

As a result, when using volumes of these types with any deployment that expects to run pods as non-root and allow securityContext.fsGroup to take care of the permissions - which is probably lots of them these days - you’ll likely see failures of this sort, and have to manually adjust permissions.

14

can you share some example for security context ?

I tried the following and still the issue persists

  • [“chown”, “-R”, “1001:1000”, “/vault/data”]
  • oc patch statefulset vault -p ‘{“spec”:{“template”:{“spec”:{“securityContext”:{“runAsUser”:1000}}}}}’
15

I have no experience of OpenShift, only Kubernetes.

The Kubernetes setting within securityContext that triggers automatic permission setting is fsGroup.

However, during the course of this thread, I learnt that the Kubernetes built-in volume support for NFS does not implement that.

You mentioned using NFS.

As I have never used NFS with either Kubernetes or OpenShift, I’m not certain what the permissions would look like by default, and whether the default permissions would even allow for further chown-ing or not.

Ultimately, the issue is that the permissions do not allow Vault to write to its storage.

I suggest that you first attempt to solve that manually - by accessing the NFS volume and seeing if you can change the permissions, and then that will prove the concept of what needs to be automated.

16

Finally, chmod +x /data did the magic and issue is resolved.