I experimented with running vault as a dev server and everything seemed to go well, so I wiped the setup and started again running vault as a production service.
I set storage to “file” with path “/home/vault/data” and made the directory and checked the perms to ensure everything is owned by vault.
I initialised my vault and captured the unsealing tokens and the root token, and I can seal and unseal and login just fine.
I just want to store simple key/value pairs for now, so tried to enable the kv engine, and I get error 400 and “failed to update mount table” like this:
# vault secrets enable kv
Error enabling: Error making API request.
URL: POST https://vault.example.com:8200/v1/sys/mounts/kv
Code: 400. Errors:
* failed to update mount table
I’ve googled for this error and found nothing except bits of source code. Even the reference to “mount table” is obscure, until I found this page which explains it a little bit: https://www.vaultproject.io/docs/secrets/
Any ideas gratefully received, thanks.
Just to check, I followed exactly the same process only using Consul for storage, and I was able to enable kv secrets engine, write a policy and then set and get data.
So my guess is that the file store backend is somehow broken?
what permissions do you have on the directory? Also are you creating the directory before starting vault or are you letting vault create the directory for you?
I have tried letting vault create the directory as well as making it for it.
the directory I set as data does get populated with files.
I also tried S3 for storage and that works fine too.
This is with vault 1.2.4 x86-64, binary downloaded from hashicorp themselves.
What do you see in the logs? If you can share that it may help isolate the issue
One of the things I would see in /var/log/message would be many lines like this:
Nov 13 23:27:28 vault-1 vault: 2019-11-13T23:27:28.098Z [ERROR] core: writing request counters to barrier: err="failed to save request counters: open /home/vault/data/sys/counters/requests/2019/_11: read-only file system"
I would also see this
Nov 14 11:46:38 vault-1 vault: 2019-11-14T11:46:38.286Z [ERROR] core: failed to persist mount table: error="open /home/vault/data/core/_mounts: read-only file system" Nov 14 11:46:38 vault-1 vault: 2019-11-14T11:46:38.286Z [ERROR] core: failed to update mount table: error="open /home/vault/data/core/_mounts: read-only file system" Nov 14 11:46:38 vault-1 vault: 2019-11-14T11:46:38.286Z [ERROR] secrets.system.system_6f80f725: mount failed: path=kv/ error="failed to update mount table"
I checked file perms carefully, as well as disabled selinux.
I’ll revert back to file storage in a while and see what else I can find.
p.s. I have this in the systemd control file:
ExecStartPre=+/usr/bin/bash -c "/sbin/setcap cap_ipc_lock=+ep $$( readlink -f /usr/local/bin/vault )"
so as to enable mlock
Hi! Looks like, if you’re on the most recent version of Vault, that error originates from here: https://github.com/hashicorp/vault/blob/8f1e9dff5d475a6eb6822fbea013a22443347f27/vault/mount.go#L523. The “open /home/vault/data/core/_mounts: read-only file system” is definitely helpful. Can you head into the /home/vault/data/core/ directory and post the result of “$ ls -lagh”? Also, what user and group are you running Vault under?
1 Like
I am running vault as user:group vault:vault.
I will get the dir permissions once I’ve re-created the vault instance with file storage.
Anyone found a solution to this? I’m facing the same issue when trying to enable aws engine.
Dec 03 18:29:28 ubuntunico vault[60379]: 2019-12-03T18:29:28.553-0300 [ERROR] core: writing request counters to barrier: err="failed to save request counters: mkdir /etc/vault.d/secrets/data/sys/counters:
Vault is running as user “vault” and I did the setup following this guide: https://learn.hashicorp.com/vault/operations/ops-deployment-guide (with file storage instead of consul). All the directories inside /etc/vault.d (including this) are owned by vault:vault
If anyone else is facing this, in my case it was because I was using this options for systemd (as the guide suggested):
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
Setting the file storage path to another directory (/vault/data for instance) solved this issue for me.