Failed to setup mount table

Vault
1

Hi Team,

We ran into a disk space on our consul masters, and after restarting vault we’re now getting this error on unseal

  • failed to setup mount table
2022-01-11T02:00:57.134Z [WARN]  core: vault is sealed
2022-01-11T02:01:34.123Z [WARN]  core: cluster listener is already started
2022-01-11T02:01:34.123Z [INFO]  core: post-unseal setup starting
2022-01-11T02:01:34.123Z [INFO]  core: loaded wrapping token key
2022-01-11T02:01:34.123Z [INFO]  core: successfully setup plugin catalog: plugin-directory="\"\""
2022-01-11T02:01:34.123Z [ERROR] core: failed to read mount table: error="decryption failed: cipher: message authentication failed"
2022-01-11T02:01:34.123Z [INFO]  core: pre-seal teardown starting
2022-01-11T02:01:34.123Z [INFO]  core: pre-seal teardown complete
2022-01-11T02:01:34.123Z [ERROR] core: post-unseal setup failed: error="failed to setup mount table"
2022-01-11T02:01:34.123Z [WARN]  core: vault is sealed

I’ve done a migration of the vault data from consul to disk and setup my own test env but the error is the same.

I’ve had a look at the files on disk and the _mount file in it doesn’t look right compared to other i’m running (permissions aren’t an issue either… I set 777 just incase for the filesystem)

# ls -al
total 164
drwxrwxrwx. 7 root root    250 Jan 11 11:43 .
drwxrwxrwx. 6 root root     56 Jan 11 11:29 ..
-rwxrwxrwx. 1 root root    133 Jan 11 11:43 _audit
-rwxrwxrwx. 1 root root    573 Jan 11 11:43 _auth
drwxrwxrwx. 3 root root     41 Jan 11 10:45 cluster
drwxrwxrwx. 2 root root     34 Jan 11 10:45 hsm
-rwxrwxrwx. 1 root root    325 Jan 11 11:43 _keyring
drwxrwxrwx. 2 root root 102400 Jan 11 11:43 leader
-rwxrwxrwx. 1 root root    133 Jan 11 11:43 _local-audit
-rwxrwxrwx. 1 root root    133 Jan 11 11:43 _local-auth
-rwxrwxrwx. 1 root root    485 Jan 11 11:43 _local-mounts
-rwxrwxrwx. 1 root root    209 Jan 11 11:43 _master
-rwxrwxrwx. 1 root root     15 Jan 11 11:43 _mounts
-rwxrwxrwx. 1 root root    169 Jan 11 11:43 _seal-config
-rwxrwxrwx. 1 root root    101 Jan 11 11:43 _shamir-kek
drwxrwxrwx. 2 root root     20 Jan 11 10:45 versions
drwxrwxrwx. 2 root root     21 Jan 11 10:45 wrapping


# cat _mounts
{"Value":null}

Looking for any tips/guidance as to how I can get this restored/accessible?

Thanks
David

2

Are those files supposed to be owned by root? Are you running consul binary as root? Normally it’s run by a consul user and the data files are owned by consul.

3

This was just my copy of it - and as it was running in a container for testing I wanted to ensure no permission issues - I ended up getting a backup from a few days ago and restored that to fix whatever caused this issue.