We are intermittently seeing permission issue in vault logs and error is as below
Sep 30 13:05:29 spanugo-vault-5bf6498685-2qbs5spanugo-vault ERROR core: writing request counters to barrier: err=“failed to save request counters: open /vault/data/backend/sys/counters/requests/2020/_09: permission denied”
Version of Vault being used is 1.5.0. Mount path for vault in our set up is /vault/data.
We are running vault as underprivileged user on docker (say clouduser), not as root. When vault is deployed “/vault/data/backend/sys/counters/requests/2020/” was having ownership of underprivileged user (clouduser) with which vault service was running as well. There was no issue in set up for couple of months as long as folder remained with clouduser ownership. However, we have noticed that recently ownership got changed to root user and root group automatically by vault service, when a new file was created in the folder.
-rwxrwxr-- 1 clouduser clouduser 81 Sep 22 16:59 _08
-rw-r----- 1 root root 77 Sep 28 23:38 _09
We saw similar issue with expire folder as well sometime ago, permissions when issue occurred was as below. As we can see below, expire folder was having root user ownership.
drwxrw-rw- 1 clouduser clouduser 0 Aug 13 07:22 counters
drwxr-x— 1 root root 0 Jan 1 1970 expire
drwx------ 1 clouduser clouduser 0 Jul 31 12:18 policy
drwx------ 1 clouduser clouduser 0 Jul 31 12:18 token
This happens when expire folder is deleted since all files deleted within it due to shorter duration of TTL and when folder is recreated again when new token is created.
We can see in the official hashicorp vault image has unprivileged user has added into root user group. As per the Hashicop documentation root user privileges not required for vault operations. But In the Dockerfile (usermod -a -G root vault) we can see this. The unprivileged user really need the sudo user privileges to perform vault operation ?
As an immediate fix, we have changed the ownership of offending file which was causing permission issue and restarted the vault and it is working fine after that.
Do you have any insight to this issue and any set up related issue which we could have done to cause this issue?