Hello,
I have 2 different Vault instances 1 pon v1.5.4 the other on v1.6.0.
On the 1.5.4 cluster audit logging is enabled at the path /var/log/vault.d/vault_audit.log it was working normally for a while, but now it seems the audit logging stops daily. I can manually disable then enable audit logging which starts it up again.
On the 1.6.0 cluster aduit is enble at the same path, but nothing is getting logged.
Any insights will be helpful.
What does the operational logging look like around the time auditing fails? E.g., syslog.
Outside of the log output…
Does it stop at the same time every day? Do you have rotation on these logs?
What does free disk space show for the mount/folder (df -h)?
I am not sure of when it actually stops. We do rotate the logs every 7 days.
There is plenty of available disk space.
Not sure as I don’t know exactly when it stops. If you have any examples of messages to look for I can query.
Doesn’t Vault immediately stop serving requests? Or do you have more than one audit device?
Troubleshooting Vault has some good suggestions, including instructions for setting your log_level – to debug in your case, I’d suggest – which is a good place to start.
We only log to file. AFAIK it immediately stops writing to the log file.
I’m referring to how Vault is designed to behave in the event that all of its audit devices are blocked.
Oh gotcha. No, Vault still serves requests
Really? I’m… speechless. Honestly, if you can provide HashiCorp with enough information to enable them to reproduce that in their own environment, you deserve a bounty!
This is fundamental functionality in Vault. If it isn’t behaving as documented, it’s unfit as an auditing platform, and many of its use cases are invalidated.
Guess I could open an issue on Github and go from there.
Had to re-provision the 1.5.4 (to 1.6.0) cluster today and now it’s behaving the same as the 1.6.0. Enabled the audit log, but it’s not logging anything.
That’s really, really odd.
Can you share your audit config, ie what you config’d with vault audit enable
as well as vault audit list output?
I configured the audit log with the following command:
vault audit enable file file_path=/var/log/vault.d/vault_audit.log
vault audit list
Path Type Description
file/ file n/a
Audit logging started working on one of our instances yesterday, but had stopped this morning. Had to disable/enable to get it working again.
Verified both clusters are now able to write audit logs, but audit has to be disabled then re-enabled daily.
I think I may have this fixed. We rotate the log every day and according to: https://www.vaultproject.io/docs/audit/file, we need to send SIGHUP to the vault process after rotating the logs.
I’ve added in the command to do that after rotating the log. Will verify the fix tomorrow.
Confirmed that my previous post fixed the issue.
Hi, is that possible to stop logging from specific policy to audit logs ?
No, I believe audit devices are an “All, or none” sort of thing; and, once enabled (whichever type you choose), they must always be available for write, or Vault will stop responding to all requests until that’s restored.