Hello,
We are using HashiCorp Vault OSS version deployed via Helm Chart. Recently, we had some errors related with the following metrics:
vault_audit_log_request_failure
vault_audit_log_response_failure
The output of both metrics is a value greater than 1. Audit device is enabled to send logs to stdout, which are collected by a logging system. Even though from my understanding based on this documentation (Audit Devices | Vault | HashiCorp Developer) Vault will stop to respond to any requests, it actually isn’t stopping.
This has happened in both version HashiCorp Vault v1.15.6 & v1.16.0
Can you please help to verify what could be the root cause of this issue?
Thank you!
What types of audit devices do you have enabled? I recently ran into this and had a few conditions lead up to it:
- Insufficient disk space - make sure you’re auto-rotating your logs with something like
logrotate and that they are indeed being cycled out as desired.
- For syslog - journald and rsyslog have rate limiting in place by default. If your environment is exceeding the rate limit syslog will not be written to and be considered blocked, once the rate limit threshold expires logs will be able to be written again until the limit is saturated
- For syslog - I forget which, but journald or rsyslog also have a maximum message size defined (I think 8KB by default). Large requests will exceed this and fail to be written to syslog. You may be able to mitigate this if it’s happening on
list operations by configuring the elide_list_responses parameter on the audit device. Otherwise you may need to increase the maximum message size to accommodate. I had to bump my config to 32KB.
Hope this helps.
1 Like
Hi @jeffsanicola
Thank you for your response!
I have enabled file type audit device and the file path is stdout. I have checked for insufficient disk space, but that is not the case.