ESO filling up audit log - how to manage logs?

We are migrating to ESO and only downside I observed is the fact that ESO is hammering vault (we have few dozen eso SA active), creating enormous amount of logs in audit log. Average rate is 30 messages per second :slight_smile:
If I am not mistaken, Vault out of the box has no means of filtering audit log (excluding what to log to be precise), so I am asking others: How are you dealing with enormous audit logs when using ESO (External Secrets Operator) with Vault?

Just to paint a picture, what used to be 6 months worth of logs, now is single day of logs thanks to ESO :slight_smile: And no, Vault user count didnt grow more than 5% if even that. Only ESO was added.

Thanks!

Are you just writing the audit logs to a file/local device or sending to some telemetry service?

If youโ€™re writing to file you will need to configure some way to rotate the logs. I have used logrotate in the past.

I already have logrotate in place. But Issue is that 4 projects out of hundred are creating over 60% of all logs - they are using pushsecret. At this rate its not sustainable and I cannot yet establish what is exactly issue with pushsecret. If I am understanding it correctly, pushsecret checks k8s secret value and compares it to secretstore value. Unless they differ, nothing is happening. But according to logs, as if pushsecrets is constantly pinging vault even though no change is detected.

I am not familiar with ESO, but I have used VSO.

There is usually some type of variable to change the frequency of polling. In VSO this parameter is refreshAfter. I am not sure what the parameter is for ESO.

Anytime the operator performs an operation on Vault it will be logged for audit purposes. If your refresh interval is short, this could generate a lot of read requests.