We are using multiple different postgres database connections on the same database secrets engine mount. We recently had an incident where a cascading effect from another k8s cluster caused a surge in requests for credentials on a specific connection (up to ~4 req/s). During this period we saw 60-90s request timings. But only on this specific connection. There was no significant increase in resource use (memory, cpu or disk) and all other operations performed flawlessly. The database itself also was not under any significant pressure and creating a another vault connection configuration to the same database with the exact same configuration we where able to read new credentials from this connections within milliseconds.
On the specific connection. Both NewUser, UpdateUser and DeleteUser metrics increased to about 60-90s on off for a period.
When the number of requests for new credentials decreased so did the timings.
In that respect everything makes sense but we would like to identify the bottleneck, if possible, in order to be able to survive such a surge in the future. Increasing the resources to Vault does not seem to be the way forward.
Any idea what the bottleneck is or how to debug the issue? What metrics should we be looking at?