Status code error 503, No server is available to handle this request


I have 4-nodes(2-sites) Vault with HA mode. The backend that i use is Consul, (4-nodes in 2-sites as well).
All of the VM’s are rhel 7.4, the vault version is 1.5.4 and the consul is 1.6.1.

My issue is, occasionally I am experiencing, “503 Service Unavailable, No server is available to handle this request.” after an api request to one of vault’s secrets, from another server (not vault architecture related).
I monitored this issue with vault audit, and i can see the request and the response there.
Using the vault UI, i see that there are around 1M http requests in a month.

This bug happens dozen times a day.

I was hoping someone might have encountered this before and can share thier thoughts on it. Thanks!