Has anyone seen this specific error message.
Cluster went from running steady state for months to unstable.
The only change being an upgrade to a go client library running on a single node in a cluster of 300 +
Hi @jkonkin can you share the error you are seeing please? Also, what version of Consul are you running?
Hi @karl-cardenas-coding
Thank you for your response, my apologies for the delayed reply
The consul-servers are running consul 1.3.1, clients are running different versions
1.3.1 and 1.9.5
It seems this is a precursor to the cluster falling over due to overloading the leader with too many write requests.
It does not seem to be running out of fds as i first suspected.
Below is the error:
2022/01/22 02:59:42 [ERR] consul.kvs: Apply failed: timed out enqueuing operation
I would be interested in your thoughts on how to mitigate.
servers are running as containers on Ubuntu 18.04.6 LTS
16 cores available but it appears the leader is only making use of 7.
I am still building tooling to measure resource useage as I do not own the hosts.
@blake I could be wrong but I wonder if this could be addressed with the changes with made in Consul 1.11. Thoughts?
are you aware of specific improvements to write performance I could reference in my pitch.