Hi All,
We have deployed our setup in Production with same resource configuration on Kubernetes environment.
On Lab and other setup, we are not observing any issues but on production with the same resource limit we are observing below errors for all most all consul sidecar container.
Due to this health check is failing for sidecar container.
2022-10-11T01:47:41.202Z [ERROR] agent.client.memberlist.lan: memberlist: Failed fallback ping: read tcp 10.x.x.x:47146->10.y.y.y:8301: i/o timeout
Consul server version: 1.8.3
Consul client version: 1.8.4
As per the documentation, there is a suspect of CPU and memory during this I/O timeout errors.
Also we have observed this issue on our setup.
But here on Production environment, there is no CPU or memory issue and still we are observing same error and clients are leaving the cluster and again joining back.