Recently I’ve upgraded the cluster from 1.0.2 to 1.2.4 and I’ve observed errors on clients saying “memberlist: Was able to connect to xxxxx but other probes failed, network may be misconfigured”.
I’ve found a part of the code that is responsible, and I’ve found some Github issues saying these might be networking issues. The thing is that I haven’t observed this kind of behavior before (this is not the first upgrade) and network wasn’t changed, meaning UDP and TCP traffic is allowed on ports 8301-8302 (and I see UDP traffic going outbound and inbound). Also worth mentioning is that Consul servers do not exhibit this kind of behavior, only clients.
Consul agents are running on Docker on AWS EC2 instances but they are using host network.
Any help is appreciated.