@lkysow Thanks for the reply. We’re running Consul 1.7.1 on K8s 1.18.3, with RHEL8.2 host images. This is a private cloud Openstack cluster. I wanted to upgrade to Consul 1.8.0 before responding. Running Consul 1.8.0, RHEL8.2 and K8s 1.18.3 now.
What we’re observing is that the “client agent” instances bind to the hostIP:8500 and everything comes up fine (three server agents and three client agents). Everything works just fine (like our existing production RHEL7.6/K8s 1.16) when the cluster is first deployed.
But with RHEL8.2 hosts, if a Consul client agent daemonset pod is deleted, the replacement pod always fails to bind to the hostIP:8500. There are no errors in the client agent logs, it all looks just fine. However, the replacement client agent does not respond to any HTTP queries at hostIP:8500.
Again noting that every thing does work just fine on first bring up. It is only when a daemonset pod is deleted and replaced, do we see the issue with the hostIP:8500 binding.
If we revert back to RHEL7.6 with K8s 1.16, the Consul client agent pods work just fine (as it has for years) and can be deleted and restarted as deployment conditions require.