Our Consul 1.7.1 is running fine on an older version of K8s (1.16.2).
Trying to test on K8s 1.18.3 cluster and having problems reaching the port 8500 on the hostIP:8500, the way a client agent would normally be reached on a worker node.
Is this expected to work on K8s 1.18.3? Is anyone doing this?
Maybe I need a later version of Consul?
Where are you running Kubernetes? This might be an issue with where you’re running Kubernetes that doesn’t support hostPorts. The next release of consul-helm will support running in hostNetwork mode which might help.
@lkysow Thanks for the reply. We’re running Consul 1.7.1 on K8s 1.18.3, with RHEL8.2 host images. This is a private cloud Openstack cluster. I wanted to upgrade to Consul 1.8.0 before responding. Running Consul 1.8.0, RHEL8.2 and K8s 1.18.3 now.
What we’re observing is that the “client agent” instances bind to the hostIP:8500 and everything comes up fine (three server agents and three client agents). Everything works just fine (like our existing production RHEL7.6/K8s 1.16) when the cluster is first deployed.
But with RHEL8.2 hosts, if a Consul client agent daemonset pod is deleted, the replacement pod always fails to bind to the hostIP:8500. There are no errors in the client agent logs, it all looks just fine. However, the replacement client agent does not respond to any HTTP queries at hostIP:8500.
Again noting that every thing does work just fine on first bring up. It is only when a daemonset pod is deleted and replaced, do we see the issue with the hostIP:8500 binding.
If we revert back to RHEL7.6 with K8s 1.16, the Consul client agent pods work just fine (as it has for years) and can be deleted and restarted as deployment conditions require.
Hmm, I’ve never seen that before. I’m not sure it’s consul-specific. Have you seen any similar issues with other Kubernetes users on GitHub?
This appears to be unrelated to Consul itself. Appears to be an issue with kube-proxy and the host machine kernel’s version of iptables not working harmoniously together. We see a stale iptables entry for the hostPort/podIpAddr when the pod is deleted. And this seems to interfere when a new client agent pod is started using the same hostPort/newPodIpAddr. Probably not many users trying this with K8s 1.18.3 on RHEL8.2 yet.