I’ve been struggling for several days with trying to get “out of cluster” nodes working with a helm-deployed version of Consul on k8s. Judging by a couple of other related topics on the forum, I’m not alone…
I’m deploying a GKE cluster and three GCE VMs using Terraform. You can find the code here: https://github.com/datawire/pro-ref-arch/tree/master/cloud-infrastructure/google-cloud-platform
I’m installing 0.8.1 of the consul-helm chart (with Consul v1.5.2) onto the Terraformed GKE cluster: https://github.com/hashicorp/consul-helm
I’ve tried following along with this tutorial, https://medium.com/hashicorp-engineering/introduction-to-hashicorp-consul-connect-with-kubernetes-d7393f798e9d and although this was helpful, I don’t think the instructions works with the current release of the helm chart?
I’m not using a “fully connected” network (as per the doc warning: https://www.consul.io/docs/platform/k8s/out-of-cluster-nodes.html), but even when switching to
host_network=true I still can’t get the out-of-cluster node to join the Consul cluster.
I have Consul running successfully on an out-of-cluster VM within the same VPC/network as the k8s cluster, and I can see that Consul successfully connects to the k8s API, as the logs show a list of Consul server node IPs was received. However, I also see an i/o failure when Consul tries to connect to the k8s node via port 8301. I’ve ssh’ed into the k8s node instance and used socat to see if anything is listening on that port on the node, and I get no response (I can, however, use socat to see the Consul agent listening to port 8500 on the same node)
Any guidance, or ideally a HashiCorp “approved” walkthrough and sample code repo, would be very much appreciated The task appears simple at first glance, but the devil very much appears to be in the details!