Connecting to existing external consul cluster from new kubernetes datacenter

Consul
1

Running into an issue where My existing cluster cannot see inside the kubernetes cluster I have. I get a 500 error and in my logs it just times out when the GET request is being performed.
I have attempted to open ports on the ingress and the datacenter shows up in my list from the webui on the external cluster. If i open the UI from my kubernetes cluster it is able to see services listed in external clusters. So the requests are not making it back in.

I notice it is trying to connect to the internal pod IP so perhaps that is the issue?

Using nginx ingress.

Any help or direction on adding a cluster in kubernetes to an existing external cluster would be great.

2

Hi Brandon, if you’ve deployed your servers inside of Kubernetes then they use their Pod IP as their WAN address. The WAN address is what the external cluster will try to use and so it makes sense that this isn’t working because obviously the Pod IP is not routable from outside Kubernetes.

Unfortunately you aren’t going to be able to use an NGINX ingress for this. Each Consul server needs to be directly routable from the other Consul servers. This means you need to make each node routable with a public IP and then use a hostPort for the WAN server ports (8300 and 8302). Then you need to set the -advertise-wan flag to the node IP for each Consul server. Or you need to create a LoadBalancer in front of each Consul server and then set the -advertise-ip to it’s IP.

Unfortunately there’s no way right now in the Helm chart to do any of this. You’re going to have to manually edit the chart yourself.

I’m actually working on this as we speak and will update this ticket with my progress: https://github.com/hashicorp/consul-helm/issues/28.

1 Like
3

Is there an example you can give perhaps or someone else doing it manually that worked?
Would there need to be a LoadBalancer for each server pod or the service as a whole?

4

There’s this PR for using a load balancer: https://github.com/hashicorp/consul-helm/pull/27. You need to have one LB per consul server pod because each server needs its own address for gossip and rpc purposes.

If you’re able to make the node IPs routable then I’ve put up this PR: https://github.com/hashicorp/consul-helm/pull/332. This will advertise the node IPs and make the WAN ports hostPorts.

5

Hello @lkysow - I’ve reached this thread as I’m investigating a similar setup. I’m sort of new to consul - so pls be gentle :slight_smile: I figured the PODIP won’t work so here’s what I did
(I’m working with AKS)

  1. Created a static IP in Azure
  2. Added advertise-wan to static IP and also enabled translate_wan_addrs
  3. Created a new Service with type: LoadBalancer with the static IP using the same selector and exposes TCP 8300, 8302 (and other ports as well). Trouble is that this only allows me to open TCP ports (or UDP ports - can’t mix the two).
  4. At this point I’m stuck - I need to open UDP as well acc to docs - but due to restrtiction above, I will need a separate Service which will be a different IP and that’ll not work since on the other end the expectation is to have a single endpoint with both TCP & UDP

I think it will help to state what I’m trying to achieve

  1. DC1 - K8s with Consul running in cluster and services with service sync catalog. This is the primary DC.
  2. DC2 - On prem env with Consul running locally.
  3. Mesh gateway routing between on prem → AKS for services

Is this a supported configuration? The docs seem to indicate that this one of the core Consul use cases, but ATM, I’m lost.

Some guidance will be very helpful.

6

Hi raghur,
We’re currently working on making this a lot easier. In our next release, we will support multiple datacenters federated through mesh gateways.

For now, each Consul server needs to be individually addressable by the other Consul servers so you can’t use a single IP address/LB for all of them unless you’re only running one server. If UDP isn’t possible you’ll get errors in the logs but consul will actually fall back to TCP and everything will still work. Thus you’ll need either one load balancer per consul server or, expose the consul servers on node ips.

7

Hi, we would like to extend our Consul Service Mesh into Kubernets Consul. Are there any tutorials available on how to connect Kubernetes Consul servers with Consul clusters outside of Kubernetes ?

8

There is the 1.8 product announcement running at the moment. I think that’s what @lkysow is talking about and you should keep up with the release notes and upcoming documentation and learning guides. :slight_smile:

9

Thanks for the info. I found the 1.8-beta download. Do you know if there is Helm chart available that enables external cluster integration ?

10

Yes! The full docs are here: https://www.consul.io/docs/k8s/installation/multi-cluster/overview

We’re just in the process of building the consul:1.8.0-beta1 Docker image so you’ll need to wait for that to be uploaded.

1 Like