Connecting a distributed service across a WAN federated Consul Cluster

Hi there,

I have two k8s consul clusters, WAN federated - dc1 and dc2 via mesh gateway.
I’ve been able to verify that they work as expected by setting up the static-client and static-server services and getting a “hello” response.

However, i want to use Consul for obviously more complicated real-world problems.

I have a distributed ethereum service, which can have peers find each other on port 4001.

What i want, is to have pods running in dc1 connect to peers on dc2.

The ultimate aim is to have redundancy across two datacenters - syncing data across all - in the hopes that should pods in dc1 fail - i have still got pods in dc2 with the latest data in it.

Following the documentation i believe that i should expose the services on both dc’s via setting up the annotation for the upstreams i.e,
consul.hashicorp.com/connect-service-upstreams: ‘ethereum-service-on-dc2:4001:dc1’
consul.hashicorp.com/connect-service-upstreams: ‘ethereum-service-on-dc1:4001:dc2’

However when this is deployed im told - “Address already in use” - which is because the sidecar proxy is trying to set the upstream as “127.0.0.1:4001” - but this is in use - as the service is running on this port to discover peers on its own dc.

So the question is - how do you expose services across datacenters on ports which are in use by the same service locally??

This isn’t what happens in the static-client/static-server example - in the example, exposing a service called static-server on dc2 - on a port 8080 is done with just the connect-inject transparent proxy, and on the client, it doesnt expose anything and just sets the upstream to the static-server service which was exposed.

I would have assumed i could do what i am attempting, for services which are distributed, and have peers, connecting via mesh gateway to other dc’s would be great.

Any explanation much appreciated, ive looked through the documentation but cannot see anything that is relevant.
Thanks

Hi, are you able to achieve this ? I am trying the same but no luck!

If you figured it out, kindly suggest the solution.

Thanks!