I am new to Consul and, typically, bitten off more than I should!
I have a Consul service mesh installation across AWS and GCP Kubernetes clusters with each cluster being a separate datacenter.
I can see the services across the mesh from each datacenter (consul catalog services) and I have confirmed that DNS routing is working.
The application is deployed using a StatefulSet that creates 3 pods in each cluster (6 in total) and I have added ‘consul.hashicorp.com/connect-inject’: ‘true’. I can see that each of the pods has the appropriate Consul sidecars.
Each pod connects to each of the other pods in a peer-to-peer network. In order to form the network I need to connect each of the pods in dc2 to each of the pods in dc1.
The problem is that each pod is exposed in Consul as an instance of a service not as a separate service. Consequently I cannot use a reference such as <service_name>.service.dc1.consul from a pod in dc2.
I have also tried ‘consul.hashicorp.com/connect-service-upstreams’ providing a local port and then changed the pod to connect to that port. The result is that the application is sent the FQDN of the local pod in dc1 (i.e. .default.svc.cluster.local) which, naturally, will not resolve in the dc2 cluster.
How do I connect from a pod in dc2 to a pod in dc1?