OK, so after a rather bumpy ride trying to get everything to play nicely with Consul, I believe I’ve almost got everything working together. However, I seem to be missing the final part of the puzzle, how to access services using service discovery.
The setup is quite complex with services in different kubernetes clusters (each cluster has its own consul datacenter). We also host services inside the service mesh, outside of the mesh, and also outside of kubernetes completely.
The idea is for external services to use consul to be able to discover services inside the cluster/mesh and vice-versa.
I have managed to get Nginx ingress controllers working for mesh services, the API gateway and mesh gateway for cross cluster in mesh traffic. Traffic works well across clusters using the mesh gateway.
The first problem that I have is that, when I try to discover a service, Consul seems to give me a list of the pods where the application is deployed. Each of these pods is in a private subnet which is not reachable from outside the cluster. Ideally they should be accessing the service is a load balanced way. Generally the services are running as CLUSTER-IP with Nginx ingress control.
The second problem I have is that the external services are running on servers with consul agents that connect to the main datacenter. These consul agents are run using docker containers, but the hostname that they provide to consul is “host.docker.internal”. Consul sees the service running and can access it’s health endpoint, but no other services can connect as they don’t know about “host.docker.internal”
Essentially at this point, no external services can discover internal kubernetes services and no services at all can discover the external services. Anything within the service mesh works well (even across datacenters and clusters), but nothing outside the mesh can discover a way in to the mesh services, even though I have both ingress and api-gateways set up and working.
I’m at a complete loss, and really need some help. Either I have set up the topology of this completely wrong, or I’m missing something simple to bring this all together.
Any help, pointers or ideas would be gratefully accepted!