I have service mesh working in Consul inside kubernetes with multiple clusters, services working both inside and outside of the service mesh. All of this is going according to plan.
I have also been able to set up the API gateway on the master consul cluster, again everything works as exepected.
All services are deployed (including consul) are deployed using helm templates. The problem comes when I try to deploy services with a HTTPRoute on the second cluster. The second cluster is only running consul as an agent. It’s not clear to me what the configuration should be with regard to setting up the gateway on the second and subsequent clusters.
For the first cluster, this is the consul API gateway config:
If I try to install a similar route on the second server I get: error: unable to recognize "test-api.yaml": no matches for kind "HTTPRoute" in version "gateway.networking.k8s.io/v1alpha2", which suggests that I need to either install the route on the first cluster and connect it to the service on the second, or install the api gateway to some extent on the second server.
I’ve tried varying different things, but always end up with an error of some description.
Hi! I haven’t personally done any testing with consul multi-clusters, so I’ll need to look into that for you, but the error you’re getting sounds like the gateway crds are missing on your second cluster.
If you run kubectl get crds while connected to your second cluster do you see httproutes.gateway.networking.k8s.io in the list?
After talking a bit with the team, it doesn’t sound like multi-cluster is currently supported with the out of the box configuration. It might work if you create a dummy service with the same name in the same cluster that the gateway lives in, so that the gateway will resolve the consul service, but it hasn’t been tested. But, in the meantime, I can raise this as a requested feature with the team.
This would be quite an important feature, ideally the option of giving a single api gateway to clients without them needing to know where the service is hosted.
Could you explain how I could implement a dummy service, at least to give it a try? I’m guessing this would have to proxy a request on to the the actual service in some way, just not sure how that would be implemented.
Most importantly, thanks for looking into it and getting back to me so quickly.
I’m currently spinning up my own set of clusters to test it out, but I believe what they were suggesting was to create test-service service on your original cluster without the route just to trigger the consul reconciliation.
I’ve tried setting up an api gateway on the second cluster, but it’s having problems with the https connection, I suspect it’s to do with the the server certificate, I copied the consul-server-cert from the main cluster and placed it in the second cluster where the agents are running.
Do you have an documentation on how you would set up an API gateway in the second cluster where everything is deployed using helm files?