Help setting up kubernetes multi-cluster API gateway with service mesh

Consul
1

I have service mesh working in Consul inside kubernetes with multiple clusters, services working both inside and outside of the service mesh. All of this is going according to plan.

I have also been able to set up the API gateway on the master consul cluster, again everything works as exepected.

All services are deployed (including consul) are deployed using helm templates. The problem comes when I try to deploy services with a HTTPRoute on the second cluster. The second cluster is only running consul as an agent. It’s not clear to me what the configuration should be with regard to setting up the gateway on the second and subsequent clusters.

For the first cluster, this is the consul API gateway config:

apiGateway:
  enabled: true
  image: "hashicorp/consul-api-gateway:0.3.0"
  logLevel: debug
  managedGatewayClass:
    enabled: true
    serviceType: LoadBalancer
    useHostPorts: true
    copyAnnotations:
      service:
        annotations: |
          - external-dns.alpha.kubernetes.io/hostname
          - external-dns.alpha.kubernetes.io/ttl
          - oci.oraclecloud.com/oci-network-security-groups
          - service.beta.kubernetes.io/oci-load-balancer-internal
          - service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode
          - service.beta.kubernetes.io/oci-load-balancer-subnet1

a test service that works as expected in this cluster:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: test-service-route
spec:
  parentRefs:
    - name: api-gateway
      namespace: consul
  hostnames:
    - "test-service.my.domain"

  rules:
    - backendRefs:
        - name: test-service
          port: 8080

If I try to install a similar route on the second server I get:
error: unable to recognize "test-api.yaml": no matches for kind "HTTPRoute" in version "gateway.networking.k8s.io/v1alpha2", which suggests that I need to either install the route on the first cluster and connect it to the service on the second, or install the api gateway to some extent on the second server.

I’ve tried varying different things, but always end up with an error of some description.

2

Hi! I haven’t personally done any testing with consul multi-clusters, so I’ll need to look into that for you, but the error you’re getting sounds like the gateway crds are missing on your second cluster.

If you run kubectl get crds while connected to your second cluster do you see httproutes.gateway.networking.k8s.io in the list?

3

Well spotted, you are correct. I’ve now installed the CRD’s and will re-test.

4

OK, now there are no errors, but it doesn’t do anything, I suspect as it’s unable to find the “Parent Ref” on the first cluster:

Name:         test-service
Namespace:    test
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1alpha2
Kind:         HTTPRoute
Metadata:
  Creation Timestamp:  2022-06-22T13:46:13Z
  Generation:          2
  Managed Fields:
    API Version:  gateway.networking.k8s.io/v1alpha2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:labels:
          .:
          f:argocd.argoproj.io/instance:
      f:spec:
        .:
        f:hostnames:
        f:parentRefs:
        f:rules:
    Manager:         argocd-application-controller
    Operation:       Update
    Time:            2022-06-22T13:46:13Z
  Resource Version:  501772
  UID:               bfced674-28d5-4672-8776-4c959f6a873b
Spec:
  Hostnames:
    test.my.domain
  Parent Refs:
    Group:      gateway.networking.k8s.io
    Kind:       Gateway
    Name:       api-gateway
    Namespace:  consul
  Rules:
    Backend Refs:
      Group:
      Kind:       Service
      Name:       test-service
      Namespace:  test
      Port:       8084
      Weight:     1
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /
Events:         <none>

If anyone has any thoughts on how to connect everything up, it would be appreciated.

5

After talking a bit with the team, it doesn’t sound like multi-cluster is currently supported with the out of the box configuration. It might work if you create a dummy service with the same name in the same cluster that the gateway lives in, so that the gateway will resolve the consul service, but it hasn’t been tested. But, in the meantime, I can raise this as a requested feature with the team.

6

This would be quite an important feature, ideally the option of giving a single api gateway to clients without them needing to know where the service is hosted.

Could you explain how I could implement a dummy service, at least to give it a try? I’m guessing this would have to proxy a request on to the the actual service in some way, just not sure how that would be implemented.

Most importantly, thanks for looking into it and getting back to me so quickly.

1 Like
7

I’m currently spinning up my own set of clusters to test it out, but I believe what they were suggesting was to create test-service service on your original cluster without the route just to trigger the consul reconciliation.

8

OK, thanks, and please keep me posted as to any ideas or feedback that you have.

9

@sarahalsmiller , did you get any further with this?

I’ve tried setting up an api gateway on the second cluster, but it’s having problems with the https connection, I suspect it’s to do with the the server certificate, I copied the consul-server-cert from the main cluster and placed it in the second cluster where the agents are running.

Do you have an documentation on how you would set up an API gateway in the second cluster where everything is deployed using helm files?