Unable to Route To Imported service via Mesh Gateway peering

Consul
1

Hi All,

I have this setup (same consul on both clusters)
Consul 1.16.X (also tried latest 1.17.x) with API Gateway, MeshGateway enabled.

1 local cluster running minikube on my MAC (using NodePort service for MeshGateway) with local-minikube as datacenter name
1 EKS cluster running on aws (either using NodePort or LB service for MeshGateway) with cloud-eks as datacenter name

I was able to peer local cluster and cloud cluster via MeshGateway (both active)
I deployed my service locally and was able to export my service (called my-service) into cloud EKS cluster.

Here’s what I deployed to EKS cloud cluster without any issue

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceResolver
metadata:
name: my-service
spec:
redirect:
peer: local-minikube
service: my-service

apiVersion: consul.hashicorp.com/v1alpha1
kind: MeshService
metadata:
name: my-mesh-service
spec:
name: my-service
peer: local-minikube

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: local-dev-route-my-service
spec:
parentRefs:

  • name: internal-api-gateway
    namespace: consul
    rules:
  • matches:
    • path:
      type: PathPrefix
      value: /api/my-service/v1
      backendRefs:
    • group: consul.hashicorp.com
      kind: MeshService
      name: my-mesh-service
      port: 3450

The issue is that when I hit: internal-api-gateway/api/my-service/v1 I always get this error (httproute is perfectly fine, no issue)

“upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: 111”

I turned on the trace for MeshGateway but didn’t see any error .

Any help will be greatly appreciated.
Thanks!