WAN Federation + Mesh Gateway Through multiple primary dc endpoints

The following consul multi-cluster configuration works for me using HAProxy Ingress controller “internal” configuration (not internet-facing), e.g. “AWS NLB internal” with listener on port 2443, distributing TCP connection to target group with HAProxy deployment configured with service in ClusterIP mode on tcp port 2443, distributing traffic to Mesh Gateway service on port tcp 2443 (I need HAProxy in-between Consul Mesh Gateways).

It works fine using e.g. AWS VPC Ppeering between Secondary dc (dc2) and Primary dc (dc1), but I want also to support other Secondaries dc using an internet-facing NLB for the same Primary dc (dc1), Secondaries are not on AWS so I cannot use internal connectivity like VPC Peering to keep private traffic from clusters.

Initially I thought that secondary could have used “primaryGateways” to connect to primary so a valid configuration could have been: all secondaries can connect to primary internal (private) or internet-facing (public) endpoints using “primaryGateways”, secondaries dc mesh gateway can be reached by primary dc as secondaries advertise its mesh gateway endpoint with wanAddress.static: “dcX.mydomain…” but defining wanAddress for primary statically make simultaneous access from internal and internet-facing impossible as it can define only one type of endpoint at a time (internal or internet-facing), so with the assumpion above I removed wanAddress stanza but doing so secondary cannot reach anymore services in primary as failover (defined with ServiceResolver).

Can someone explane detailts pf connection logic between mesh gateways of 2 consul datacenter ?
I there a configuration settings allowing “Primary dc Mesh Gateway” to be contacted on multiple endpoints simultaneously (NLB internal + NLB internet-facing) ? Something as Multiple wanAddres, or something else ?

The objective is to connect to the “**same Primary dc mesh gateway **”, multiple secondaries dc mesh gateways through internal or internet-facing simultaneously.

Primary dc (dc1) Mesh Gateway config

  meshGateway:
    enabled: true
    replicas: 2
    wanAddress:
      source: Static
      static: "dc1.mydomain.internal.private"
      port: 2443
    service:
      type: ClusterIP
      port: 2443

Secondary dc (dc2) Mesh Gateway config

  global:
    federation:
      enabled: true
      primaryDatacenter: dc1
      primaryGateways: ["dc1.mydomain.internal.private:2443"]
  meshGateway:
    enabled: true
    replicas: 2
    wanAddress:
      source: Static
      static: "dc2.mydomain.internal.private"
      port: 2443
    service:
      type: ClusterIP
      port: 2443

If I’m not missing something, lack of support for “multiple Primary mesh-gateway listening endpoints” seems to be a limitation for native consul mesh gateway too. E.g. Only one mesh gateway service definition on primary dc mesh gateway will supporto only one scenario: internal or internet-facing.

Hybrid (internal / internet-facing) use case is useful to manage multi-cluster federation between private (local subnets) and public (Internet) secondary connected consul clusters (e.g. intra or cross-cloud provider, on-prem) allowing to have safer private ones.

Probably only a limitation in k8s consul helm chart, making these scenarios not fully automated.

Not sure if allowing multiple independent “mesh gateway services” definisions in helm chart for Primary dc mesh gateway can fulfill this scenario.