Help with communication across multiple clusters

I am attempting to connect two clusters using service mesh, and appear to be having some issues. It’s not clear to me exactly what the approach is, and where I’m going wrong.

The first cluster works well and as expected. I then attempted to connect a second cluster following the instructions given here:

Again, this worked as expected with the services showing up in the consul UI. The problem was that none of the cross cluster services are able to communicate with each other using the service mesh.

I then started to look at the following for creating a mesh gateway in each of the two clusters: HashiCorp Learn

The configuration is completely different to the first example, to the point where I can’t seem to get anything to now work as expected. In the mesh gateways, there is a new datacentre with a server, in the first example there was only a connection to an external server.

Importantly, I need to implement acls and gossip encryption, which worked fine in the first example, but not at all with mesh gateways.

Does anyone have any guidance?

For reference, here is my current configuration for the second cluster (which doesn’t work):

    name: consul-test
    datacenter: dc2
      enabled: true
      enabled: true
        secretName: consul-federation
        secretKey: caCert
        secretName: consul-federation
        secretKey: caKey
      enabled: true
      primaryDatacenter: dc1
    ## For production turn on ACLs and gossipEncryption:
#    acls:
#      manageSystemACLs: true
#      bootstrapToken:
#        secretName: consul-bootstrap-acl-token
#        secretKey: token
#      replicationToken:
#        secretName: consul-acl-replication-acl-token
#        secretKey: token
      secretName: consul-gossip-encryption-key
      secretKey: key
    replicas: 1
      - type: secret
        name: consul-federation
          - key: serverConfigJSON
            path: config.json
        load: true
    exposeGossipAndRPCPorts: true # exposes the server gossip and RPC ports as hostPorts
      # Configures the server gossip port
        # Note that this needs to be different than 8301, to avoid conflicting with the client gossip hostPort
        port: 9301
      runAsNonRoot: false
      runAsUser: 0
    enabled: true
    exposeGossipPorts: true # exposes client gossip ports as hostPorts
    hostNetwork: true
    dnsPolicy: ClusterFirstWithHostNet
    join: ["provider=k8s host_network=true kubeconfig=/consul/userconfig/consul-kubeconfig/kubeconfig namespace=\"consul\" label_selector=\"app=consul,component=server\""]
      - type: secret
        name: consul-kubeconfig
        load: false
    # This method will inject the sidecar container into Pods:
    enabled: true
    # But not by default, only do this for Pods that have the explicit annotation:
    # "true"
    default: false
    enabled: true
    enabled: true

I have found the page: Federation Between Kubernetes Clusters | Consul by HashiCorp which has helped to resolve the problems.

In particular, in case anyone else has issues, the part relating the configuration of the second cluster, ACLs and federation.