Mesh Gateways were designed to solve the exact connectivity challenges you described. Mesh Gateways are the only proxies which need to be exposed on the public Internet. All inter-DC service-to-service communication will flow across a single IP & port pair between gateways.
Gateways can be horizontally scaled to support higher network throughput, or high availability. See the Scaling Mesh Gateways thread for a recent conversation on this.
Consul 1.8 (currently in beta) further simplifies inter-DC communication by also forwarding the WAN federation traffic between Consul servers across the gateways. See the following links for the announcement of this feature, and instructions on deploying gateways in Kubernetes (other platforms are also supported).
- Announcing Consul 1.8
- Connecting Kubernetes Clusters with HashiCorp Consul WAN Federation
- Secure and Route Service Mesh Communication Across Kubernetes Clusters
Let me know if you have any additional questions. 