Mesh gateway in mixed k8s + VM datacenter

we are trying to set up a consul service mesh for mixed environment:VM + k8s. Everything is located in our infra - we are trying to keep all in one consul datacenter - without federation set up - so we’re using external servers for k8s.

It generally works - but there is a problem with connections from services outside of k8s to services inside the k8s. As services inside k8s are publishing internal IPs which are unreachable from outside the k8s cluster.
We are trying to solve it with using mesh gateway - set up in k8s and use it in remote mode by services outside of k8s cluster. But we can’t get it to work.

Would really appreciate some ideas on how to implement it.

on VM:

service {
  name = "web"
  port = 32768
  connect {
    sidecar_service {
      proxy {
        upstreams = [
            destination_name = "static-server2"
            local_bind_port = 1234
                mesh_gateway = {
                    mode = "remote"

envoy is running

consul connect envoy -sidecar-for web

in k8s:

consul-consul-mesh-gateway-6f459676b4-spj2w           1/1     Running

consul-consul-mesh-gateway       LoadBalancer,   443:30988/TCP   174m


If I understand correctly after the envoy on VM is started it should redirect connections for localhost:1234 to mesh-gateway which will redirect them further to the destination - static server in this case.

tcp   LISTEN 0      4096*     users:(("envoy",pid=5438,fd=25))

it seems to listen, but don’t send anything to the mesh-gateway .

What are we missing? (probably something dumb…)


curl: (56) Recv failure: Connection reset by peer

[2023-05-23 13:15:31.855][5455][debug][conn_handler] [source/server/] [C73] **new connection from**
[2023-05-23 13:15:35.133][5438][debug][main] [source/server/] flushing stats
[2023-05-23 13:15:36.857][5455][debug][pool] [source/common/conn_pool/] [C74] connect timeout
[2023-05-23 13:15:36.857][5455][debug][connection] [source/common/network/] [C74] closing data_to_write=0 type=1
[2023-05-23 13:15:36.857][5455][debug][connection] [source/common/network/] [C74] closing socket: 1
[2023-05-23 13:15:36.858][5455][debug][pool] [source/common/conn_pool/] [C74] client disconnected, failure reason: 
[2023-05-23 13:15:36.858][5455][debug][filter] [source/common/tcp_proxy/] [C73] connect timeout
[2023-05-23 13:15:36.858][5455][debug][filter] [source/common/tcp_proxy/] [C73] **Creating connection to cluster dd412229~static-server2.default.dc1.internal.ad9c9e3b-7324-9a17-a952-58cefa707a77.consul**

Hi @antonof.k,

Welcome to HashiCorp Forums!

Did you explore your pods’ options to have a routable IP address from your VM subnet? I think that will be the easiest option. The requirement would be similar to what is documented here: Single Consul Datacenter in Multiple Kubernetes Clusters - Kubernetes | Consul | HashiCorp Developer

If you want the services in the Consul Mesh to be accessed by external clients, you should use an Ingress Gateway or Consul API Gateway. This is specifically geared towards clients coming from outside the Mesh.


hei. thanks.
yup i’ve seen this.

the problem with that topology is this requirement -
This deployment topology requires that the Kubernetes clusters have a flat network for both pods and nodes so that pods or nodes from one cluster can connect to pods or nodes in another.

But pods in k8s and VMs are in different networks, and VMs can’t reach the pods.
That’s why we are trying to use mesh gateway - for traffic from VMs to pods.
Basically we can make the mesh gateway reachable by exposing it as a NodePort and proxy traffic inside. But at the current momment we can’t find a correct configuration to firect the raffic through it.

Also, we’ve tried to make wan federation work:

  name: consul
  datacenter: dc2
    enabled: true
      secretName: consul-federation
      secretKey: caCert
      secretName: consul-federation
      secretKey: caKey

  # Delete this acls section if ACLs are disabled.
    manageSystemACLs: true
      secretName: consul-federation
      secretKey: replicationToken

    enabled: true
    primaryDatacenter: dc1
    primaryGateways: [""]
  # Delete this gossipEncryption section if gossip encryption is disabled.
    secretName: consul-gossip
    secretKey: gossipEncryptionKey

  enabled: true
  enabled: true
    source: Static

k8s@k8s:~/consul$ k get all
NAME                                               READY   STATUS    RESTARTS   AGE
pod/consul-webhook-cert-manager-5dd6848777-clwl8   1/1     Running   0          118s
pod/consul-server-0                                1/1     Running   0          118s
pod/consul-connect-injector-7cd4c467c5-rz848       1/1     Running   0          118s
pod/consul-mesh-gateway-74cf79b554-jz8qf           1/1     Running   0          118s

NAME                              TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                                                            AGE
service/consul-server             ClusterIP      None             <none>         8501/TCP,8502/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP   119s
service/consul-dns                ClusterIP    <none>         53/TCP,53/UDP                                                                      119s
service/consul-ui                 ClusterIP    <none>         443/TCP                                                                            119s
service/consul-connect-injector   ClusterIP   <none>         443/TCP                                                                            119s
service/consul-mesh-gateway       LoadBalancer   443:31816/TCP                                                                      119s

NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/consul-webhook-cert-manager   1/1     1            1           118s
deployment.apps/consul-connect-injector       1/1     1            1           118s
deployment.apps/consul-mesh-gateway           1/1     1            1           118s

NAME                                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/consul-webhook-cert-manager-5dd6848777   1         1         1       118s
replicaset.apps/consul-connect-injector-7cd4c467c5       1         1         1       118s
replicaset.apps/consul-mesh-gateway-74cf79b554           1         1         1       118s

NAME                             READY   AGE
statefulset.apps/consul-server   1/1     118s

but the problem is, consul server on VM can’t reach server in k8s as it tries to reach it directly and not through the mesh gateway.

consul@consul-server-001:/etc/consul.d$ consul members -wan
Node                   Address          Status  Type    Build   Protocol  DC   Partition  Segment
consul-server-0.dc2  alive   server  1.15.1  2         dc2  default    <all>
consul-server-001.dc1    alive   server  1.14.5  2         dc1  default    <all>

wan: memberlist: Suspect consul-server-0.dc2 has failed, no acks received
wan: memberlist: Failed to send gossip to read tcp> read>
wan: memberlist: Failed to send gossip to read tcp> read>
wan: memberlist: Failed to send gossip to read tcp> read>
wan: memberlist: Failed to send gossip to read tcp> read>
wan: memberlist: Failed to send UDP compound ping and suspect message to read tcp 10.0>
[ERROR] agent.proxycfg: Failed to handle update from watch: kind=mesh-gateway proxy=gateway-primary service_id=gateway-primary id=mesh-gateway:dc2 error="error filling agent cache: Remote DC has no server currently reachable"

not sure how to solve it.
it seems like mesh gateway from Vm can connect to mesh-gateway:dc2 but then there is no proxying to the k8s consul server :face_with_raised_eyebrow:

Just for the record.

After a lot of testing two things made the difference - changing consul version on the primary DC - it was lower then on the secondary.
And fixing the policy on the anonymous token, for some reason it wasn’t applied there.