# Error Wan federation between VMs and Kubernetes

**URL:** <https://discuss.hashicorp.com/t/error-wan-federation-between-vms-and-kubernetes/11918>\
**Category:** Consul\
**Tags:** k8s\
**Created:** [July 24, 2020, 2:23am UTC](https://discuss.hashicorp.com/t/error-wan-federation-between-vms-and-kubernetes/11918 "2020-07-24T02:23:32Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![YaoTu](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/yaotu/32/9749_2.png) [@YaoTu](https://discuss.hashicorp.com/u/YaoTu)\
**Post date:** [July 24, 2020, 2:23am UTC](https://discuss.hashicorp.com/t/error-wan-federation-between-vms-and-kubernetes/11918/1 "2020-07-24T02:23:32Z")

</div>

Hello!

I was following this guide  
[WAN Federation Through Mesh Gateways - VMs and Kubernetes | Consul | HashiCorp Developer](https://www.consul.io/docs/k8s/installation/multi-cluster/vms-and-kubernetes)

But the federation failed.  
Log from the consul server in Kubernetes:

> 2020-07-23T13:39:00.306Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: yao-dc1-server-0.dc1 10.1.168.149  
> 7/23/2020 9:39:00 AM 2020-07-23T13:39:00.306Z [INFO] agent.server: Handled event for server in area: event=member-join server=yao-dc1-server-0.dc1 area=wan  
> 7/23/2020 9:39:00 AM 2020-07-23T13:39:00.401Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.149:8302: EOF  
> 7/23/2020 9:39:00 AM 2020-07-23T13:39:00.799Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.149:8300 datacenter=dc1 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 7/23/2020 9:39:00 AM 2020-07-23T13:39:00.902Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.149:8302: EOF  
> 7/23/2020 9:39:01 AM 2020-07-23T13:39:01.743Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.149:8300 datacenter=dc1 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 7/23/2020 9:39:01 AM 2020-07-23T13:39:01.901Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.149:8302: EOF  
> 7/23/2020 9:39:02 AM 2020-07-23T13:39:02.691Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.149:8300 datacenter=dc1 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 7/23/2020 9:39:04 AM 2020-07-23T13:39:04.900Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ack: EOF from=10.42.1.112:55954  
> 7/23/2020 9:39:05 AM 2020-07-23T13:39:05.133Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.149:8300 datacenter=dc1 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 7/23/2020 9:39:09 AM 2020-07-23T13:39:09.369Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.149:8300 datacenter=dc1 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 7/23/2020 9:39:11 AM 2020-07-23T13:39:11.669Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.149:8300 datacenter=dc1 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 7/23/2020 9:39:12 AM 2020-07-23T13:39:12.998Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.149:8300 datacenter=dc1 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 7/23/2020 9:39:13 AM 2020-07-23T13:39:13.399Z [INFO] agent.server.memberlist.wan: memberlist: Suspect yao-dc1-server-0.dc1 has failed, no acks received  
> 7/23/2020 9:39:13 AM 2020-07-23T13:39:13.501Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: EOF  
> 7/23/2020 9:39:44 AM 2020-07-23T13:39:44.703Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.149:8300 datacenter=dc1 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 7/23/2020 9:39:44 AM 2020-07-23T13:39:44.808Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ack: EOF from=10.42.1.112:55954  
> 7/23/2020 9:39:44 AM 2020-07-23T13:39:44.808Z [WARN] agent.server.memberlist.wan: memberlist: Refuting a suspect message (from: yao-dc1-server-0.dc1)  
> 7/23/2020 9:39:44 AM 2020-07-23T13:39:44.902Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.149:8302: EOF  
> 7/23/2020 9:39:58 AM 2020-07-23T13:39:58.401Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: EOF  
> 7/23/2020 9:39:59 AM 2020-07-23T13:39:59.013Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.149:8300 datacenter=dc1 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 7/23/2020 9:40:03 AM 2020-07-23T13:40:03.401Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: EOF  
> 7/23/2020 9:40:34 AM 2020-07-23T13:40:34.808Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ack: EOF from=10.42.1.112:55954  
> 7/23/2020 9:40:35 AM 2020-07-23T13:40:35.638Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.149:8300 datacenter=dc1 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 7/23/2020 9:40:36 AM 2020-07-23T13:40:35.999Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.149:8300 datacenter=dc1 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 7/23/2020 9:40:39 AM 2020-07-23T13:40:39.614Z [ERROR] agent.server.memberlist.wan: memberlist: Push/Pull with yao-dc1-server-0.dc1 failed: EOF  
> 7/23/2020 9:40:40 AM 2020-07-23T13:40:40.630Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=dc1  
> 7/23/2020 9:40:43 AM 2020-07-23T13:40:43.401Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: EOF  
> 7/23/2020 9:40:47 AM 2020-07-23T13:40:47.425Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=dc1  
> 7/23/2020 9:40:47 AM 2020-07-23T13:40:47.750Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=dc1  
> 7/23/2020 9:40:48 AM 2020-07-23T13:40:48.499Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: EOF  
> 7/23/2020 9:40:49 AM 2020-07-23T13:40:49.726Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=dc1  
> 7/23/2020 9:40:50 AM 2020-07-23T13:40:50.297Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=dc1  
> 7/23/2020 9:40:51 AM 2020-07-23T13:40:51.602Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=dc1

Netstat of vm consul server:

> Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name  
> tcp 0 0 127.0.0.1:8600 0.0.0.0:\* LISTEN 999 37945 3462/consul  
> tcp 0 0 10.1.168.149:8443 0.0.0.0:\* LISTEN 1000 37288 3497/envoy  
> tcp 0 0 127.0.0.1:8443 0.0.0.0:\* LISTEN 1000 37281 3497/envoy  
> tcp 0 0 127.0.0.1:19005 0.0.0.0:\* LISTEN 1000 37269 3497/envoy  
> tcp 0 0 127.0.0.1:8500 0.0.0.0:\* LISTEN 999 37947 3462/consul  
> tcp 0 0 127.0.0.1:8501 0.0.0.0:\* LISTEN 999 37948 3462/consul  
> tcp 0 0 127.0.0.1:8502 0.0.0.0:\* LISTEN 999 37951 3462/consul  
> tcp 0 0 0.0.0.0:22 0.0.0.0:\* LISTEN 0 16412 1236/sshd  
> tcp6 0 0 :::8300 :::\* LISTEN 999 37186 3462/consul  
> tcp6 0 0 :::8301 :::\* LISTEN 999 37189 3462/consul  
> tcp6 0 0 :::8302 :::\* LISTEN 999 37187 3462/consul  
> tcp6 0 0 :::22 :::\* LISTEN 0 16421 1236/sshd  
> udp 0 0 0.0.0.0:68 0.0.0.0:\* 0 12064 850/dhclient  
> udp 0 0 127.0.0.1:8600 0.0.0.0:\* 999 37944 3462/consul  
> udp6 0 0 :::8301 :::\* 999 37190 3462/consul  
> udp6 0 0 :::8302 :::\* 999 37188 3462/consul

I can nc to port 8300 and port 8302 on the vm consul server from Kubernetes consul server.

Log from vm consul server:

> 2020-07-23T13:39:01.381Z [INFO] agent: Synced service: service=gateway-secondary  
> 2020-07-23T13:39:01.394Z [INFO] agent: Synced service: service=gateway-secondary  
> 2020-07-23T13:39:01.464Z [INFO] agent.server: federation state anti-entropy synced  
> 2020-07-23T13:39:03.864Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-server-0.dc2 has failed, no acks received  
> 2020-07-23T13:39:04.161Z [INFO] agent.server.serf.wan: serf: attempting reconnect to consul-server-0.dc2 10.42.1.114:8302  
> 2020-07-23T13:39:04.262Z [WARN] agent.server.memberlist.wan: memberlist: Refuting a suspect message (from: yao-dc1-server-0.dc1)  
> 2020-07-23T13:39:04.262Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc2 10.42.1.114  
> 2020-07-23T13:39:04.262Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc2 area=wan  
> 2020-07-23T13:39:06.008Z [INFO] agent: Synced check: check=service:gateway-secondary  
> 2020-07-23T13:39:06.051Z [INFO] agent.server: federation state anti-entropy synced  
> 2020-07-23T13:39:48.864Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-server-0.dc2 has failed, no acks received  
> 2020-07-23T13:40:18.865Z [INFO] agent.server.memberlist.wan: memberlist: Marking consul-server-0.dc2 as failed, suspect timeout reached (0 peer confirmations)  
> 2020-07-23T13:40:18.865Z [INFO] agent.server.serf.wan: serf: EventMemberFailed: consul-server-0.dc2 10.42.1.114  
> 2020-07-23T13:40:18.865Z [INFO] agent.server: Handled event for server in area: event=member-failed server=consul-server-0.dc2 area=wan  
> 2020-07-23T13:40:28.865Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-server-0.dc2 has failed, no acks received  
> 2020-07-23T13:40:34.263Z [INFO] agent.server.serf.wan: serf: attempting reconnect to consul-server-0.dc2 10.42.1.114:8302  
> 2020-07-23T13:40:34.359Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc2 10.42.1.114  
> 2020-07-23T13:40:34.359Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc2 area=wan  
> 2020-07-23T13:41:18.865Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-server-0.dc2 has failed, no acks received  
> 2020-07-23T13:41:48.865Z [INFO] agent.server.memberlist.wan: memberlist: Marking consul-server-0.dc2 as failed, suspect timeout reached (0 peer confirmations)  
> 2020-07-23T13:41:48.865Z [INFO] agent.server.serf.wan: serf: EventMemberFailed: consul-server-0.dc2 10.42.1.114  
> 2020-07-23T13:41:48.865Z [INFO] agent.server: Handled event for server in area: event=member-failed server=consul-server-0.dc2 area=wan  
> 2020-07-23T13:42:03.864Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-server-0.dc2 has failed, no acks received  
> 2020-07-23T13:42:04.359Z [INFO] agent.server.serf.wan: serf: attempting reconnect to consul-server-0.dc2 10.42.1.114:8302  
> 2020-07-23T13:42:04.459Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc2 10.42.1.114  
> 2020-07-23T13:42:04.460Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc2 area=wan  
> 2020-07-23T13:42:53.864Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-server-0.dc2 has failed, no acks received  
> 2020-07-23T13:43:23.864Z [INFO] agent.server.memberlist.wan: memberlist: Marking consul-server-0.dc2 as failed, suspect timeout reached (0 peer confirmations)

Log from vm envoy mesh gateway:

> ==\> Registered service: gateway-secondary  
> [2020-07-23 13:39:01.423][3497][info][main] [external/envoy/source/server/server.cc:251] initializing epoch 0 (hot restart version=disabled)  
> [2020-07-23 13:39:01.423][3497][info][main] [external/envoy/source/server/server.cc:253] statically linked extensions:  
> [2020-07-23 13:39:01.423][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.thrift\_proxy.transports: auto, framed, header, unframed  
> [2020-07-23 13:39:01.423][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.stats\_sinks: envoy.dog\_statsd, envoy.metrics\_service, envoy.stat\_sinks.hystrix, envoy.statsd  
> [2020-07-23 13:39:01.423][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.opencensus, envoy.tracers.xray, envoy.zipkin  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.udp\_listeners: raw\_udp\_listener  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.access\_loggers: envoy.file\_access\_log, envoy.http\_grpc\_access\_log, envoy.tcp\_grpc\_access\_log  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.thrift\_proxy.filters: envoy.filters.thrift.rate\_limit, envoy.filters.thrift.router  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.retry\_priorities: envoy.retry\_priorities.previous\_priorities  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.retry\_host\_predicates: envoy.retry\_host\_predicates.omit\_canary\_hosts, envoy.retry\_host\_predicates.previous\_hosts  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.dubbo\_proxy.serializers: dubbo.hessian2  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.resolvers: envoy.ip  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.filters.udp\_listener: envoy.filters.udp\_listener.udp\_proxy  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.filters.network: envoy.client\_ssl\_auth, envoy.echo, envoy.ext\_authz, envoy.filters.network.dubbo\_proxy, envoy.filters.network.kafka\_broker, envoy.filters.network.local\_ratelimit, envoy.filters.network.mysql\_proxy, envoy.filters.network.rbac, envoy.filters.network.sni\_cluster, envoy.filters.network.thrift\_proxy, envoy.filters.network.zookeeper\_proxy, envoy.http\_connection\_manager, envoy.mongo\_proxy, envoy.ratelimit, envoy.redis\_proxy, envoy.tcp\_proxy  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.resource\_monitors: envoy.resource\_monitors.fixed\_heap, envoy.resource\_monitors.injected\_resource  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext\_authz, envoy.fault, envoy.filters.http.adaptive\_concurrency, envoy.filters.http.dynamic\_forward\_proxy, envoy.filters.http.grpc\_http1\_reverse\_bridge, envoy.filters.http.grpc\_stats, envoy.filters.http.header\_to\_metadata, envoy.filters.http.jwt\_authn, envoy.filters.http.on\_demand, envoy.filters.http.original\_src, envoy.filters.http.rbac, envoy.filters.http.tap, envoy.grpc\_http1\_bridge, envoy.grpc\_json\_transcoder, envoy.grpc\_web, envoy.gzip, envoy.health\_check, envoy.http\_dynamo\_filter, envoy.ip\_tagging, envoy.lua, envoy.rate\_limit, envoy.router, envoy.squash  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.transport\_sockets.upstream: envoy.transport\_sockets.alts, envoy.transport\_sockets.raw\_buffer, envoy.transport\_sockets.tap, envoy.transport\_sockets.tls, raw\_buffer, tls  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.transport\_sockets.downstream: envoy.transport\_sockets.alts, envoy.transport\_sockets.raw\_buffer, envoy.transport\_sockets.tap, envoy.transport\_sockets.tls, raw\_buffer, tls  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.clusters: envoy.cluster.eds, envoy.cluster.logical\_dns, envoy.cluster.original\_dst, envoy.cluster.static, envoy.cluster.strict\_dns, envoy.clusters.aggregate, envoy.clusters.dynamic\_forward\_proxy, envoy.clusters.redis  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.filters.listener: envoy.listener.http\_inspector, envoy.listener.original\_dst, envoy.listener.original\_src, envoy.listener.proxy\_protocol, envoy.listener.tls\_inspector  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.health\_checkers: envoy.health\_checkers.redis  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.dubbo\_proxy.protocols: dubbo  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.grpc\_credentials: envoy.grpc\_credentials.aws\_iam, envoy.grpc\_credentials.default, envoy.grpc\_credentials.file\_based\_metadata  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.dubbo\_proxy.filters: envoy.filters.dubbo.router  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.thrift\_proxy.protocols: auto, binary, binary/non-strict, compact, twitter  
> [2020-07-23 13:39:01.424][3497][info][main] [external/envoy/source/server/server.cc:255] envoy.dubbo\_proxy.route\_matchers: default  
> [2020-07-23 13:39:01.482][3497][warning][misc] [external/envoy/source/common/protobuf/utility.cc:441] Using deprecated option ‘envoy.api.v2.Cluster.hosts’ from file cluster.proto. This configuration will be removed from Envoy soon. Please see [Deprecated — envoy 1.29.0-dev-2de016 documentation](https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated) for details.  
> [2020-07-23 13:39:01.482][3497][warning][misc] [external/envoy/source/common/protobuf/utility.cc:441] Using deprecated option ‘envoy.api.v2.Cluster.tls\_context’ from file cluster.proto. This configuration will be removed from Envoy soon. Please see [Deprecated — envoy 1.29.0-dev-2de016 documentation](https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated) for details.  
> [2020-07-23 13:39:01.482][3497][info][main] [external/envoy/source/server/server.cc:336] admin address: 127.0.0.1:19005  
> [2020-07-23 13:39:01.485][3497][info][main] [external/envoy/source/server/server.cc:455] runtime: layers: - name: static\_layer  
> static\_layer:  
> envoy.deprecated\_features:envoy.config.trace.v2.ZipkinConfig.HTTP\_JSON\_V1: true  
> envoy.deprecated\_features:envoy.config.filter.network.http\_connection\_manager.v2.HttpConnectionManager.Tracing.operation\_name: true  
> envoy.deprecated\_features:envoy.api.v2.Cluster.tls\_context: true  
> [2020-07-23 13:39:01.485][3497][info][config] [external/envoy/source/server/configuration\_impl.cc:62] loading 0 static secret(s)  
> [2020-07-23 13:39:01.485][3497][info][config] [external/envoy/source/server/configuration\_impl.cc:68] loading 1 cluster(s)  
> [2020-07-23 13:39:01.498][3497][info][upstream] [external/envoy/source/common/upstream/cluster\_manager\_impl.cc:167] cm init: initializing cds  
> [2020-07-23 13:39:01.501][3497][info][config] [external/envoy/source/server/configuration\_impl.cc:72] loading 0 listener(s)  
> [2020-07-23 13:39:01.501][3497][info][config] [external/envoy/source/server/configuration\_impl.cc:97] loading tracing configuration  
> [2020-07-23 13:39:01.502][3497][info][config] [external/envoy/source/server/configuration\_impl.cc:116] loading stats sink configuration  
> [2020-07-23 13:39:01.502][3497][info][main] [external/envoy/source/server/server.cc:550] starting main dispatch loop  
> [2020-07-23 13:39:01.506][3497][info][upstream] [external/envoy/source/common/upstream/cds\_api\_impl.cc:74] cds: add 1 cluster(s), remove 1 cluster(s)  
> [2020-07-23 13:39:01.532][3497][info][upstream] [external/envoy/source/common/upstream/cds\_api\_impl.cc:90] cds: add/update cluster ‘dc2.internal.9560fc88-3471-6a9c-c64c-9174e676b207.consul’  
> [2020-07-23 13:39:01.532][3497][info][upstream] [external/envoy/source/common/upstream/cluster\_manager\_impl.cc:145] cm init: initializing secondary clusters  
> [2020-07-23 13:39:01.533][3497][info][upstream] [external/envoy/source/common/upstream/cluster\_manager\_impl.cc:171] cm init: all clusters initialized  
> [2020-07-23 13:39:01.533][3497][info][main] [external/envoy/source/server/server.cc:529] all clusters initialized. initializing init manager  
> [2020-07-23 13:39:01.536][3497][info][upstream] [external/envoy/source/server/lds\_api.cc:73] lds: add/update listener ‘lan:127.0.0.1:8443’  
> [2020-07-23 13:39:01.536][3497][warning][misc] [external/envoy/source/common/protobuf/utility.cc:441] Using deprecated option ‘envoy.api.v2.listener.Filter.config’ from file listener\_components.proto. This configuration will be removed from Envoy soon. Please see [Deprecated — envoy 1.29.0-dev-2de016 documentation](https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated) for details.  
> [2020-07-23 13:39:01.537][3497][info][upstream] [external/envoy/source/server/lds\_api.cc:73] lds: add/update listener ‘wan:10.1.168.149:8443’  
> [2020-07-23 13:39:01.537][3497][info][config] [external/envoy/source/server/listener\_manager\_impl.cc:707] all dependencies initialized. starting workers  
> [2020-07-23 13:39:01.561][3497][info][upstream] [external/envoy/source/common/upstream/cds\_api\_impl.cc:74] cds: add 1 cluster(s), remove 1 cluster(s)  
> [2020-07-23 13:39:01.562][3497][warning][misc] [external/envoy/source/common/protobuf/utility.cc:441] Using deprecated option ‘envoy.api.v2.listener.Filter.config’ from file listener\_components.proto. This configuration will be removed from Envoy soon. Please see [Deprecated — envoy 1.29.0-dev-2de016 documentation](https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated) for details.  
> [external/envoy/source/server/drain\_manager\_impl.cc:68] shutting down parent after drain

Mesh gateway log in Kubernetes:

> [2020-07-23 04:05:55.200][1][info][upstream] [source/common/upstream/cds\_api\_impl.cc:93] cds: add/update cluster ‘dc1.internal.9560fc88-3471-6a9c-c64c-9174e676b207.consul’  
> 7/23/2020 12:05:55 AM [2020-07-23 04:05:55.307][1][warning][misc] [bazel-out/k8-opt/bin/source/extensions/common/\_virtual\_includes/utility\_lib/extensions/common/utility.h:65] Using deprecated extension name ‘envoy.listener.tls\_inspector’ for ‘envoy.filters.listener.tls\_inspector’. This name will be removed from Envoy soon. Please see [Deprecated — envoy 1.29.0-dev-2de016 documentation](https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated) for details.  
> 7/23/2020 12:05:55 AM [2020-07-23 04:05:55.400][1][info][upstream] [source/server/lds\_api.cc:76] lds: add/update listener ‘default:10.42.1.112:8443’  
> 7/23/2020 12:09:17 AM [2020-07-23 04:09:17.760][1][info][upstream] [source/common/upstream/cds\_api\_impl.cc:77] cds: add 4 cluster(s), remove 1 cluster(s)  
> 7/23/2020 12:09:26 AM [2020-07-23 04:09:26.111][1][info][upstream] [source/common/upstream/cds\_api\_impl.cc:77] cds: add 4 cluster(s), remove 1 cluster(s)

The Helm config for Kubernetes cluster:

 ![image](https://global.discourse-cdn.com/hashicorp/original/2X/1/175f1ff6fb3861ad7f24c269dbd0fbd9788d11eb.png)

I only bootstrapped one server. Also disabled acl and gossip encryption.  
Instead of using load balancer, I used NodePort service to expose the mesh gateway.

My vm server config:

> cert\_file = “/home/ubuntu/dc1-server-consul-0.pem”  
> key\_file = “/home/ubuntu/dc1-server-consul-0-key.pem”  
> ca\_file = “/home/ubuntu/consul-agent-ca.pem”  
> primary\_gateways = [“10.1.168.145:32001”]  
> server = true  
> bootstrap\_expect = 1  
> datacenter = “dc1”  
> data\_dir = “/opt/consul”  
> enable\_central\_service\_config = true  
> primary\_datacenter = “dc2”  
> connect {  
> enabled = true  
> enable\_mesh\_gateway\_wan\_federation = true  
> }  
> verify\_incoming\_rpc = true  
> verify\_outgoing = true  
> verify\_server\_hostname = true  
> ports {  
> https = 8501  
> http = 8500  
> grpc = 8502  
> }

The command I used to launch mesh gateway on vm server:

> consul connect envoy -mesh-gateway -register -service “gateway-secondary” -address “127.0.0.1:8443” -wan-address “10.1.168.149:8443” -admin-bind  
> 127.0.0.1:19005 -grpc-addr=[https://127.0.0.1:8502](https://127.0.0.1:8502) -ca-file=/home/ubuntu/consul-agent-ca.pem

From the vm server I was able to show the services in kubernetes cluster, however from the kuberenetes consul server when I query the services in vm cluster I got:

> Error listing services: Unexpected response code: 500 (Remote DC has no server currently reachable)

I followed this guide to set up consul on vm

> **[Deployment Guide | Consul | HashiCorp Developer](https://developer.hashicorp.com/consul/tutorials/production-deploy/deployment-guide)**
>
> Deploy your first three-server Consul datacenter in accordance with the Reference Architecture on Linux or Windows.

Can someone please help me see what’s wrong here?

---

<div class="post-metadata">

**Author:** ![WannaBeGeekster](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/wannabegeekster/32/9875_2.png) [@WannaBeGeekster](https://discuss.hashicorp.com/u/WannaBeGeekster)\
**Post date:** [July 27, 2020, 11:13pm UTC](https://discuss.hashicorp.com/t/error-wan-federation-between-vms-and-kubernetes/11918/2 "2020-07-27T23:13:33Z")

</div>

I am having similar issues. The documentation page for WAN Federation is not available anymore. Is there any guide or other documentation that can be used on how to set WAN federation up with consul 1.8 between VMs and Kubernetes clusters?

---

<div class="post-metadata">

**Author:** ![lkysow](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/lkysow/32/1105_2.png) [@lkysow](https://discuss.hashicorp.com/u/lkysow)\
**Post date:** [July 27, 2020, 11:25pm UTC](https://discuss.hashicorp.com/t/error-wan-federation-between-vms-and-kubernetes/11918/3 "2020-07-27T23:25:57Z")

</div>

@WannaBeGeekster There are these docs: [https://www.consul.io/docs/k8s/installation/multi-cluster/overview](https://www.consul.io/docs/k8s/installation/multi-cluster/overview), [https://www.consul.io/docs/k8s/installation/multi-cluster/vms-and-kubernetes](https://www.consul.io/docs/k8s/installation/multi-cluster/vms-and-kubernetes)

---

<div class="post-metadata">

**Author:** ![lkysow](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/lkysow/32/1105_2.png) [@lkysow](https://discuss.hashicorp.com/u/lkysow)\
**Post date:** [July 27, 2020, 11:26pm UTC](https://discuss.hashicorp.com/t/error-wan-federation-between-vms-and-kubernetes/11918/4 "2020-07-27T23:26:42Z")

</div>

@YaoTu sorry you’re having these issues. We’re going to try and reproduce them and get back to you.

---

<div class="post-metadata">

**Author:** ![YaoTu](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/yaotu/32/9749_2.png) [@YaoTu](https://discuss.hashicorp.com/u/YaoTu)\
**Post date:** [July 28, 2020, 1:49am UTC](https://discuss.hashicorp.com/t/error-wan-federation-between-vms-and-kubernetes/11918/5 "2020-07-28T01:49:16Z")

</div>

Hi @lkysow, thank you for your response.

Interestingly, I can reproduce this issue in WAN federation of vm clusters. I thought this might be a network issue of my environment, but somehow I was able to connect kubernetes clusters following this guide.

> **[Secure service mesh communication across Kubernetes clusters | Consul |...](https://developer.hashicorp.com/consul/tutorials/kubernetes/kubernetes-mesh-gateways)**
>
> Secure service-to-service communication across multiple Kubernetes clusters with Consul's mesh gateway feature.

Just want to share how I re-created this issue in vm clusters.

I used two virtual machines whose security group allows all IPv4 and IPv6 traffic to and from each other. They’re called dc3-server and dc4-server.

**Set up for dc3-server:**

I followed the [deployment guide](https://learn.hashicorp.com/consul/datacenter-deploy/deployment-guide) to set up consul. I didn’t generate the gossip encryption key.

Here is the configuration for the server:

> datacenter = “dc3”  
> data\_dir = “/opt/consul”  
> ca\_file = “/etc/consul.d/consul-agent-ca.pem”  
> cert\_file = “/etc/consul.d/dc3-server-consul-0.pem”  
> key\_file = “/etc/consul.d/dc3-server-consul-0-key.pem”  
> verify\_incoming = true  
> verify\_outgoing = true  
> verify\_server\_hostname = true  
> server = true  
> primary\_datacenter = “dc3”  
> bootstrap\_expect = 1  
> connect {  
> enabled = true  
> enable\_mesh\_gateway\_wan\_federation = true  
> }  
> enable\_central\_service\_config = true  
> ports {  
> grpc = 8502  
> https = 8501  
> }

For the systemd configuration I changed Type to exec as instructed by the guide.

Command to run mesh gateway **On dc3-server**

> sudo consul connect envoy -mesh-gateway -register -service gateway-primary -address 10.1.169.25:443 -wan-address 10.1.169.25:8443 -admin-bind 127.0.0.1:19005 -grpc-addr=[https://127.0.0.1:8502](https://127.0.0.1:8502) -ca-file=/etc/consul.d/consul-agent-ca.pem -expose-servers

Log from consul in dc3-server:

> 2020-07-28T01:25:50.368Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-primary id=federation-state-list-mesh-gateways error=“error filling agent cache: No cluster leader”  
> 2020-07-28T01:25:50.425Z [ERROR] agent.anti\_entropy: failed to sync remote state: error=“No cluster leader”  
> 2020-07-28T01:25:50.425Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-primary id=consul-server-list error=“error filling agent cache: No cluster leader”  
> 2020-07-28T01:25:50.432Z [ERROR] agent: error handling service update: error=“error watching service config: No cluster leader”  
> 2020-07-28T01:25:50.449Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-primary id=roots error=“error filling agent cache: No cluster leader”  
> 2020-07-28T01:25:50.537Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-primary id=service-list error=“error filling agent cache: No cluster leader”  
> 2020-07-28T01:25:50.726Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-primary id=service-resolvers error=“error filling agent cache: No cluster leader”  
> 2020-07-28T01:25:51.841Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=  
> 2020-07-28T01:25:51.841Z [INFO] agent.server.raft: entering candidate state: node=“Node at 10.1.169.25:8300 [Candidate]” term=9  
> 2020-07-28T01:25:51.855Z [INFO] agent.server.raft: election won: tally=1  
> 2020-07-28T01:25:51.855Z [INFO] agent.server.raft: entering leader state: leader=“Node at 10.1.169.25:8300 [Leader]”  
> 2020-07-28T01:25:51.856Z [INFO] agent.server: cluster leadership acquired  
> 2020-07-28T01:25:51.856Z [INFO] agent.server: New leader elected: payload=yao-dc3-server  
> 2020-07-28T01:25:52.204Z [INFO] agent.leader: started routine: routine=“federation state anti-entropy”  
> 2020-07-28T01:25:52.204Z [INFO] agent.leader: started routine: routine=“federation state pruning”  
> 2020-07-28T01:25:52.205Z [INFO] agent.leader: started routine: routine=“CA root pruning”  
> 2020-07-28T01:25:53.061Z [INFO] agent: Synced node info  
> 2020-07-28T01:25:57.360Z [INFO] agent.server.gateway\_locator: new cached locations of mesh gateways: primary=[10.1.169.25:8443] local=[10.1.169.25:443]  
> 2020-07-28T01:26:28.850Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: yao-dc4-server.dc4 10.1.168.162  
> 2020-07-28T01:26:28.851Z [INFO] agent.server: Handled event for server in area: event=member-join server=yao-dc4-server.dc4 area=wan  
> 2020-07-28T01:26:28.861Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.162:8302: EOF  
> 2020-07-28T01:26:29.361Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.162:8302: EOF  
> 2020-07-28T01:26:29.862Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.162:8302: EOF  
> 2020-07-28T01:26:30.362Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.162:8302: EOF  
> 2020-07-28T01:26:30.861Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.162:8302: EOF  
> 2020-07-28T01:26:33.842Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ack: EOF from=10.1.169.25:43494  
> 2020-07-28T01:26:38.362Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: EOF  
> 2020-07-28T01:26:39.340Z [WARN] agent.server.memberlist.wan: memberlist: Refuting a suspect message (from: yao-dc4-server.dc4)  
> 2020-07-28T01:26:39.361Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.162:8302: EOF  
> 2020-07-28T01:26:39.862Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.162:8302: EOF  
> 2020-07-28T01:26:40.361Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.162:8302: EOF  
> 2020-07-28T01:26:40.862Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.168.162:8302: EOF  
> 2020-07-28T01:26:43.843Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ack: EOF from=10.1.169.25:43494  
> 2020-07-28T01:26:45.715Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:45.715Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-primary id=mesh-gateway:dc4 error=“error filling agent cache: rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:45.717Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:45.717Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-primary id=mesh-gateway:dc4 error=“error filling agent cache: rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:45.718Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:45.719Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-primary id=mesh-gateway:dc4 error=“error filling agent cache: rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:45.719Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:45.720Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-primary id=mesh-gateway:dc4 error=“error filling agent cache: rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:47.967Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:47.967Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-primary id=mesh-gateway:dc4 error=“error filling agent cache: rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:47.968Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:47.970Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:47.971Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:48.362Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: EOF  
> 2020-07-28T01:26:49.150Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:50.796Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:54.685Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:54.686Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-primary id=mesh-gateway:dc4 error=“error filling agent cache: rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:54.687Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:54.688Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:54.690Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:55.472Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:57.092Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:57.270Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.168.162:8300 datacenter=dc4 method=Internal.ServiceDump error=“rpc error getting client: failed to get conn: EOF”  
> 2020-07-28T01:26:57.631Z [ERROR] agent.server.memberlist.wan: memberlist: Push/Pull with yao-dc4-server.dc4 failed: EOF  
> 2020-07-28T01:26:58.361Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: EOF  
> 2020-07-28T01:26:58.842Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ack: EOF from=10.1.169.25:43494

netstat from dc3-server:

 ![image](https://global.discourse-cdn.com/hashicorp/original/2X/b/b1743dca7bd7a62a73af9c1a931e77f933b7dfb5.png)

**Set up for dc4-server:**

Configuration:

> datacenter = “dc4”  
> data\_dir = “/opt/consul”  
> ca\_file = “/etc/consul.d/consul-agent-ca.pem”  
> cert\_file = “/etc/consul.d/dc4-server-consul-0.pem”  
> key\_file = “/etc/consul.d/dc4-server-consul-0-key.pem”  
> verify\_incoming = true  
> verify\_outgoing = true  
> verify\_server\_hostname = true  
> server = true  
> bootstrap\_expect = 1  
> primary\_gateways = [“10.1.169.25:8443”]  
> primary\_datacenter = “dc3”  
> enable\_central\_service\_config = true  
> connect {  
> enabled = true  
> enable\_mesh\_gateway\_wan\_federation = true  
> }  
> enable\_central\_service\_config = true  
> ports {  
> grpc = 8502  
> https = 8501  
> }

Command to run mesh gateway **on dc4-server** :

> sudo consul connect envoy -mesh-gateway -register -service gateway-secondary -address 10.1.168.162:443 -wan-address 10.1.168.162:8443 -admin-bind 127.0.0.1:19005 -grpc-addr=[https://127.0.0.1:8502](https://127.0.0.1:8502) -ca-file=/etc/consul.d/consul-agent-ca.pem

Log from consul on dc4-server:

> 2020-07-28T01:29:59.922Z [ERROR] agent.anti\_entropy: failed to sync remote state: error=“No cluster leader”  
> 2020-07-28T01:30:00.030Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-secondary id=service-list error=“error filling agent cache: No cluster leader”  
> 2020-07-28T01:30:00.063Z [ERROR] agent: error handling service update: error=“error watching service config: No cluster leader”  
> 2020-07-28T01:30:00.084Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-secondary id=roots error=“error filling agent cache: No cluster leader”  
> 2020-07-28T01:30:00.129Z [ERROR] agent.proxycfg: watch error: service\_id=gateway-secondary id=service-resolvers error=“error filling agent cache: No cluster leader”  
> 2020-07-28T01:30:01.865Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=  
> 2020-07-28T01:30:01.865Z [INFO] agent.server.raft: entering candidate state: node=“Node at 10.1.168.162:8300 [Candidate]” term=11  
> 2020-07-28T01:30:01.881Z [INFO] agent.server.raft: election won: tally=1  
> 2020-07-28T01:30:01.881Z [INFO] agent.server.raft: entering leader state: leader=“Node at 10.1.168.162:8300 [Leader]”  
> 2020-07-28T01:30:01.882Z [INFO] agent.server: cluster leadership acquired  
> 2020-07-28T01:30:01.882Z [INFO] agent.server: New leader elected: payload=yao-dc4-server  
> 2020-07-28T01:30:02.098Z [INFO] agent.server.connect: initialized secondary datacenter CA with provider: provider=consul  
> 2020-07-28T01:30:02.098Z [INFO] agent.leader: started routine: routine=“config entry replication”  
> 2020-07-28T01:30:02.099Z [INFO] agent.leader: started routine: routine=“federation state replication”  
> 2020-07-28T01:30:02.099Z [INFO] agent.leader: started routine: routine=“federation state anti-entropy”  
> 2020-07-28T01:30:02.099Z [INFO] agent.leader: started routine: routine=“secondary CA roots watch”  
> 2020-07-28T01:30:02.099Z [INFO] agent.leader: started routine: routine=“intention replication”  
> 2020-07-28T01:30:02.099Z [INFO] agent.leader: started routine: routine=“secondary cert renew watch”  
> 2020-07-28T01:30:02.099Z [INFO] agent.leader: started routine: routine=“CA root pruning”  
> 2020-07-28T01:30:02.101Z [INFO] agent.server.gateway\_locator: will dial the primary datacenter using our local mesh gateways if possible  
> 2020-07-28T01:30:02.232Z [INFO] agent: Synced node info  
> 2020-07-28T01:30:02.810Z [INFO] agent.server.memberlist.wan: memberlist: Suspect yao-dc3-server.dc3 has failed, no acks received  
> 2020-07-28T01:30:06.809Z [INFO] agent.server.gateway\_locator: new cached locations of mesh gateways: primary=[10.1.169.25:8443] local=  
> 2020-07-28T01:30:17.810Z [INFO] agent.server.memberlist.wan: memberlist: Suspect yao-dc3-server.dc3 has failed, no acks received  
> 2020-07-28T01:30:32.810Z [INFO] agent.server.memberlist.wan: memberlist: Marking yao-dc3-server.dc3 as failed, suspect timeout reached (0 peer confirmations)  
> 2020-07-28T01:30:32.810Z [INFO] agent.server.serf.wan: serf: EventMemberFailed: yao-dc3-server.dc3 10.1.169.25  
> 2020-07-28T01:30:32.810Z [INFO] agent.server: Handled event for server in area: event=member-failed server=yao-dc3-server.dc3 area=wan  
> 2020-07-28T01:30:37.811Z [INFO] agent.server.memberlist.wan: memberlist: Suspect yao-dc3-server.dc3 has failed, no acks received  
> 2020-07-28T01:30:52.811Z [INFO] agent.server.serf.wan: serf: attempting reconnect to yao-dc3-server.dc3 10.1.169.25:8302  
> 2020-07-28T01:30:52.816Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: yao-dc3-server.dc3 10.1.169.25  
> 2020-07-28T01:30:52.816Z [INFO] agent.server: Handled event for server in area: event=member-join server=yao-dc3-server.dc3 area=wan  
> 2020-07-28T01:31:17.810Z [INFO] agent.server.memberlist.wan: memberlist: Suspect yao-dc3-server.dc3 has failed, no acks received  
> 2020-07-28T01:31:47.809Z [INFO] agent.server.memberlist.wan: memberlist: Suspect yao-dc3-server.dc3 has failed, no acks received  
> 2020-07-28T01:31:47.811Z [INFO] agent.server.memberlist.wan: memberlist: Marking yao-dc3-server.dc3 as failed, suspect timeout reached (0 peer confirmations)

Netstat from dc4-server:

 ![image](https://global.discourse-cdn.com/hashicorp/original/2X/f/fee4495f640fd57ded5f737cafe161d3f72b67f8.png)

In both servers envoy mesh gateways didn’t complain anything.

I followed the order: primary dc server → primary mesh gateway → secondary dc server → secondary gateway.  
Hope this information helps

---

<div class="post-metadata">

**Author:** ![rboyer](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/rboyer/32/884_2.png) [@rboyer](https://discuss.hashicorp.com/u/rboyer)\
**Post date:** [July 28, 2020, 2:44pm UTC](https://discuss.hashicorp.com/t/error-wan-federation-between-vms-and-kubernetes/11918/6 "2020-07-28T14:44:58Z")

</div>

For the VM-only scenario you described can you double check that the TLS SAN fields are correct? There is an additional SAN field required for servers when they are WAN federated via mesh gateways: [https://www.consul.io/docs/connect/gateways/wan-federation-via-mesh-gateways#tls](https://www.consul.io/docs/connect/gateways/wan-federation-via-mesh-gateways#tls)

---

<div class="post-metadata">

**Author:** ![YaoTu](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/yaotu/32/9749_2.png) [@YaoTu](https://discuss.hashicorp.com/u/YaoTu)\
**Post date:** [July 29, 2020, 12:58am UTC](https://discuss.hashicorp.com/t/error-wan-federation-between-vms-and-kubernetes/11918/7 "2020-07-29T00:58:36Z")

</div>

Hi @rboyer, I checked the SAN fields in both servers’ TLS.

Looks like they have the SAN fields required:

dc4-server:

> X509v3 Subject Alternative Name:  
> DNS:yao-dc4-server.server.dc4.consul, DNS:server.dc4.consul, DNS:localhost, IP Address:127.0.0.1

dc3- server:

> X509v3 Subject Alternative Name:  
> DNS:yao-dc3-server.server.dc3.consul, DNS:server.dc3.consul, DNS:localhost, IP Address:127.0.0.1

---

<div class="post-metadata">

**Author:** ![YaoTu](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/yaotu/32/9749_2.png) [@YaoTu](https://discuss.hashicorp.com/u/YaoTu)\
**Post date:** [August 2, 2020, 3:14pm UTC](https://discuss.hashicorp.com/t/error-wan-federation-between-vms-and-kubernetes/11918/8 "2020-08-02T15:14:40Z")

</div>

Ok I figured out what’s wrong. I didn’t add the -expose-servers flag when I run the vm mesh gateway. As noted in this document:  
[https://www.consul.io/docs/connect/gateways/wan-federation-via-mesh-gateways](https://www.consul.io/docs/connect/gateways/wan-federation-via-mesh-gateways)