Error WAN Federation beween vms and EKS

  • I create EKS consul 1.8 with Helm chart (eks-dev) and create a mesh gateway with AWS load balancer.

meshGateway:
enabled: true
globalMode: local
replicas: 3
wanAddress:
source: “Service”
port: 443
static: “”
service:
enabled: true
type: LoadBalancer
port: 443
nodePort: null
annotations: |
service.beta.kubernetes.io/aws-load-balancer-type: nlb
additionalSpec: null
imageEnvoy: envoyproxy/envoy-alpine:v1.14.2
hostNetwork: false
dnsPolicy: null
consulServiceName: “mesh-gateway”
containerPort: 8443
hostPort: null
resources:
requests:
memory: “100Mi”
cpu: “100m”
limits:
memory: “100Mi”
cpu: “100m”
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: {{ template “consul.name” . }}
release: “{{ .Release.Name }}”
component: mesh-gateway
topologyKey: kubernetes.io/hostname
tolerations: null
nodeSelector: null
priorityClassName: “”
annotations: null

When I check:
nc -v 443
Connection to 443 port [tcp/https] succeeded!

I create another region follow https://www.consul.io/docs/k8s/installation/multi-cluster/vms-and-kubernetes.

I config in VMs (do-singapore):

{
“addresses”: {
“dns”: “0.0.0.0”,
“grpc”: “0.0.0.0”,
“http”: “0.0.0.0”,
“https”: “0.0.0.0”
},
“advertise_addr”: “10.130.118.203”,
“advertise_addr_wan”: “”,
“bind_addr”: “0.0.0.0”,
“bootstrap”: false,
“bootstrap_expect”: 3,
“ca_file”: “/etc/consul/ssl/ca.crt”,
“cert_file”: “/etc/consul/ssl/server.crt”,
“client_addr”: “0.0.0.0”,
“data_dir”: “/var/consul”,
“datacenter”: “do-singapore”,
“disable_update_check”: false,
“domain”: “consul1.do-singapore.consul.uizadev.io”,
“enable_local_script_checks”: true,
“enable_script_checks”: false,
“encrypt”: “”,
“key_file”: “/etc/consul/ssl/server.key”,
“log_file”: “/var/log/consul/consul.log”,
“log_level”: “INFO”,
“log_rotate_bytes”: 0,
“log_rotate_duration”: “24h”,
“log_rotate_max_files”: 0,
“node_name”: “consul1”,
“performance”: {
“leave_drain_time”: “5s”,
“raft_multiplier”: 1,
“rpc_hold_timeout”: “7s”
},
“ports”: {
“dns”: 8600,
“grpc”: 8502,
“http”: 8500,
“https”: 8501,
“serf_lan”: 8301,
“serf_wan”: 8302,
“server”: 8300
},
“raft_protocol”: 3,
“retry_interval”: “30s”,
“retry_join”: [
“10.130.119.102”,
“10.130.118.203”,
“10.130.118.183”
],
“retry_max”: 0,
“server”: true,
“tls_min_version”: “tls12”,
“tls_prefer_server_cipher_suites”: false,
“translate_wan_addrs”: true,
“enable_central_service_config”: true,
“ui”: true,
“verify_incoming”: false,
“verify_incoming_https”: false,
“verify_incoming_rpc”: true,
“verify_outgoing”: true,
“verify_server_hostname”: false,
“primary_datacenter”: “eks-dev”,
“primary_gateways”: [" < load balancer dns > :443"],
“connect”: {
“enabled”: true,
“enable_mesh_gateway_wan_federation”: true
}
}

we got the error:

2020-07-01T04:37:41.788Z [WARN] agent: (WAN) couldn’t join: number_of_nodes=0 error="1 error occurred:
* Failed to join 192.0.2.2: read tcp 128.199.208.31:33014->54.151.170.130:443: read: connection reset by peer

Someone can help me about this problem?

Hi,
It looks like you haven’t enabled federation or TLS in your Kubernetes Helm cnfig. You need to use the configuration from the docs: https://www.consul.io/docs/k8s/installation/multi-cluster/kubernetes#primary-datacenter

This is likely why the gateway is giving you a connection reset by peer.

I have solved this problem. Thank you @lkysow but EKS has been enabled federation. In VMs, we need to config:

“verify_incoming_rpc”: true
“verify_outgoing”: true
“verify_server_hostname”: true

Sorry, so is everything working now?

Note that that config is what’s described in the docs: https://www.consul.io/docs/k8s/installation/multi-cluster/vms-and-kubernetes