Consul: "Catalog.ServiceNodes" RPC failed and use of closed network connection

Consul
1

Hello,

I have issue to connect consul / nomad clients in on-prem network on consul / nomad servers on Azure.

Consul log:

Sep 17 14:28:20 onprem.agent consul[397908]: 2019/09/17 14:28:20 [WARN] agent: Check "_nomad-check-24418e6b43798938532fab839c68b6688fef10e6" is now critical
Sep 17 14:28:30 onprem.agent consul[397908]: 2019/09/17 14:28:30 [ERR] yamux: Failed to read stream data: read tcp 10.70.x.x:47481->10.180.x.x:8300: read: connection reset by peer
Sep 17 14:28:30 onprem.agent consul[397908]: 2019/09/17 14:28:30 [ERR] consul: "Catalog.ServiceNodes" RPC failed to server 10.180.x.x:8300: rpc error making call: EOF
Sep 17 14:28:30 onprem.agent consul[397908]: 2019/09/17 14:28:30 [ERR] yamux: Failed to write header: write tcp 10.70.x.x:47481->10.180.x.x:8300: use of closed network connection
Sep 17 14:28:30 onprem.agent consul[397908]: 2019/09/17 14:28:30 [ERR] http: Request GET /v1/catalog/service/nomad?dc=az1&near=_agent&stale=&tag=rpc&wait=2000ms, error: rpc error making call: EOF from=127.0.0.1:60304
Sep 17 14:28:30 onprem.agent consul[397908]: 2019/09/17 14:28:30 [WARN] agent: Check "_nomad-check-24418e6b43798938532fab839c68b6688fef10e6" is now critical

Nomad log:

Sep 17 14:29:28 onprem.agent nomad[398060]: 2019-09-17T14:29:28.078-0300 [WARN ] client.server_mgr: no servers available
Sep 17 14:29:30 onprem.agent nomad[398060]: 2019-09-17T14:29:30.655-0300 [ERROR] http: request failed: method=GET path=/v1/agent/health?type=client error="{"client":{"ok":false,"message":"no known servers"}}" code=500

Consul server configs:

server = true
bootstrap_expect = {{bootstrap_expect}}
ui = true

addresses {
http = "{{ansible_default_ipv4.address}}"
}

Consul configs (All Nodes):

datacenter = "dc1"

performance {
    raft_multiplier = 1
}

retry_join = [{{consul_servers_ips}}]
bind_addr = "{{ansible_default_ipv4.address}}"

The Consul and Nomad servers in on Azure network and nomad clients is on onprem network.

Azure network have this network: 10.180.x.x

On-prem network have this network: 10.70.x.x

2
  1. Is your on-premise network on some sort of direct vpn tunnel via the on-premise firewall <-> Azure infra?
    OR
  2. do you login into your Azure VMs via a public IP ?

Case 1: ports could be blocked.
Case 2: you may need to fiddle with the “translate wan addresses” settings…

HTH.