Problem with service mesh when the host is behind nat

Nomad
1

Hi folks,

I’m using consul + nomad, the latest versions.

I first installed consul and after nomad. I followed the docs: Connect Nodes into a Cluster | Nomad - HashiCorp Learn

I’m testing the infra with fake-service and countdash (Consul Service Mesh | Nomad by HashiCorp). My infra has
3 VMs on oracle cloud
3 VMs + 2 VMs on premise in two distinct networks.

The VMs on premise has public IPv4 and service mesh works well.
The VMs on oracle cloud has private IPv4, so has a NAT between the service mesh. I think NAT is the problem.

There are some kind of conflict between private and public IP:

2022-07-05T04:51:48.333Z [INFO]  agent: Synced service: service=_nomad-task-ff4ca4b2-d379-93c3-8920-7ca53c478064-group-api-count-api-9001
2022-07-05T04:51:48.366Z [INFO]  agent: Synced service: service=_nomad-task-ff4ca4b2-d379-93c3-8920-7ca53c478064-group-api-count-api-9001-sidecar-proxy
2022-07-05T04:51:53.209Z [WARN]  agent: Check socket connection failed: check=service:_nomad-task-ff4ca4b2-d379-93c3-8920-7ca53c478064-group-api-count-api-9001-sidecar-proxy:2 error="dial tcp 10.0.0.11:29137: connect: connection refused"
2022-07-05T04:51:53.209Z [WARN]  agent: Check is now critical: check=service:_nomad-task-ff4ca4b2-d379-93c3-8920-7ca53c478064-group-api-count-api-9001-sidecar-proxy:2
2022-07-05T04:51:57.757Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with edge2 failed: dial tcp 191.36.8.27:8301: i/o timeout
2022-07-05T04:52:03.225Z [INFO]  agent: Synced check: check=service:_nomad-task-ff4ca4b2-d379-93c3-8920-7ca53c478064-group-api-count-api-9001-sidecar-proxy:2
2022-07-05T04:52:37.758Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with edge1 failed: dial tcp 191.36.8.26:8301: i/o timeout
2022-07-05T04:53:17.759Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with fog2 failed: dial tcp 191.36.15.212:8301: i/o timeout
2022-07-05T04:54:06.524Z [ERROR] agent.server.memberlist.wan: memberlist: Conflicting address for cloud3.cloud. Mine: 193.123.119.125:8302 Theirs: 10.0.0.13:8302 Old state: 0
2022-07-05T04:54:06.524Z [WARN]  agent.server.serf.wan: serf: Name conflict for 'cloud3.cloud' both 193.123.119.125:8302 and 10.0.0.13:8302 are claiming
2022-07-05T04:54:24.689Z [ERROR] agent.server.memberlist.wan: memberlist: Conflicting address for cloud3.cloud. Mine: 193.123.119.125:8302 Theirs: 10.0.0.13:8302 Old state: 0
2022-07-05T04:54:24.689Z [WARN]  agent.server.serf.wan: serf: Name conflict for 'cloud3.cloud' both 193.123.119.125:8302 and 10.0.0.13:8302 are claiming
2022-07-05T04:54:57.765Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with fog1 failed: dial tcp 191.36.15.211:8301: i/o timeout
2022-07-05T04:55:06.526Z [ERROR] agent.server.memberlist.wan: memberlist: Conflicting address for cloud3.cloud. Mine: 193.123.119.125:8302 Theirs: 10.0.0.13:8302 Old state:

Consul configuration:

root@node1:/home/ubuntu# cat /etc/consul.d/consul.hcl 
datacenter = "cloud"
data_dir = "/opt/consul"
node_name = "cloud1"
client_addr = "0.0.0.0"
ui_config {
  enabled = true
}
server = true
bind_addr = "0.0.0.0"
advertise_addr  = "193.123.108.17"
bootstrap_expect = 3
ports {
  grpc  = 8502
}
connect {
  enabled = true
}

Nomad configuration:

root@node1:/home/ubuntu# cat /etc/nomad.d/nomad.hcl 
datacenter = "cloud"
data_dir = "/opt/nomad"
name = "cloud1"
advertise {
http = "193.123.108.17"
rpc  = "193.123.108.17"
serf = "193.123.108.17"
}
server {
  enabled = true
  bootstrap_expect = 3
}
client {
  enabled       = true
}

Someone could help me?