How Nomad Client search Nomad Servers through Consul?

Can you tell me how Nomad searches through Consul Nomad servers and, respectively, the cluster to connect to?

I encountered a problem that the Nomad client was connected to a PROD cluster, then a network problem happened and Nomad started searching for the cluster through Consul:

2022-08-17T01:06:09.232092+00:00 nomad[104240]: 2022-08-17T01:06:09.217Z [ERROR] client.rpc: error performing RPC to server: error="rpc error: EOF" rpc=Node.UpdateStatus server=10.175.92.183:4647
2022-08-17T01:06:09.245962+00:00 nomad[104240]: 2022-08-17T01:06:09.245Z [ERROR] client: error querying node allocations: error="rpc error: EOF"
2022-08-17T01:06:09.273003+00:00 nomad[104240]: 2022-08-17T01:06:09.272Z [ERROR] client: error heartbeating. retrying: error="failed to update status: rpc error: EOF" period=1.98873694s
2022-08-17T01:06:09.308976+00:00 nomad[104240]: 2022-08-17T01:06:09.308Z [DEBUG] client.consul: bootstrap contacting Consul DCs: consul_dcs=[megatron_cref]
2022-08-17T01:06:09.699962+00:00 nomad[104240]: 2022-08-17T01:06:09.699Z [INFO]  client.consul: discovered following servers: servers=[10.175.108.182:4647, 10.175.108.183:4647, 10.175.108.181:4647]
2022-08-17T01:06:09.700444+00:00 nomad[104240]: 2022-08-17T01:06:09.699Z [DEBUG] client.server_mgr: new server list: new_servers=[10.175.108.181:4647, 10.175.108.182:4647, 10.175.108.183:4647] old_servers=[10.175.92.182:4647, 10.175.92.181:4647, 10.175.92.183:4647]

But in the same data center consul are nomad servers on STG and PROD

And as you can see the consul gave nomad servers from STG. Can someone explain how this happened and what request Nomad sends to consul to find nomad cluster?

Hi @falseroses , do you have client_auto_join configured? If so, the Nomad Client will query Consul for live Nomad servers - first in its local DC, then in other DCs.

If your Nomad clusters are all configured as the same region there is nothing to prevent such cross-connection.