How Nomad Client search Nomad Servers through Consul?

Nomad
1

Can you tell me how Nomad searches through Consul Nomad servers and, respectively, the cluster to connect to?

I encountered a problem that the Nomad client was connected to a PROD cluster, then a network problem happened and Nomad started searching for the cluster through Consul:

2022-08-17T01:06:09.232092+00:00 nomad[104240]: 2022-08-17T01:06:09.217Z [ERROR] client.rpc: error performing RPC to server: error="rpc error: EOF" rpc=Node.UpdateStatus server=10.175.92.183:4647
2022-08-17T01:06:09.245962+00:00 nomad[104240]: 2022-08-17T01:06:09.245Z [ERROR] client: error querying node allocations: error="rpc error: EOF"
2022-08-17T01:06:09.273003+00:00 nomad[104240]: 2022-08-17T01:06:09.272Z [ERROR] client: error heartbeating. retrying: error="failed to update status: rpc error: EOF" period=1.98873694s
2022-08-17T01:06:09.308976+00:00 nomad[104240]: 2022-08-17T01:06:09.308Z [DEBUG] client.consul: bootstrap contacting Consul DCs: consul_dcs=[megatron_cref]
2022-08-17T01:06:09.699962+00:00 nomad[104240]: 2022-08-17T01:06:09.699Z [INFO]  client.consul: discovered following servers: servers=[10.175.108.182:4647, 10.175.108.183:4647, 10.175.108.181:4647]
2022-08-17T01:06:09.700444+00:00 nomad[104240]: 2022-08-17T01:06:09.699Z [DEBUG] client.server_mgr: new server list: new_servers=[10.175.108.181:4647, 10.175.108.182:4647, 10.175.108.183:4647] old_servers=[10.175.92.182:4647, 10.175.92.181:4647, 10.175.92.183:4647]

But in the same data center consul are nomad servers on STG and PROD

And as you can see the consul gave nomad servers from STG. Can someone explain how this happened and what request Nomad sends to consul to find nomad cluster?

2

Hi @falseroses , do you have client_auto_join configured? If so, the Nomad Client will query Consul for live Nomad servers - first in its local DC, then in other DCs.

If your Nomad clusters are all configured as the same region there is nothing to prevent such cross-connection.

3

Okay, but if we don’t use the consul block and the client_auto_join option is not configured, does it still automatically work as “true”?

bind_addr = "0.0.0.0"
region             = "global"
datacenter         = "kiev"
data_dir           = "/var/lib/nomad/"
log_level          = "DEBUG"
leave_on_interrupt = "true"
leave_on_terminate = "true"
client {
    enabled = true
      servers = ["10.175.108.181", "10.175.108.182", "10.175.108.183"]
      max_kill_timeout = "350s"
    network_speed = "10000"
    reserved {
       memory = "1170"
    }
    meta {
      "env" = "kiev"      "app" = "mongodb"
      "hostname" = "kiev-db-mongodb-01"
    }
}
advertise {
  http = "10.175.107.101:4646"
  rpc  = "10.175.107.101:4647"
  serf = "10.175.107.101:4648"
}
telemetry {
    publish_allocation_metrics = true
  publish_node_metrics = true
      prometheus_metrics = true
  }
plugin "docker" {
  config {
    volumes {
      enabled = true
    }
  }
}
4

Ah, as I understand it, yes.

A default consul stanza is automatically merged with all Nomad agent configurations. These sane defaults automatically enable Consul integration if Consul is detected on the system. This allows for seamless bootstrapping of the cluster with zero configuration. To put it another way: if you have a Consul agent running on the same host as the Nomad agent with the default configuration, Nomad will automatically connect and configure with Consul.