Client not able to connect to nomad cluster

Nomad
1

Hi Team i am facing issue trying out nomad and looking for help

I have created 3 server node and 3 client nodes on ubuntu 24 VM’s , my all 3 master node are able to talk to each other however client nodes are not able to connect with server nodes. (please note that i am trying with bare minimum, No Consul , No Docker installed - only client server communication)
and yes i can ping from each other, as they all are in the same network.

ubuntu24@master1:~$ nomad server members
Name            Address        Port  Status  Leader  Raft Version  Build  Datacenter  Region
master1.global  192.168.64.10  4648  alive   false   3             1.8.3  dc1         global
master2.global  192.168.64.11  4648  alive   false   3             1.8.3  dc1         global
master3.global  192.168.64.12  4648  alive   true    3             1.8.3  dc1         global

my server configuration looks like this


server {
  enabled          = true
  bootstrap_expect = 3
  server_join {
    retry_join = ["192.168.64.10", "192.168.64.11", "192.168.64.12"]
    retry_max  = 3
    retry_interval = "5s"
  }
}

bind_addr = "0.0.0.0"
data_dir = "/opt/nomad"
log_level = "INFO"

leave_on_interrupt = true
leave_on_terminate = true

and my client configuration looks like this

client {
  enabled = true
  servers = ["192.168.64.10:4647,192.168.64.11:4647,192.168.64.12:4647"]
}

bind_addr = "0.0.0.0"
data_dir = "/opt/nomad/data"

# Optional logging configuration
log_level = "INFO"

Nomad version details

ubuntu24@master1:~$ nomad -v
Nomad v1.8.3
BuildDate 2024-08-13T07:37:30Z
Revision 63b636e5cbaca312cf6ea63e040f445f05f00478

My service logs are showing this

Aug 27 13:38:24 worker1 nomad[1834]:     2024-08-27T13:38:24.583Z [WARN]  client.server_mgr: no servers available
Aug 27 13:38:24 worker1 nomad[1834]:     2024-08-27T13:38:24.869Z [WARN]  client.server_mgr: no servers available
Aug 27 13:38:24 worker1 nomad[1834]:     2024-08-27T13:38:24.869Z [ERROR] client.rpc: error performing RPC to server, deadline exceeded, cannot retry: error="no servers" rpc=Node.Register
Aug 27 13:38:24 worker1 nomad[1834]:     2024-08-27T13:38:24.871Z [ERROR] client: error discovering nomad servers: error="client.consul: unable to query Consul datacenters: Get \"http://127.0.0.1:8500/v1/catalog/d>

Every help appreciated, thanks in advance

2

Hi. It still tries to use consul. Check if you have consul block in logs, or explicitly add consul_client_join consul Block - Agent Configuration | Nomad | HashiCorp Developer as false.

Client nomad block also uses same server_joij retry_join block, try it too client Block - Agent Configuration | Nomad | HashiCorp Developer .

1 Like
4

Thanks for your reply @Kamilcuk , I have not installed consul and as per the little document i went through, i understand that consul is optional, so i do not want to use it.

I am manually providing my ip’s in the configuration file for worker/client node, if it is still trying to use consult, as per the attached error logs is there way i can ask nomad to Not to use consul at this moment? (assuming that is something creating a problem)

5

finally found the issue, it was a silly mistake (since the configuration file was generated via ChatGPT, it did a mistake. :smiling_face_with_tear:

correct way to add server in configuration
servers = ["192.168.64.10:4647", "192.168.64.11:4647", "192.168.64.12:4647"]

wrong way
servers = ["192.168.64.10:4647, 192.168.64.11:4647 , 192.168.64.12:4647"]

1 Like
6

This topic was automatically closed 62 days after the last reply. New replies are no longer allowed.