Hello.
I have a problem with all nodes: No Cluster Leader. Before I could create a file peer.json in /opt/nomad/server/raft/, but now when I create the file and then create after restarting nomad the nomad stops working. I have 3 nodes. I’ve created a file on the first node and this one doesn’t work. In other nodes, I get the error about missing a leader.
From the second node:
nomad server members
Name Address Port Status Leader Protocol Build Datacenter Region
prod-node0.global 172.18.0.13 4648 failed false 2 1.2.0 dc1 global
prod-node1.global 172.18.0.11 4648 alive false 2 1.2.0 dc1 global
prod-node2.global 172.18.0.17 4648 alive false 2 1.2.0 dc1 global
Error determining leaders: 1 error occurred:
* Region "global": Unexpected response code: 500 (No cluster leader)
From the first node:
nomad server members
Error querying servers: Get "http://127.0.0.1:4646/v1/agent/members": dial tcp 127.0.0.1:4646: connect: connection refused
Logs:
Dec 16 09:33:10 prod-node0 consul[696]: 2021-12-16T09:33:10.319+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Deregister server=172.18.0.12:8300 error="rpc error making call: No cluster lea
der"
Dec 16 09:33:10 prod-node0 consul[696]: 2021-12-16T09:33:10.320+0300 [WARN] agent: Deregistering service failed.: service=_nomad-server-fwzylv3q3e37q4emdznarxxnzyct7v3h error="rpc error making call: No clus
ter leader"
Dec 16 09:33:10 prod-node0 consul[696]: 2021-12-16T09:33:10.320+0300 [ERROR] agent: failed to sync changes: error="rpc error making call: No cluster leader"
Dec 16 09:33:10 prod-node0 consul[696]: 2021-12-16T09:33:10.319+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Deregister server=172.18.0.12:8300 error="rpc error mak
ing call: No cluster leader"
Dec 16 09:33:10 prod-node0 consul[696]: 2021-12-16T09:33:10.320+0300 [WARN] agent: Deregistering service failed.: service=_nomad-server-fwzylv3q3e37q4emdznarxxnzyct7v3h error="rpc er
ror making call: No cluster leader"
Dec 16 09:33:10 prod-node0 consul[696]: 2021-12-16T09:33:10.320+0300 [ERROR] agent: failed to sync changes: error="rpc error making call: No cluster leader"
Dec 16 09:33:17 prod-node0 consul[696]: 2021-12-16T09:33:17.527+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Register server=172.18.0.12:8300 error="rpc error making call: No cluster leade
r"
Dec 16 09:33:17 prod-node0 consul[696]: 2021-12-16T09:33:17.527+0300 [WARN] agent: Syncing service failed.: service=_nomad-task-3d0da439-36c1-4e08-b428-b515ff2d58f1-gitea-job-gitea-lb error="rpc error makin
g call: No cluster leader"
Dec 16 09:33:17 prod-node0 consul[696]: 2021-12-16T09:33:17.527+0300 [ERROR] agent: failed to sync changes: error="rpc error making call: No cluster leader"
Dec 16 09:33:17 prod-node0 consul[696]: 2021-12-16T09:33:17.527+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Register server=172.18.0.12:8300 error="rpc error makin
g call: No cluster leader"
Dec 16 09:33:17 prod-node0 consul[696]: 2021-12-16T09:33:17.527+0300 [WARN] agent: Syncing service failed.: service=_nomad-task-3d0da439-36c1-4e08-b428-b515ff2d58f1-gitea-job-gitea-l
b error="rpc error making call: No cluster leader"
Dec 16 09:33:17 prod-node0 consul[696]: 2021-12-16T09:33:17.527+0300 [ERROR] agent: failed to sync changes: error="rpc error making call: No cluster leader"
Dec 16 09:33:24 prod-node0 consul[696]: 2021-12-16T09:33:24.549+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Register server=172.18.0.12:8300 error="rpc error making call: No cluster leade
r"
Dec 16 09:33:24 prod-node0 consul[696]: 2021-12-16T09:33:24.549+0300 [WARN] agent: Syncing service failed.: service=_nomad-task-3d0da439-36c1-4e08-b428-b515ff2d58f1-gitea-job-gitea-lb error="rpc error makin
g call: No cluster leader"
Dec 16 09:33:24 prod-node0 consul[696]: 2021-12-16T09:33:24.549+0300 [ERROR] agent: failed to sync changes: error="rpc error making call: No cluster leader"
Dec 16 09:33:24 prod-node0 consul[696]: 2021-12-16T09:33:24.549+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Register server=172.18.0.12:8300 error="rpc error makin
g call: No cluster leader"
Dec 16 09:33:24 prod-node0 consul[696]: 2021-12-16T09:33:24.549+0300 [WARN] agent: Syncing service failed.: service=_nomad-task-3d0da439-36c1-4e08-b428-b515ff2d58f1-gitea-job-gitea-l
b error="rpc error making call: No cluster leader"
Dec 16 09:33:24 prod-node0 consul[696]: 2021-12-16T09:33:24.549+0300 [ERROR] agent: failed to sync changes: error="rpc error making call: No cluster leader"
Dec 16 09:33:30 prod-node0 consul[696]: 2021-12-16T09:33:30.837+0300 [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=172.18.0.12:8300 error="rpc error making call: No cluster lead
er"
Dec 16 09:33:30 prod-node0 consul[696]: 2021-12-16T09:33:30.837+0300 [ERROR] agent: Coordinate update error: error="rpc error making call: No cluster leader"
Dec 16 09:33:30 prod-node0 consul[696]: 2021-12-16T09:33:30.837+0300 [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=172.18.0.12:8300 error="rpc error maki
ng call: No cluster leader"
Dec 16 09:33:30 prod-node0 consul[696]: 2021-12-16T09:33:30.837+0300 [ERROR] agent: Coordinate update error: error="rpc error making call: No cluster leader"
Dec 16 09:33:32 prod-node0 consul[696]: 2021-12-16T09:33:32.166+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Deregister server=172.18.0.12:8300 error="rpc error making call: No cluster lea
der"
Dec 16 09:33:32 prod-node0 consul[696]: 2021-12-16T09:33:32.166+0300 [WARN] agent: Deregistering service failed.: service=_nomad-server-an5m2ytvifeudttvxqaikfncz63zzrja error="rpc error making call: No clus
ter leader"
Dec 16 09:33:32 prod-node0 consul[696]: 2021-12-16T09:33:32.166+0300 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error making call: No cluster leader"
Dec 16 09:33:32 prod-node0 consul[696]: 2021-12-16T09:33:32.166+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Deregister server=172.18.0.12:8300 error="rpc error mak
ing call: No cluster leader"
Dec 16 09:33:32 prod-node0 consul[696]: 2021-12-16T09:33:32.166+0300 [WARN] agent: Deregistering service failed.: service=_nomad-server-an5m2ytvifeudttvxqaikfncz63zzrja error="rpc er
ror making call: No cluster leader"
Dec 16 09:33:32 prod-node0 consul[696]: 2021-12-16T09:33:32.166+0300 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error making call: No cluster leader"
Dec 16 09:33:36 prod-node0 consul[696]: 2021-12-16T09:33:36.890+0300 [INFO] agent.client: New leader elected: payload=prod-master
Dec 16 09:33:36 prod-node0 consul[696]: 2021-12-16T09:33:36.890+0300 [INFO] agent.client: New leader elected: payload=prod-master
Dec 16 09:33:39 prod-node0 consul[696]: 2021-12-16T09:33:39.563+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Deregister server=172.18.0.12:8300 error="rpc error making call: node is not th
e leader"
Dec 16 09:33:39 prod-node0 consul[696]: 2021-12-16T09:33:39.564+0300 [WARN] agent: Deregistering service failed.: service=_nomad-server-fwzylv3q3e37q4emdznarxxnzyct7v3h error="rpc error making call: node is
not the leader"
Dec 16 09:33:39 prod-node0 consul[696]: 2021-12-16T09:33:39.564+0300 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error making call: node is not the leader"
Dec 16 09:33:39 prod-node0 consul[696]: 2021-12-16T09:33:39.563+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Deregister server=172.18.0.12:8300 error="rpc error mak
ing call: node is not the leader"
Dec 16 09:33:39 prod-node0 consul[696]: 2021-12-16T09:33:39.564+0300 [WARN] agent: Deregistering service failed.: service=_nomad-server-fwzylv3q3e37q4emdznarxxnzyct7v3h error="rpc er
ror making call: node is not the leader"
Dec 16 09:33:39 prod-node0 consul[696]: 2021-12-16T09:33:39.564+0300 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error making call: node is not the leader"
Dec 16 09:34:02 prod-node0 sshd[1273478]: Connection closed by 172.18.0.2 port 33414 [preauth]
Dec 16 09:34:04 prod-node0 consul[696]: 2021-12-16T09:34:04.897+0300 [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=172.18.0.12:8300 error="rpc error making call: No cluster lead
er"
Dec 16 09:34:04 prod-node0 consul[696]: 2021-12-16T09:34:04.897+0300 [ERROR] agent: Coordinate update error: error="rpc error making call: No cluster leader"
Dec 16 09:34:04 prod-node0 consul[696]: 2021-12-16T09:34:04.897+0300 [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=172.18.0.12:8300 error="rpc error maki
ng call: No cluster leader"
Dec 16 09:34:04 prod-node0 consul[696]: 2021-12-16T09:34:04.897+0300 [ERROR] agent: Coordinate update error: error="rpc error making call: No cluster leader"
Dec 16 09:34:05 prod-node0 consul[696]: 2021-12-16T09:34:05.953+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Register server=172.18.0.12:8300 error="rpc error making call: No cluster leade
r"
Dec 16 09:34:05 prod-node0 consul[696]: 2021-12-16T09:34:05.953+0300 [WARN] agent: Syncing service failed.: service=_nomad-task-3d0da439-36c1-4e08-b428-b515ff2d58f1-gitea-job-gitea-lb error="rpc error makin
g call: No cluster leader"
Dec 16 09:34:05 prod-node0 consul[696]: 2021-12-16T09:34:05.953+0300 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error making call: No cluster leader"
Dec 16 09:34:05 prod-node0 consul[696]: 2021-12-16T09:34:05.953+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Register server=172.18.0.12:8300 error="rpc error makin
g call: No cluster leader"
Dec 16 09:34:05 prod-node0 consul[696]: 2021-12-16T09:34:05.953+0300 [WARN] agent: Syncing service failed.: service=_nomad-task-3d0da439-36c1-4e08-b428-b515ff2d58f1-gitea-job-gitea-l
b error="rpc error making call: No cluster leader"
Dec 16 09:34:05 prod-node0 consul[696]: 2021-12-16T09:34:05.953+0300 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error making call: No cluster leader"
Dec 16 09:34:28 prod-node0 consul[696]: 2021-12-16T09:34:28.289+0300 [INFO] agent.client: New leader elected: payload=prod-master
Dec 16 09:34:28 prod-node0 consul[696]: 2021-12-16T09:34:28.289+0300 [INFO] agent.client: New leader elected: payload=prod-master
Dec 16 09:34:30 prod-node0 consul[696]: 2021-12-16T09:34:30.730+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Deregister server=172.18.0.12:8300 error="rpc error making call: node is not th
e leader"
Dec 16 09:34:30 prod-node0 consul[696]: 2021-12-16T09:34:30.730+0300 [WARN] agent: Deregistering service failed.: service=_nomad-server-hpgreurwpow55vxs7pqkpgekcvozygxs error="rpc error making call: node is
not the leader"
Dec 16 09:34:30 prod-node0 consul[696]: 2021-12-16T09:34:30.730+0300 [ERROR] agent: failed to sync changes: error="rpc error making call: node is not the leader"
Dec 16 09:34:30 prod-node0 consul[696]: 2021-12-16T09:34:30.730+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Deregister server=172.18.0.12:8300 error="rpc error mak
ing call: node is not the leader"
Dec 16 09:34:30 prod-node0 consul[696]: 2021-12-16T09:34:30.730+0300 [WARN] agent: Deregistering service failed.: service=_nomad-server-hpgreurwpow55vxs7pqkpgekcvozygxs error="rpc er
ror making call: node is not the leader"
Dec 16 09:34:30 prod-node0 consul[696]: 2021-12-16T09:34:30.730+0300 [ERROR] agent: failed to sync changes: error="rpc error making call: node is not the leader"
Dec 16 09:34:37 prod-node0 consul[696]: 2021-12-16T09:34:37.829+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Deregister server=172.18.0.12:8300 error="rpc error making call: No cluster lea
der"
Dec 16 09:34:37 prod-node0 consul[696]: 2021-12-16T09:34:37.829+0300 [WARN] agent: Deregistering service failed.: service=_nomad-server-hpgreurwpow55vxs7pqkpgekcvozygxs error="rpc error making call: No clus
ter leader"
Dec 16 09:34:37 prod-node0 consul[696]: 2021-12-16T09:34:37.829+0300 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error making call: No cluster leader"
Dec 16 09:34:37 prod-node0 consul[696]: 2021-12-16T09:34:37.829+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.Deregister server=172.18.0.12:8300 error="rpc error mak
ing call: No cluster leader"
Dec 16 09:34:37 prod-node0 consul[696]: 2021-12-16T09:34:37.829+0300 [WARN] agent: Deregistering service failed.: service=_nomad-server-hpgreurwpow55vxs7pqkpgekcvozygxs error="rpc er
ror making call: No cluster leader"
Dec 16 09:34:37 prod-node0 consul[696]: 2021-12-16T09:34:37.829+0300 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error making call: No cluster leader"
Peer.json:
["172.16.0.13:4648", "172.18.0.11:4648", "172.18.0.17:4648"]