Cluster does not longer startup, node services restarting immediately due to segmentation fault.
Nomad version: 1.11.1-1
OS: Linux master 6.12.63+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.63-1 (2025-12-30) x86_64 GNU/Linux (Debian Trixie)
This is a three-node cluster, both nodes failing instantly on the join of the second node. First node startup is fine, upon join of another node a segmentation fault is reported.
192.168.0.20 - first node, running
192.168.0.21 - second node, started after first node
192.168.0.22 - cluster member, not running
Any idea how to recover from this error?
Log messages:
Summary
Feb 06 16:33:52 master nomad[8819]: 2026-02-06T16:33:52.545-0500 [INFO] nomad: serf: EventMemberJoin: compute1.global 192.168.0.21
Feb 06 16:33:52 master nomad[8819]: 2026-02-06T16:33:52.546-0500 [INFO] nomad: adding server: name=compute1.global addr=192.168.0.21:4647 dc=home
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.078-0500 [ERROR] client.rpc: error performing RPC to server: error=“rpc error: failed to get conn: dial tcp 192.168.0.22:4647: connect: connection refused” rpc=Node.Register server=192.168.0.22:4647
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.078-0500 [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error=“rpc error: failed to get conn: dial tcp 192.168.0.22:4647: connect: connection refused” rpc=Node.Register server=192.168.0.22:4647
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.078-0500 [ERROR] client: error registering: error=“rpc error: failed to get conn: dial tcp 192.168.0.22:4647: connect: connection refused”
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.465-0500 [WARN] nomad.raft: Election timeout reached, restarting election
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.465-0500 [INFO] nomad.raft: entering candidate state: node=“Node at 192.168.0.20:4647 [Candidate]” term=53383
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.466-0500 [ERROR] nomad.raft: failed to make requestVote RPC: target=“{Voter cf113de4-b47a-90aa-0bb5-1ebaab4e115b 192.168.0.22:4647}” error=“dial tcp 192.168.0.22:4647: connect: connection refused” term=53383
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.467-0500 [INFO] nomad.raft: pre-vote successful, starting election: term=53383 tally=2 refused=1 votesNeeded=2
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.469-0500 [ERROR] nomad.raft: failed to make requestVote RPC: target=“{Voter cf113de4-b47a-90aa-0bb5-1ebaab4e115b 192.168.0.22:4647}” error=“dial tcp 192.168.0.22:4647: connect: connection refused” term=53383
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.483-0500 [INFO] nomad.raft: election won: term=53383 tally=2
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.484-0500 [INFO] nomad.raft: entering leader state: leader=“Node at 192.168.0.20:4647 [Leader]”
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.484-0500 [INFO] nomad.raft: added peer, starting replication: peer=da0080b4-5f97-d465-d128-884640c8be10
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.484-0500 [INFO] nomad.raft: added peer, starting replication: peer=cf113de4-b47a-90aa-0bb5-1ebaab4e115b
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.485-0500 [INFO] nomad: cluster leadership acquired
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.486-0500 [ERROR] nomad.raft: failed to appendEntries to: peer=“{Voter cf113de4-b47a-90aa-0bb5-1ebaab4e115b 192.168.0.22:4647}” error=“dial tcp 192.168.0.22:4647: connect: connection refused”
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.487-0500 [WARN] nomad.raft: appendEntries rejected, sending older logs: peer=“{Voter da0080b4-5f97-d465-d128-884640c8be10 192.168.0.21:4647}” next=2807585
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.492-0500 [INFO] nomad.raft: pipelining replication: peer=“{Voter da0080b4-5f97-d465-d128-884640c8be10 192.168.0.21:4647}”
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.518-0500 [ERROR] nomad.raft: failed to appendEntries to: peer=“{Voter cf113de4-b47a-90aa-0bb5-1ebaab4e115b 192.168.0.22:4647}” error=“dial tcp 192.168.0.22:4647: connect: connection refused”
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.579-0500 [ERROR] nomad.raft: failed to appendEntries to: peer=“{Voter cf113de4-b47a-90aa-0bb5-1ebaab4e115b 192.168.0.22:4647}” error=“dial tcp 192.168.0.22:4647: connect: connection refused”
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.722-0500 [ERROR] nomad.raft: failed to heartbeat to: peer=192.168.0.22:4647 backoff time=10ms error=“dial tcp 192.168.0.22:4647: connect: connection refused”
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.723-0500 [ERROR] nomad.raft: failed to appendEntries to: peer=“{Voter cf113de4-b47a-90aa-0bb5-1ebaab4e115b 192.168.0.22:4647}” error=“dial tcp 192.168.0.22:4647: connect: connection refused”
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.903-0500 [ERROR] nomad.raft: failed to appendEntries to: peer=“{Voter cf113de4-b47a-90aa-0bb5-1ebaab4e115b 192.168.0.22:4647}” error=“dial tcp 192.168.0.22:4647: connect: connection refused”
Feb 06 16:33:53 master nomad[8819]: 2026-02-06T16:33:53.988-0500 [WARN] nomad.raft: failed to contact: server-id=cf113de4-b47a-90aa-0bb5-1ebaab4e115b time=504.350242ms
Feb 06 16:33:54 master nomad[8819]: 2026-02-06T16:33:54.037-0500 [ERROR] nomad.raft: failed to heartbeat to: peer=192.168.0.22:4647 backoff time=10ms error=“dial tcp 192.168.0.22:4647: connect: connection refused”
Feb 06 16:33:54 master nomad[8819]: 2026-02-06T16:33:54.080-0500 [ERROR] nomad.raft: failed to appendEntries to: peer=“{Voter cf113de4-b47a-90aa-0bb5-1ebaab4e115b 192.168.0.22:4647}” error=“dial tcp 192.168.0.22:4647: connect: connection refused”
Feb 06 16:33:54 master nomad[8819]: 2026-02-06T16:33:54.084-0500 [INFO] nomad: eval broker status modified: paused=false
Feb 06 16:33:54 master nomad[8819]: 2026-02-06T16:33:54.084-0500 [INFO] nomad: blocked evals status modified: paused=false
Feb 06 16:33:54 master nomad[8819]: 2026-02-06T16:33:54.241-0500 [ERROR] nomad.raft: failed to heartbeat to: peer=192.168.0.22:4647 backoff time=10ms error=“dial tcp 192.168.0.22:4647: connect: connection refused”
Feb 06 16:33:54 master nomad[8819]: 2026-02-06T16:33:54.341-0500 [ERROR] nomad.raft: failed to appendEntries to: peer=“{Voter cf113de4-b47a-90aa-0bb5-1ebaab4e115b 192.168.0.22:4647}” error=“dial tcp 192.168.0.22:4647: connect: connection refused”
Feb 06 16:33:54 master nomad[8819]: 2026-02-06T16:33:54.361-0500 [ERROR] nomad.raft: failed to heartbeat to: peer=192.168.0.22:4647 backoff time=20ms error=“dial tcp 192.168.0.22:4647: connect: connection refused”
Feb 06 16:33:54 master nomad[8819]: 2026-02-06T16:33:54.405-0500 [WARN] nomad.raft: failed to contact: server-id=cf113de4-b47a-90aa-0bb5-1ebaab4e115b time=921.073821ms
Feb 06 16:33:54 master nomad[8819]: panic: runtime error: invalid memory address or nil pointer dereference
Feb 06 16:33:54 master nomad[8819]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x193b951]
Feb 06 16:33:54 master nomad[8819]: goroutine 772 [running]:
Feb 06 16:33:54 master nomad[8819]: github.com/hashicorp/nomad/nomad/structs.(*DeploymentState).Copy(…)
Feb 06 16:33:54 master nomad[8819]: github.com/hashicorp/nomad/nomad/structs/deployment.go:307
Feb 06 16:33:54 master nomad[8819]: github.com/hashicorp/nomad/nomad/structs.(*Deployment).Copy(0xc002ba8160)
Feb 06 16:33:54 master nomad[8819]: github.com/hashicorp/nomad/nomad/structs/deployment.go:153 +0x1b1
Feb 06 16:33:54 master nomad[8819]: github.com/hashicorp/nomad/nomad.evaluatePlanPlacements(0xc0018a6fc0, 0xc001800f30, 0xc001bd4990, {0x43ab910, 0xc000c2e720})
Feb 06 16:33:54 master nomad[8819]: github.com/hashicorp/nomad/nomad/plan_apply.go:479 +0x70
Feb 06 16:33:54 master nomad[8819]: github.com/hashicorp/nomad/nomad.evaluatePlan(0xc0018a6fc0, 0xc001800f30, 0xc001bd4990, {0x43ab910, 0xc000c2e720})
Feb 06 16:33:54 master nomad[8819]: github.com/hashicorp/nomad/nomad/plan_apply.go:468 +0x274
Feb 06 16:33:54 master nomad[8819]: github.com/hashicorp/nomad/nomad.(*planner).planApply(0xc000c23640)
Feb 06 16:33:54 master nomad[8819]: github.com/hashicorp/nomad/nomad/plan_apply.go:165 +0x3bf
Feb 06 16:33:54 master nomad[8819]: created by github.com/hashicorp/nomad/nomad.(*Server).establishLeadership in goroutine 716
Feb 06 16:33:54 master nomad[8819]: github.com/hashicorp/nomad/nomad/leader.go:407 +0x17f
Feb 06 16:33:54 master systemd[1]: nomad.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 06 16:33:54 master systemd[1]: nomad.service: Failed with result ‘exit-code’.