Hi, I have an environment where I have 1 Nomad Server and 8 Nomad Clients. Nomad Server is running on Linux and Clients are Windows machines. I have nomad services which are executed raw_exec. All apps are in .NET and they are working on Windows Clients. I have an issue with Nomad Server. Currently I’m using the latest version 1.10.5. Nomad server is not starting. Below I change public rpc address and I attached the log from start:
2025-10-10T06:08:37.133396+00:00 systemd[1]: Starting nomad.service - Nomad…
2025-10-10T06:08:37.181987+00:00 nomad[4141]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
2025-10-10T06:08:37.182194+00:00 nomad[4141]: ==> Loaded configuration from /etc/nomad.d/nomad.hcl
2025-10-10T06:08:37.182234+00:00 nomad[4141]: ==> Starting Nomad agent…
2025-10-10T06:08:37.541924+00:00 nomad[4141]: ==> Nomad agent configuration:
2025-10-10T06:08:37.542026+00:00 nomad[4141]: Advertise Addrs: HTTP: 10.0.0.19:4646; RPC: X.X.X.X:4647; Serf: 10.0.0.19:4648
2025-10-10T06:08:37.542049+00:00 nomad[4141]: Bind Addrs: HTTP: [0.0.0.0:4646]; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
2025-10-10T06:08:37.542068+00:00 nomad[4141]: Client: false
2025-10-10T06:08:37.542085+00:00 nomad[4141]: Log Level: DEBUG
2025-10-10T06:08:37.542103+00:00 nomad[4141]: Node Id: e1de4d5c-7e61-facf-e546-310db1761e8c
2025-10-10T06:08:37.542137+00:00 nomad[4141]: Region: global (DC: dc1)
2025-10-10T06:08:37.542155+00:00 nomad[4141]: Server: true
2025-10-10T06:08:37.542183+00:00 nomad[4141]: Version: 1.10.5
2025-10-10T06:08:37.542201+00:00 nomad[4141]: ==> Nomad agent started! Log data will stream in below:
2025-10-10T06:08:37.542220+00:00 nomad[4141]: 2025-10-10T06:08:37.183Z [DEBUG] nomad: issuer not set; OIDC Discovery endpoint for workload identities disabled
2025-10-10T06:08:37.542244+00:00 nomad[4141]: 2025-10-10T06:08:37.193Z [INFO] nomad: setting up raft bolt store: no_freelist_sync=false
2025-10-10T06:08:37.542267+00:00 nomad[4141]: 2025-10-10T06:08:37.199Z [INFO] nomad.raft: starting restore from snapshot: id=2-32791-1759890807932 last-index=32791 last-term=2 size-in-bytes=1001033
2025-10-10T06:08:37.542289+00:00 nomad[4141]: 2025-10-10T06:08:37.234Z [INFO] nomad.raft: snapshot restore progress: id=2-32791-1759890807932 last-index=32791 last-term=2 size-in-bytes=1001033 read-bytes=1001033 percent-complete=“100.00%”
2025-10-10T06:08:37.542312+00:00 nomad[4141]: 2025-10-10T06:08:37.234Z [INFO] nomad.raft: restored from snapshot: id=2-32791-1759890807932 last-index=32791 last-term=2 size-in-bytes=1001033
2025-10-10T06:08:37.542338+00:00 nomad[4141]: 2025-10-10T06:08:37.538Z [INFO] nomad.raft: initial configuration: index=1 servers=“[{Suffrage:Voter ID:f082e3a9-2829-0ec1-d53d-6c089a6eb166 Address:10.0.0.19:4647}]”
2025-10-10T06:08:37.542361+00:00 nomad[4141]: 2025-10-10T06:08:37.539Z [INFO] nomad.raft: entering follower state: follower=“Node at 10.0.0.19:4647 [Follower]” leader-address= leader-id=
2025-10-10T06:08:37.542383+00:00 nomad[4141]: 2025-10-10T06:08:37.539Z [INFO] nomad: serf: EventMemberJoin: .global 10.0.0.19
2025-10-10T06:08:37.542404+00:00 nomad[4141]: 2025-10-10T06:08:37.539Z [INFO] nomad: starting scheduling worker(s): num_workers=2 schedulers=[“sysbatch”, “service”, “batch”, “system”, “_core”]
2025-10-10T06:08:37.542435+00:00 nomad[4141]: 2025-10-10T06:08:37.539Z [DEBUG] nomad: started scheduling worker: id=11d16c65-0f3d-fab6-a5be-4a469b642dc9 index=1 of=2
2025-10-10T06:08:37.542457+00:00 nomad[4141]: 2025-10-10T06:08:37.539Z [DEBUG] nomad: started scheduling worker: id=4ab9515c-a8d5-571c-ab0b-03f275f88afb index=2 of=2
2025-10-10T06:08:37.542501+00:00 nomad[4141]: 2025-10-10T06:08:37.539Z [INFO] nomad: started scheduling worker(s): num_workers=2 schedulers=[“sysbatch”, “service”, “batch”, “system”, “_core”]
2025-10-10T06:08:37.543137+00:00 nomad[4141]: 2025-10-10T06:08:37.539Z [WARN] nomad: serf: Failed to re-join any previously known node
2025-10-10T06:08:37.543171+00:00 nomad[4141]: 2025-10-10T06:08:37.539Z [WARN] agent: not registering Nomad HTTPS Health Check because verify_https_client enabled
2025-10-10T06:08:37.543214+00:00 nomad[4141]: 2025-10-10T06:08:37.539Z [DEBUG] worker: running: worker_id=11d16c65-0f3d-fab6-a5be-4a469b642dc9
2025-10-10T06:08:37.543236+00:00 nomad[4141]: 2025-10-10T06:08:37.540Z [DEBUG] worker: running: worker_id=4ab9515c-a8d5-571c-ab0b-03f275f88afb
2025-10-10T06:08:37.543259+00:00 nomad[4141]: 2025-10-10T06:08:37.540Z [INFO] nomad: adding server: server=“.global (Addr: 10.0.0.19:4647) (DC: dc1)”
2025-10-10T06:08:37.543282+00:00 nomad[4141]: 2025-10-10T06:08:37.541Z [DEBUG] http: UI is enabled
2025-10-10T06:08:37.543301+00:00 nomad[4141]: 2025-10-10T06:08:37.541Z [DEBUG] nomad.keyring.replicator: starting encryption key replication
2025-10-10T06:08:37.543347+00:00 systemd[1]: Started nomad.service - Nomad.
2025-10-10T06:08:39.205810+00:00 nomad[4141]: 2025-10-10T06:08:39.205Z [WARN] nomad.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-10T06:08:39.205941+00:00 nomad[4141]: 2025-10-10T06:08:39.205Z [INFO] nomad.raft: entering candidate state: node=“Node at 10.0.0.19:4647 [Candidate]” term=22235
2025-10-10T06:08:39.205991+00:00 nomad[4141]: 2025-10-10T06:08:39.205Z [DEBUG] nomad.raft: pre-voting for self: term=22235 id=f082e3a9-2829-0ec1-d53d-6c089a6eb166
2025-10-10T06:08:39.206021+00:00 nomad[4141]: 2025-10-10T06:08:39.205Z [DEBUG] nomad.raft: calculated votes needed: needed=1 term=22235
2025-10-10T06:08:39.206043+00:00 nomad[4141]: 2025-10-10T06:08:39.205Z [DEBUG] nomad.raft: pre-vote received: from=f082e3a9-2829-0ec1-d53d-6c089a6eb166 term=22235 tally=0
2025-10-10T06:08:39.206063+00:00 nomad[4141]: 2025-10-10T06:08:39.205Z [DEBUG] nomad.raft: pre-vote granted: from=f082e3a9-2829-0ec1-d53d-6c089a6eb166 term=22235 tally=1
2025-10-10T06:08:39.206083+00:00 nomad[4141]: 2025-10-10T06:08:39.205Z [INFO] nomad.raft: pre-vote successful, starting election: term=22235 tally=1 refused=0 votesNeeded=1
2025-10-10T06:08:39.229035+00:00 nomad[4141]: 2025-10-10T06:08:39.228Z [DEBUG] nomad.raft: voting for self: term=22235 id=f082e3a9-2829-0ec1-d53d-6c089a6eb166
2025-10-10T06:08:39.322060+00:00 nomad[4141]: 2025-10-10T06:08:39.321Z [DEBUG] nomad.raft: vote granted: from=f082e3a9-2829-0ec1-d53d-6c089a6eb166 term=22235 tally=1
2025-10-10T06:08:39.322175+00:00 nomad[4141]: 2025-10-10T06:08:39.322Z [INFO] nomad.raft: election won: term=22235 tally=1
2025-10-10T06:08:39.322367+00:00 nomad[4141]: 2025-10-10T06:08:39.322Z [INFO] nomad.raft: entering leader state: leader=“Node at 10.0.0.19:4647 [Leader]”
2025-10-10T06:08:39.322692+00:00 nomad[4141]: 2025-10-10T06:08:39.322Z [INFO] nomad: cluster leadership acquired
2025-10-10T06:08:39.394061+00:00 nomad[4141]: panic: runtime error: invalid memory address or nil pointer dereference
2025-10-10T06:08:39.394180+00:00 nomad[4141]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x98 pc=0x179a7d2]
2025-10-10T06:08:39.394219+00:00 nomad[4141]: goroutine 30 [running]:
2025-10-10T06:08:39.394415+00:00 nomad[4141]: ``github.com/hashicorp/nomad/nomad/structs.(*Evaluation).ShouldEnqueue(0xc000c95c20?)
2025-10-10T06:08:39.394447+00:00 nomad[4141]: #``011github.com/hashicorp/nomad/nomad/structs/structs.go:12621`` +0x12
2025-10-10T06:08:39.394468+00:00 nomad[4141]: ``github.com/hashicorp/nomad/nomad.(*nomadFSM).applyAllocClientUpdate``(0xc0007f5340, 0x9, {0xc0015c1001, 0x2f16, 0x2f16}, 0x8318)
2025-10-10T06:08:39.394495+00:00 nomad[4141]: #``011github.com/hashicorp/nomad/nomad/fsm.go:1027`` +0x8a5
2025-10-10T06:08:39.394693+00:00 nomad[4141]: ``github.com/hashicorp/nomad/nomad.(*nomadFSM).Apply``(0x1430690?, 0xc0015ca000)
2025-10-10T06:08:39.394760+00:00 nomad[4141]: #``011github.com/hashicorp/nomad/nomad/fsm.go:286`` +0x20b
2025-10-10T06:08:39.394796+00:00 nomad[4141]: ``github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc00158ef60)
2025-10-10T06:08:39.394833+00:00 nomad[4141]: #``011github.com/hashicorp/raft@v1.7.3/fsm.go:107`` +0x1fe
2025-10-10T06:08:39.394890+00:00 nomad[4141]: ``github.com/hashicorp/raft.(*Raft).runFSM.func2(...)
2025-10-10T06:08:39.394940+00:00 nomad[4141]: #``011github.com/hashicorp/raft@v1.7.3/fsm.go:130
2025-10-10T06:08:39.394974+00:00 nomad[4141]: ``github.com/hashicorp/raft.(*Raft).runFSM(0xc00079adc8)
2025-10-10T06:08:39.395017+00:00 nomad[4141]: #``011github.com/hashicorp/raft@v1.7.3/fsm.go:246`` +0x9cd
2025-10-10T06:08:39.395071+00:00 nomad[4141]: ``github.com/hashicorp/raft.(*raftState).goFunc.func1()
2025-10-10T06:08:39.395112+00:00 nomad[4141]: #``011github.com/hashicorp/raft@v1.7.3/state.go:149`` +0x50
2025-10-10T06:08:39.395149+00:00 nomad[4141]: created by ``github.com/hashicorp/raft.(*raftState).goFunc`` in goroutine 1
2025-10-10T06:08:39.395199+00:00 nomad[4141]: #``011github.com/hashicorp/raft@v1.7.3/state.go:147`` +0x76
2025-10-10T06:08:39.398376+00:00 systemd[1]: nomad.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
2025-10-10T06:08:39.398469+00:00 systemd[1]: nomad.service: Failed with result ‘exit-code’.
I’s the second time when it appears. Previous time around 2-3 weeks ago I solved it by removing server data directory but I had to recreate tokens and policies restart nomad service on all clients to rejoin etc. I wonder if there is a better solution to this problem 
I will be very grateful for any help.