Nomad won't start

Nomad
1

Hi,

I have a simple 1 host with nomad running as client and server, for some reason, it won’t start and I can’t understand the logs, could someone please advice on what could go wrong?

Aug 31 19:41:21 nid001311 systemd[1]: Reloading nomad.service.
Aug 31 19:41:21 nid001311 systemd[1]: Reloaded nomad.service.
Aug 31 19:41:21 nid001311 nomad[75579]: ==> Caught signal: hangup
Aug 31 19:41:21 nid001311 nomad[75579]: ==> Reloading configuration...
Aug 31 19:41:21 nid001311 nomad[75579]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.725+0200 [DEBUG] agent: starting reload of agent config
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.726+0200 [DEBUG] agent: starting reload of server config
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.729+0200 [INFO]  nomad: reloading server connections due to configuration changes
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.730+0200 [ERROR] nomad.rpc: failed to accept RPC conn: error="accept tcp [::]:4647: use of closed network connection" delay=5ms
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.730+0200 [INFO]  nomad.rpc: closing server RPC connection
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.730+0200 [DEBUG] nomad: finished reloading server connections
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.730+0200 [DEBUG] agent: starting reload of client config
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.731+0200 [INFO]  client.fingerprint_mgr: reloading fingerprinter: fingerprinter=cni
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.731+0200 [DEBUG] client.fingerprint_mgr: CNI config dir is not set or does not exist, skipping: cni_config_dir=/opt/cni/config
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.731+0200 [INFO]  agent: reloading HTTP server with new TLS configuration
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.731+0200 [DEBUG] http: shutting down http server
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.731+0200 [ERROR] client.rpc: error performing RPC to server: error="rpc error: EOF" rpc=Node.GetClientAllocs server=148.187.113.147:4647
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.732+0200 [DEBUG] http: UI is enabled
Aug 31 19:41:21 nid001311 nomad[75579]:     2023-08-31T19:41:21.732+0200 [DEBUG] http: UI is enabled
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:1: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:2: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:3: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:4: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:38: Unknown key name 'StartLimitIntervalSec' in section 'Service', ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:1: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:2: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:3: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:4: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:1: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:2: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:3: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:4: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:1: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:2: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:3: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: /etc/systemd/system/nomad.service:4: Assignment outside of section. Ignoring.
Aug 31 19:41:22 nid001311 systemd[1]: Stopping nomad.service...
Aug 31 19:41:22 nid001311 nomad[75579]: ==> Caught signal: interrupt
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.365+0200 [INFO]  agent: requesting shutdown
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.365+0200 [INFO]  client: shutting down
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.365+0200 [INFO]  client.plugin: shutting down plugin manager: plugin-type=device
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.365+0200 [INFO]  client.plugin: plugin manager finished: plugin-type=device
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.365+0200 [INFO]  client.plugin: shutting down plugin manager: plugin-type=driver
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.385+0200 [INFO]  client.plugin: plugin manager finished: plugin-type=driver
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.385+0200 [INFO]  client.plugin: shutting down plugin manager: plugin-type=csi
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.385+0200 [INFO]  client.plugin: plugin manager finished: plugin-type=csi
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.385+0200 [DEBUG] client.server_mgr: shutting down
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.386+0200 [INFO]  nomad: shutting down server
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.386+0200 [DEBUG] nomad.autopilot: state update routine is now stopped
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.386+0200 [DEBUG] nomad: shutting down leader loop
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.386+0200 [DEBUG] nomad.autopilot: autopilot is now stopped
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.386+0200 [WARN]  nomad: serf: Shutdown without a Leave
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.386+0200 [DEBUG] worker: stopped: worker_id=e199740c-3844-ab10-8957-212548540e55
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.386+0200 [DEBUG] worker: stopped: worker_id=7502e55d-514a-69d2-cc99-d70b0213bc72
...
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.386+0200 [DEBUG] worker: stopped: worker_id=a519cb66-5f0e-8006-9321-ca1a7934bc8d
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.386+0200 [DEBUG] worker: stopped: worker_id=09635d36-468e-6e26-f5ee-9f50b46513c4
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.387+0200 [INFO]  agent: shutdown complete
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.387+0200 [DEBUG] worker: stopped: worker_id=a4cd9f47-3e0a-c3c0-7f80-d9abaccc20a5
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.387+0200 [DEBUG] worker: stopped: worker_id=3fc8328f-082f-26e4-a668-1810a766340d
...
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.387+0200 [DEBUG] worker: stopped: worker_id=dd0446c8-534c-d724-0497-3eab5360ffc2
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.387+0200 [DEBUG] worker: stopped: worker_id=8a0c1828-9b51-575c-417a-32f9c510c93c
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.386+0200 [INFO]  nomad: cluster leadership lost
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.386+0200 [DEBUG] worker: stopped: worker_id=9dfecaa4-802c-2778-4eb1-03a8e8910120
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.386+0200 [DEBUG] worker: stopped: worker_id=d92a5692-222c-91d8-ce44-57a0d3300e3e
...
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.387+0200 [DEBUG] worker: stopped: worker_id=6b5e8804-4a5b-1551-8669-7e70b2357b4b
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.387+0200 [DEBUG] http: shutting down http server
Aug 31 19:41:22 nid001311 nomad[75579]:     2023-08-31T19:41:22.387+0200 [DEBUG] worker: stopped: worker_id=d21dfb0b-a730-24fe-0a64-4d5d0c834c5a
Aug 31 19:41:22 nid001311 systemd[1]: nomad.service: Main process exited, code=exited, status=1/FAILURE
Aug 31 19:41:22 nid001311 systemd[1]: nomad.service: Failed with result 'exit-code'.
Aug 31 19:41:22 nid001311 systemd[1]: Stopped nomad.service.
Aug 31 19:41:22 nid001311 systemd[1]: Started nomad.service.
Aug 31 19:41:22 nid001311 nomad[81789]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Aug 31 19:41:22 nid001311 nomad[81789]: ==> Loaded configuration from /etc/nomad.d/client.hcl, /etc/nomad.d/nomad.hcl, /etc/nomad.d/server.hcl
Aug 31 19:41:22 nid001311 nomad[81789]: ==> Starting Nomad agent...
Aug 31 19:41:23 nid001311 systemd[1]: Reloading nomad.service.
Aug 31 19:41:23 nid001311 systemd[1]: Reloaded nomad.service.
Aug 31 19:41:23 nid001311 systemd[1]: nomad.service: Succeeded.

thank you

2

Hi @masuberu,

Do you have the Nomad agent logs available from disk to read through?

Thanks,
jrasell and the Nomad team

3

Hi, I run Nomad through systemd, are you referring to all the journalctld logs or something else?

$ journalctl -u nomad | cat
-- Logs begin at Thu 2023-08-31 18:49:59 CEST, end at Fri 2023-09-01 20:14:01 CEST. --
...
Sep 01 18:49:35 nid001311 systemd[1]: Started nomad.service.
Sep 01 18:49:35 nid001311 nomad[120103]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Sep 01 18:49:35 nid001311 nomad[120103]: ==> Loaded configuration from /etc/nomad.d/client.hcl, /etc/nomad.d/nomad.hcl, /etc/nomad.d/server.hcl
Sep 01 18:49:35 nid001311 nomad[120103]: ==> Starting Nomad agent...
Sep 01 18:49:45 nid001311 nomad[120103]: ==> Nomad agent configuration:
Sep 01 18:49:45 nid001311 nomad[120103]:        Advertise Addrs: HTTP: nid001311:4646; RPC: nid001311:4647; Serf: nid001311:4648
Sep 01 18:49:45 nid001311 nomad[120103]:             Bind Addrs: HTTP: [0.0.0.0:4646]; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
Sep 01 18:49:45 nid001311 nomad[120103]:                 Client: true
Sep 01 18:49:45 nid001311 nomad[120103]:              Log Level: DEBUG
Sep 01 18:49:45 nid001311 nomad[120103]:                Node Id: 91bec387-3b7e-98e7-a498-b82822d31153
Sep 01 18:49:45 nid001311 nomad[120103]:                 Region: global (DC: moleson-tds)
Sep 01 18:49:45 nid001311 nomad[120103]:                 Server: true
Sep 01 18:49:45 nid001311 nomad[120103]:                Version: 1.6.1
Sep 01 18:49:45 nid001311 nomad[120103]: ==> Nomad agent started! Log data will stream in below:
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.734+0200 [INFO]  nomad: setting up raft bolt store: no_freelist_sync=false
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.738+0200 [INFO]  nomad.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:12c9a06b-8fd9-6d07-646c-74b2b15b3dcf Address:148.187.113.147:4647}]"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.738+0200 [INFO]  nomad.raft: entering follower state: follower="Node at 148.187.113.147:4647 [Follower]" leader-address= leader-id=
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.738+0200 [WARN]  nomad: memberlist: Binding to public address without encryption!
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.738+0200 [INFO]  nomad: serf: EventMemberJoin: nid001311.global 148.187.113.147
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.738+0200 [INFO]  nomad: starting scheduling worker(s): num_workers=256 schedulers=["service", "batch", "system", "sysbatch", "_core"]
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.738+0200 [DEBUG] nomad: started scheduling worker: id=b69dc503-13f5-bb6a-d5c6-7ca8a84884a4 index=1 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.738+0200 [DEBUG] nomad: started scheduling worker: id=d95f6bf5-4cf3-7551-a93a-b44b5622593d index=2 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] nomad: started scheduling worker: id=e29c12bc-503f-1cbe-e858-875d383fa819 index=3 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] nomad: started scheduling worker: id=096ddd50-cc04-f25e-0041-5d0e8a3c7426 index=4 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] nomad: started scheduling worker: id=791806b4-1f63-b304-4cf3-ccd39a9fa053 index=5 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] nomad: started scheduling worker: id=c5e836f8-1367-97d2-8a91-3393ef3c5a1e index=6 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] nomad: started scheduling worker: id=54f5cb31-4c51-1d9d-7037-b72d0d2495d5 index=7 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] nomad: started scheduling worker: id=0636a429-8a09-57d9-d38f-d3e67170cb38 index=8 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [WARN]  nomad: serf: Failed to re-join any previously known node
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] nomad: started scheduling worker: id=034ee4e8-fa17-5a81-b460-883a83674984 index=9 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] worker: running: worker_id=e29c12bc-503f-1cbe-e858-875d383fa819
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] worker: running: worker_id=d95f6bf5-4cf3-7551-a93a-b44b5622593d
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] nomad: started scheduling worker: id=e6ac96d2-e058-2779-6188-3a691c46d896 index=10 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] nomad: started scheduling worker: id=4bd5a9f1-6609-375d-8a7d-aae2367c8ba7 index=11 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] nomad: started scheduling worker: id=3583de32-357f-bc5d-359d-6f667a9b343d index=12 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] worker: running: worker_id=c5e836f8-1367-97d2-8a91-3393ef3c5a1e
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] worker: running: worker_id=3583de32-357f-bc5d-359d-6f667a9b343d
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] worker: running: worker_id=e6ac96d2-e058-2779-6188-3a691c46d896
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] worker: running: worker_id=096ddd50-cc04-f25e-0041-5d0e8a3c7426
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] worker: running: worker_id=54f5cb31-4c51-1d9d-7037-b72d0d2495d5
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] worker: running: worker_id=0636a429-8a09-57d9-d38f-d3e67170cb38
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] worker: running: worker_id=034ee4e8-fa17-5a81-b460-883a83674984
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.739+0200 [DEBUG] worker: running: worker_id=b69dc503-13f5-bb6a-d5c6-7ca8a84884a4
...
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [DEBUG] nomad: started scheduling worker: id=6ed92d97-e463-6070-4aeb-a2620d8351de index=253 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [DEBUG] worker: running: worker_id=6ed92d97-e463-6070-4aeb-a2620d8351de
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [DEBUG] nomad: started scheduling worker: id=aca7e6ef-bd67-f633-398a-64d727e1ff7b index=254 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [DEBUG] nomad: started scheduling worker: id=ee403a3b-6b97-b44b-c7f4-f761cde15c18 index=255 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [DEBUG] worker: running: worker_id=aca7e6ef-bd67-f633-398a-64d727e1ff7b
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [DEBUG] nomad: started scheduling worker: id=55ce4802-1319-4ebc-a6a6-28c65621d1da index=256 of=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [INFO]  nomad: started scheduling worker(s): num_workers=256 schedulers=["service", "batch", "system", "sysbatch", "_core"]
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [DEBUG] worker: running: worker_id=ee403a3b-6b97-b44b-c7f4-f761cde15c18
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [DEBUG] worker: running: worker_id=55ce4802-1319-4ebc-a6a6-28c65621d1da
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [WARN]  agent: not registering Nomad HTTPS Health Check because verify_https_client enabled
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [DEBUG] nomad.keyring.replicator: starting encryption key replication
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [INFO]  nomad: adding server: server="nid001311.global (Addr: 148.187.113.147:4647) (DC: moleson-tds)"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.748+0200 [WARN]  agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/scratch/shared/nomad_server/nid001311/plugins
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.749+0200 [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/scratch/shared/nomad_server/nid001311/plugins
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.750+0200 [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.750+0200 [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.750+0200 [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.750+0200 [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.750+0200 [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.778+0200 [INFO]  client: using state directory: state_dir=/scratch/shared/nomad_server/nid001311/client
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.779+0200 [INFO]  client: using alloc directory: alloc_dir=/scratch/shared/nomad_server/nid001311/alloc
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.779+0200 [INFO]  client: using dynamic ports: min=20000 max=32000 reserved=""
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.779+0200 [INFO]  client.cpuset.v1: initialized cpuset cgroup manager: parent=/nomad cpuset=0-255
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.828+0200 [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters=["arch", "bridge", "cgroup", "cni", "consul", "cpu", "host", "landlock", "memory", "network", "nomad", "plugins_cni", "signal", "storage", "vault", "env_aws", "env_gce", "env_azure", "env_digitalocean"]
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.834+0200 [INFO]  client.fingerprint_mgr.cgroup: cgroups are available
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.834+0200 [DEBUG] client.fingerprint_mgr: CNI config dir is not set or does not exist, skipping: cni_config_dir=/opt/cni/config
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.834+0200 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup initial_period=15s
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.835+0200 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul initial_period=15s
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.864+0200 [DEBUG] client.fingerprint_mgr.cpu: detected CPU model: name="AMD EPYC 7742 64-Core Processor"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.864+0200 [DEBUG] client.fingerprint_mgr.cpu: detected CPU frequency: mhz=2250
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.864+0200 [DEBUG] client.fingerprint_mgr.cpu: detected CPU core count: EXTRA_VALUE_AT_END=256
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.864+0200 [DEBUG] client.fingerprint_mgr.cpu: client configuration reserves these cores for node: cores=[]
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.864+0200 [DEBUG] client.fingerprint_mgr.cpu: set of reservable cores available for tasks: cores=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255]
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.871+0200 [WARN]  client.fingerprint_mgr.landlock: failed to fingerprint kernel landlock feature: error="function not implemented"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.876+0200 [DEBUG] client.fingerprint_mgr.network: link speed detected: interface=hsn0 mbits=200000
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.877+0200 [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=hsn0 IP=148.187.113.147
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.879+0200 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=lo
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.879+0200 [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/lo/speed device=lo
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.879+0200 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=lo mbits=1000
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.887+0200 [WARN]  client.fingerprint_mgr.cni_plugins: failed to read CNI plugins directory: cni_path=/opt/cni/bin error="open /opt/cni/bin: no such file or directory"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:35.888+0200 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault initial_period=15s
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:37.212+0200 [WARN]  nomad.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:37.212+0200 [INFO]  nomad.raft: entering candidate state: node="Node at 148.187.113.147:4647 [Candidate]" term=4
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:37.215+0200 [DEBUG] nomad.raft: voting for self: term=4 id=12c9a06b-8fd9-6d07-646c-74b2b15b3dcf
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:37.217+0200 [DEBUG] nomad.raft: calculated votes needed: needed=1 term=4
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:37.217+0200 [DEBUG] nomad.raft: vote granted: from=12c9a06b-8fd9-6d07-646c-74b2b15b3dcf term=4 tally=1
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:37.217+0200 [INFO]  nomad.raft: election won: term=4 tally=1
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:37.217+0200 [INFO]  nomad.raft: entering leader state: leader="Node at 148.187.113.147:4647 [Leader]"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:37.218+0200 [INFO]  nomad: cluster leadership acquired
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:37.228+0200 [INFO]  nomad: eval broker status modified: paused=false
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:37.228+0200 [DEBUG] nomad.autopilot: autopilot is now running
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:37.228+0200 [DEBUG] nomad.autopilot: state update routine is now running
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:37.228+0200 [INFO]  nomad: blocked evals status modified: paused=false
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:41.891+0200 [DEBUG] client.fingerprint_mgr.env_gce: could not read value for attribute: attribute=machine-type error="Get \"http://169.254.169.254/computeMetadata/v1/instance/machine-type\": dial tcp 169.254.169.254:80: i/o timeout (Client.Timeout exceeded while awaiting headers)"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:41.891+0200 [DEBUG] client.fingerprint_mgr.env_gce: error querying GCE Metadata URL, skipping
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:43.891+0200 [DEBUG] client.fingerprint_mgr.env_azure: could not read value for attribute: attribute=compute/azEnvironment error="Get \"http://169.254.169.254/metadata/instance/compute/azEnvironment?api-version=2019-06-04&format=text\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [DEBUG] client.fingerprint_mgr.env_digitalocean: failed to request metadata: attribute=region error="Get \"http://169.254.169.254/metadata/v1/region\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs=["arch", "bridge", "cgroup", "cpu", "host", "network", "nomad", "signal", "storage"]
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [INFO]  client.plugin: starting plugin manager: plugin-type=csi
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [INFO]  client.plugin: starting plugin manager: plugin-type=driver
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [INFO]  client.plugin: starting plugin manager: plugin-type=device
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [DEBUG] client.device_mgr: exiting since there are no device plugins
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=raw_exec health=healthy description=Healthy
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=driver
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=device
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=device
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [DEBUG] client.driver_mgr.docker: using client connection initialized from environment: driver=docker
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=qemu health=undetected description=""
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=exec health=healthy description=Healthy
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [ERROR] client.driver_mgr.docker: failed to list pause containers for recovery: driver=docker error="Get \"http://unix.sock/containers/json?filters=%7B%22label%22%3A%5B%22com.hashicorp.nomad.alloc_id%22%5D%7D\": dial unix /var/run/docker.sock: connect: no such file or directory"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [DEBUG] client.driver_mgr.docker: could not connect to docker daemon: driver=docker endpoint=unix:///var/run/docker.sock error="Get \"http://unix.sock/version\": dial unix /var/run/docker.sock: connect: no such file or directory"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.892+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=docker health=undetected description="Failed to connect to docker daemon"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.900+0200 [DEBUG] client.server_mgr: new server list: new_servers=[0.0.0.0:4647, 148.187.113.147:4647, 148.187.113.147:4647] old_servers=[]
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.975+0200 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=java health=healthy description=Healthy
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.975+0200 [DEBUG] client.driver_mgr: detected drivers: drivers="map[healthy:[raw_exec exec java] undetected:[qemu docker]]"
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.975+0200 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=driver
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.975+0200 [INFO]  client: started client: node_id=5ebd2c38-6fd4-5f12-a569-df0f9b03bcf5
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.975+0200 [WARN]  agent: not registering Nomad HTTPS Health Check because verify_https_client enabled
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.975+0200 [DEBUG] client: updated allocations: index=1 total=0 pulled=0 filtered=0
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.975+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=0
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.976+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=0 errors=0
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.976+0200 [DEBUG] http: UI is enabled
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.976+0200 [DEBUG] http: UI is enabled
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.980+0200 [INFO]  agent.joiner: starting retry join: servers=nid001311
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.981+0200 [DEBUG] nomad: memberlist: Initiating push/pull sync with:  148.187.113.147:4648
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.981+0200 [DEBUG] nomad: memberlist: Stream connection from=148.187.113.147:44032
Sep 01 18:49:45 nid001311 nomad[120103]:     2023-09-01T18:49:45.982+0200 [INFO]  agent.joiner: retry join completed: initial_servers=1 agent_mode=server
Sep 01 18:49:46 nid001311 nomad[120103]:     2023-09-01T18:49:46.000+0200 [DEBUG] client.server_mgr: new server list: new_servers=[148.187.113.147:4647] old_servers=[0.0.0.0:4647, 148.187.113.147:4647, 148.187.113.147:4647]
Sep 01 18:49:46 nid001311 nomad[120103]:     2023-09-01T18:49:46.000+0200 [INFO]  client: node registration complete
Sep 01 18:49:53 nid001311 nomad[120103]:     2023-09-01T18:49:53.731+0200 [DEBUG] client: state changed, updating node and re-registering
Sep 01 18:49:53 nid001311 nomad[120103]:     2023-09-01T18:49:53.735+0200 [INFO]  client: node registration complete
Sep 01 18:49:53 nid001311 nomad[120103]:     2023-09-01T18:49:53.737+0200 [DEBUG] client: state updated: node_status=ready
Sep 01 18:50:35 nid001311 systemd[1]: /etc/systemd/system/nomad.service:1: Assignment outside of section. Ignoring.
Sep 01 18:50:35 nid001311 systemd[1]: /etc/systemd/system/nomad.service:2: Assignment outside of section. Ignoring.
Sep 01 18:50:35 nid001311 systemd[1]: /etc/systemd/system/nomad.service:3: Assignment outside of section. Ignoring.
Sep 01 18:50:35 nid001311 systemd[1]: /etc/systemd/system/nomad.service:4: Assignment outside of section. Ignoring.
Sep 01 18:50:35 nid001311 systemd[1]: /etc/systemd/system/nomad.service:38: Unknown key name 'StartLimitIntervalSec' in section 'Service', ignoring.
Sep 01 18:50:45 nid001311 systemd[1]: Reloading nomad.service.
Sep 01 18:50:45 nid001311 systemd[1]: Reloaded nomad.service.
Sep 01 18:50:45 nid001311 nomad[120103]: ==> Caught signal: hangup
Sep 01 18:50:45 nid001311 nomad[120103]: ==> Reloading configuration...
Sep 01 18:50:45 nid001311 nomad[120103]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Sep 01 18:50:45 nid001311 nomad[120103]:     2023-09-01T18:50:45.308+0200 [DEBUG] agent: starting reload of server config
Sep 01 18:50:45 nid001311 nomad[120103]:     2023-09-01T18:50:45.310+0200 [DEBUG] agent: starting reload of client config
Sep 01 18:50:45 nid001311 nomad[120103]:     2023-09-01T18:50:45.310+0200 [INFO]  client.fingerprint_mgr: reloading fingerprinter: fingerprinter=cni
Sep 01 18:50:45 nid001311 nomad[120103]:     2023-09-01T18:50:45.310+0200 [DEBUG] client.fingerprint_mgr: CNI config dir is not set or does not exist, skipping: cni_config_dir=/opt/cni/config
Sep 01 18:50:47 nid001311 systemd[1]: /etc/systemd/system/nomad.service:1: Assignment outside of section. Ignoring.
Sep 01 18:50:47 nid001311 systemd[1]: /etc/systemd/system/nomad.service:2: Assignment outside of section. Ignoring.
Sep 01 18:50:47 nid001311 systemd[1]: /etc/systemd/system/nomad.service:3: Assignment outside of section. Ignoring.
Sep 01 18:50:47 nid001311 systemd[1]: /etc/systemd/system/nomad.service:4: Assignment outside of section. Ignoring.
Sep 01 18:50:47 nid001311 systemd[1]: /etc/systemd/system/nomad.service:38: Unknown key name 'StartLimitIntervalSec' in section 'Service', ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: /etc/systemd/system/nomad.service:1: Assignment outside of section. Ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: /etc/systemd/system/nomad.service:2: Assignment outside of section. Ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: /etc/systemd/system/nomad.service:3: Assignment outside of section. Ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: /etc/systemd/system/nomad.service:4: Assignment outside of section. Ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: /etc/systemd/system/nomad.service:1: Assignment outside of section. Ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: /etc/systemd/system/nomad.service:2: Assignment outside of section. Ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: /etc/systemd/system/nomad.service:3: Assignment outside of section. Ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: /etc/systemd/system/nomad.service:4: Assignment outside of section. Ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: /etc/systemd/system/nomad.service:1: Assignment outside of section. Ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: /etc/systemd/system/nomad.service:2: Assignment outside of section. Ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: /etc/systemd/system/nomad.service:3: Assignment outside of section. Ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: /etc/systemd/system/nomad.service:4: Assignment outside of section. Ignoring.
Sep 01 18:50:48 nid001311 systemd[1]: Stopping nomad.service...
Sep 01 18:50:48 nid001311 nomad[120103]: ==> Caught signal: interrupt
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.091+0200 [INFO]  agent: requesting shutdown
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.091+0200 [INFO]  client: shutting down
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.091+0200 [INFO]  client.plugin: shutting down plugin manager: plugin-type=device
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.091+0200 [INFO]  client.plugin: plugin manager finished: plugin-type=device
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.091+0200 [INFO]  client.plugin: shutting down plugin manager: plugin-type=driver
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.105+0200 [INFO]  client.plugin: plugin manager finished: plugin-type=driver
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.105+0200 [INFO]  client.plugin: shutting down plugin manager: plugin-type=csi
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.105+0200 [INFO]  client.plugin: plugin manager finished: plugin-type=csi
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [DEBUG] client.server_mgr: shutting down
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [INFO]  nomad: shutting down server
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [DEBUG] nomad: shutting down leader loop
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [DEBUG] nomad.autopilot: state update routine is now stopped
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [DEBUG] nomad.autopilot: autopilot is now stopped
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [WARN]  nomad: serf: Shutdown without a Leave
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [DEBUG] worker: stopped: worker_id=ab42baac-0ec5-684a-1277-a6f223d1295b
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [DEBUG] worker: stopped: worker_id=7349351e-ca7b-ade7-eea3-d127176a8fdc
...
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [DEBUG] worker: stopped: worker_id=26114754-7b60-ff9a-a2f9-d2b36974863a
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [DEBUG] worker: stopped: worker_id=d2751722-195f-78cf-1679-bf7dd02255c5
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [INFO]  nomad: cluster leadership lost
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [DEBUG] worker: stopped: worker_id=b3517d98-e1b9-b855-9cc9-daf77b43f76b
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [DEBUG] worker: stopped: worker_id=3a746653-ab48-c6f8-fe6d-8543f1f6d18d
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [DEBUG] worker: stopped: worker_id=f5cbdf18-43f9-404b-63fe-a06c802214ad
...
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.107+0200 [DEBUG] worker: stopped: worker_id=b5e1b708-f736-a0b9-9667-21a27b56a40c
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.107+0200 [DEBUG] worker: stopped: worker_id=5477dede-97da-4f80-ed9d-521ec5a68702
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.107+0200 [DEBUG] worker: stopped: worker_id=0946f5c0-ba55-17e9-9ea6-b54778193344
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.107+0200 [DEBUG] worker: stopped: worker_id=3ac90af7-f237-8d5a-e42c-29832639a5e3
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.107+0200 [DEBUG] worker: stopped: worker_id=789d9772-f2ae-e38c-7fe0-bc4bb8b60196
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.107+0200 [DEBUG] worker: stopped: worker_id=5fd10bc8-671f-99e2-8397-16940b707abf
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.106+0200 [DEBUG] worker: stopped: worker_id=6179714c-feeb-0c86-4ecf-71b2ab2ef15d
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.107+0200 [INFO]  agent: shutdown complete
Sep 01 18:50:48 nid001311 nomad[120103]:     2023-09-01T18:50:48.109+0200 [DEBUG] http: shutting down http server
Sep 01 18:50:48 nid001311 systemd[1]: nomad.service: Main process exited, code=exited, status=1/FAILURE
Sep 01 18:50:48 nid001311 systemd[1]: nomad.service: Failed with result 'exit-code'.
Sep 01 18:50:48 nid001311 systemd[1]: Stopped nomad.service.
Sep 01 18:50:48 nid001311 systemd[1]: Started nomad.service.
Sep 01 18:50:48 nid001311 nomad[126330]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Sep 01 18:50:48 nid001311 nomad[126330]: ==> Loaded configuration from /etc/nomad.d/client.hcl, /etc/nomad.d/nomad.hcl, /etc/nomad.d/server.hcl
Sep 01 18:50:48 nid001311 nomad[126330]: ==> Starting Nomad agent...
Sep 01 18:50:48 nid001311 systemd[1]: Reloading nomad.service.
Sep 01 18:50:48 nid001311 systemd[1]: Reloaded nomad.service.
Sep 01 18:50:48 nid001311 systemd[1]: nomad.service: Succeeded.
4

Hi @masuberu,

The log line below indicates something is sending an interrupt to the Nomad agent. I have not seen any further log entries which might indicates a problem.

==> Caught signal: interrupt

Does Nomad start OK if you run the application directly without using systemd?

Thanks,
jrasell and the Nomad team

5

Hi @masuberu,

My guess is that you have set the systemd Service type to notify, which is why the systemd is issuing an interrupt, as Nomad doesn’t notify systemd about a successful start.

Could you share the output of systemctl cat nomad?

If your Nomad systemd unit files [Service] block has Type=notify:

  • remove it
  • run systemctl daemon-reload
  • run systemctl restart nomad

This should fix the issue.

ref: Nomad Deployment Guide | Nomad | HashiCorp Developer

6

sorry for the late reply, I don’t have Type=notify instead I have Type=forking

$ systemctl cat slurmd
# /usr/lib/systemd/system/slurmd.service
[Unit]
Description=Slurm node daemon
After=munge.service network.target remote-fs.target
#ConditionPathExists=/etc/slurm/slurm.conf

[Service]
Type=forking
EnvironmentFile=-/etc/sysconfig/slurmd
ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/var/run/slurmd.pid
KillMode=process
LimitNOFILE=131072
LimitMEMLOCK=infinity
LimitSTACK=infinity
Delegate=yes
TasksMax=infinity

[Install]
WantedBy=multi-user.target

does this change anything?

7

Hi @masuberu,

It should be Type=simple (the default type).

Could you remove that line, do a systemctl daemon-reload and restart nomad (systemctl restart nomad) and that should fix the issue.

8

sorry my bad, I checked the wrong systemd unit file (slurmd instead of nomad). Nomad systemd unit file does not have the Type parameter so it is working as expected