Node pool creation at first nomad server startup / server provisioning

Hello all,

I want to use the node pool feature to “tag” my nomad client. (Nomad 1.7.1)
Reading the official documentation i got the command to create a node pool.

I am currently using cloud-init to provision a machine, so after installing nomad i play the two following commands :

  • systemctl enable nomad --now
  • nomad node pool apply -address=“http://10.0.0.4:4646” /run/lb-pool.nomad.hcl

I also tried to start the service via 2 commands instead of using --now (with the same results) :

  • systemctl enable nomad
  • systemctl start nomad

10.0.0.4 here represents the local machine but nomad is exposed on this interface so i specify it.

Unfortunately, the second command is played while the nomad service is still «starting» and the nomad service not yet exposed on port 4646. Hence, the following logs :

Dec 12 14:52:30 subnetrouter-dev systemd[1]: Started nomad.service - “HashiCorp Nomad”.
Dec 12 14:52:31 subnetrouter-dev cloud-init[2369]: Error applying node pool: Put “http://10.0.0.4:4646/v1/node/pools”: dial tcp 10.0.0.>
Dec 12 14:52:31 subnetrouter-dev cloud-init[2369]: 2023-12-12 13:52:31,025 - cc_scripts_user.py[WARNING]: Failed to run module scripts->
Dec 12 14:52:31 subnetrouter-dev cloud-init[2369]: 2023-12-12 13:52:31,025 - util.py[WARNING]: Running module scripts-user (<module 'cl>
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: ==> WARNING: mTLS is not configured - Nomad is not secure without mTLS!
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: ==> Loaded configuration from /etc/nomad.d/nomad.hcl
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: ==> Starting Nomad agent…
Dec 12 14:52:31 subnetrouter-dev cloud-init[15530]: #############################################################
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: ==> Nomad agent configuration:
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: Advertise Addrs: HTTP: 10.0.0.4:4646; RPC: 10.0.0.4:4647; Serf: 10.0.0.4:4648
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: Bind Addrs: HTTP: [10.0.0.4:4646]; RPC: 10.0.0.4:4647; Serf: 10.0.0.4:4648
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: Client: false
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: Log Level: INFO
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: Node Id: 7472c7f4-6203-f6c1-4028-e8f7099b1cd0
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: Region: global (DC: dc1)
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: Server: true
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: Version: 1.7.1
Dec 12 14:52:31 subnetrouter-dev nomad[15513]: ==> Nomad agent started! Log data will stream in below:

First idea to solve the problem

Note :
My systemctl service file is quite simple and contain the following line

ExecStart=/usr/bin/nomad agent -config /etc/nomad.d

So, my first idea was to add my node pool file (/run/lb-pool.nomad.hcl) to the nomad configuration directory … but i had the following error

Dec 12 15:54:22 subnetrouter-dev nomad[15792]: ==> Error loading configuration from > /etc/nomad.d: Error loading /etc/nomad.d/lb-pool.nomad.hcl: unexpected keys lb, node_pool

It seems (from the error) that node pool configuration is not considered nomad client/server configuration so cannot be used here.
I also had a look at the nomad client command line options … to no avail.

I finally also tried to play with the systemctl ExecStartPost option … with the same result again.

Second idea

I thought about having a kind of “sleep [INT]” between my 2 commands to give the nomad agent some time to start … but don’'t like this idea at all.

Other ideas ?

Did someone encounter the same issue or have an idea of how to solve this «elegantly» ?
Any help/suggestion would be greatly appreciated ?

Best regards,
Brahim