I am setting up a Kafka cluster using Nomad and I need to specify the peer node information for the Kafka configuration, which is passed in as a docker environment variable. I would like to retrieve this information dynamically within the job specification, rather than hardcoding it, if that’s possible.
Can anyone recommend the best way to get the peer node information (hostnames or IP addresses), i.e. of other nodes forming the cluster, when writing a job specification in Nomad? I’m using Nomad together with Consul for service discovery if that’s relevant.
I would love to see this covered in a learn guide (or whatever we’re calling them these days). Wanting to run services as an HA cluster in Nomad is, I would think, super common or something that we would all want to be super common. The HA Proxy guide uses a somewhat bespoke pattern that leans on passing a Consul DNS SRV host to the HA Proxy’s internal configuration file syntax and not some of the more broadly useful details around what Nomad can output about service internals. It’s super cool, and good to know that’s a tool in the belt, but probably not as applicable to just iterating over the allocated service ips as this example does.
Replying to my own solution: This leads to absolute chaos when trying to do a rolling restart, since the restart triggers a change in the service registration, which means the template changes, which causes an infinite feedback-loop of job restarts.