Referencing each node in HA service using Nomad job template variables

Nomad
1

Hello there.

I am attempting to create a Nomad job for a Redpanda HA cluster. The internal config files, similar to how we would set up bootstrap addresses for other Consul or Nomad server nodes, need to know how to reach the other nodes (I have a group with a count of 3). I’m having trouble figuring out how to describe this in the Nomad templates; I would think that I could reference the allocations by index (0-2) but I can’t find any primitives that looks like they would do the trick and it seems like the references in the template would be hard to know at the time of launch.

I am able to see things like node.unique.id or NOMAD_IP_label, but those are all in reference to the current node. I need to be able to point to the other allocations

I’m using Consul, so maybe there’s some service-level environment variables there I can tap into.

Has anyone done this? Any help would be appreciated.

2

I had a similar question:

Although not discussed there, I came to realise that the solution here was to use a consul_template lookup.

Have you had a look at this tutorial?

1 Like
3

Thanks @brucellino1 ! Keep forgetting the whole or Consul-Template is packaged in there.

4

I think this is what I’m looking for…

5

I also thought that was what I needed, but there is a Catch-22. You need to look up services in the catalog in order to register the same services in the catalog. This is a self-referential loop, and if I remember correctly, you can’t ask the job to find all other instances of it’s own services. You can only find other services which already exist in the catalog.

One of the hacks I thought of trying was to register so-called “pilot” services – dummy services which register themselves independently, reserving the place of the actual service which will be placed later. I thought of using the prestart lifecycle for this, but never actually got around it it.

6

It seems like using the independent Consul-Template binary as you originally suggested, rather than what is baked into Nomad would be better then, as it can listen to the catalog and update itself as task statuses change.

Did you have more luck on that front?

7

@brucellino1 one other possible path… Not sure if you tried this.

I was looking at this HAProxy tutorial where they set up cluster, similar to what I’m trying to do, and it looks like you can use this syntax to return all the addresses available for a service.

The output might need some massaging, but it looks like it could be just the thing.

1 Like
8

and it looks like you can use this syntax to return all the addresses available

Same Catch-22 as mentioned by @brucellino1

You need to look up services in the catalog in order to register the same services in the catalog.

9

Actually, using a template block with consul lookup seems to work! E.g.

      template {
        env = true
        destination = "docker_env"
        data = <<EOF
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=
  {{- range $i, $e := service "kafka-broker" -}}
    {{- if eq $i 0 -}}
      {{$i}}@{{ .Address }}:{{ .Port }}
    {{- else -}}
      ,{{$i}}@{{ .Address }}:{{ .Port }}
    {{- end -}}
  {{- end}}
        EOF
      }
2 Likes
10

Beautiful, I’ll give it a whirl. Thanks @ignatius.public!

11

I later found that change_mode = "noop" is probably not what you want, so I removed it from the previous post.

1 Like