Need to interpolate IP:Port of created containers in template within same group

Hi Everyone.

I want to start by saying I have searched these forums to find an answer before posting, so if this was already answered I apologize for the redundant thread.

Summary
I am attempting to deploy a Redis cluster (6.2.6) using a nomad Job. There is a challenge with Redis around clustering, where you have to explicitly cluster using IP:Port of all nodes. Assuming the cluster was up and everything is announced in Consul, this would work perfectly

template {
  destination   = "local/create_cluster.sh"
  data          = <<-EOF
    redis-cli --cluster-replicas 1 --cluster-yes --cluster create \
    {{- range service  "redis" }}
    {{ .Address }}:{{ .Port }} \
    {{- end }}
  EOF
}

to generate:

redis-cli --cluster-replicas 1 --cluster-yes --cluster create \
10.0.0.1:6379 \
10.0.0.2:6379 \
10.0.0.3:6379 \
10.0.0.4:6379 \
10.0.0.5:6379 \
10.0.0.6:6379 \

I am currently using 2 tasks in 1 group:

  1. Deploy all redis containers, ready to create a cluster, or be added to one
  2. A post-start ephemeral sidecar that runs a bash script to handle the cluster create or cluster add-node logic.

The script from the post-start ephemeral sidecar is where the interpolation of IP:Port needs to be, specifically for cluster create. This is not a problem when needing to cluster add-node since I can query consul for any cluster member, and can self discover my (container) IP:Port. Below is the entire (pre-interpolation) bash script:

#!/bin/bash
# Give everything a few seconds to start
sleep 5
echo "SCRIPT: Get a REDIS host other than 'me'"

my_host_ip={{ env "attr.unique.network.ip-address" }}
my_redis_port={{ env "NOMAD_PORT_db"}}

# Get the redis host and db port for index 0
{{- with $instances := service "redis" }}
{{- with $first := index $instances 0 }}
redis_host={{- $first.Address }}
redis_port={{- $first.Port }}
{{- end}}{{- end}}

# Making sure we hit a different host ensures we dont hit 'this' container
if [[ $my_host_ip == $redis_host ]]; then
  # Get the redis host and db port for index 1
  echo "My IP ($my_host_ip) and the discovered REDIS IP ($redis_host) are the same"
  {{- with $instances := service "redis" }}
  {{- with $first := index $instances 1 }}
  redis_host={{- $first.Address }}
  redis_port={{- $first.Port }}
  {{- end}}{{- end}}
  echo "SCRIPT: Fetched new IP for a REDIS host: $redis_host"
fi

echo "SCRIPT: Discovering if cluster exists already"

# Check for existing cluster (array length > 1)
readarray -t nodes < <(echo "CLUSTER NODES" | redis-cli -h $redis_host -p $redis_port)

if [[ $${#nodes[@]} -eq 1 ]]; then create_cluster=true; else create_cluster=false; fi

if $create_cluster; then
  echo "SCRIPT: Create Cluster: True"
  echo "SCRIPT: Creating Cluster...\n"

if [ "$NOMAD_ALLOC_INDEX" == "6"]; then
  # This fails to interpolate because these services dont exist in consul yet
  redis-cli --cluster-replicas 1 --cluster-yes --cluster create \
    {{- range service  "redis" }}
    {{ .Address }}:{{ .Port }} \
    {{- end }}
  fi
else
  echo "SCRIPT: Create Cluster: False"
  echo "SCRIPT: Adding self to existing cluster at $redis_host\n"
  redis-cli --cluster add-node --slave \
  $${my_host_ip}:$${my_redis_port} $${redis_host}:$${redis_port}
fi

echo "Clustering Operations Complete"

Problem
Unfortunately This cannot be used for the cluster create portion since the timing of generating the template comes before the services are announced in consul. Circular dependency of sorts. The template renders, but there is no IP:Port list of nodes… it’s just empty (working as intended)

So far I have been unable to find a way to express the IP:Port of the created containers using only information available to the environment at runtime, when the template is interpolated.

Desired Solution
Any method I can use to correctly generate the list of IP:Port of the created containers from within the same Job group (Or a seperate group if needed) for the cluster create portion of this script. Any other ideas or suggestions are appreciated as well. TIA!

Full Job Spec

job "redis" {
  datacenters = ["dc"]
  type        = "service"

  group "cluster" {
    count = 6

    network {
      mode = "host"
      port "db" { to = 6349 }
      port "cluster" { to = 16379 }
    }

    task "cluster_nodes" {
      driver = "docker"

      config {
        image    = "redis:6.2"
        hostname = "alloc${NOMAD_ALLOC_INDEX}"

        ports = ["db", "cluster"]

        command = "redis-server"
        args    = ["/etc/redis.conf"]

        mount {
          type     = "bind"
          source   = "local/redis.conf"
          target   = "/etc/redis.conf"
          readonly = true
        }
      }

      service {
        name = "redis"
        port = "db"

        check {
          type     = "tcp"
          port     = "db"
          interval = "15s"
          timeout  = "3s"
        }
      }
      service {
        name = "redis-cluster"
        port = "cluster"

        check {
          type     = "tcp"
          port     = "cluster"
          interval = "15s"
          timeout  = "3s"
        }
      }

      # https://redis.io/docs/manual/scaling/#redis-cluster-configuration-parameters
      # https://redis.io/docs/manual/config/
      # https://raw.githubusercontent.com/redis/redis/6.2/redis.conf
      template {
        destination   = "local/redis.conf"
        change_mode   = "signal"
        change_signal = "SIGHUP"
        data          = <<-EOF
          appendonly yes
          port       6379

          cluster-enabled           yes
          cluster-config-file       nodes.conf
          cluster-node-timeout      5000
          cluster-announce-ip       {{ env "attr.unique.network.ip-address" }}
          cluster-announce-port     {{ env "NOMAD_PORT_db" }}
          cluster-announce-bus-port {{ env "NOMAD_PORT_cluster" }}
        EOF
      }
    }

    task "manage_cluster" {
      driver = "docker"

      lifecycle { # https://www.nomadproject.io/docs/job-specification/lifecycle
        hook    = "poststart"
        sidecar = false # Run as Ephemeral (throw away)
      }

      config {
        image    = "redis:6.2"
        hostname = "alloc${NOMAD_ALLOC_INDEX}"

        ports = ["db", "cluster"]

        command = "/bin/bash"
        args    = ["/opt/manage_cluster.sh"]

        mount {
          type     = "bind"
          source   = "local/manage_cluster.sh"
          target   = "/opt/manage_cluster.sh"
          readonly = true
        }
      }

      template {
        destination   = "local/manage_cluster.sh"
        change_mode   = "signal"
        change_signal = "SIGHUP"
        data          = <<-EOF
          #!/bin/bash
          # In nomad, you escape a '$'{} with a double $$ when you dont want Nomad to interpret it

          # Give everything a few seconds to start
          sleep 5
          echo "SCRIPT: Get a REDIS host other than 'me'"

          my_host_ip={{ env "attr.unique.network.ip-address" }}
          my_redis_port={{ env "NOMAD_PORT_db"}}

          # Get the redis host and db port for index 0
          {{- with $instances := service "redis" }}
          {{- with $first := index $instances 0 }}
          redis_host={{- $first.Address }}
          redis_port={{- $first.Port }}
          {{- end}}{{- end}}

          # Making sure we hit a different host ensures we dont hit 'this' container
          if [[ $my_host_ip == $redis_host ]]; then
            # Get the redis host and db port for index 1
            echo "My IP ($my_host_ip) and the discovered REDIS IP ($redis_host) are the same"
            {{- with $instances := service "redis" }}
            {{- with $first := index $instances 1 }}
            redis_host={{- $first.Address }}
            redis_port={{- $first.Port }}
            {{- end}}{{- end}}
            echo "SCRIPT: Fetched new IP for a REDIS host: $redis_host"
          fi

          echo "SCRIPT: Discovering if cluster exists already"

          # Check for existing cluster (array length > 1)
          readarray -t nodes < <(echo "CLUSTER NODES" | redis-cli -h $redis_host -p $redis_port)

          if [[ $${#nodes[@]} -eq 1 ]]; then create_cluster=true; else create_cluster=false; fi

          if $create_cluster; then
            echo "SCRIPT: Create Cluster: True"
            echo "SCRIPT: Creating Cluster...\n"

            # IMPROVEMENT: make implicit rather than explicit
            # Only run once; on alloc 6 only (because last)
            if [ "$NOMAD_ALLOC_INDEX" == "6"]; then
            redis-cli --cluster-replicas 1 --cluster-yes --cluster create \
              {{- range service  "redis" }}
              {{ .Address }}:{{ .Port }} \
              {{- end }}
            fi
          else
            echo "SCRIPT: Create Cluster: False"
            echo "SCRIPT: Adding self to existing cluster at $redis_host\n"
            redis-cli --cluster add-node --slave \
            $${my_host_ip}:$${my_redis_port} $${redis_host}:$${redis_port}
          fi

          echo "Clustering Operations Complete"
        EOF
      }
    }
  }
}
1 Like