Hi Everyone.
I want to start by saying I have searched these forums to find an answer before posting, so if this was already answered I apologize for the redundant thread.
Summary
I am attempting to deploy a Redis cluster (6.2.6) using a nomad Job. There is a challenge with Redis around clustering, where you have to explicitly cluster using IP:Port
of all nodes. Assuming the cluster was up and everything is announced in Consul, this would work perfectly
template {
destination = "local/create_cluster.sh"
data = <<-EOF
redis-cli --cluster-replicas 1 --cluster-yes --cluster create \
{{- range service "redis" }}
{{ .Address }}:{{ .Port }} \
{{- end }}
EOF
}
to generate:
redis-cli --cluster-replicas 1 --cluster-yes --cluster create \
10.0.0.1:6379 \
10.0.0.2:6379 \
10.0.0.3:6379 \
10.0.0.4:6379 \
10.0.0.5:6379 \
10.0.0.6:6379 \
I am currently using 2 tasks in 1 group:
- Deploy all redis containers, ready to create a cluster, or be added to one
- A
post-start
ephemeral sidecar that runs a bash script to handle thecluster create
orcluster add-node
logic.
The script from the post-start
ephemeral sidecar is where the interpolation of IP:Port
needs to be, specifically for cluster create
. This is not a problem when needing to cluster add-node
since I can query consul for any
cluster member, and can self discover my
(container) IP:Port
. Below is the entire (pre-interpolation) bash script:
#!/bin/bash
# Give everything a few seconds to start
sleep 5
echo "SCRIPT: Get a REDIS host other than 'me'"
my_host_ip={{ env "attr.unique.network.ip-address" }}
my_redis_port={{ env "NOMAD_PORT_db"}}
# Get the redis host and db port for index 0
{{- with $instances := service "redis" }}
{{- with $first := index $instances 0 }}
redis_host={{- $first.Address }}
redis_port={{- $first.Port }}
{{- end}}{{- end}}
# Making sure we hit a different host ensures we dont hit 'this' container
if [[ $my_host_ip == $redis_host ]]; then
# Get the redis host and db port for index 1
echo "My IP ($my_host_ip) and the discovered REDIS IP ($redis_host) are the same"
{{- with $instances := service "redis" }}
{{- with $first := index $instances 1 }}
redis_host={{- $first.Address }}
redis_port={{- $first.Port }}
{{- end}}{{- end}}
echo "SCRIPT: Fetched new IP for a REDIS host: $redis_host"
fi
echo "SCRIPT: Discovering if cluster exists already"
# Check for existing cluster (array length > 1)
readarray -t nodes < <(echo "CLUSTER NODES" | redis-cli -h $redis_host -p $redis_port)
if [[ $${#nodes[@]} -eq 1 ]]; then create_cluster=true; else create_cluster=false; fi
if $create_cluster; then
echo "SCRIPT: Create Cluster: True"
echo "SCRIPT: Creating Cluster...\n"
if [ "$NOMAD_ALLOC_INDEX" == "6"]; then
# This fails to interpolate because these services dont exist in consul yet
redis-cli --cluster-replicas 1 --cluster-yes --cluster create \
{{- range service "redis" }}
{{ .Address }}:{{ .Port }} \
{{- end }}
fi
else
echo "SCRIPT: Create Cluster: False"
echo "SCRIPT: Adding self to existing cluster at $redis_host\n"
redis-cli --cluster add-node --slave \
$${my_host_ip}:$${my_redis_port} $${redis_host}:$${redis_port}
fi
echo "Clustering Operations Complete"
Problem
Unfortunately This cannot be used for the cluster create
portion since the timing of generating the template comes before the services are announced in consul. Circular dependency of sorts. The template renders, but there is no IP:Port
list of nodes… it’s just empty (working as intended)
So far I have been unable to find a way to express the IP:Port
of the created containers using only information available to the environment at runtime, when the template is interpolated.
Desired Solution
Any method I can use to correctly generate the list of IP:Port
of the created containers from within the same Job group (Or a seperate group if needed) for the cluster create
portion of this script. Any other ideas or suggestions are appreciated as well. TIA!
Full Job Spec
job "redis" {
datacenters = ["dc"]
type = "service"
group "cluster" {
count = 6
network {
mode = "host"
port "db" { to = 6349 }
port "cluster" { to = 16379 }
}
task "cluster_nodes" {
driver = "docker"
config {
image = "redis:6.2"
hostname = "alloc${NOMAD_ALLOC_INDEX}"
ports = ["db", "cluster"]
command = "redis-server"
args = ["/etc/redis.conf"]
mount {
type = "bind"
source = "local/redis.conf"
target = "/etc/redis.conf"
readonly = true
}
}
service {
name = "redis"
port = "db"
check {
type = "tcp"
port = "db"
interval = "15s"
timeout = "3s"
}
}
service {
name = "redis-cluster"
port = "cluster"
check {
type = "tcp"
port = "cluster"
interval = "15s"
timeout = "3s"
}
}
# https://redis.io/docs/manual/scaling/#redis-cluster-configuration-parameters
# https://redis.io/docs/manual/config/
# https://raw.githubusercontent.com/redis/redis/6.2/redis.conf
template {
destination = "local/redis.conf"
change_mode = "signal"
change_signal = "SIGHUP"
data = <<-EOF
appendonly yes
port 6379
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-announce-ip {{ env "attr.unique.network.ip-address" }}
cluster-announce-port {{ env "NOMAD_PORT_db" }}
cluster-announce-bus-port {{ env "NOMAD_PORT_cluster" }}
EOF
}
}
task "manage_cluster" {
driver = "docker"
lifecycle { # https://www.nomadproject.io/docs/job-specification/lifecycle
hook = "poststart"
sidecar = false # Run as Ephemeral (throw away)
}
config {
image = "redis:6.2"
hostname = "alloc${NOMAD_ALLOC_INDEX}"
ports = ["db", "cluster"]
command = "/bin/bash"
args = ["/opt/manage_cluster.sh"]
mount {
type = "bind"
source = "local/manage_cluster.sh"
target = "/opt/manage_cluster.sh"
readonly = true
}
}
template {
destination = "local/manage_cluster.sh"
change_mode = "signal"
change_signal = "SIGHUP"
data = <<-EOF
#!/bin/bash
# In nomad, you escape a '$'{} with a double $$ when you dont want Nomad to interpret it
# Give everything a few seconds to start
sleep 5
echo "SCRIPT: Get a REDIS host other than 'me'"
my_host_ip={{ env "attr.unique.network.ip-address" }}
my_redis_port={{ env "NOMAD_PORT_db"}}
# Get the redis host and db port for index 0
{{- with $instances := service "redis" }}
{{- with $first := index $instances 0 }}
redis_host={{- $first.Address }}
redis_port={{- $first.Port }}
{{- end}}{{- end}}
# Making sure we hit a different host ensures we dont hit 'this' container
if [[ $my_host_ip == $redis_host ]]; then
# Get the redis host and db port for index 1
echo "My IP ($my_host_ip) and the discovered REDIS IP ($redis_host) are the same"
{{- with $instances := service "redis" }}
{{- with $first := index $instances 1 }}
redis_host={{- $first.Address }}
redis_port={{- $first.Port }}
{{- end}}{{- end}}
echo "SCRIPT: Fetched new IP for a REDIS host: $redis_host"
fi
echo "SCRIPT: Discovering if cluster exists already"
# Check for existing cluster (array length > 1)
readarray -t nodes < <(echo "CLUSTER NODES" | redis-cli -h $redis_host -p $redis_port)
if [[ $${#nodes[@]} -eq 1 ]]; then create_cluster=true; else create_cluster=false; fi
if $create_cluster; then
echo "SCRIPT: Create Cluster: True"
echo "SCRIPT: Creating Cluster...\n"
# IMPROVEMENT: make implicit rather than explicit
# Only run once; on alloc 6 only (because last)
if [ "$NOMAD_ALLOC_INDEX" == "6"]; then
redis-cli --cluster-replicas 1 --cluster-yes --cluster create \
{{- range service "redis" }}
{{ .Address }}:{{ .Port }} \
{{- end }}
fi
else
echo "SCRIPT: Create Cluster: False"
echo "SCRIPT: Adding self to existing cluster at $redis_host\n"
redis-cli --cluster add-node --slave \
$${my_host_ip}:$${my_redis_port} $${redis_host}:$${redis_port}
fi
echo "Clustering Operations Complete"
EOF
}
}
}
}