Since Nomad does not seem to be able to advertise multiple ports for a service, I found a workaround for my scenario. I am defining the second dynamic port and then saving it to the metadata. Consul is able to pass that metadata on to Prometheus where I am able to use relabel_configs
to have Prometheus scrape the correct port dynamically while still using consul_sd_configs
.
job "awesome-api" {
group "awesome-group" {
network {
port "http" { host_network = "private" }
port "metrics" { host_network = "private" }
}
service {
name = "awesome-api"
port = "http"
}
meta {
metrics-port = "${NOMAD_PORT_metrics}"
}
task "gateway" {
driver = "exec"
config {
command = "${NOMAD_ALLOC_DIR}/api/app"
args = ["-config", "config.json"]
}
template {
destination = "config.json"
data = <<-EOT
{
"http_address" : "{{ env "NOMAD_IP_http" }}:{{ env "NOMAD_PORT_http" }}",
"metrics_address" : ":{{ env "NOMAD_PORT_metrics" }}",
}
EOT
}
}
}
}
Prometheus config snippet:
...
- job_name: 'all-services-consul-sd'
metrics_path: "/"
consul_sd_configs:
- server: 'localhost:8500'
relabel_configs:
- source_labels: [__meta_consul_service_address,__meta_consul_service_metadata_metrics_port]
separator: ';'
regex: (.*);(\d{4,5})
target_label: '__address__'
replacement: '$1:$2'
action: 'replace'
...