I think that this is more an issue with Nomad than Consul, but I’m not certain.
I’ve tried following the steps shown on this blog to add a service to the Consul Connect mesh but expose a single path for metrics to other hosts.
Where it seems to fail is that I can curl all paths on the Nomad bind port with no issues. I also see no mention in the headers of them going through the proxy. This may be intended, so I generally have set the host network for my ports to bind only to lo so that they are private and can only be reached through Consul Connect.
The issue when I do this is that even when exposing a path via the connect stanza, it’s not accessible. Nomad 1.4 doesn’t seem to show me the bindings for tasks, so I can’t see the ports there, but when I look into Consul, I can see that the main service is bound to a port on my loopback interface and that the Connect Sidecar is bound to another. If I curl the sidecar port with the /metrics path, I get an empty reply from the server.
It says on the Consul documentation that the Expose functionality is designed for something like metrics scraping, but it doesn’t seem like there is a convenient way to use it for that. Even if I manage to get the sidecar to respond to the /metrics path, I’m unable to set a meta tag on the consul service indicating sidecar host and port.
How are these capabilities supposed to function?
This is my job configuration:
variable "config_data" {
type = string
description = "Plain text config file for blocky"
}
job "blocky" {
datacenters = ["dc1"]
type = "system"
priority = 100
update {
max_parallel = 1
auto_revert = true
}
group "blocky" {
network {
mode = "bridge"
port "dns" {
static = "53"
}
port "api" {
host_network = "loopback"
to = "4000"
}
}
service {
name = "blocky-dns"
port = "dns"
}
service {
name = "blocky-api"
port = "api"
meta {
metrics_addr = "${NOMAD_ADDR_api}"
}
tags = [
"traefik.enable=true",
]
connect {
sidecar_service {
proxy {
local_service_port = 400
expose {
path {
path = "/metrics"
protocol = "http"
local_path_port = 4000
listener_port = "api"
}
}
upstreams {
destination_name = "redis"
local_bind_port = 6379
}
}
}
sidecar_task {
resources {
cpu = 50
memory = 20
memory_max = 50
}
}
}
check {
name = "api-health"
port = "api"
type = "http"
path = "/"
interval = "10s"
timeout = "3s"
}
}
task "blocky" {
driver = "docker"
config {
image = "ghcr.io/0xerr0r/blocky"
ports = ["dns", "api"]
mount {
type = "bind"
target = "/app/config.yml"
source = "app/config.yml"
}
}
resources {
cpu = 50
memory = 50
memory_max = 100
}
template {
data = var.config_data
destination = "app/config.yml"
splay = "1m"
}
}
}
}