I got almost all I want working…
The last thing I need is communication between consul services
So I defined a job with network bridge and a service → connect → sidecar_service {} to test it out. The service is called “node-exporter-nginx”.
Deployed a curl to test upstream communication… which fails with a 503 error.
I did allow curl → node-exporter-nginx connection using Intentions.
The service itself is healthy and I can access just fine from the host using consul dns for example.
Where should I be looking for the root cause?
How can I test that consul connect is working correctly, etc?
job "curl" {
datacenters = ["dc1"]
type = "service"
group "demo" {
count = ${replica_count}
shutdown_delay = "30s"
network {
mode = "bridge"
}
service {
name = "curl"
provider = "consul"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "node-exporter-nginx"
local_bind_port = 9113
config {
protocol = "http"
}
}
}
}
}
tags = [
"primary",
]
}
task "curl" {
driver = "docker"
config {
image = "curlimages/curl:latest"
command = "sh"
args = ["-c", "sleep infinity"]
}
env {
NETWORK_IP2 = "$${attr.unique.network.ip-address}"
DRIVER_IP2 = "$${attr.driver.docker.bridge_ip}"
}
}
}
}
~ $ netstat -tunpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.2:19001 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:9113 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:27402 0.0.0.0:* LISTEN -
~ $ curl -v 127.0.0.1:9113
* Trying 127.0.0.1:9113...
* Connected to 127.0.0.1 (127.0.0.1) port 9113
* using HTTP/1.x
> GET / HTTP/1.1
> Host: 127.0.0.1:9113
> User-Agent: curl/8.13.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 503 Service Unavailable
< content-length: 98
< content-type: text/plain
< date: Sat, 14 Jun 2025 13:46:34 GMT
< server: envoy
<
* Connection #0 to host 127.0.0.1 left intact
upstream connect error or disconnect/reset before headers. reset reason: remote connection failure
