I have grafana Loki running as a nomad service which gets connected to clients via consul proxy.
Main Runtime service
service {
name = "runtime"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "loki"
local_bind_port = 18092
}
}
}
}
}
Loki Service
service {
name = "loki"
port = 3100
check {
name = "Loki instance health check"
type = "http"
path = "/ready"
interval = "10s"
timeout = "2s"
expose = true
check_restart {
limit = 3
grace = "2m"
}
}
connect {
sidecar_service {}
sidecar_task {
resources {
cpu = 80
memory = 80
}
}
}
}
Everything works correctly, but when I stop the Loki service, I receive a ‘connection reset’ error from the Consul proxy, even after reducing the client connection timeout to 1 second (default connection timeout for the Consul proxy is 5 seconds). In this case, I expect an HTTP connection timeout instead of an immediate connection reset by the proxy.
# wget --timeout=1 http://127.0.0.1:18092/ready
Connecting to 127.0.0.1:18092 (127.0.0.1:18092)
wget: error getting response: Connection reset by peer
As you can see above, the Consul proxy reset the connection even though the client (“wget”) has 1-second timeout.
Could someone please help with this? Am I missing some configuration in the nomad service spec?