In a Connect-enabled Nomad cluster, is it possible to have a Nomad task listen on a node’s loopback/localhost interface for localhost-only interactions with other host-local services that are not managed by Nomad (and so, does not have access to the Connect network segments)?
Also, is it possible for traffic to flow in the opposite direction: can a Nomad task (with Connect enabled) reach a host service that is listening on the host’s loopback/localhost interface?
One obvious use case is monitoring Consul using Telegraf, where Telegraf is run as a Nomad system job using the Docker driver.
Based on the Monitor Consul with Telegraf guide, I’m trying to get Consul, a host systemd service, to reach Telegraf. The only way I was able to do that is to “publish” Telegraf’s statsd port (8125) to the host as shown in the Web frontend example of Nomad Consul Connect integration:
job "telegraf" {
type = "system"
group "telegraf" {
network {
mode = "bridge"
port "statsd" {
static = 8125
to = 8125
}}
...
}}
The problem with this approach is that port 8125 is now exposed on the network, not only the node’s localhost, allowing anything on the network to submit arbitrary metrics.
The other issue is for Telegraf to reach Consul’s http API to monitor Consul checks. Consul API is listening on the node’s localhost, which, AFAICT, is unreachable from a Connect-enabled task.
I’m running Telegraf as a Connect-enabled Nomad job so it can securely connect to the rest of the monitoring stack (Influxdb, Riemann, etc.), which are all Connect-enabled Nomad jobs.
So, how can Telegraf monitor Consul in this case while still keeping security tight? And more generally, how can Connect tasks interact with local node services?