Consul Connect transparent_proxy breaks Envoy metrics scraping

[sacpk]

Hi all,

I’m running Consul + Nomad with Connect sidecars. I have a Nomad job like this (simplified):

service {
  name = "bar"
  port = 9090

  meta {
    envoy_metrics_port = "${NOMAD_HOST_PORT_envoy_metrics}"
  }

  connect {
    sidecar_service {
      proxy {
        config {
          envoy_prometheus_bind_addr = "0.0.0.0:9102"
        }
        transparent_proxy {
          no_dns = true
        }
      }
    }
  }
}

1. Without the transparent_proxy block, Prometheus can scrape http://<node-ip>:<dynamic-port>/metrics correctly.
2. Once I enable transparent_proxy { no_dns = true }, Prometheus fails with:

Error scraping target: Get “``http://10.0.x.x:23139/metrics”:`` EOF

• Prometheus itself is also running in Nomad with transparent_proxy enabled.

• I have intentions set up to allow Prometheus → bar.

Question:
Does transparent_proxy mode currently support scraping Envoy proxy metrics (via envoy_prometheus_bind_addr)?
If not, what’s the recommended way to expose Envoy/sidecar metrics when using transparent proxy? Should these metrics be scraped via the sidecar service through Consul DNS instead of the host IP/port?

Thanks!

[Material-Scientist]

Works for me. Here’s my config:

    network {
      mode = "bridge"
      port "envoy-metrics" {}
    }
    service {
      name     = "n8n"
      port     = "5678"
      meta {
        envoy_metrics_port = "${NOMAD_HOST_PORT_envoy_metrics}" # make envoy metrics port available in Consul
      }
      connect {
        sidecar_service {
          proxy {
            transparent_proxy {}
            expose {
              path {
                path = "/metrics"
                protocol = "http"
                local_path_port = 9102
                listener_port = "envoy-metrics"
              }
            }
          }
        }
      }
      check {
        expose   = true
        type     = "http"
        path     = "/healthz"
        interval = "5s"
        timeout  = "1s"
      }
    }

Prometheus yml:

  - job_name: consul-connect-envoy
    scheme: http
    #tls_config:
    #  ca_file: '/etc/prometheus/ca.pem'
    #  cert_file: '/etc/prometheus/cert.pem'
    #  key_file: '/etc/prometheus/key.pem'
    metrics_path: "/metrics"
    consul_sd_configs:
    - server: '{{ env "attr.unique.network.ip-address" }}:8501'
      token: '{{ env "secrets/consul_management_token" }}'
      scheme: 'https'
      tls_config:
        ca_file: '/etc/prometheus/ca.pem'
        cert_file: '/etc/prometheus/cert.pem'
        key_file: '/etc/prometheus/key.pem'
    relabel_configs:
      - source_labels: [__meta_consul_service]
        regex: (.+)-sidecar-proxy
        action: drop
      - source_labels: [__meta_consul_service_metadata_envoy_metrics_port]
        regex: ^$
        action: drop
      - source_labels: [__meta_consul_service_metadata_envoy_metrics_port]
        regex: (\d+)
        action: keep
      - source_labels: [__address__, __meta_consul_service_metadata_envoy_metrics_port]
        regex: (.+?)(?::\d+)?;(\d+)
        replacement: $${1}:$${2}
        target_label: __address__

proxy-defaults:

Kind      = "proxy-defaults"
Name      = "global"
Config {
  local_connect_timeout_ms = 1000
  handshake_timeout_ms     = 10000
  #protocol = "http"
  envoy_prometheus_bind_addr = "127.0.0.1:9102"
  #envoy_stats_bind_addr = "127.0.0.1:9102"
  #envoy_telemetry_collector_bind_socket_dir = "local"
}

But I’m not using no_dns = true

[sacpk]

@Material-Scientist
Initially, I was using multiple port configurations in my Nomad job like this:

network {
  mode = "bridge"
  port "http" {
    to = 9090
  }

  port "envoy_metrics" {
    to = 9102
  }
}

When I removed the http port configuration and instead added the expose configuration (as shown in your example), Prometheus started scraping the Envoy metrics correctly.

Here’s my full Nomad job for reference:

job "foo" {
  namespace = "MyNS"
  datacenters = ["myDC"]
  type = "service"

  group "foo" {
    count = 1

    network {
      mode = "bridge"  
      port "http" {
        to     = 9090
      }    
      port "envoy_metrics" {
        to = 9102
      }       
    }        


    service {
      name = "foo"
      port = 9090
      


      meta {
        envoy_metrics_port = "${NOMAD_HOST_PORT_envoy_metrics}"
      }
      
      check {
        expose   = true
        type     = "http"
        path     = "/health"
        interval = "30s"
        timeout  = "5s"
      }
       
      connect {
        sidecar_service {
           proxy {
          config {
              envoy_prometheus_bind_addr = "0.0.0.0:9102"
            }
             expose {
              path {
                path = "/metrics"
                protocol = "http"
                local_path_port = 9102
                listener_port = "envoy_metrics"
              }
            }
          transparent_proxy {
              no_dns = true
            }
        }         
        }
      }
    }        
    
    task "foo" {
      driver = "docker"

      config {
        image   = "nicholasjackson/fake-service:v0.26.0"
      }
      env {
        UPSTREAM_URIS = "http://127.0.0.1:9091"
        NAME = "foo"
        MESSAGE = "foo service"
        ERROR_RATE = "0.2"
        ERROR_DELAY = "0.3s"
        TIMING_VARIANCE = "10"
      }
    }
  }
}

Thanks !