Consul 2.0.0 Telemetry Collector not forwarding Envoy and API Gateway metrics to OpenTelemetry Collector

Consul
1

Hello,

I’m deploying Consul Helm chart 2.0.0 on Minikube and I’m trying to export Envoy sidecar and API Gateway metrics to an OpenTelemetry Collector using the built-in Telemetry Collector.

My setup:

  • Consul Helm chart: 2.0.0

  • Minikube

  • OpenTelemetry Collector receiving OTLP HTTP on port 4318

  • Prometheus scraping metrics exposed by the OTel Collector

Consul configuration:

global:
  name: consul
  datacenter: minikube

  tls:
    enabled: true
    enableAutoEncrypt: true

  acls:
    manageSystemACLs: true

  gossipEncryption:
    autoGenerate: true

  metrics:
    enabled: true
    enableAgentMetrics: true
    enableTelemetryCollector: true

server:
  replicas: 1

client:
  enabled: true

connectInject:
  enabled: true
  cni:
    enabled: true

telemetryCollector:
  enabled: true
  customExporterConfig: |
    {"http_collector_endpoint": "http://otel-collector-opentelemetry-collector.observability:4318"}

ui:
  enabled: true
  service:
    type: ClusterIP

  metrics:
    enabled: true
    baseURL: "http://prometheus-kube-prometheus-prometheus.observability:9090"

  dashboardURLTemplates:
    service: "http://localhost:3000/d/heHhNSFf6Na8vIZWRs8H/envoy-global?orgId=1&var-service={{Service.Name}}&var-namespace={{Service.Namespace}}&var-dc={{Datacenter}}"

OTel Collector configuration:

mode: deployment

image:
  repository: otel/opentelemetry-collector-contrib

ports:
  prometheus:
    enabled: true
    containerPort: 8889
    servicePort: 8889
    protocol: TCP

serviceMonitor:
  enabled: true
  metricsEndpoints:
    - port: prometheus
      path: /metrics
      interval: 15s

config:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318

  exporters:
    prometheus:
      endpoint: 0.0.0.0:8889
      enable_open_metrics: true

    otlphttp/jaeger:
      endpoint: http://jaeger:4318
      tls:
        insecure: true

  processors:
    batch: { }

  service:
    pipelines:
      traces:
        receivers: [ otlp ]
        processors: [ batch ]
        exporters: [ otlphttp/jaeger ]

      metrics:
        receivers: [ otlp ]
        processors: [ batch ]
        exporters: [ prometheus ]

What I observe:

  • Application metrics sent directly to the OTel Collector are visible in Prometheus.

  • Telemetry Collector starts successfully and reports no export errors.

  • Telemetry Collector logs show metrics being received.

  • I can manually access Envoy metrics on port 20200 from injected sidecars.

  • However, no Envoy or API Gateway metrics appear in Prometheus through the Telemetry Collector pipeline.

Has anyone successfully used Consul Telemetry Collector in Consul 2.0.0 to export Envoy/API Gateway metrics to an OpenTelemetry Collector?

Are there any additional settings required to enable Envoy metric collection, or is this a known limitation/issue in Consul 2.0.0?

Thanks.

2

I think it would help to first clarify the intended metrics collection model.

Are you trying to use a pull-based Prometheus model, where Prometheus or an OpenTelemetry Collector with a Prometheus receiver scrapes Envoy/API Gateway metrics from the exposed metrics endpoint, for example port 20200?

Or are you trying to use a push/forwarding model, where Envoy or Consul components send metrics into the Consul Telemetry Collector, and then the Consul Telemetry Collector exports them to your OpenTelemetry Collector?

The reason I ask is that being able to manually curl metrics from port 20200 mainly validates the pull-based Prometheus path. It does not prove those Envoy/API Gateway metrics are wired into the Consul Telemetry Collector forwarding path.

If your goal is pull-based collection, I would focus on the scrape annotations, sidecar/gateway metrics settings, and the OpenTelemetry Collector Prometheus receiver configuration.

If your goal is push/forwarding via the Consul Telemetry Collector, I would verify whether the injected sidecar and API Gateway Envoy bootstrap are actually configured to send stats to the telemetry collector, not only expose Prometheus metrics locally.

3

Thanks for the clarification.

My goal was mainly to understand what the intended metrics flow is when using the Consul Telemetry Collector. I wasn’t sure whether the expected approach is for the Telemetry Collector to collect and forward Envoy/API Gateway metrics, or whether those metrics are expected to be scraped directly by Prometheus (or an OpenTelemetry Collector with a Prometheus receiver).

For reference, this is how I currently have Consul configured:

global:
  name: consul
  datacenter: minikube

  tls:
    enabled: true
    enableAutoEncrypt: true

  acls:
    manageSystemACLs: true

  gossipEncryption:
    autoGenerate: true

  metrics:
    enabled: true
    enableAgentMetrics: true

server:
  replicas: 1

client:
  enabled: true

connectInject:
  enabled: true
  cni:
    enabled: true

ui:
  enabled: true

  metrics:
    enabled: true
    baseURL: "http://prometheus-kube-prometheus-prometheus.observability:9090"  dashboardURLTemplates:
    service: "http://localhost:3000/d/adwqlwl/application-performance-monitoring?orgId=1&from=now-15m&to=now&timezone=browser&var-service={{Service.Name}}"

At the moment I’m using a pull-based approach with PodMonitor resources that scrape the Envoy metrics endpoint on port 20200:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: consul-sidecar-metrics
  namespace: observability
spec:
  namespaceSelector:
    any: true
  selector:
    matchExpressions:
      - key: consul.hashicorp.com/connect-inject-status
        operator: Exists
  podMetricsEndpoints:
    - targetPort: 20200
      path: /metrics
      interval: 5s
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: consul-api-gateway-metrics
  namespace: observability
spec:
  namespaceSelector:
    matchNames:
      - consul
  selector:
    matchLabels:
      component: api-gateway
  podMetricsEndpoints:
    - targetPort: 20200
      path: /metrics
      interval: 5s

With this setup I’m able to collect both sidecar and API Gateway metrics in Prometheus.

However, I’ve noticed that the API Gateway metrics still do not appear in the Consul UI, even though the gateway exposes metrics on port 20200 and Prometheus successfully scrapes them. Because of that, I’m not sure whether I’m missing some additional configuration, whether the metrics exposed by the gateway are not the ones expected by the UI, or whether the Consul UI relies on a different metrics collection path.

image
image2332×796 76.6 KB

So my original question was really about understanding the recommended and supported architecture around the Consul Telemetry Collector. If the Telemetry Collector is not intended to ingest Envoy/API Gateway metrics automatically, then using PodMonitor seems to be the correct approach. I just wanted to confirm whether that’s the expected design.