Consul Connect - Updating the proxy-defaults global config for metrics & tracing

Consul
, ,
1

I’ve updated the consul connect proxy-defaults to use envoy_dogstatsd_url envoy_extra_static_clusters_json and envoy_tracing_json as follows based on Nicolas Jackson’s fake service with Datadog :

{
    "Kind": "proxy-defaults",
    "Name": "global",
    "TransparentProxy": {},
    "Config": {
        "envoy_dogstatsd_url": "udp://127.0.0.1:8125",
        "envoy_extra_static_clusters_json": "{\"connect_timeout\":\"3.000s\",\"dns_lookup_family\":\"V4_ONLY\",\"lb_policy\":\"ROUND_ROBIN\",\"load_assignment\":{\"cluster_name\":\"datadog_8126\",\"endpoints\":[{\"lb_endpoints\":[{\"endpoint\":{\"address\":{\"socket_address\":{\"address\":\"localhost\",\"port_value\":8126,\"protocol\":\"TCP\"}}}}]}]},\"name\":\"datadog_8126\",\"type\":\"STRICT_DNS\"}",
        "envoy_tracing_json": "{\"http\":{\"config\":{\"collector_cluster\":\"datadog_8126\",\"service_name\":\"envoy\"},\"name\":\"envoy.tracers.datadog\"}}"
    },
    "MeshGateway": {},
    "Expose": {},
    "CreateIndex": 2178817,
    "ModifyIndex": 2178817
}

I think the config looks right. , but since deploying this update a few hours ago, I have yet to find any envoy.* metrics in Datadog other than the one service I had previosly instrumented them for. (This one service has been sending envoy metrics since before the update as it is an ec2 instnace with a datadog agent that has an envoy check configured to run against http://localhost:19000/stats to gather them. )

What is needed to get this new config to begin gathering metrics from the other services proxy sidecars? I’ve issued a SIGHUP to test, but maybe the services need to re-register with consul?

3

Hi @mlindes,

If you are using Consul 1.10 with a recent version of Envoy, you’ll need to update the escape-hatch config to be compatible with xDS version 3, per the upgrade guideance at Consul 1.10: Envoy xDS Protocol Upgrades.

Here’s an example proxy-defaults config which uses the updated syntax.

Kind = "proxy-defaults"
Name = "global"

Config = {
  envoy_extra_static_clusters_json = <<EOL
    {
      "name": "datadog_8126",
      "type": "STRICT_DNS",
      "connect_timeout": "3.000s",
      "dns_lookup_family": "V4_ONLY",
      "lb_policy": "ROUND_ROBIN",
      "load_assignment": {
        "cluster_name": "datadog_8126",
        "endpoints": [
          {
            "lb_endpoints": [
              {
                "endpoint": {
                  "address": {
                    "socket_address": {
                      "address": "127.0.0.1",
                      "port_value": 8126,
                      "protocol": "TCP"
                    }
                  }
                }
              }
            ]
          }
        ]
      }
    }
  EOL

  envoy_tracing_json = <<EOL
  {
    "http": {
      "name": "envoy.tracers.datadog",
      "typed_config": {
        "@type": "type.googleapis.com/envoy.config.trace.v3.DatadogConfig",
        "collector_cluster": "datadog_8126",
        "service_name": "envoy"
      }
    }
  }
  EOL
}
4

Thanks @blake . After updating the escape hatch config to be compatible with xDS v3, I’m getting some metrics… but I’m not convinced I’m getting them all.

I’m also confused why editing the tracing endpoint on port 8126 impacts the envoy_dogstatsd_url endpoint on udp:8125.

I’m going over the envoy docs in some more detail but would love some additional guidance/explanation…