Consul api-gateway tracing

jrudowski · January 15, 2024, 11:57am

Hi all,

I wanted to check if there are any plans for the near future to make random sampling for traces in Envoy configurable in an easy way - with special attention to api-gateway / ingress proxies?

So far I see that RandomSampling in Consul’s configuration is set to 0, disabling tracing effectively - this is the default opposite to Envoy’s own setting, which is 100% by default.
What is the reason for disabling it in Consul Connect’s Envoys with such a change?
Referring to this code piece:

github.com

hashicorp/consul/blob/v1.17.0/agent/xds/listeners.go#L2477


      
          		AccessLog:  accessLogs,
          		StatPrefix: makeStatPrefix(opts.statPrefix, opts.filterName),
          		CodecType:  envoy_http_v3.HttpConnectionManager_AUTO,
          		HttpFilters: []*envoy_http_v3.HttpFilter{
          			router,
          		},
          		Tracing: &envoy_http_v3.HttpConnectionManager_Tracing{
          			// Don't trace any requests by default unless the client application
          			// explicitly propagates trace headers that indicate this should be
          			// sampled.
          			RandomSampling: &envoy_type_v3.Percent{Value: 0.0},
          		},
          		// Explicitly enable WebSocket upgrades for all HTTP listeners
          		UpgradeConfigs: []*envoy_http_v3.HttpConnectionManager_UpgradeConfig{
          			{UpgradeType: "websocket"},
          		},
          	}
          
          	if opts.tracing != nil {
          		cfg.Tracing = opts.tracing
          	}

I see that right now it’s expected for Envoy to receive “x-client-trace-id” header set, to enable tracing.
The problem is that in the environment where I have Consul Connect mesh the requests are not coming with such header and I, unfortunately, have no way to inject it before the traffic reaches “api-gateway”.

I know that I could use customized proxy-defaults, overriding the default random sampling - but that would require also customizing a lot of things, that I don’t need nor want customized, if I understand things correctly. And those defaults would also affect sidecar proxies, messing with our current configs, so I’d have to add perhaps another customized configurations for listeners etc. on sidecars.
Doesn’t seem particularly user-friendly to me right now.

Another relatively simple option I see - but also not the greatest one all in all - compiling custom consul binary, with RandomSampling switched back to 100.
Or maybe even adding some more logic/configurability ourselves to the code, but that’s - again - not really such a great way to have to maintain by ourselves custom code for otherwise little changes we need.

It would be nice to have that racing a bit better configurable out of the box, I guess.

Or am I still missing some options, that might be easy to implement, to have tracing from everywhere in the mesh, including gateways?
Open to suggestions in my “quest for an easy tracing enabling”.

On GitHub I saw some suggested changes, but still in “Open” state, perhaps abandoned already, with latest entries from around 2 years ago - but would be really glad to see if there are any changes related to tracing configuration in the coming versions.

Kind regards
Jacek Rudowski