Specifying service tags in upstreams while preserving L7 features

Sort of like this thread: Upstream with service tags

In my environment, we have hundreds of environments. We want to be able to specify that services within one environment can find other instances of the same service within that environment. Before consul, we’ve used a subdomain to manage this, e.g. rx3-someservice.testing.covermymeds.com routes only to the instances of someservice within the rx3 environment.

I can see that we could give our services some ServiceMeta like environment=rx3, then create a prepared query using regexp+templating to resolve rx3-someservice to someservice where ServiceMeta.environment == rx3

However, my hunch is that using a prepared query precludes me from being able to use the L7 routing features (something I also want to take advantage of). If that isn’t true, I would love some information about how those features interact with each other (does the prepared query come before L7 routing? After?)

If not, the suggestion was to use a service resolver. Is it possible to force a service resolver from the upstream definition? i.e. for my rx3 instances, I would define my upstreams as pointing to the rx3 subset.

Otherwise, I’m not quite sure how to make this work. I can see that generally you’re meant to use a service-router to select the appropriate subset, but in my case there are no query parameters, headers, or path elements that would indicate the right subset. I need to define the criteria at the upstream definition level rather than by selecting something about the query my app is producing.

One thought I had was to define the upstreams with a prefix, sort of like how the prepared query interface works, and then use a regex within the router definition. e.g.

Name = (.*)-(.*)
Destination {
  Service = "${Matches[2]}"
  ServiceSubset = "${Matches[1]}"
}

Hi @chrisjohnson,

You’re correct that prepared query upstreams are not compatible with the L7 routing features. Consul proxies all traffic as pure TCP when the upstream is a prepared query destination – a limitation that seems to be missing from our docs. :thinking:

Unfortunately this is not currently possible. You would need to provide some information in the request such as a header or query parameter containing the target environment, and then use that to route to a service subset which scopes services based on the environment tag.

There’s an issue / feature request on GitHub about adding zone-aware routing to Consul (Provide a way to configure Zone aware routing using Envoy Proxy · Issue #7401 · hashicorp/consul · GitHub) which feels like it could be applicable to your use case. Each environment could potentially be a “zone” in Consul, and then Envoy could be configured to prefer routing to instances within the same zone instead of anywhere in the data center.