Consul Connect dynamic upstreams

Consul
1

Hi,

We have an application split up in several microservices.

  1. One “central” microservice must be able to realtime discover all microservices filtered by a specific tag (let’s say a tag “plugin”).

  2. Then this “central” microservice must be able to call (http) these “plugin” microservices when needed

For the “discovery”, no problem; we’re using the Consul registry/discovery API and it’s perfectly working.

For the “calls”, we want to use Consul Connect and route the calls through the Mesh using Envoy.

Currently, if we want the “central” microservice to be able to call a “plugin” microservice, we need to:

  • update its Proxy configuration to add the “plugins” upstreams
  • update the app conf to indicate that plugin x is reachable using the envoy port xx

It’ absolutely not dynamic… :frowning_face:

Our goal is to deploy plugins without any action on the Central Service

Below a simplified example of our issue:

{
  "ID": "app-central-1-proxy",
  "Name": "app-central-proxy",
  "Kind": "connect-proxy",
  "Proxy": {
    "DestinationServiceName": "app-central",
    "DestinationServiceId": "app-central-1",
    "LocalServicePort": 7000,
    "Upstreams": [
      {
        "DestinationType": "service",
        "DestinationName": "app-plugin-x",
        "LocalBindPort": 8001
      },
      {
        "DestinationType": "service",
        "DestinationName": "app-plugin-y",
        "LocalBindPort": 8002
      },
      {
        "DestinationType": "service",
        "DestinationName": "app-plugin-z",
        "LocalBindPort": 8003
      },
      {
       ...
      }
    ]
  },
  "Port": 21000
}

This kind of use-case looks pretty common; what’s the best way to deal with?

1 Like
2

I thought the whole point of a service mesh sidecar proxy implementation was that it could intercept outbound traffic from the service and decide what to do with it.
If we have to update every service to connect to localhost and manually configure the proxy ports… this seems like an overcomplication… istio and aws app mesh both handle this for you, and you just reference virtual services by host name.

Short version: I’m right there with you on needing this dynamic upstream functionality, and I’m surprised this isn’t how it is supposed to work out of the box, with consul providing the discoverability and connect linking the services.

2 Likes
3

+100 agree with the OP and @jcampbell . There seems to be either:

  • a gap in how the Connect proxy upstream configuration is documented
  • a gap in the intended functionality altogether

Typical service meshes might need to support at least 50+ different services that need to communicate with one another. I’m thinking of contemporary microservice architectures in a production environment.

If that’s the case, I’m unsure of how Connect would allow for:

  • variable # of upstreams from the perspective of each of these service nodes
  • abstracting the networking from the service developers, such that they would not need to know which local port to call when calling out to a particular upstream.

Isn’t Connect meant to simplify this interface so that service “foo” only needs to call service “bar” via something like:

GET localhost:{common_egress_port}/bar/resource

The example in the Connect/Envoy section of the docs provides the counting and dashboard services - thus, only two nodes that need to communicate. The configuration in this example is clear, but seemingly does not scale to real-world application in the examples mentioned in this thread.

I’d love to know if we’re misunderstanding the documentation, or overlooking some implied dynamic behavior that handles the cases we’re thinking of. If not, it’d be great to get clarification on if we’re simply jumping the gun on things that are on the Connect roadmap + allow us to look for other options that satisfy our use case in the time being.

1 Like
4

Hi @jcampbell and @parkedwards,

You are not misunderstanding the documentation. Currently Consul requires that upstream destinations and ports be manually declared, and it does not support transparent traffic redirection. We’re using the GitHub issue below to track this feature request.

The good news is that this feature is on our near term roadmap, and is targeted for Consul 1.10.

I hope this helps. Let me know if you have any other questions.

5

Thanks for the confirmation, Blake