Hello,
I have a restricted environment in which we have to install consul based services (basically porting aws environment where we rely on Consul for everything, into customers on-prem, which allows only ports 443 and TLS communication across all his VMs), but since Consul requires 8300, 8301, 8600, it seems like an impossible task.
Maybe someone has ideas on how is it possible to run Consul based infra and services over 443 only?
I was thinking about utilizing the service mesh, Envoy based, for our application itself. But in order for service mesh to work, consul agents and masters have to connect first, but if they’ll grab the 443 for their connection, how Envoy would listen on this port then, for the mesh purposes and the applications connections. Some chicken and egg problem… if only I was able to setup a mesh and then put consul to work over that mesh - but I need consul to manage the Envoys … tricky stuff… Is overlay network the only way? (organizing the overlay using some Weave or Calico, and running Consul and all other parts on top of it, but then it’s like building half-kubernetes from scratch, reinvent the wheel)
You should able to utilize Consul’s WAN federation via mesh gateways to federate two Consul clusters via mesh gateways over port 443.
See Federation Between VMs and Kubernetes for a high level walkthrough of this process. This process can be adapted to establish federation between two VM-based environments.
Thanks for the suggestion!
Can I somehow utilize this method to replace the LAN gossip and communication between consul agents to use the proxy too?
The environment will be internally restrictive, each VM in a pool of 10-20 VMs that are going to form the infra, will not be reachable on ports 8000+ that are required for agent<->master and agent<->agent connections, so they could only somehow communicate using Envoy from the beginning. But I don’t see an option to run it like that. Seems like they can run mesh-gateway for connections to some external locations, like another cluster or external resource, but how to direct the ‘administrative’ consul traffic through the mesh starting from first connection of agent to master/peer?
To be honest, I don’t think what you ask is easily achievable. The only way I can see this work (and this just pure speculation at this point), is by treating each VM as a one-node Consul-cluster in each own datacenter, and use the mesh-gateway functionality.
Yeah seems like an overlay network that runs through some tunnel encapsulation on port 443, is a better way, probably only one. Or a manual setup of Envoy and its proxying settings, so it’ll receive all traffic on 443 whether it’s consul one or application, and reshuffle it from there to the local containers - but then I need to manage Envoy myself and losing the point of Consul Connect feature
… tricky stuff… Is overlay network the only way? (organizing the overlay using some Weave or Calico, and running Consul and all other parts on top of it, but then it’s like building half-kubernetes from scratch, reinvent the wheel)