Connecting load balancer frontend with backend services securely

Hey there! I hope there’s a simple solution for this, as I’ve sort of reached the limit of my understanding for how to glue this all together.

Right now I have a fairly simple setup: I have Traefik fronting HTTP(S) traffic to the various services running in my Nomad cluster. I’m using the consul catalog method to let Traefik know what runs where.

End user ↔ Traefik ↔ nomad service

This works great, except that the second connection - between Traefik and the backing service - isn’t secured. Traefik does terminate TLS, but the reverse proxied connection just ends up being proxied to a plain HTTP listener wherever that service may be running.

End user ⬅ TLS ➡ Traefik ⬅ Plain HTTP ➡ nomad service

Note that I do have Traefik running as a Nomad service as well. What is the simplest - or best - way to secure that reverse proxied connection?

Consul connect is, of course, the canonical way to secure connections between services, but Traefik is filling it its service catalog based upon the address advertised in consul catalog. That’s a plain IP address of the Nomad host and Traefik as of the time of this writing isn’t consul connect-aware.

Maybe there’s a way to leverage consul connect to do this while still getting the convenient automatically populated service catalog? I’m actually happy to swap out that Traefik layer with some other HTTP(S) reverse proxy if that makes this possible, the primary reasons I’m relying on Traefik are for a) the automatic TLS and b) the automatic reverse proxy functionality (routing requests for foo.bar.local to the right nomad service).

What are the options to ensure the request is secure/encrypted from the user request all the way to the nomad service listener?

Something I’ve had great success with is using Vault’s PKI engine with Traefik. Simply trust the CA (you can pull it directly in a template stanza, or throw it into a consul/vault key, CAs can be public) and use the cert on your nomad service using a {{ with secret "pki/issue/foo.bar.local" }}{{....}}{{ end }} (see PKI - Secrets Engines | Vault by HashiCorp).

If you want, you can even issue a traefik.bar.local cert and present that to foo.bar.local and have full mutual TLS internally as well.

1 Like

Have absolutely no experience with this, but maybe Gateways | Consul by HashiCorp can help?