Intention doesn't seem to work: Traefik can't proxy requests to any services

Consul
1

I have traefik and an apache server running on nomad but all requests going to traefik result in 502 bad gateway. I suspect that mtls is failing somewhere. I also have a tcp service behind traefik that also cannot be reached.

Some details

  • traefik is connect enabled (services show up in dashboard just fine)
  • traefik and apache have sidecars running
  • intention has been created: consul intention create traefik apache

Traefik job: https://termbin.com/vnkf
Apache job: https://termbin.com/0bbe (details don’t really matter, but it has a sidecar running) (fwiw the postgres upstream doesn’t work either… connection is immediately reset when contacted)

Any help debugging this would be appreciated.

2

Traefik log message:

time=“2022-09-16T23:36:39Z” level=debug msg=“‘502 Bad Gateway’ caused by: EOF”

3

I took a wireshark capture of a connection to one of the sidecars. The proxy accepts the connection but never sends any data back. Not even to finish the tcp handshake.

4

I’m not 100% sure why this fixes it, but something to do with this doesn’t sit well with the proxy.

network {
  mode = "bridge"
  port "http" { to = 8080 }
}

Changed to:

network {
  mode = "bridge"
  port "http" { }
}

…and made use of NOMAD_PORT_http in apache config.

Replicated this fix with some other jobs too. Upstreams also work now.