There are a few ways you could figure this out. Since you are using Envoy, if you have the admin interface bound to localhost:19000 which is the default then the stats endpoint should give some information about where the connection could be going wrong.
The envoy docs are pretty decent with regards to documenting the stats. If there was a problem with the mTLS connection to the proxy then that should show up in the listener stats. If the problem were between Envoy and the application then the general statistics should help to detect that. In particular the upstream_cx_* ones should give insights about the connection.
Another thing I personally like to do is inject a container running tcpdump or tshark into the network namespace of the Envoy container.
The docker image I use for this is from this dockerfile:
FROM alpine:latest
RUN apk add --no-cache tshark
ENTRYPOINT ["/usr/bin/tshark"]
CMD []
Then you can run it like: docker run -ti --rm --network container:<envoy container name or id> tshark <tshark arguments>
Some helpful tshark arguments I use are:
-
-V- This outputs much more in depth packet decoding and will show TLS information as well as any TCP/UDP/HTTP(s) information for the unencrypted side of the proxy. -
port <application port> and host <IP of the proxied application>- To debug a connection between the proxy and one endpoint. -
port <public listener port>- To debug the main proxy listener. This will be a mTLS connection.