I have prometheus deployed using the standard helm chart inside a kuberenetes cluster (Azure AKS). It happily connects to all upstream services (both inside and outside of the service mesh), but then after a random amount of time all attempts at an external connection through the proxy receive a connection refused error.
I have created a bug issue: K8S Prometheus deployed inside consul service mesh get connection refused on all outbound connections are a random amount of time · Issue #4141 · hashicorp/consul-k8s · GitHub with much more detail.
Here is an example log entry from the prometheus application:
caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: failed to list *v1.Service: Get \"https://10X.X.X:443/api/v1/namespaces/monitoring/services?limit=500&resourceVersion=0\": dial tcp 10.X.X.X:443: connect: connection refused"
One interesting thing is that if I exclude a port from the proxy consul.hashicorp.com/transparent-proxy-exclude-outbound-ports: "10250" then all calls to that port work without problems, so this only affects calls over the transparent proxy. Also, this can work perfectly for hours or even days before randomly failing.