Consul Transparent proxy and k8s network policies

Consul
1

Hi all,
By default transparent proxy (TP) limits inbound traffic to meshed only, however outbound traffic is not limited and default is allow ALL for meshed apps.

We want to control outbound traffic for our meshed applications and allow only specific particular destinations of course including other meshed applications.

According the Consul documentations to limit outbound traffic with TP we need to create a

apiVersion: consul.hashicorp.com/v1alpha1
 kind: Mesh
 metadata:
   name: mesh
 spec:
   transparentProxy:
     meshDestinationsOnly: true

This will allow outbound only to the meshed apps.

And according the same docs in order to bypass TP (this is useful in case if we need to have any not meshed apps as destination for our meshed apps) we need to add specific annotations to our deployments

consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: Provides the ability to exclude a list of ports for inbound traffic that the service exposes from redirection
consul.hashicorp.com/transparent-proxy-exclude-outbound-ports: Provides the ability to exclude a list of ports for outbound traffic that the service exposes from redirection.
consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs: Provides the ability to exclude a CIDR that the service communicates with for outbound requests from redirection.
consul.hashicorp.com/transparent-proxy-overwrite-probes (default: true): Provides the ability to overwrite HTTP health probes of a container to point them to Envoy. This allows Pods that have health probes — such as liveness and readiness probes — to still function even if traffic redirection is enabled.

My main question is how to limit outbound traffic for the meshed apps (with TP enabled), but without using “Mesh” and “Annotations” mentioned earlier. This means that we want to keep TP outbound as default (Allow all) and control outbound traffic with native k8s network policies. For this to work we need to know which outbound ports and to where should be opened for the meshed app.

If I do not enable transparent proxy the following network policy works fine. It limits outbound traffic only to meshed apps (of course intention should exist) and DNS. If we use same policy with TP enabled outbound traffic to meshed apps become unavailable.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: np-out-allow
spec:
  podSelector: {}
  egress:
  - to: []
    ports:
    - protocol: TCP
      port: 53
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 8501                     
  - to:
    - namespaceSelector:
        matchLabels: 
          app: consul
      podSelector:
        matchLabels:
          app: "consul"
    ports: 
    - protocol: TCP
      port: 8443
    - protocol: TCP
      port: 8502
    - protocol: UDP
      port: 8301
    - protocol: TCP
      port: 8600
    - protocol: UDP
      port: 8600
    - protocol: TCP
      port: 8600
    - protocol: TCP
      port: 8080   
    - protocol: TCP
      port: 21000
  policyTypes:
  - Egress

Thank you in advance

2

Hi @andriktr,

On Kubernetes, the proxies communicate using port 20000, not 21000. I believe if you update this your NetworkPolicy to permit port 20000, outbound connectivity to meshed apps should work.