Preventing egress / outbound traffic from container, except via sidecar gateway/proxy

Nomad
1

I would like to run a container image, without giving it full access to the network.
It should have no access to the network, except via a sidecar gateway/proxy that implements firewall/access control.

In Docker-Compose, you can use “networks”.
But it doesn’t seem that Nomad offers similar functionality.

Example: Using Envoy for egress

In Kubernetes this would be configured with NetworkPolicy, I think?

Can something like this be achieved with Nomad?

I was thinking about the following options:

  • use CNI plugins, e.g. bridge, macvlan, firewall, etc (but I can’t find any relevant examples on usage)
  • write my own CNI plugin to configure iptables, create networks/bridges
  • use a socket file that is shared between container and sidecar, so container has no network access
  • use task lifecycle hooks to add/remove iptables/networks/bridges
  • use podman driver and rewrite it to support custom syntax to configure network/bridge/iptables
  • use in-container iptables is not relevant, because it would require extensive customization of third party container images
2

If the container doesn’t need any access at all, you could set the network mode to “none”.

group "container" {
  network {
    mode = "none"
  }
}
3

Unfortunately, not a solution. The ideal configuration would be like docker-compose, where the networks can be specified on each container:

networks:
  internet: {}
  no-internet:
    internal: true
    
services:
  proxy:
    networks:
      – no-internet
      – internet

  protected-container:
    networks:
      – no-internet
    links:
      – proxy
4

what I do for this is that I insert iptables rules in the NOMAD-ADMIN chain of my Nomad nodes where I allow only minimal connections I know all containers should have, like

      -N NOMAD-ADMIN
      -N LOGDENY
      # Connexions établies
      -A NOMAD-ADMIN -m state --state RELATED,ESTABLISHED -j ACCEPT
      # Des nodes vers les containers
      -A NOMAD-ADMIN -s 10.99.9.0/16 -d 172.26.64.0/20 -m state --state NEW -j ACCEPT
      # Des containers vers les nodes
      -A NOMAD-ADMIN -s 172.26.64.0/20 -d 10.99.9.0/24 -m state --state NEW -j ACCEPT
      # Des containers vers les DNS Gandi pour les challenges ACME
      -A NOMAD-ADMIN -s 172.26.64.0/20 ! -d 172.26.64.0/20 -m state --state NEW -p udp --dport 53 -j ACCEPT
      -A NOMAD-ADMIN -s 172.26.64.0/20 ! -d 172.26.64.0/20 -m state --state NEW -p tcp --dport 53 -j ACCEPT
      # Des containers vers l'API TrueNAS pour le stockage CSI
      -A NOMAD-ADMIN -s 172.26.64.0/20 -d 10.99.3.27 -m state --state NEW -p tcp -m multiport --dports 80,443 -j ACCEPT
      # Des VPN vers les services
      -A NOMAD-ADMIN -s 10.99.20.0/24,10.99.23.0/24,192.168.7.0/24 -p tcp --dport 20000:32000 -m state --state NEW -j ACCEPT
      -A NOMAD-ADMIN -s 10.99.20.0/24,10.99.23.0/24,192.168.7.0/24 -p udp --dport 20000:32000 -m state --state NEW -j ACCEPT
      # Entre les containers
      -A NOMAD-ADMIN -s 172.26.64.0/20 -d 172.26.64.0/20 -m state --state NEW -j ACCEPT
      # Input Traefik
      -A NOMAD-ADMIN -d 172.26.64.0/20 -p tcp -m multiport --dports 5443,5080 -m state --state NEW -j ACCEPT

Then I deploy a set of special, unrestricted nodes (without rules in NOMAD-ADMIN) for consul mesh gateways (I’m using the new node_pool feature of 1.6 for this). And all outbound trafic from containers must go through a published service exposed by one of those egress gateways

1 Like
5

I am not familiar with any feature that mimics the “link” functionality with docker compose. All tasks within a group are going to share the same network. There are a couple avenues to look to accomplish this. I haven’t tested some of them myself, so no promises :slight_smile:

  • Use the firewall CNI plugin
  • SBR could possibly work
  • @dbd suggestion would work and you could run those iptables commands as a pretask, however I’d make sure to consider the security management considerations of that. Meaning, you will need to use raw_exec driver, so you’ll need to tightly control who has access to that job and Nomad in general.
    *And lastly, Consul service Mesh. This is honestly the best route. It has the best security mechanisms, and it scales much much better than docker compose. It’s honestly not that bad to set up.

Hope this helps!