Trying to do something which I thought would be quite simple.
I have a nomad cluster running, with all nodes having only a single public network interface and the docker bridge interface. All intra-cluster comms is secured with TLS and all intra-service comms is secured using consul service mesh. So everything is encrypted end to end, but going over the public internet.
I have a consul ingress running on one of the machines in my cluster at https://<server_public_ip>:<random_nomad_assigned_port> and I have traefik running on every single worker instance as well.
What I am trying to do is setup UFW (or any software based firewall, obviously they all use IPTABLES under the hood) to block off access to any port on my worker instances except traefik on 443 to the public, but allow access to any port, by other members of the cluster.
The UFW rules are respected for anything run outside of nomad (i.e nomad and consul) and I have also managed to get container just managed by docker to work with UFW by following this: How to manage IPTables rules with UFW and Docker
However anything that is run by nomad, doesn’t respect the UFW rules and is completely accessible to the world. I am assuming the issue is something to do with how CNI modifies IP tables rules.
Does anyone have any suggestions on how to get this to work? I am keen to avoid using a separate VPN or overlay network between the cluster, as that is all being handled by consul service mesh.