Issue accessing service when using static ports

Nomad
1

Hi,

I’m having an issue currently when using traefik as an ingress proxy to my services. I have several ports defined with a static port which worked up untill a few days ago. What I find weird is that if i comment out the static portion in the ports stanza it is now accessible (through the randomly selected port) but putting it back to static (80,443,8081,ect) it’s inaccessible.

I’m using consul connect and also have traefik set to native mode to make use of the consul connect functionality. I’ve spent this weekend trying to find out whats going on but to no avail so was hoping to see if anyone else has any ideas on what could be happening?

2

Hi @CarbonCollins, this sounds like a strange one. Do you have a reproduction you could share so that I can test this?

I would suggest checking the iptables rules on the machine(s) that run the Traefik application as this sounds like there are potentially orphaned rules affecting routing. Without a way to reproduce the problem, I can’t think of anything else currently that might be causing this.

Thanks,
jrasell and the Nomad team

3

Hi, I spent the rest of yesterday debugging further and its looking less like a Nomad issue now and more like a docker issue :sweat_smile: its looking like traffic is not crossing the docker bridge from the host network (still cant figure out why though!) I can access the service if i use the docker container ip but not if I use the host ip…

I did end up checking iptables during this search and ended up flushing all iptables rules and adding allow all rules as it was looking like a routing/firewall issue but it did not seem to change anything…

Due to this all though i’m not sure how to reproduce it… It is however occuring on at least two of my nodes.

4

So good news, seems like I missed some chains in iptables (CNI-DN, CNI-HOSTPORT-DNAT, ect) and had not flushed them all… after actually flushing all the chains and then and clearing the persisted rules (they had also been persisted so were being loaded when I rebooted) I could start to access services again. Not sure if its linked to this issue: iptables entries are not reconciled · Issue #6385 · hashicorp/nomad · GitHub. but in either case the steps in that issue helped me find that iptables -L does not list all rules at least :sweat_smile: