I just realised a bunch of my clients didn’t have the consul client port 8301 open, but everything seemed to work fine for a long time and I did not know. For my own education sake, how would I have been negatively affected by this?
If everything was working fine, that would mean that you had
8301/udp open but
8301/tcp wasn’t. Consul uses
8301/udp for Serf Health Checks and would only use
8301/tcp as a fallback.
If you are sure that both UDP and TCP
8301 were closed, then you probably had the nodes constantly fail the Serf Health Check and getting kicked out of the cluster, but then because these nodes could talk to other nodes on port
8301/udp, they would join back to the Serf Pool.
You will be able to clearly see the above activity in the Consul logs.
Sorry for the late reply. Thanks for explaining what I would be observing with those ports closed @Ranjandas!