Ok so I have confirmed that there is no firewall blocking any ports.
Another thing I have noticed is that if I consider only logs with word error in them and then see those on timeline then each lump of instability starts with messages like
2020-12-02T09:54:44.326Z [WARN] agent: error getting server health from server: server=consul-server-1 error="context deadline exceeded"
2020-12-02T09:54:44.326Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
This is actually mentioned in troubleshooting section of consul. We will try implementing monitoring as advised there.
It is a new installation with barely any load so monitoring was not setup yet. Let’s see.