These are very early steps for me with Consul, mind.
So, I’ve spun up a ““datacenter”” cluster of 5 server agents. I’ve disabled serf_wan port (-1) on purpose, and I’m using retry_join. All the agents have regular/public IPv4 addressing (although not Internet routable), and they span 3 different continents with latencies ranging from 40 to 220ms between them, depending on the source and destination locations.
They join and the UI shows them as expected. I added a key to the KV store, and all nodes respond with the value. I bring two nodes down, and the DNS service records update accordingly. A Vault cluster is also being monitored. I have no standalone agents yet though, but so far, it looks good!
But, am I subverting Consul?! I’m asking this because all the documentation is telling me that this shouldn’t happen, that all the nodes should live in the same subnet, and that at most WAN federation can be used via Serf Wan, which is able to sync services status, but will not sync the KV store.
But so far I seem to be getting away with it, so I wonder, what kind of internals in Consul would actually work against this kind of deployment of a “datacenter” in a global WAN environment? What could go wrong here?
Your considerations would be very much appreciated.