I want to create a Consul cluster with some Raspberry Pi computers and a central server on the Internet. Each of the RPi computers will be on a different LAN.
NOTE: I appreciate that having a single Consul server is not suitable for production, but this is an idea I am playing with.
I have configured the Consul server on the Internet and have enabled ACLs for both of the software. I have installed Consul on the RPi computers and indeed they can communicate with the central server, but the health check is flip-flopping from healthy to not-healthy.
Looking into this I think it is because the address that the client has advertised the LAN address to the server rather than its WAN address. I have tried to use the advertise_addr_wan and translate_wan_addrs options but this has not helped.
I have been looking at the architecture diagrams for Consul and see that it is possible to connect servers across the WAN from different data-centres but not (obviously) clients. Do I need to run a server and a client process on each of the RPi so that each one is its own Data Centre and then connect the DC servers together across the WAN?
After I have set this up I want to configure Nomad to use Consul and have a Nomad agent on each of the client RPi computers. I assume that I will encounter similar issues with IP addressing. Am I correct?
I do not want to resort to having to run a VPN for all the traffic between the nodes nor do I want to install Docker and run Kubernetes because the app I will be deploying is not natively in a container.
Any ideas on how to accomplish this is very gratefully received.