Hi all,
since I could not find any good examples on the internet how to set up a HA cluster with Nomad and Consul, I would like to share my current setup. Maybe someone can benefit from it.
Hightlights are:
- Full high-availability with 3 nodes, any one node can go down with the cluster healing itself
- Ingress management with keepalived and Consul Connect ingress gateway, plus UDP ingress management with NGINX
- Observation of nearly everything, including traffic between services and a lot of other metrics
- Online backups of the databases via Nomad Actions
- Distribution of the worker jobs across all compute nodes for Immich
The setup is split into three repositories
- hashilab-core - basic HA setup
- hashilab-support - additional services
- hashilab-apps - end-user applications
Comments are totally welcome! There are still some rough edges, but I’m quite happy with the setup right now.