Dear community,
having to maintain a bare metal kubernetes cluster at work, I am looking for an alternative to control my smart home. I would like to have a solution that lets me focus on the applications themselves, comes with less overhead to maintain and runs locally. Ideally, it works as simple as docker-compose.
In the past, I have been using docker-compose which will be my basis for the new setup. Compose has served me well with the exception of resiliency (running on one server only). Since docker swarm has no future, I have been looking for alternatives. Being new to nomad, I am hoping that this is the right path.
I started this thread, hoping that you can point me in the right direction, validating my assumptions.
Requirements
I would like to have some resiliency in my services with two servers handling the requests so in case one goes down, the other one can take over all other services (short downtimes would be acceptable).
Since it is a two people home only, the load will be quite low. It is more the reliability I am concerned about.
Some cluster services should be accessible from outside the home while other services should only be accessible from the internal home network.
Since the servers are running a smart home solution, they need access to the local network and its devices. Even access to Zigbee (via USB plugged into one server) and Bluetooth is required.
Persistent storage to run (small) databases and store some data.
Infrastructure as code, version controlled. Ideally, Gitops driven.
All persistent data and databases backed up to Backblaze B2.
Monitoring, centralized logging and alerting.
HTTPS for all services through a reverse proxy.
Selective external access to services and the bare metal servers through some type of VPN.
All secrets are encrypted inside my configurations & git repositories.
Some software I intent to run on the cluster:
- home assistant
- nextcloud
- nodered
- freshrss
- paperless-ng
How I am planing on achieving the setup
I have 5 Intel NUCs at my disposal, 3 of them configured with additional SSDs. A NAS provides storage over NFS.
From what I understand, the mimimum number of servers I need for this setup is 5: 3 servers running Nomad, Consul and Vault (server mode) and two servers running Nomad (client mode).
Using cloud-init, I would prepare the 5 hosts with Ubuntu Server 20.04 and give them IP addresses from my local DNS pool (statically assigned). Also, the servers will be controllable through SSH (key provided with cloud-init).
For easy setup, I am planning on using hashi-up. This should give me the base installation with Nomad, Consul and Vault up and running in HA mode. Two servers would be set up as workers (nomad client).
The setup would be done with Task due to simplicity.
So far, I feel this setup is straight forward. Now things become less clear…
How stable are the Nomad Packs ? Should I rather go down this path for all the software available and only create my own jobs where needed?
Ingress & reverse proxy: I am thinking of following the guide Load Balancing with Traefik with input from this blog post. Or would Fabio be a simpler solution (I have never used it)? Traefik should be able to provide ssl certificates so that I do not need another piece of software.
DNS: inital setup would be done with Terraform on Cloudflare.
Persistent storage: no idea yet. Ideally, I can have a solution that leverages the fast SSDs from my three NUCs and for slower, larger storage leveraging my NAS through nfs. Since I have no knowledge in this area, I prefer a simple setup.
Monitoring and alerting: Using Prometheus to Monitor Nomad Metrics. Not sure how to monitor my own services though. What configuration would be required?
Not sure how to set up Grafana though. Any hints?
Logging: Logging on Nomad and log aggregation with Loki
Backups: looks like I would need to build something myself. Thinking of using restic like the gentleman in this repo: hydra/terraform/modules/restic at master · mr-karan/hydra · GitHub. This is what it does: hydra/foss-united-apr-2021.md at master · mr-karan/hydra · GitHub
VPN: I am thinking of using Tailscale or Zerotier to connect securely to some services on my home cluster by running two proxy instances like the gentleman overhere: hydra/foss-united-apr-2021.md at master · mr-karan/hydra · GitHub. I am not sure on the details though since networking is not my strong suite. Should be interesting…
USB: looks like Nomad provides this access in beta mode. No idea how to hook into Bluetooth, though.
For kubernetes I found a great resource to set up a cluster for home usage: https://k8s-at-home.com/. Is there anything like it for nomad and the hashicorp stack?
I am looking forward to your ideas and inputs.