Hi @balupton I am doing something similar to you. It’s an ongoing project I started over the holidays: https://hashiatho.me
In my experience so far (which, I should say is closer to “beginner” than “knowledgeable”) you need something get the hardware to a “terraformable” state. In my understanding that basically means that you need to expose terraform providers, ie APIs. Since you’re starting from bare metal which has no APIs, there is a bootstrap problem. I got around this by using Ansible to deploy the intial layers of things I want to Terraform later, e.g. bare-bones Consul and Nomad clusters: Playbook
You can terraform these after they have been deployed, but to get them to a deployed state, you need something that can talk to the lower level of the stack, i.e. the actual OS. You could try using something like the Linux provider, but this provider uses SSH - so you anyway need to have a working OS and SSH user on the remote end. Again, we are back to a bootstrap problem.
There are several options available to bootstrap the local “datacentre” hardware (where datacentre is your bunch of pis), but I would highly recommend Ansible due to it’s readability and availability of community resources. The other options include:
Maas: a “metal as a service” product from Canonical.
Digital Rebar - will do bare-metal deploys of OS for you and works ok with Ansible and Terraform.
Foreman - a bit heavy for RPis, with a few external dependencies, but very complete.
For a small local setup, all of these are huge overkill – unless you’re actually trying to learn how to boostrap and manage an entire datacentre. If you goal is just to get something up to use then I would suggest going the simpler route with plain Ansible playbooks.
One thing I haven’t tried yet is to use Packer to provision the boot images for your pis, and pre-configure your Consul and Nomad configurations on them with appropriate provisioners. Then, when you boot up the pis, they should already have podman installed, and should auto-join Consul and Nomad clusters as desired. It’s not Terraform, but at least it’s simple