Newbie Qs about glusterfs

Slowly setting up a raspi cluster with ansible and nomad.
Is there a best practice for glusterfs?

I have a choice to run glusterfs on each node through systemd or run glusterd system job on the appropriate nodes.

AFAIK, there’s not established best practice around this or any specific reason why one should generally do it one way or the other.

That being said, while you could run glusterfs server as a nomad job, I think it’s better advised to run it outside of nomad; if you wish to use any of your gluster volumes as nomad host volumes, nomad will not start when they’re not mounted. I think you should run it as a systemd job directly on the host.

While nothing’s stopping you from having nomad server/nomad client/gluster server all on the same host, you should ideally have the glusterfs servers on dedicated hosts.

For your scenario, this means that if you’re running glusterfs replicated and want a proper HA cluster, you need a minimum of 10 raspberry pis:

  • 3 consul servers
  • 3 nomad servers
  • 3 glusterfs servers (one of which can be arbiter)
  • 1 nomad client for running jobs

If you want to compromise on any of these, that would be running nomad clients on either of gluster servers or nomad servers (so minimum of 9). At the very least, don’t have the same host run consul server/nomad server/glusterfs server.

glusterfs adds a quite significant overhead and you’ll get disappointing results running anything else on your gluster servers on raspberry pis.

(I personally have a cluster of raspberry pis so I’m speaking from experience here. For my use-cases, rpi 4b doesn’t cut it for glusterfs servers)

Also do make sure you’re running arm64 and not arm32 whether it’s for clients or servers - glusterfs is buggy and unsupported on arm32.