Corrupt Databases and Storage

After some days of digging into Nomad I managed to create a nice on-prem cluster with 3 servers and 4 clients. So far it has been a very enjoyable ride, and I am actually finally thinking of converting my Old School Docker Compose-Stack towards Nomad.

Only thing that won’t work quite well is Storage and Databases. I managed so install a simple NFS-Server on a separate VM which I then mounted on all my clients:

  host_volume "mysql-nfs" {
    path      = "/mnt/docker/mysql"
    read_only = false
  }

This worked after some tweaking regarding docker permissions, so I started doing some failover-tests like what happens If one client loses Power and the Database (MySQL) is suddenly restarting on another Client. You might have guessed it already: It didn’t take me too long to break / corrupt the database.

So I was thinking of migrating to something like GlusterFS and using CSI since I want to stay On-Prem. But will this actually solve this kind of issue?

I did some reading on Databases and Docker in general and came to the conclusion that at least relational databases are kind of a big no no regarding Containers. So should I simply stop using Databases in Containers, or is there a more suitable solution in order to make this setup more robust?

The reason I am asking is I don’t want to sink more hours into something that might be the wrong approach in general (meaning more complex storage).

Please guide me a little here, thanks.