Nomad as a mid-tier orchestrator


I’ve recently fallen in love with the Hashistack, and I’m working on integrating into a few different places. The main test drive right now is seeing how well it works to host services for an open-source video game I’m working on, Veloren. We have a ton of different items that need centralized hosting, and Nomad feels so refreshing after trying to learn k8s.

We have a breadth of items that we plan on running, including web sites, forums, auth tools, CDNs, game servers, and probably a lot more. Currently, we use Docker Compose to run most of our stuff. This works well enough, but doesn’t scale well, and doesn’t offer great deployment options.

We recently provisioned a dedicated server in Germany that we’ll be using to host our smaller services, and play around with. Note, this dedicated server just serves an OS on the metal, not a hypervisor to run multiple VMs. We don’t have a massive budget for servers, we only have what we get from our Open Collective, which is around $200 USD monthly. We spend about a third to a half on servers. So, our Nomad server will be a single node for the time being.

I know this means that we run the risk of losing availability should our single server go down. But for the time being, this is all we’ll have provisioned. I’ve read many places that it’s not good to run dev Nomad/Consul in production, since if we over-provision we’ll have a fun time with OOM.

So my current game plan is to host several Vagrant-managed VMs on our server. I’m thinking 3 Nomad server VMs, and 2 Nomad client VMs. This will be so we can simulate a larger cluster (which we’ll inevitably have in the future), specifically with inner-VM traffic. Further, the VMs should restrict resources to make sure that our clients can freely take as many jobs as they like without worrying about bricking the box. The server VMs would have quite limited resources, but there would be 3 as to simulate leader election and stuff.

In the future, we’ll hopefully host game servers in geo-dependent data centers. The ease of Nomad orchestrating this multi-cluster fleet seems like one of the biggest advantages compared to k8s. So with our “proof of concept”/temp production server, we’d want to try and get everything simulating as if it was a much larger cluster.

So, for the questions. I’d love to get some insight into some Nomad things. Primarily, does this pattern (Vagrant VMs hosting production Nomad/Consul) make sense? Are there any potential foot guns that I haven’t considered yet? Any thoughts about where we are as a project would also be great, I’m sure people who work with massive clouds will have a lot of thoughts on our setup.

We try to be super transparent with our infrastructure/finances/roadmap, so if there is anything that would provide more information, I’d be happy to go over it!