Nomad as a mid-tier orchestrator

Nomad
1

Hello!

I’ve recently fallen in love with the Hashistack, and I’m working on integrating into a few different places. The main test drive right now is seeing how well it works to host services for an open-source video game I’m working on, Veloren. We have a ton of different items that need centralized hosting, and Nomad feels so refreshing after trying to learn k8s.

We have a breadth of items that we plan on running, including web sites, forums, auth tools, CDNs, game servers, and probably a lot more. Currently, we use Docker Compose to run most of our stuff. This works well enough, but doesn’t scale well, and doesn’t offer great deployment options.

We recently provisioned a dedicated server in Germany that we’ll be using to host our smaller services, and play around with. Note, this dedicated server just serves an OS on the metal, not a hypervisor to run multiple VMs. We don’t have a massive budget for servers, we only have what we get from our Open Collective, which is around $200 USD monthly. We spend about a third to a half on servers. So, our Nomad server will be a single node for the time being.

I know this means that we run the risk of losing availability should our single server go down. But for the time being, this is all we’ll have provisioned. I’ve read many places that it’s not good to run dev Nomad/Consul in production, since if we over-provision we’ll have a fun time with OOM.

So my current game plan is to host several Vagrant-managed VMs on our server. I’m thinking 3 Nomad server VMs, and 2 Nomad client VMs. This will be so we can simulate a larger cluster (which we’ll inevitably have in the future), specifically with inner-VM traffic. Further, the VMs should restrict resources to make sure that our clients can freely take as many jobs as they like without worrying about bricking the box. The server VMs would have quite limited resources, but there would be 3 as to simulate leader election and stuff.

In the future, we’ll hopefully host game servers in geo-dependent data centers. The ease of Nomad orchestrating this multi-cluster fleet seems like one of the biggest advantages compared to k8s. So with our “proof of concept”/temp production server, we’d want to try and get everything simulating as if it was a much larger cluster.

So, for the questions. I’d love to get some insight into some Nomad things. Primarily, does this pattern (Vagrant VMs hosting production Nomad/Consul) make sense? Are there any potential foot guns that I haven’t considered yet? Any thoughts about where we are as a project would also be great, I’m sure people who work with massive clouds will have a lot of thoughts on our setup.

We try to be super transparent with our infrastructure/finances/roadmap, so if there is anything that would provide more information, I’d be happy to go over it!

Thanks!

3 Likes
2

Hi @AngelOnFira :wave:

That’s a pretty cool use case, and we would love to hear updates as the project progresses :grinning_face_with_smiling_eyes:

As you mentioned, this is not really something we would recommend for production. But since you are already running in a single host it might be a good start (this is kind of how I run my homelab :grin:).

I think that the advantages of using VMs in this case is that you would be able to create immutable images, and so recover more quickly if something goes wrong.

Network and disk performance are probably the main things you would want to lookout for. If the servers start to run slow, or are not able to communicate in a timely manner you could end up with flappy leadership elections. I would recommend you to try load testing as much as possible.

You should probably watchout for OOMs and swaps as well, as your hypervisor might over-provision your VMs to try to increase hardware usage. So your VM might think it has 4GB, but in reality the physical memory has run out and the OS is swapping the excess memory to disk. This could have a significant impact on the Nomad servers, since they try to keep as much as possible in-memory.

I don’t have much to say on this, other than it’s always hard to start something new, so I wish you the best of luck :four_leaf_clover: