Seeking advice on replacing manually maintained architecture with Nomad

[grenadejumper]

Hi all. I am seeking general advice and resources for an architecture I am trying to set up for my company’s web applications.

I am familiar with some DevOps technologies like Docker, GitHub Actions and Vercel, but am struggling to piece together a full picture for how I can accomplish setting up an architecture with Nomad (and the rest of the Hashicorp suite) for our web app.

First I’ll give a quick background and how I imagine solving this, then I’ll list some key challenges I am hoping some of you may have some hints or answers to. Also feel free to let me know if you have some general thoughts on the architecture described here.

Our company has a giant monolithic .Net Framework 4.5 web app that we host on our own WMs (in Azure). To each WM we deploy multiple instances of this web app, each with its specific configuration in a web.config-file, and its own domain name. Each instance uses a database hosted on another WM (also in Azure).

The problem is this doesn’t scale very well, and we use a lot of time for upgrades and administration. We were hoping to hoping to find something other than Kubernetes due to its steep learning curve and possible issues containerizing this web app. Nomad and its Windows IIS Task Driver looks like it would be a good fit for this. Here is the parts we need as I see it:

• Terraform provisions the needed VMs and installs Nomad and other software.
• Nomad runs the websites using the Windows IIS Task Driver
• Consul for service discovery
• Possibly we’ll need Vault for secrets?

Here are some challenges I am unable to find answers to:

• I need to install .net framework 4.5, and possibly other more obscure software on servers. Can Terraform do this, or do I need to combine it with something like Ansible?
• Can we somehow do canary deployments using the Windows IIS Task driver? Maybe in some sort of combination with an nginx in front of it or something?
• There will be multiple IIS websites and application pools, each with their own domain name and a Web.config containing general config, secrets and bindings. What is the best way to manage this?
• Each update to the app will be accompanied by a database migration that will need to be applied at approximately the time when we are updating the application code (served by the Windows IIS task driver). How do we run these migrations?
• I am also truggling a bit to find the best way to run databases in Nomad. Preferably we’d like to spin those up in the same manner as other nodes in Nomad, almost like a Database-as-a-service, in the same way that is possible in Portworx (from what I understand).

Sorry for the longform post, but all help is greatly appreciated.

[dky]

Hello, Here’s my stab at your question based on my experience with the Hashicorp toolset + DevOps.

• I need to install .net framework 4.5, and possibly other more obscure software on servers. Can Terraform do this, or do I need to combine it with something like Ansible?

Yes - This sounds like a provisioning task best left to Ansible or building your base image with Packer.

• Can we somehow do canary deployments using the Windows IIS Task driver? Maybe in some sort of combination with an nginx in front of it or something?

Do you have a load balancer in front of your existing apps? Have you looked at Consul + Connect? Connect and service mesh have these capabilities using Envoy.

• There will be multiple IIS websites and application pools, each with their own domain name and a Web.config containing general config, secrets and bindings. What is the best way to manage this?

Since you mentioned you have experience with Docker, have you thought about breaking the monolith into individual Docker containers? That sounds like it might be a lot easier to deploy than the method you have now.

• Each update to the app will be accompanied by a database migration that will need to be applied at approximately the time when we are updating the application code (served by the Windows IIS task driver). How do we run these migrations?

This is a grey area for me but sounds like a task for CICD - potentially look at their new tool called waypoint and see if you can build some sort of CICD pipeline to do this?

• I am also truggling a bit to find the best way to run databases in Nomad. Preferably we’d like to spin those up in the same manner as other nodes in Nomad, almost like a Database-as-a-service, in the same way that is possible in Portworx (from what I understand).

The way I treat databases is they are long lived. Aka why not run them as containers inside of docker with some type of persisted volume but use Nomad as the scheduler for this logic? Aside from schema updates once deployed databases typically don’t go away, not sure what the use case is for wanting have databases that are ephemeral from what it sounds.

Hope this helps!

[grenadejumper]

Thanks for replying! This does indeed help

@dky

• I need to install .net framework 4.5, and possibly other more obscure software on servers. Can Terraform do this, or do I need to combine it with something like Ansible?

Yes - This sounds like a provisioning task best left to Ansible or building your base image with Packer.

Great, I’ll do some more investigations on both of these options.

@dky

• Can we somehow do canary deployments using the Windows IIS Task driver? Maybe in some sort of combination with an nginx in front of it or something?

Do you have a load balancer in front of your existing apps? Have you looked at Consul + Connect? Connect and service mesh have these capabilities using Envoy.

We don’t have any load balancers in our current infrastructure. Thanks for the suggestion, will take a look at how Consul + Connect can help.

@dky

• There will be multiple IIS websites and application pools, each with their own domain name and a Web.config containing general config, secrets and bindings. What is the best way to manage this?

Since you mentioned you have experience with Docker, have you thought about breaking the monolith into individual Docker containers? That sounds like it might be a lot easier to deploy than the method you have now.

We have not tried to dockerize the app yet, but that might be the best way to start our way to a more manageable and scaleable infrastructure. We fear that some parts of it might require some heavy changes (dependecies, upgrade procedures etc), but I guess there is no way around it.

@dky

• Each update to the app will be accompanied by a database migration that will need to be applied at approximately the time when we are updating the application code (served by the Windows IIS task driver). How do we run these migrations?

This is a grey area for me but sounds like a task for CICD - potentially look at their new tool called waypoint and see if you can build some sort of CICD pipeline to do this?

We are using GitHub Actions as our CICD tool for our simpler apps. Was considering perhaps running the migrations as a Nomad-job as a step in a GitHub Actions pipeline. Just not sure how to handle any errors that might occur. I guess this will be a general issue, no matter what tech we choose.

@dky

• I am also truggling a bit to find the best way to run databases in Nomad. Preferably we’d like to spin those up in the same manner as other nodes in Nomad, almost like a Database-as-a-service, in the same way that is possible in Portworx (from what I understand).

The way I treat databases is they are long lived. Aka why not run them as containers inside of docker with some type of persisted volume but use Nomad as the scheduler for this logic? Aside from schema updates once deployed databases typically don’t go away, not sure what the use case is for wanting have databases that are ephemeral from what it sounds.

I am looking to run them as containers inside Nomad. What I’d like is for them is for the system to be able to handle a complete failure of the node thats running it - and spin up another instance quickly - with all the data still there. This requires that the data is safely isolated from the node that failed. I am unfortunately not that familiar with my options for how to achieve this - so this might all be much simpler to implement then I am visioning.

Again, thanks for taking the time to reply. Lots of good suggestions I’ll take a deep dive into the resources for the projects you mentioned.