Disclaimer: I’m currently starting to evaluate Nomad so I still haven’t heavily researched this and there may be something I’m missing. If that’s the case, please do point me in the direction of said docs
Background: A client is in the throws of implementing an orchestrator (currently they do it manually still) and on the table they have ECS, EKS and I’ve added Nomad too. We’re doing a cost analysis, learning curve, pros and cons etc. One thing that I think may throw a wrench in my subtle plan to possibly introduce Nomad as a strong contender, is the fact the client would like to introduce a GitOps workflow (just one of the reasons that ECS is already being discarded).
Question: I’m curious to know if there is some way of adopting a GitOps workflow with Nomad, as in having some agent monitoring a repo and reconciling what is in the repo with what tasks are running on the Nomad cluster.
Well, most likely the reconciliation loop. So a workflow similar to:
Developer has a new version of app. Submits a PR with changes to app, configs etc.
PR is merged and pipeline builds artefacts, new manifests (in this case it would be a job file), and pushes to git repo.
An agent in Nomad server would pick up on the changed job spec and run an apply.
Personaly, I think one of the benefits of GitOps oeprator running in the cluster is this reconciliation loop, where you know that the manifest checked in to git, is what you’ll have running on your cluster, regardless if someone changed something ad-hoc.
From my initial assessment, it seems as though deploys (apply) will only, always be triggered, be it manually by a developer/operator, or by a pipeline.
We’re building Nomad Pack, in part, to support CI/CD pipelines. It’s early days, so I’m not sure if it will or won’t help you where you are today, but we’d love to hear your assessment of it. Our expectation is GitHub actions (or something similar) invokes the pack on merge, and then reconciles the new job spec.
If you have any feedback, feel free to post it here or in the repository.
Thanks @runeron.
Considered Waypoint and will do a little testing, but that won’t help with reconciliation loop (at least don’t think so. Maybe after some testing I could be proven wrong).
Guess that I’ll have to add “lack of GitOps Operator” to the cons list for now, but will surely continue pursuing this.
No, you are reading right. You can put feedback here, or raise an issue in the repo.
The idea is that you create templates instead of job files, and then define variables that get injected. You can check in any non-secret variables via an overrides.hcl file and inject any secrets you need from the GitAction. The GitAction executes on merge and voila.
Based on your response, I sense there are features you need that we don’t yet have, and rather than anticipate them, I’d love to hear what they are. As I mentioned, this is in its early stages (tech_preview_1), so please let us know what’s missing. You could be shaping the future.
I just happened to read this blogpost (from Febuary), and thought I’d just add it to this post. Gives a basic introduction of Nomad, and a brief comparison with Kubernetes.
Doesn’t include much about the GitOps-parts, which was your question, but still a nice read when evaluating it vs other orchestrators.
Thanks. I had already stumbled upon this and is prooving quite useful in this early stage.
I know this is totally off topic, but it was the one thing that stuck with me the most when reading this. Nomad is indeed simple with few moving parts, until you need a little more from it (like persistent storage etc.). So, the fact that there is no managed service yet (looking at you HCP), this means that running and maintaining Nomad isn’t quite as simple as one may initially think, because you need to manage Vault, Consul, some storage layer, potentially CNI etc. Obviously, if I were managing k8s myself, then the complexity of Nomad pales in comparison.
Yes, I believe this is an important observation, and something to consider.
For good & bad, Nomad lets you pick and choose the parts you want to use. You want Consul & Vault? It’s very easy to integrate. You don’t want to use it, or don’t need it? No problem. You don’t want to use containers? No problem; you can schedule basically anything in any way you want (see task drivers). You can even try out scheduling remote tasks with AWS ECS.
Yes, you have a single binary, and a single’ish configuration-source (comparable to k3s, i think?), but it is still as complex as you make it. If your setup requires autoscaling, csi-storage, cni-networking, monitoring, etc, you need to understand it & configure it. You can evolve your cluster with new features as you go along, though.
I initially evaluated (self-deployed) k8s vs Nomad a few years back as well, for modernizing a legacy IT-environment. One of my primary reasons for ending up with Nomad back then was the simplicity of maintaining it (was a lot less features then as well). Kubernetes seemed straight-forward to set up (e.g. rancher, ansible, etc), but I was very worried about managing it in case (…when…) something would eventually break in production.
Yes, would be GREAT with a hosted solution in the near future.
I wholeheartedly agree, that Nomad vs. self-managed k8s, wins, hands-down! The reality I’m evaluating now though, is managed k8s, vs a self-managed Nomad cluster, with all the baggage and complexity modern day apps inherintly carry.
One thing is for sure, even if I end up going k8s route for this particular client and use case, I’ll surely be messing around with Nomad on my own time, because it does look very interesting and promosing
For other tools, like GitHub Actions, you can use Levant or Nomad Pack (still in tech preview) to handle the job deployment part since they have Docker images available.
There’s also a community list of CI/CD tools that integrate with Nomad:
Hello @pslobo, thanks for asking this question, I had exactly the same. I share with you the focus on the reconciliation loop part of gitops. I really want to use Nomad and I am willing to go the extra mile and more, but I still don’t see an answer to the gitops approach.
The Terraform provider doesn’t seem to be an answer, since the documentation itself Terraform Registry states:
This can be used to initialize your cluster with system jobs, common services, and more. In day to day Nomad use it is common for developers to submit jobs to Nomad directly, such as for general app deployment. In addition to these apps, a Nomad cluster often runs core system services that are ideally setup during infrastructure creation. This resource is ideal for the latter type of job, but can be used to manage any job within Nomad.
I think here we are looking for the “apps” type of jobs…
@pslobo@marco-m
Just to be clear; you are talking about “continuous reconciliation”, right? E.g. like I think crossplane has an agent to ensure there is no “drift” vs. what is in your single source of truth (i.e. git). At least that is how I initially understood the question.
Reconciliation, as in “update state based on changes in git” is no problem with Nomad, and can be done in many ways. And Nomad will also countinously ensure the running-state is what you have defined in the job-specification (you can add inn consul health-checks, etc.).
Examples:
Someone manually sets a job to “stopped” from nomad cli/ui, then it will not restart by itself, even if is should be running, according to git.
Job fails due to some issue on nomad host, a coupled service, database, resource-starvation, or similar. Then Nomad would of course try to restart and/or reschedule it automatically, based on job-settings. Same as kxs
Thanks for the links. The Internet Archive ones I had already seen and do seen and in their case, makes total sense the change (at least the way they framed it). The other links, I’ll check them out in more detail soon.
