GitOps Workflow with Nomad

Thanks @runeron.
Considered Waypoint and will do a little testing, but that won’t help with reconciliation loop (at least don’t think so. Maybe after some testing I could be proven wrong).

Guess that I’ll have to add “lack of GitOps Operator” to the cons list for now, but will surely continue pursuing this.

So, first glance from GitHub, this seems more like Helm for Nomad, or am I reading it wrong.

No, you are reading right. You can put feedback here, or raise an issue in the repo.

The idea is that you create templates instead of job files, and then define variables that get injected. You can check in any non-secret variables via an overrides.hcl file and inject any secrets you need from the GitAction. The GitAction executes on merge and voila.

Based on your response, I sense there are features you need that we don’t yet have, and rather than anticipate them, I’d love to hear what they are. As I mentioned, this is in its early stages (tech_preview_1), so please let us know what’s missing. You could be shaping the future.


1 Like

I am just seeing this comment. You are correct, we don’t (yet) have an Operator SDK equivalent if that’s what you are referring to.


Some of my Waypoint colleagues provided this feedback to share with you.

"a quick answer: yes! waypoint + nomad + gitops is going to work. Once Nomad on-demand runner support is in, it will be easier than ever to do this.

The gitops flow in waypoint also matches their ‘reconciliation’ loop they mention."


Thanks for this information @DerekStrickland, will surely come in handy.

As for Nomad Pack, I’ll include it in my testing and learning and will get back with any feedback I may have.

I just happened to read this blogpost (from Febuary), and thought I’d just add it to this post. Gives a basic introduction of Nomad, and a brief comparison with Kubernetes.

Doesn’t include much about the GitOps-parts, which was your question, but still a nice read when evaluating it vs other orchestrators.


Thanks. I had already stumbled upon this and is prooving quite useful in this early stage.

I know this is totally off topic, but it was the one thing that stuck with me the most when reading this. Nomad is indeed simple with few moving parts, until you need a little more from it (like persistent storage etc.). So, the fact that there is no managed service yet (looking at you HCP), this means that running and maintaining Nomad isn’t quite as simple as one may initially think, because you need to manage Vault, Consul, some storage layer, potentially CNI etc. Obviously, if I were managing k8s myself, then the complexity of Nomad pales in comparison.

1 Like

Yes, I believe this is an important observation, and something to consider.

  • For good & bad, Nomad lets you pick and choose the parts you want to use. You want Consul & Vault? It’s very easy to integrate. You don’t want to use it, or don’t need it? No problem. You don’t want to use containers? No problem; you can schedule basically anything in any way you want (see task drivers). You can even try out scheduling remote tasks with AWS ECS.

  • Yes, you have a single binary, and a single’ish configuration-source (comparable to k3s, i think?), but it is still as complex as you make it. If your setup requires autoscaling, csi-storage, cni-networking, monitoring, etc, you need to understand it & configure it. You can evolve your cluster with new features as you go along, though.

I initially evaluated (self-deployed) k8s vs Nomad a few years back as well, for modernizing a legacy IT-environment. One of my primary reasons for ending up with Nomad back then was the simplicity of maintaining it (was a lot less features then as well). Kubernetes seemed straight-forward to set up (e.g. rancher, ansible, etc), but I was very worried about managing it in case (…when…) something would eventually break in production.

Yes, would be GREAT with a hosted solution in the near future.

Note: posted from wrong account :smiley:


I wholeheartedly agree, that Nomad vs. self-managed k8s, wins, hands-down! The reality I’m evaluating now though, is managed k8s, vs a self-managed Nomad cluster, with all the baggage and complexity modern day apps inherintly carry.

One thing is for sure, even if I end up going k8s route for this particular client and use case, I’ll surely be messing around with Nomad on my own time, because it does look very interesting and promosing :slight_smile:

1 Like

Hi @pslobo :wave:

There’s a lot of good content and information in this thread, but I would like to send two more that you may find useful.

I don’t really know of any out-of-the-box solution for this, but I think it would be possible to integrate Nomad into existing CI/CD pipelines.

For example, here are some resources on integrating Nomad and GitLab from one of our past webinars and from how the Internet Archive uses Nomad:

For other tools, like GitHub Actions, you can use Levant or Nomad Pack (still in tech preview) to handle the job deployment part since they have Docker images available.

There’s also a community list of CI/CD tools that integrate with Nomad:


Hello @pslobo, thanks for asking this question, I had exactly the same. I share with you the focus on the reconciliation loop part of gitops. I really want to use Nomad and I am willing to go the extra mile and more, but I still don’t see an answer to the gitops approach.

The Terraform provider doesn’t seem to be an answer, since the documentation itself states:

This can be used to initialize your cluster with system jobs, common services, and more. In day to day Nomad use it is common for developers to submit jobs to Nomad directly, such as for general app deployment. In addition to these apps, a Nomad cluster often runs core system services that are ideally setup during infrastructure creation. This resource is ideal for the latter type of job, but can be used to manage any job within Nomad.

I think here we are looking for the “apps” type of jobs…

1 Like

@pslobo @marco-m
Just to be clear; you are talking about “continuous reconciliation”, right? E.g. like I think crossplane has an agent to ensure there is no “drift” vs. what is in your single source of truth (i.e. git). At least that is how I initially understood the question.

Reconciliation, as in “update state based on changes in git” is no problem with Nomad, and can be done in many ways. And Nomad will also countinously ensure the running-state is what you have defined in the job-specification (you can add inn consul health-checks, etc.).


  • Someone manually sets a job to “stopped” from nomad cli/ui, then it will not restart by itself, even if is should be running, according to git.
  • Job fails due to some issue on nomad host, a coupled service, database, resource-starvation, or similar. Then Nomad would of course try to restart and/or reschedule it automatically, based on job-settings. Same as kxs
  • A nomad client fails; horizontal autoscaler can create a new node, if you use it.

Any use-cases would be nice, as I find the term “reconciliation-loop” to be a bit unclear, but maybe that is just me :slight_smile:

1 Like

Thanks for the links. The Internet Archive ones I had already seen and do seen and in their case, makes total sense the change (at least the way they framed it). The other links, I’ll check them out in more detail soon.

@runeron that’s exactly right, and I’d wager a bet that it’s the same thing @marco-m is after. In k8s world, this would be something like ArgoCD or Flux CD (already mentioned by you), where an operator deployed to the cluster monitors a got repo (our single source of truth) and ensures that in the event there is some external change, then the state is reconciled with git repo.


  • A new version of the app is deployed and possibly k8s manifests are updated too. This goes through a rigorous testing and PR and gets merged. The gitops operator sees there is a new version and deploys these updates. Later, there is an issue, and someone starts debugging, changing k8s deployment configs etc. If they forget to submit an updated manifest with any changes they made, the operator will override those, as the git repo is the source of truth. This therefore ensures and enforces to some extent, that developers always commit any ad-hoc changes they may have made.

In Nomad, I’d think something akin to that would be cool. Possibly something in the server stanza config stating which repo to watch, and all job specs get picked up from that repo. Obviously, that would render the nomad cli/ui as useless as the reconciliation would override it.

1 Like

fully agree. Would be nice to have something like Flux for Nomad as well. We have our home-rolled shell scripts for this, but far from ideal.

Just as an update for the community, Nomad On Demand runners for Waypoint are now available.

Here are some useful links that might help you get started.

Using Waypoint Runners to Enable GitOps Workflows blog
Waypoint Runners Documentation
Video explaining Waypoint GitOps architecture
Waypoint Nomad On Demand Runner PR

Also, worth mentioning is our upcoming HashiTalks Deploy event which has some relevant sessions.


The problem is that HashiCorp and many others think that GitOps is a one-way thing where a git commit triggers a pipeline that then deploy something. Thats only half the solution.

As some here has mentioned, you need a controller that keeps looking for changes, and enforces the state in git in your cluster.

Jonas, one of my friends, are working on this (also mentioned here) here : GitHub - jonasvinther/nomad-gitops-operator: A GitOps operator for Hashicorp Nomad and is a POC right now. But you can deploy it and it will deploy the hcl in the remote repository, and keep polling for changes.

We have a lot of ideas for the controller, but its still developed in our spare time. It’s not finished, and all contributions are welcome.

We tried Waypoint, but it’s basically doing the same as a Jenkins would do. Pull, build, test and deploy. We want to separate CI and CD from deployment, making releasing/deployment a simple business decision and done by a simple git operation.


Templates are not considered source of truth. What are generated from these templates are, in GitOps.


I’m also very interested in gitops for nomad/waypoint.

nomad-gitops-operator seems to be an interesting solution as a workaround. I did find an issue where deleted jobs aren’t stopped. jobs that no longer existed should be stopped or deleted · Issue #2 · jonasvinther/nomad-gitops-operator · GitHub

I had asked this on waypoint github issue but is not currently on their roadmap.