GitOps Workflow with Nomad

@runeron that’s exactly right, and I’d wager a bet that it’s the same thing @marco-m is after. In k8s world, this would be something like ArgoCD or Flux CD (already mentioned by you), where an operator deployed to the cluster monitors a got repo (our single source of truth) and ensures that in the event there is some external change, then the state is reconciled with git repo.

Example:

  • A new version of the app is deployed and possibly k8s manifests are updated too. This goes through a rigorous testing and PR and gets merged. The gitops operator sees there is a new version and deploys these updates. Later, there is an issue, and someone starts debugging, changing k8s deployment configs etc. If they forget to submit an updated manifest with any changes they made, the operator will override those, as the git repo is the source of truth. This therefore ensures and enforces to some extent, that developers always commit any ad-hoc changes they may have made.

In Nomad, I’d think something akin to that would be cool. Possibly something in the server stanza config stating which repo to watch, and all job specs get picked up from that repo. Obviously, that would render the nomad cli/ui as useless as the reconciliation would override it.

1 Like

+1
fully agree. Would be nice to have something like Flux for Nomad as well. We have our home-rolled shell scripts for this, but far from ideal.

Just as an update for the community, Nomad On Demand runners for Waypoint are now available.

Here are some useful links that might help you get started.

Using Waypoint Runners to Enable GitOps Workflows blog
Waypoint Runners Documentation
Video explaining Waypoint GitOps architecture
Waypoint Nomad On Demand Runner PR

Also, worth mentioning is our upcoming HashiTalks Deploy event which has some relevant sessions.

4 Likes

The problem is that HashiCorp and many others think that GitOps is a one-way thing where a git commit triggers a pipeline that then deploy something. Thats only half the solution.

As some here has mentioned, you need a controller that keeps looking for changes, and enforces the state in git in your cluster.

Jonas, one of my friends, are working on this (also mentioned here) here : GitHub - jonasvinther/nomad-gitops-operator: A GitOps operator for Hashicorp Nomad and is a POC right now. But you can deploy it and it will deploy the hcl in the remote repository, and keep polling for changes.

We have a lot of ideas for the controller, but its still developed in our spare time. It’s not finished, and all contributions are welcome.

We tried Waypoint, but it’s basically doing the same as a Jenkins would do. Pull, build, test and deploy. We want to separate CI and CD from deployment, making releasing/deployment a simple business decision and done by a simple git operation.

4 Likes

Templates are not considered source of truth. What are generated from these templates are, in GitOps.

2 Likes

I’m also very interested in gitops for nomad/waypoint.

nomad-gitops-operator seems to be an interesting solution as a workaround. I did find an issue where deleted jobs aren’t stopped. jobs that no longer existed should be stopped or deleted · Issue #2 · jonasvinther/nomad-gitops-operator · GitHub

I had asked this on waypoint github issue but is not currently on their roadmap.

3 Likes

If anybody is still interested in this. We built & run https://github.com/nomad-ops/nomad-ops. Fits our use case so far. Happy to hear your thoughts.

1 Like