HI There I have a curious question. Will DAG support be made available for job scheduling?
Where each DAG can be mapped at the Task and Group level in 1 job.
2 Likes
There’s a ticket to integrate with Airflow.
Depending on your specific needs, you can use lifecycle hooks right now.
This is really a must have feature to me. Airflow is not the easiest tool to manage and develop with. We need something like Argo Workflows. All the other tools suggest to be deployed to Kubernetes in production. I don’t want to manage a kubernetes cluster to run some containerized ETL’s or some batch job in Nomad. I just need to be able to define some order in the jobspec file. But well, the Airflow integration should solve the problem.
Airflow integration is cool, but would be nice to have a solution for smaller scale or lighter setups, since Nomad shines in this way. I’ve been using Kestra in a single node setup for over a year now and think it has a lot of potential for this case. Easy to manage and very supportive of s3 and container based setups. a basic trigger/depends_on feature in Nomad would be great too but it may not be so simple to implement. I’ll report back after I try running Kestra in a Nomad cluster-- seems like it should work, though
1 Like