Daggyr DAG Work Orchestration Engine

Hacker News

Share card

Daggyr DAG Work Orchestration Engine

Hello all, I've been seeing a lot of interest in work orchestration frameworks, and thought I'd show off my own take. It's the project I've used to learn rust (entire project is async rust using tokio and actix), but it's been really useful for my main work, too. Daggyr is an engine for executing tasks as quickly as possible. It can run tasks on flexible back-ends (slurm, local threads, etc), and report updates / events to trackers (currently only in-memory, working on postgres and mongodb). There's no UI, just a REST interface. It doesn't do scheduling (there are tons of tools for that). It doesn't currently have multiple users / auth, but I'm working on that. It does support pipelines defined in JSON, task parameterization, a very simple and fast REST interface for job submission and tracking, and resuming failed runs from where they left off. Why another one of these things? 1. I couldn't get behind Airflow's weird scheduling opinions. 2. Dagster, Airflow, and Prefect rely on tight integration with python. A lot of my work flows are calling custom binaries, so python provides no value, and it's also a barrier to defining jobs. 3. Daggyr can scale to pipelines of hundreds of thousands of tasks. 4. Daggyr is stupidly simple to run: no external services or dependencies (unless you want them). Simple run the binary and send some work. Long story short, it does what it says on the tin, but there's more work to be done. I'd really appreciate any feedback. No idea if I'm going to try and commercialize it, but if you'd like to see a feature ASAP, I'm always open to sponsored feature development! edit: formatting