dagronDurable DAG workflows
for code in any language.
For work that is expensive to lose: training runs, fine-tuning, batch inference, agent loops. Define the graph in plain YAML; dagron validates it and runs each task the moment its dependencies succeed. An eight-hour step holds a heartbeat lease, so it is never mistaken for a dead one, and a retry resumes from its own checkpoint instead of starting over. The tasks are whatever you already run — a Python script, a Node job, a Spark submit, a container image — in any language; only the graph is dagron's. No control plane, no cluster to operate, a database as the only state.
Apache-2.0 Any language, any image SQLite or Postgres Kubernetes-native MCP server — 42 tools
| Time | Workflow | Task | Runner | Status |
|---|---|---|---|---|
| 02:00 | NIGHTLY-ETL | LOAD | DEFAULT | RUNNING |
| 02:15 | FEATURE-BUILD | FEATURES | DOCKER | CACHE HIT |
| 03:00 | LLM-FINETUNE | TRAIN | GPU-A100 | RUNNING |
| 03:40 | WAREHOUSE-ROLLUP | ROLLUP | POOL:ETL | QUEUED |
| 04:05 | CDC-ORDERS | INGEST | STREAM | RETRY 2/3 |
| 04:30 | MODEL-PUBLISH | PUBLISH | APPROVAL | AWAITING OK |
| 05:00 | SENSOR-BACKFILL | PARSE | DEFAULT | DEAD-LETTER |
Waiting / retrying Failed → dead-letter Succeeded illustrative board — not live data
The eight-hour step
is the hard part.
Anything can run a graph of short tasks. The work that costs money runs for hours on hardware you are paying for by the second, gets preempted, and must not silently begin again at zero. That is the case dagron is built around, and every mechanism below is in the open build.
Seven hours in, the node this task was running on drops. The lease stops being renewed, the scheduler reclaims the task and retries it — and because the task had reported a checkpoint, the retry starts at step 4,200 rather than at zero. The alternative is not a slower run; it is seven hours of paid GPU time spent twice.
A long task is never mistaken for a dead one.
A running task renews a heartbeat lease every 10 seconds under a triple-guarded claim, so a step may run for hours without the scheduler deciding its worker died and handing the work to someone else.
A retry picks up where the failure happened.
A task reports a checkpoint over HTTP or by file convention; on retry dagron hands it back as DAGRON_RESUME_FROM. The difference between a retry that costs a minute and one that costs a night is whether the engine remembers where you were.
Route the GPUs; cap what shares them.
Tasks carry a runner class, so the fine-tune lands on the A100 pool and the CSV parse does not. Named concurrency pools bound how many run at once, and dispatch priority decides who goes first when the pool is full.
Retries budgeted by fault class, not by count.
A node with failing ECC memory and a malformed input are not the same failure and must not get the same three attempts. Retry budgets are set per fault class, so an infrastructure fault can be retried elsewhere while a bad input stops immediately.
A conversation is a run.
The MCP server exposes 42 tools over stdio (author, run, wait, recover, inspect), so an agent drives dagron without a human dropping to curl. An agent loop is an ordinary workflow: each turn is its own child run and state is artifacts rather than process memory, so the conversation is durable, inspectable and resumable like anything else on the board.
Everything to run workflows.
Nothing you don't need.
A lean trade: plain YAML, a database, and your code exactly as it already runs — durable orchestration without the operational weight of Airflow or a Celery fleet.
A graph of tasks, and what each one waits for.
The spec is validated and cycle-checked server-side before a single task runs, so a broken graph fails at submission rather than halfway through a nightly load.
A failed task retries. A parked task holds nothing.
Per-task max_attempts with exponential backoff, timeouts and trigger rules, and when: conditions combine with and, or and in. Approval gates and wait sensors park a run while it occupies no worker slot.
Hand work to Spark or a warehouse. Hold nothing while it runs.
A task with defer: submits a job and parks on its handle, so a six-hour Spark job costs a row, not a worker. If every scheduler dies mid-job, the next one resumes the poll, and a cancelled run sends the job the cancel call you configure. dagron-step-spark and dagron-step-sql do the submitting.
Runs stream into a live DAG view.
Inspect the graph and read per-task logs while the work is still moving — over the same engine the CLI drives.
The storage backend is a build flag.
Start on an embedded SQLite file, then move to Postgres for multi-node without touching the workflow.
Three executors, routed by runner class.
Run tasks in-process, as Docker containers, or as one-shot Kubernetes Pods — and send the expensive hardware only the steps that need it.
Reconciled from a repository, or driven by an agent.
A dedicated worker reconciles versioned workflows from a repository; the MCP server exposes submit, inspect and cancel to an AI agent over the same authenticated API the console uses.
Lineage that downstream work can act on.
produces: records which run wrote which dataset. Sense one, fire a workflow when it updates, or wait for all of several. With OPENLINEAGE_URL set, each run's inputs and outputs reach your lineage backend.
Every series documented. Eight dashboards ready.
Prometheus metrics for runs, tasks, run time per workflow, backlog and the engine process, a guide to querying and alerting on each, and eight Grafana dashboards that load with no plugins.
Concurrency caps, dispatch order, and a cache key.
Cap concurrency per pool, order the ready queue by dispatch priority, and memoize a step so an unchanged key never runs again.
Write the stage once. Import it everywhere.
A saved workflow that declares templates: is a library: use: brings its templates into another workflow by name. A template defined twice — in the spec and a library, or in two libraries — is refused rather than silently overridden.
Watch a run
Author and launch
Operate
Measure
One binary at the edge.
The same YAML in the cluster.
The spec in the middle does not change between them. Only what it is pointed at does.
One process, one file
A static binary against a local SQLite file. State is local, so the site keeps running while disconnected and resumes from its own database after a reboot.
- State
- site.db (SQLite)
- Executor
- local subprocess
- Arch
- arm64 · amd64 · armv7
- Operates
- nothing
# workflow.yaml — unchanged in both columns name: nightly-etl tasks: - name: extract command: ["extract.sh"] max_attempts: 3 - name: transform depends_on: [extract] - name: load depends_on: [transform] produces: ["s3://dw/daily"]
Multi-node, Kubernetes
The same spec against Postgres, dispatching one-shot Pods by runner class, installed from the OCI Helm chart.
- State
- Postgres
- Executor
- Kubernetes Pods
- Install
- helm · OCI chart
- Operates
- no control plane
Six shapes it fits.
dagron scales down as well as it scales out: the workflow you run on a gateway is the workflow you run on Postgres and Kubernetes, unchanged.
AI & ML workloads
Training, fine-tuning, batch inference, agent-driven runs
- The durability mechanisms are above: heartbeat leases, checkpoint resume, GPU routing and fault-class retry budgets. See the eight-hour step.
- Gang scheduling co-schedules N ranks all-or-nothing for distributed training Enterprise; the open build runs the ranks as ordinary tasks.
- Eval gates as approvals. A run parks in awaiting_approval until a metric threshold or a person releases it, holding no worker while it waits.
- The YAML is the whole interface: the spec beside this row is the entire task.
Data engineering
Nightly loads, backfills, warehouse pipelines
- Schedules and backfills with cron, plus a dead-letter queue you can inspect and redrive.
- Fair sharing. Concurrency pools and dispatch priority stop a wide backfill starving the nightly load.
- Lineage that means something. produces: records which run wrote which dataset; downstream work can wait on it or fire from it.
- Reuse whole pipelines as a single step with template: — a DAG of DAGs, expanded at run creation.
- Spark and warehouse jobs that hold no worker. defer: parks the task on the remote job and survives a scheduler crash; {{ ds }} names the partition.
Data science
Feature builds, evaluation, reproducible notebooks-as-jobs
- Don't recompute what hasn't changed. cache: memoizes a successful step by key; a later run with the same key reuses the output and skips execution entirely.
- A human in the loop. An approval gate parks the run, holding no worker, until someone approves or rejects it.
- Per-task environments. Each step can be its own container image with its own CPU and memory request.
Event-driven & streaming
CDC, sensor feeds, per-event pipelines
- A stream of workflows. Follow an NDJSON file or FIFO: each line submits a run; a directory of shards splits across consumers by lease.
- Exactly once. The run and the source offset commit in the same transaction, so a crash can neither duplicate a run nor lose one.
- Poison lines don't stall the feed: they dead-letter, and you redrive them once fixed.
- Managed broker connectors for Redis, SQS, Kafka and NATS Enterprise; the file and FIFO source is open.
Edge & IoT
Gateways, retail sites, vehicles, remote plant
- Nothing to operate. One static binary and a SQLite file. No control plane, no broker, no cluster.
- Small enough to co-locate. Published for arm64, amd64 and armv7, the 32-bit gateways already in the field. The engine container idles around 10 MB resident, and the whole stack on a Pi 4 peaks at ~340 MB of 3.7 GB (measured).
- One line turns the host into an edge host. profile: edge presets the constrained-host knobs; dagron config prints every one with where it came from, so a preset never hides a value from you.
- Admission gates, not crashes. DAGRON_PRESSURE_FILE lets a thermal, battery or maintenance daemon stop new work by touching a file, and DAGRON_MIN_FREE_BYTES holds a free-disk floor (64 MiB default) so a full flash refuses admission instead of corrupting a run.
- MQTT is the broker at the edge, and it is in this build. SOURCE=mqtt subscribes a topic and turns messages into runs, behind a Cargo feature rather than an edition, so a stock binary links no broker client and --features mqtt is all it takes.
- The clock is not assumed. A gateway boots without RTC or network and lies about the time, so every run carries a confidence (synced, drifted or unknown, defaulting to unknown because that is the honest answer without evidence) for your pipeline to act on.
- Signed workflow bundles. A definition pushed to a unit is ed25519-verified against DAGRON_BUNDLE_PUBKEYS before anything runs, fail-closed: unset the keys and the route answers 501 rather than accepting an unsigned spec. Verification ships in every build.
- Survives the uplink. State is local, so a site keeps running while disconnected and resumes from its own database after a reboot.
- Lint before you ship. dagron validate parses, expands and graph-checks a spec offline, with no server and no network.
Lightweight compute
Small teams, side services, anything that doesn't deserve Airflow
- Zero infra to start. dagron dev brings up the engine, the management API and Swagger on localhost.
- No idle daemon. A one-shot run drains and exits; it only stays resident when you configure it to.
- Room to grow. Move the same YAML to Postgres for multi-node: a build flag, not a rewrite.
The concepts map.
Your code comes with you.
The question that decides an evaluation is not "is it faster", it is "what does switching cost, and what do I stop operating". Here is the whole answer: what carries over unchanged, what you no longer run, and the one genuine loss.
What carries over: your code, unchanged. Every task keeps the command, the container image or the pod it already runs — Python, Node, Java, a Spark submit, in whatever language it is written. What changes is only the graph around it: a dagron workflow is data, not a program. There is no PythonOperator and no DAG file that executes at parse time. Your Python still runs, as a command, a container or a pod; what stops being Python is the graph. That is the trade the whole design rests on: a spec that cannot run arbitrary code at parse time is one that dagron validate can check offline, and that a signed bundle can be verified before it is applied.
The switch itself is transcription, not a rewrite. There is no Airflow converter, and this page will not pretend otherwise — but the port above is by hand because it is mostly copying: each operator becomes a task naming the command or image it already runs, one workflow at a time, and dagron validate checks each one before it ever runs. In return the scheduler, webserver, worker fleet and broker in the last row of the table are things you stop operating. Coming from Argo Workflows there is a converter: the open build ships an importer that converts an entrypoint dag template into a dagron spec. The command is below. Temporal and Prefect are a different shape of tool (durable execution, and a Python-native framework); neither has an importer here.
# the importer is a workspace binary, not in the release # archives or any image — run it from a clone $ cargo run -p dagron-import \ -- argo my-argo-workflow.yaml > workflow.yaml # it refuses what it cannot map, so check what came out $ dagron validate workflow.yaml $ dagron dev workflow.yaml
One entrypoint dag template, whose tasks reference container templates, becomes a dagron spec with command, docker_image and depends_on. Steps templates, artifacts, parameters and when-expressions are reported as errors, not dropped: the importer would rather stop than hand you a spec that runs and is quietly wrong.
What the scheduler costs,
measured beside Airflow.
The plate at the top of this page comes from one run. This is the rig it ran on, what it showed, and what it cannot tell you.
- Cluster
- Amazon EKS, Kubernetes 1.36, three t4g.xlarge nodes: 12 vCPU and 48 GiB in total, arm64.
- Schedulers
- Three dagron 0.9.3 engines (the released image), then three Airflow 3.3.2 schedulers, on the same nodes one after the other. Never both at once, so neither took CPU from the other.
- Load
- The same for both: 29 runs submitted at one a second, each a small ETL graph.
- Tasks
- The same synthetic ETL image on both, sleep-dominated, about 26 seconds a task. dagron ran each task as one pod. Airflow ran each as two: a KubernetesExecutor worker pod, and the task pod it launches.
- Datastore
- Postgres inside the cluster, for both.
- Settings
- Each system's defaults. Airflow was not tuned.
- Readings
- CPU and memory are the peak over the run, summed over the three scheduler pods, from cAdvisor. Submit latency is the median time for the API to accept a run, timed by a client inside the cluster.
- When
- 2 October 2026. One run of each.
What it does not tell you
- The plate shows Airflow's scheduler alone. Airflow 3 parses DAGs in a separate DAG processor, which a dagron engine does itself. Counting it, the Airflow side reads 1.08 vCPU and about 1.9 GiB. Its two API servers, which every task also talks to, used a further 0.25 vCPU and about 0.6 GiB.
- It is not a throughput result. Both finished all 29 runs: dagron in about two minutes, Airflow in about eleven, because at its defaults Airflow launched more pods than twelve vCPUs could schedule. With both held to twelve concurrent tasks, dagron finished 239 runs in 17 minutes and Airflow had finished 75 after 30, when that run was stopped. Much of that gap is this rig's two pods per Airflow task, not Airflow's scheduler, so this page quotes no throughput multiple.
- The work is synthetic, and it ran once. With sleeping tasks and an in-cluster database, absolute durations mean little, and a single run says nothing about variance. Only the cost of the two schedulers is comparable.
- You cannot re-run it yet. The harness behind these readings is not in the public repository, so for now they are our numbers rather than ones you can check. The Raspberry Pi benchmark is the one published with its rig.
Airflow is the better answer when
- your DAGs are Python and should stay Python: the graph is the part dagron takes away;
- you depend on its providers, the operators and hooks that dagron does not have;
- the work is calendar ETL, where data intervals and catchup semantics are the product;
- you need a large community to hire from, and a long record of running in production.
Look at dagron when
- the scheduler's own cost is a budget line, or it has to fit on a small machine;
- you want dispatch bounded and load shed at the API by default, instead of pods piling up pending;
- you are not on Kubernetes, or want the same YAML at the edge and in the cluster;
- a long step has to survive a lost worker and resume from its own checkpoint.
Neither list is a migration plan. There is no Airflow converter, so a port is done by hand, one workflow at a time — each task keeping the command or image it already runs.
Free and open.
Enterprise when you scale.
The core engine, UI, and Helm chart are Apache-2.0 and free forever. The Enterprise edition adds the identity, tenancy, and operations layer teams need in production.
in this edition· not in this edition
| Capability | Open Source | Enterprise |
|---|---|---|
| DAG engine, retries, scheduling, web UI | ||
| SQLite / Postgres, Docker & Kubernetes executors | ||
| Approval gates, GitOps sync, MCP, dead-letter queue | ||
| Datasets — produces:, lineage, sensors, triggers on one or several datasets, OpenLineage | ||
| Pools, dispatch priority, result memoization, wait sensors | ||
| Remote jobs — defer:, the Spark & SQL steps, cancel teardown | ||
| Streaming ingestion — file/FIFO source, sharded consumers | ||
| External dataset events — CDC, object-store notifications, other orchestrators | ||
| Managed broker sources — Redis, SQS, Kafka, NATS | ||
| Gang scheduling — all-or-nothing co-scheduled ranks | ||
| Encryption at rest — envelope / BYOK-KMS + key rotation | ||
| SSO — OIDC, SAML, LDAP / Active Directory | ||
| Multi-tenant control plane & tenant router | ||
| RBAC roles + full audit trail | ||
| Auto-backfill & self-healing reruns | ||
| GitOps operator + Workflow/CronWorkflow CRDs | ||
| Air-gapped install & signed offline licensing | ||
| Usage metering & billing export | ||
| Priority support & SLA |
Running dagron across teams?
Enterprise adds single sign-on, a multi-tenant control plane with per-tenant isolation, RBAC and audit, air-gapped deployment, and a support agreement — self-hosted on your infrastructure. Metered or flat-license pricing.
The same engine.
Someone else on call.
A managed dagron: we operate the control plane, the datastore and the upgrades, and you keep writing the same YAML against the same API. The workflows you run on your laptop today are the workflows that run there — moving is a connection string, not a rewrite.
- Status
- In development. No date announced — when there is one, it will be here.
- Self-hosted
- Unaffected. The engine, console and Helm chart stay Apache-2.0 and free, exactly as they are now.
Your pipelines,
run by the engine itself.
CI built on dagron rather than beside it: a pipeline is YAML that lives with the code, a push or a pull request becomes a run on the same engine this page describes, and the result goes back to the commit as a status. A GitHub Actions workflow comes over with the importer; what has no equivalent here is named in the file it writes, not dropped quietly.
- Status
- In development, being readied for release. No date announced — when there is one, it will be here.
- Today
- The engine it runs on is 0.10.0, below. Nothing about self-hosted dagron changes when CI lands.
Three ways in.
Smallest first.
The first one needs no cluster, no repository and no container runtime: one binary and a file. Start there; the other two are the same engine with more of it.
# one binary. no clone, no runtime. # grab your platform build from # github.com/lucheeseng827/dagron $ chmod +x dagron # run the graph below, live UI on :8787 $ ./dagron dev workflow.yaml
# published images — builds nothing $ git clone \ https://github.com/lucheeseng827/dagron $ cd dagron $ docker compose \ -f compose.quickstart.yaml up -d # console at /, API at /api — one port http://localhost:8080
# multi-arch images + OCI chart $ helm install dagron \ oci://registry-1.docker.io/mancube/dagron # or point it at your own Postgres $ helm install dagron \ oci://…/mancube/dagron \ --set postgres.enabled=false \ --set externalDatabaseSecret.name=\ dagron-db
The whole of workflow.yaml. Nothing is elided; this is a complete, valid spec.
name: finetune tasks: - name: prepare command: ["./prepare.sh"] - name: train needs: [prepare] command: ["./train.sh"] runner_class: gpu-a100 timeout_secs: 86400 max_attempts: 3 # retry gets DAGRON_RESUME_FROM - name: evaluate needs: [train] command: ["./eval.sh"]
Questions, answered.
What is dagron?
dagron is an open-source DAG workflow engine and scheduler. Define a workflow as a graph of tasks in YAML; dagron validates it and runs each task as soon as its dependencies succeed — concurrently, with retries and exponential backoff. The tasks are the commands, containers and pods you already have, in any language, with a database as the only state.
How is dagron different from Airflow?
dagron is a lightweight Airflow alternative: plain YAML and one engine to operate instead of a scheduler, webserver, worker fleet and broker. Your tasks keep running as the local commands, Docker containers, or Kubernetes pods they already are — in whatever language they are written. No control plane, no cluster to operate — an embedded SQLite file works to start, and you switch to Postgres for multi-node.
Is dagron free and open source?
Yes. The core engine, web UI, and Helm chart are Apache-2.0 and free to self-host indefinitely. An optional Enterprise edition adds SSO (SAML/OIDC/LDAP), a multi-tenant control plane, RBAC and audit, air-gapped install, and support.
How do I install dagron?
Install the Helm chart from Docker Hub (OCI): helm install dagron oci://registry-1.docker.io/mancube/dagron. To try it locally, docker compose -f compose.quickstart.yaml up -d pulls published images and builds nothing. See the get-started section.
What happens to a running task if the worker dies?
A running task holds a lease it renews every 10 seconds. If the worker stops renewing, the scheduler reclaims the task and retries it — and if that task reported a checkpoint, it is handed back as DAGRON_RESUME_FROM rather than starting over. A slow task is not a dead task: the lease is what tells them apart, which is why an eight-hour step is not reclaimed at minute fifty-nine. See the eight-hour step.
How hard is the port from Airflow?
The concepts map almost one to one — DAG to workflow, operator to a task with a command or
an image, sensor to type: wait, XCom to artifacts and datasets, pool to pool, and
the mapping table is on this page. Your code carries over unchanged:
every task keeps the command, container or pod it already runs, in whatever language it is
written — what stops being Python (or any language) is only the graph, which becomes
YAML data instead of a program. There is no Airflow converter, so that port
is by hand, one workflow at a time — transcription, checked by
dagron validate as you go. Coming from Argo Workflows there is a converter — the
open build ships an importer that errors on what it cannot map rather than dropping it silently.
Can an AI agent drive dagron?
Yes. dagron-mcp is a Model Context Protocol server over stdio exposing 42 tools — register and update workflows, run them, long-poll to terminal, rerun, resubmit, inspect logs and artifacts — so an agent does the work without a human dropping to curl. It is the callee, not the agent: it holds no model and runs no loop. An agent conversation is expressed as an ordinary run, so each turn is a child run and state is artifacts rather than process memory.
What is not in the open build?
External dataset events, managed broker sources (Kafka, NATS, SQS, Redis), fleet management across many units, gang scheduling, SSO, RBAC with an audit trail, and envelope/BYOK-KMS encryption. The open build does not fail silently on any of them — it refuses with an error naming what is missing and the open path that does work. Everything in the eight-hour step is in the open build.
How does dagron make money, and what does that mean for the open build?
Open core. The engine, console, SDKs and Helm chart are Apache-2.0 and free to self-host indefinitely — that is the whole product for a single team, not a trial. Revenue comes from the Enterprise edition (the identity, tenancy and operations layer in the table above, self-hosted on your infrastructure) and, in future, dagron Cloud, a managed dagron with no announced date. The line between them is stated in the code itself: the open build names what it will not do rather than hiding it.