Scaling & HA

How dagron scales

Every dagron scheduler replica runs the same binary with the same configuration. Replicas do not know about each other and hold no coordination state: they meet only in the database. Three properties of the claim protocol do all the work:

• Disjoint claims. On a Postgres backend, each replica claims ready tasks with FOR UPDATE SKIP LOCKED. Two replicas claiming concurrently lock disjoint rows, so no task is ever dispatched twice — without a leader, a heartbeat table, or any election on the hot path.
• Lease recovery. A claim takes a 30-second lease on the task. If the claiming replica dies, the lease expires and any surviving replica resets the task to ready on its next tick. By default workers renew a running task's lease every 10 seconds (TASK_LEASE_HEARTBEAT), so a long task is never mistaken for a dead one.
• Fencing. The claim carries a version that acts as a fencing token: a zombie executor that finishes after its lease was reclaimed cannot write a stale result back.

Replicas wake on Postgres LISTEN/NOTIFY rather than polling, and the reconcile loop is idempotent and database-driven — there is no in-memory state that matters, which is what makes replicas interchangeable. Adding a replica therefore adds throughput and removes a single point of failure in the same move. This is the same multiple-active-scheduler model Apache Airflow 2.x uses for its scheduler.

Deployment configurations

dagron is the same binary everywhere; a "setup" is which backend and components you wire behind it. There are two anchors — Basic and HA — with a scale-out step between them and a sovereign variant beyond.

For air-gapped installs of the shipped configurations, the Helm chart's global.imageRegistry value relocates every image in the chart to a private mirror with a single value.

Scaling out the engine

The Helm chart (on Artifact Hub) exposes the scale-relevant values directly. The engine image must be built with the postgres, ops, and kubernetes cargo features — the default is a SQLite-only build, which cannot do multi-replica claiming.

engine:
  replicas: 3             # identical schedulers, active-active
  workerCount: 8          # per-replica task concurrency (WORKER_COUNT)
  executor: k8s           # each task runs as a one-shot Pod
  maxInflightRuns: 2000   # admission cap: POST /runs sheds load with 429
                          # + Retry-After above it; 0 disables

postgres:
  enabled: false          # the bundled Postgres is for throwaway testing;
                          # bring a managed database instead
externalDatabaseSecret:
  name: dagron-db         # pre-existing Secret holding the connection string,
  key: DATABASE_URL       # so the password never lands in helm history

$ kubectl -n dagron create secret generic dagron-db \
    --from-literal=DATABASE_URL='postgres://dagron:REAL_PW@db.internal:5432/dagron?sslmode=require'
$ helm upgrade --install dagron oci://registry-1.docker.io/mancube/dagron \
    -n dagron --create-namespace -f values.yaml

Chart defaults are deliberately small: engine.replicas: 1, engine.workerCount: 4, engine.maxInflightRuns: 64. Scaling the fleet is a values change and a helm upgrade — because replicas coordinate purely through the database, adding one requires no rebalancing, no peer discovery, and no config on the existing replicas. Environment knobs the chart does not model can be appended verbatim via engine.extraEnv.

Three things to keep in view as replicas grows:

• Connection budget. Each engine holds a Postgres pool of DB_MAX_CONNECTIONS (default 8, minimum 2). Replicas × pool size — plus anything else on the database — must stay under the server's max_connections.
• Executor placement. With executor: k8s, each task is a one-shot Pod, so fan-out scales on the cluster and its autoscaler rather than on the scheduler process. The scheduler replica itself stays small.
• Identical configuration. Values that shape claiming — POOLS in particular — must be identical across replicas, since any replica may claim any task.

Scaling ingestion

Submissions scale independently of task dispatch. The open streaming source (SOURCE=stream) follows an NDJSON file or FIFO; pointing STREAM_PATH at a directory switches to sharded multi-consumer mode — each *.ndjson file is a partition, split across engines via per-partition leases, each shard with its own exactly-once cursor. Cap how many shards one engine holds with STREAM_MAX_PARTITIONS so capacity spreads across consumers instead of one engine hoarding every shard. Whatever the source, MAX_INFLIGHT_RUNS is the admission valve: overflow stays buffered at the source, never in the scheduler.

Scaling the database

The engine is not the component that grows. Postgres is the shared source of truth, the throughput ceiling, and — until you replicate it — the single point of failure. The HA topology is standard Postgres practice, not anything dagron-specific:

• Primary + synchronous standby with automated failover: Patroni, RDS Multi-AZ, or Cloud SQL HA.
• PgBouncer in transaction mode as the fleet's connection count grows.
• Read replicas for the read-mostly UI/API paths (run lists, graphs, logs), which tolerate replica lag.

What happens during a failover

When the primary fails and a standby is promoted, no operator action is needed on the dagron side:

1. Scheduler connections drop; in-flight claim and mark transactions error.
2. The reconcile loop is idempotent and database-driven, so each replica simply retries its tick against the new primary. No in-memory state is lost because there is none that matters.
3. Any task whose completion did not commit before failover keeps its running lease; after the lease expires it is reclaimed and re-run (at-least-once).
4. The singleton leader's session is gone, so a standby replica acquires leadership on the new primary.

PgBouncer and the LISTEN session

PgBouncer's transaction pooling cannot serve a session-scoped LISTEN. dagron has a split-DSN seam for exactly this: point DATABASE_URL at PgBouncer for pooled transactions, and set DATABASE_LISTEN_URL to the direct Postgres endpoint for the reconcile loop's LISTEN session. The same pair exists on dagron-api for its SSE listener.

Backups and restore procedures for the state database are covered on the Admin & maintenance page.

Concurrency controls within the engine

Inside each replica, and across the fleet, these environment variables govern how much runs at once. The full reference lives on the CLI & configuration page; these are the scale-relevant ones.

Two per-workflow controls complement the fleet-wide caps: a task-level priority orders dispatch among simultaneously-ready tasks (a tiebreak that never overrides dependencies), and a DAG-level max_active_runs caps how many runs of one workflow may be running at once — further fires are held back and the API returns 429.

Cluster singletons & the leader

A few jobs must run exactly once cluster-wide, not once per replica: firing cron schedules (CRON_CONFIG), firing database-backed UI schedules (DB_SCHEDULES), retention GC (GC_RETENTION_SECS), and the stale-ready alert. With N active schedulers these would otherwise fire N times — N duplicate cron runs, N racing deleters.

dagron gates them behind a single leader elected with a lightweight, session-scoped lock in Postgres (an advisory lock — no new infrastructure, since every replica already holds a Postgres connection). One replica wins and runs the singleton steps in its reconcile tick; the others skip those steps and keep doing data-plane work at full speed. The leadership lease length is LEADER_LEASE_SECS (default 30).

When the leader dies — process death, or a network partition from the database — its session ends, Postgres releases the lock, and a standby replica acquires leadership on its next attempt. There is no lease table to garbage-collect and nothing to page on: schedule firing pauses for at most the takeover window, then resumes on the new leader.

Limits

Details that matter when you are near the ceiling:

• The budget counts leaf tasks after expansion, not the width you asked for. A fan-out of width 100,000 with one upstream and one downstream task is 100,002 leaves and is refused; the widest such fan-out that fits is 99,998.
• Prefer with_items templates over spelled-out task lists. In the project's offline measurements, the template form of the same graph was materially cheaper to parse and submit — the size of the YAML body, not the expanded task list, dominates submission cost.
• The body travels with every submission. dagron has no registration step: POST /runs carries the full spec each time, so very large specs pay their parse-and-expand cost on every run and re-run.
• Sub-workflow nesting is capped at SUBWORKFLOW_MAX_DEPTH (default 8); a trigger at the cap fails that task with a message naming the depth, and the rest of the run proceeds under normal failure handling.

Validating your sizing

The engine binary is not the thing you size. The engine container idles around 10 MB resident — the footprint of a deployment is dominated by Postgres, the executor, and any ingestion component, never by the engine. Capacity planning is therefore: size Postgres, size where tasks actually run, then add cheap scheduler replicas for HA and throughput.

The tiers below are the project's own rough guidance — starting points to validate with a load test, not promises. Real numbers depend on DAG width, task duration, executor, and history retention.

On node sizing: a scheduler replica is a few MiB of RAM plus whatever its WORKER_COUNT local tasks need — and with EXECUTOR=kubernetes, tasks run as pods elsewhere, so the replica stays tiny. Size nodes for Postgres connections and headroom, not for the binary.

Postgres is the number to watch: run/task history and the per-task claim rate drive its load. Whatever tier you pick, confirm it under your own workload shape before committing — the repository ships a load-test harness with config-driven profiles (sustained, spike, soak, ramp), Grafana dashboards, and chaos scripts that kill a scheduler mid-run to verify lease recovery. Stepping engine.replicas during a run reads out your horizontal control-plane headroom directly.

Beyond one team Enterprise

Everything above is Apache-2.0 and complete on its own. For fleets shared across teams, dagron Enterprise adds a scaling layer on the same engine: managed broker sources (Redis, SQS, Kafka, NATS) for queue-driven ingestion Enterprise, gang scheduling for all-or-nothing co-scheduled ranks Enterprise, and a multi-tenant control plane with per-tenant isolation Enterprise. The open-vs-enterprise line is drawn in the project README.