Admin & maintenance

What you are operating

A full dagron stack is at most five moving parts, and only one of them holds state:

• engine — the dagron binary: scheduler daemon plus task runner. Optionally serves the ops API at API_ADDR (built with the ops feature).
• dagron-api — the authenticated UI/REST gateway. Postgres-only, stateless, listens on PORT (default 8080).
• frontend — the web console, default port 3000.
• GitOps worker — optional; reconciles workflow definitions from registered git repos. Off by default in the Helm chart (gitops.enabled: false).
• Postgres or SQLite — the state. All coordination between engines, workers, and dagron-api goes through the database, not RPC.

There are two HTTP surfaces with two trust levels, and keeping them straight is most of the security story:

Deployment shapes

Three ways to run it; pick by how much you are operating. See the overview for the tour and Scaling & HA for multi-node.

Compose (local/dev, full UI stack)

podman compose up --build from the module root (docker compose works the same) brings up postgres → engine (ops API on 8787, cluster-internal) → dagron-api (127.0.0.1:8080) → frontend (127.0.0.1:3000). Sign in at http://localhost:3000 with the seeded admin (admin@local / dagron-admin). Before any non-local deploy: change DAGRON_JWT_SECRET (≥ 32 chars), change the admin password, and leave DAGRON_COOKIE_SECURE at its default (true).

Helm (Kubernetes)

The chart is on Artifact Hub. The admin-relevant values:

Single binary

The default build (sqlite,ops) needs no other infrastructure: dagron dev for a resident server with Swagger on 127.0.0.1:8787/docs, or dagron <file.yaml> for a one-shot run that exits when the run drains. Setting any of API_ADDR, CRON_CONFIG, GC_RETENTION_SECS, or DB_SCHEDULES=1 keeps the process resident; with none set, a file run drains and exits 0. Flags and env vars are on CLI & configuration.

What the state is

Back up the database; everything else is either an env var to keep safe or a re-creatable input. Executors, workers, and dagron-api are stateless — they coordinate only through the DB.

What is not state: the engine/worker/dagron-api processes; WORKFLOW_DIR (an input directory — the specs it ingests are stored in the DB); GitOps-synced repos (the repo is the source of truth and re-syncs). Two directories sit in between: the artifact store (DAGRON_ARTIFACT_DIR or object store) holds run outputs and checkpoints tasks wrote, and the GC archive (GC_ARCHIVE_DIR/GC_ARCHIVE_URL) holds runs the retention GC moved out of the DB — back both up if you cannot recompute their contents. Enterprise builds can encrypt artifacts at rest under a KEK provider; that key material (or the KMS key + version it references) must survive alongside the database, or restored ciphertext artifacts are unreadable even after a clean DB restore.

Monitoring & health

• Liveness: GET /healthz on both the engine ops API and dagron-api (no auth; no DB touch on the latter).
• Metrics: Prometheus text on the engine at GET <API_ADDR>/metrics — task counters (dispatched/succeeded/failed/retried), the reconcile-tick latency histogram, and live DB gauges (all scheduler_*). dagron-api re-surfaces JSON counts at GET /api/metrics for the UI, and GET /api/health returns a rich status for dashboards: {api, edition, db, scheduler_leader, leader_holder, active_runs, awaiting_approvals, dead_letters}. It never 500s — a DB outage answers db: "error" so the outage itself is reportable.
• Capacity: engine defaults are WORKER_COUNT=16 concurrent tasks per node and MAX_INFLIGHT_RUNS=64 active runs. Above the cap, POST /runs answers 429 with Retry-After — dagron sheds load explicitly, it does not silently drop.

Alert on:

• failed runs trending up;
• dead_letters count > 0 and growing — submissions are being rejected; redrive or fix them;
• the reconcile-tick histogram's upper buckets filling — a pegged tick shows up under load;
• Postgres availability.

A ready-to-run Prometheus + Grafana stack with a bundled dashboard ships in examples/monitoring.

Backups

SQLite

$ sqlite3 workflow.db ".backup 'workflow-backup.db'"   # safe while running (WAL)

Never copy workflow.db alone from a running daemon — the -wal/-shm sidecars carry committed data that has not been checkpointed. Either use .backup, or stop the daemon and copy all three files.

Postgres

The floor is a scheduled pg_dump -Fc — nightly, plus before every upgrade. The repo ships wrapper scripts that write timestamped custom-format dumps, refuse to leave a truncated file looking like a backup, and restrict permissions on the output:

$ ./scripts/backup-postgres.sh                       # uses $DATABASE_URL
$ ./scripts/backup-postgres.sh -d /var/backups/dagron
$ PGPASSWORD=… ./scripts/backup-postgres.sh -H db -U dagron -n workflow

# Writes <dir>/workflow-<UTC timestamp>.dump in pg_dump custom format (-Fc).
# -k/--retain N deletes dumps older than N days.

Keep the dumps somewhere that is not the database host. The restore script insists on the drop-and-recreate path — without it, pg_restore layers the dump on top of whatever is already there and you get a "successful" restore of a corrupt database:

# rehearse into a scratch DB (safe)
$ ./scripts/restore-postgres.sh -f workflow-2026….dump -n workflow_dr_test --create

# real recovery into a fresh database
$ ./scripts/restore-postgres.sh -f workflow-2026….dump -n workflow --create

# --create DROPs the target first, WITH (FORCE), so it succeeds while
# clients are connected — and prompts for the database name before doing it.

Restoring is only half of it. Afterwards, start one engine against the database and confirm it reaches worker pool ready with exit 0 — migrations are applied at engine startup and there is no dagron migrate command, so a starting engine is the only proof the schema is usable. The restore script prints row counts (runs, tasks, workflows, users, migrations) to compare against the source: a restore reporting zero users or zero workflows restored a partial dump. A backup you have not restored is not a backup — rehearse this against a scratch database; it takes minutes.

Point-in-time recovery

A nightly dump means up to 24 h of lost runs. For a real RPO, use continuous WAL archiving. Three AWS shapes, in increasing order of ops burden:

• RDS / Aurora PostgreSQL — managed PITR, ~5 min RPO; automated backups are the WAL archiving. The default choice.
• CloudNativePG on EKS — WAL plus base backups to S3, declarative manifests, seconds-to-minutes RPO; use it when you want the database next to the workload.
• pgBackRest on EC2 — self-managed Postgres shipping to S3; highest burden, for an existing EC2 estate or requirements RDS won't meet.

Whichever you pick: the bundled chart's Postgres is a test shape with no backups — production starts with postgres.enabled=false and an external database. And after any PITR restore, the dagron-specific steps still apply: repoint DATABASE_URL, start one engine first, make sure DAGRON_ENV_SECRET_KEY matches the era of the restored data, and cancel any runs that must not re-execute before the engine reclaims their stale leases. Full walkthrough with terraform wiring and verified restore drills: BACKUP_AWS.md.

Day-to-day operations

Dead letters

The dead-letter queue holds submissions that never became runs — a payload arrived from a source, could not be turned into a workflow, and was parked so ingestion could carry on. Unparseable specs park immediately; transient create_run failures are retried and park after the configured number of attempts. A run that failed is not a dead letter — its recovery lives on the run itself (rerun, retry, clear).

Every dead letter carries the original payload, the last error, the source, and the failure count. Inspect and act from the console (Dead letters in the sidebar) or over REST:

$ curl -s localhost:8080/api/dead-letters -b cookies.txt          # list
$ curl -s -X POST localhost:8080/api/dead-letters/<id>/redrive -b cookies.txt
$ curl -s -X DELETE localhost:8080/api/dead-letters/<id> -b cookies.txt

The same three operations exist unauthenticated on the engine ops API (GET /dead-letters, POST /dead-letters/{id}/redrive, DELETE /dead-letters/{id}) for in-cluster tooling. Redrive fixes nothing by itself — it re-attempts the same payload, so it only helps once the cause is gone; redriving into an unfixed cause just parks it again with the failure count one higher.

The retry policy — how many delivery attempts before parking — defaults to the engine's DEAD_LETTER_MAX_ATTEMPTS env value (default 3) and can be overridden at runtime, no restart:

$ curl -s localhost:8080/api/settings/dead-letters -b cookies.txt
$ curl -s -X PUT localhost:8080/api/settings/dead-letters -b cookies.txt \
    -H 'content-type: application/json' -d '{"max_attempts":5}'

max_attempts counts total attempts including the first — 1 parks on the first failure, 5 allows 4 retries. 0 is rejected with 400, and the PUT requires an admin session. It does not apply to parse failures, which are deterministic and park on the first attempt regardless. For watching depth, GET /api/health returns dead_letters — the number to alert on.

Stuck or wrong runs

The run and task controls, available on both surfaces (paths shown for dagron-api; the engine ops API has the same operations without the /api prefix):

GET /api/approvals lists every task parked in awaiting_approval, oldest first — the human-in-the-loop worklist. A task that ran twice after a crash or restart is expected crash-recovery behaviour, not a bug to fix here: leases expire (default 30 s) and a surviving node reclaims and re-dispatches, with version fencing rejecting the stale attempt's write. Make tasks idempotent.

API tokens for CI

POST /api/login mints a session token meant for a browser. Automation should use a personal access token instead: named, individually revocable, optionally expiring.

# Create one (requires a password session — see below).
$ curl -s -X POST localhost:8080/api/tokens -b cookies.txt \
    -H 'content-type: application/json' \
    -d '{"name":"nightly-ci","expires_in_days":90}'
# -> 201 {"id":"…","name":"nightly-ci","prefix":"dgp_…","token":"dgp_…","expires_at":"…"}

# Use it anywhere the session bearer would go — no cookie jar, no password.
$ curl -s localhost:8080/api/runs -H "Authorization: Bearer $DAGRON_TOKEN"

# List (prefix, last use, expiry — never the secret) and revoke.
$ curl -s localhost:8080/api/tokens -b cookies.txt
$ curl -s -X DELETE localhost:8080/api/tokens/<id> -b cookies.txt

Three properties to know before handing these out:

1. Copy the token at creation. Only its SHA-256 is stored; no endpoint can show it again, and a database dump yields no working credential.
2. Token management requires a password session. Requests to /api/tokens carrying a dgp_ bearer get 403 — a token that could mint tokens would replace itself faster than you could revoke it.
3. A token carries its owner's permissions, read live. There is no per-token scoping yet; a token belonging to an admin is an admin. Give automation its own user, and check last_used_at before revoking to see whether a token is still wired into something.

Secrets and environment variables

Two layers, both living in a named environment a workflow opts into with environment: <name>: plain variables (substituted into the spec, stored unencrypted) and secrets (AES-256-GCM encrypted at rest, decrypted only at task dispatch). One shared key on both dagron-api (encrypts on write) and the engine (decrypts at dispatch):

$ export DAGRON_ENV_SECRET_KEY="$(openssl rand -base64 32)"   # same value for api + engine

Without a key, writing a secret returns 503. Values are write-only: GET /api/environments lists variables in full but secrets by name only — values never leave the server. The Helm chart wires the key via envSecrets.key / envSecrets.existingSecret; compose sets one already. Remember the red-lamp notice above: this key is state your database backups do not contain.

Other admin-only routes on dagron-api worth knowing: POST /api/users and GET /api/users (user management), GET /api/audit (audit rows), and the instance-wide notification settings under /api/settings/notifications — all require the admin group. Where artifact encryption at rest is configured Enterprise, POST /api/artifacts/rotate (admin only) re-keys every artifact from the previous KEK to the current one; quiesce artifact writes while it runs, since rotation does not coordinate with concurrent PUTs.

Security posture

• Two listeners, two trust levels. The engine ops API is unauthenticated by design — bind API_ADDR to localhost or a cluster-private interface only. The public edge is dagron-api: argon2 password hashing, HS256 session JWT in an HttpOnly dagron_session cookie or Bearer header, sessions defaulting to 7 days.
• Secrets: DAGRON_JWT_SECRET must be ≥ 32 chars (startup-enforced); DATABASE_URL credentials are redacted before logging; GITHUB_TOKEN (git sync) is read from env, never stored in the DB.
• TLS is out of scope for the binaries — terminate it in front (ingress or reverse proxy). The session cookie is Secure by default; only relax it (DAGRON_COOKIE_SECURE=false) for plain-HTTP local dev.
• CORS on dagron-api is currently permissive (dev posture) — restrict to the frontend origin for production exposure.
• Tasks are arbitrary subprocesses. Whoever can submit a workflow can run commands as the engine's user (LocalExecutor) or in containers — the docker/kubernetes executors are the isolation boundary to prefer for untrusted definitions. Agent access goes through the JWT-gated edge only.

Vulnerability disclosure: SECURITY.md.

Troubleshooting

Symptom-first. The full table lives in OPERATIONS.md; these are the ones that page people.