Orchestrator DLQ management

Harn records failed trigger deliveries in trigger.dlq. The portal exposes that event-log topic at /dlq so operators can inspect, replay, purge, and export dead-letter entries without hand-reading JSONL or SQLite rows.

DLQ management panel

Portal workflow#

The /dlq route lists pending entries with:

trigger_id
event_id
failed_at
last_error
retry_count
derived error_class

Filters are applied server-side for trigger, provider, error class, state, text, and time range. Selecting a row opens the full event payload, headers, attempt history, and matching predicate lifecycle trace.

Actions:

Replay starts a trigger replay job for the original event id.
Replay with drift accept uses the same replay path but marks the portal job as an accepted-drift replay.
Purge appends a discarded tombstone to trigger.dlq after confirmation.
Export fixture downloads a harn.dlq.fixture.v1 JSON fixture for conformance and regression tests.

Bulk operations use the current filters and are capped by the admin API. Replay bulk jobs are rate-limited, and purge bulk jobs are intended for narrow filters such as error_class=unknown with an age window.

Admin API#

The portal server exposes the DLQ admin API:

GET  /api/dlq
GET  /api/dlq/{entry_id}
POST /api/dlq/{entry_id}/replay
POST /api/dlq/{entry_id}/replay-drift-accept
POST /api/dlq/{entry_id}/purge
GET  /api/dlq/{entry_id}/export
POST /api/dlq/bulk/replay
POST /api/dlq/bulk/purge

The list endpoint returns normalized entries, error-class groups, active spike alerts, and configured alert destinations.

Error classes#

DLQ records are tagged when moved into the queue. Older records are classified when read by the portal.

Class	Meaning
`provider_5xx`	Upstream provider outage or server error.
`predicate_panic`	Trigger predicate crashed or threw.
`handler_panic`	Handler crashed or threw.
`handler_timeout`	Handler exceeded a wall-clock deadline.
`auth_failed`	Downstream authentication or authorization rejection.
`budget_exhausted`	Trigger or predicate budget short-circuited dispatch.
`unknown`	Fallback for uncategorized errors.

Alerts#

Per-trigger DLQ alerts are configured in harn.toml:

[[triggers]]
id = "cake-classifier"
kind = "webhook"
provider = "github"
handler = "handlers.classify"

[[triggers.dlq_alerts]]
destinations = [
  { kind = "slack", channel = "#ops", webhook_url_env = "OPS_SLACK_WEBHOOK" },
  { kind = "webhook", url = "https://alerts.example.com/harn" },
]
threshold = { entries_in_1h = 5, percent_of_dispatches = 20.0 }

The portal surfaces active spike alerts when matching DLQ volume crosses the configured entries_in_1h threshold. percent_of_dispatches is preserved in the API response for integrations that compare DLQ volume to dispatch totals.

Edit this page on GitHub