@spec error_groups(keyword()) :: [map()]

Failures grouped by what went wrong: error type, message with numbers replaced by <n> (so "timeout after 1503ms" and "after 1499ms" group), and the step it happened at. Biggest group first.

[%{workflow: "fulfill_order", step: "ship_order", type: "RuntimeError",
   message: "carrier API returned <n>", count: 41,
   first_at: ~U[...], last_at: ~U[...], rolled_back: 41}]

@spec failing_workflows(keyword()) :: [map()]

Workflows with failures in the window, worst first, each with its biggest error group:

[%{workflow: "fulfill_order", failed: 41, finished: 660, failure_rate: 0.062,
   last_failed_at: ~U[...], top_error: %{type:, message:, step:, count:}}]

@spec keys(keyword()) :: map()

Workflow keys over the window, for one workflow with :workflow or for all:

%{
  keyed: true,          # the workflow has started a run with a key
  held: 1_284,          # keys held right now
  readded: 3_912,       # starts that found their key taken
  dropped: 41,          # ...whose input differed and was dropped
  errors: 12,           # ...answered with {:error, {:key_taken, id}}
  replaced: 3,          # ...that cancelled the holder (:cancel_existing)
  sent: 9,              # ...sent to the holder as an event ({:send_event, _})
  debounced: 30,        # ...that gave a debounced run newer input
  handed_off: 87,       # keys a start took from a finished run
  timeline: [%{at: ~U[...], returned: 160, errors: 1, replaced: 0,
               sent: 2, debounced: 5}, ...],
  top: [%{key: "article:17", readded: 48, dropped: 0,
          holder_id: "…", holder_status: :completed}, ...]
}

The counts come from Durable.Activity, which keeps a week, so they cover at most the last 7 days. :bucket_seconds and :limit (top keys, default

10. are optional.

@spec overview(keyword()) :: map()

Headline numbers for the window:

%{
  started: 12_482,
  finished: %{completed: 12_031, failed: 180, compensated: 34, cancelled: 21, ...},
  failure_rate: 0.017,            # failed-like ÷ finished
  duration_ms: %{p50: 380, p95: 4_210, max: 11_800},   # completed runs
  queue_wait_ms: %{p50: 120, p95: 1_300},             # scheduled → started
  backlog: 37,                    # ready to start right now
  waiting: %{input: 9, event: 96, sleeping: 23}       # right now
}

@spec queue_history(keyword()) :: %{required(String.t()) => [map()]}

Each queue's depth per minute over the window, from Durable.Queue.Sampler, oldest first. Minutes with no sample (no node was up) are absent.

%{"default" => [%{at: ~U[2026-10-16 09:00:00Z], ready: 31, scheduled: 4,
                  running: 8, waiting: 120}, ...]}

@spec queues(keyword()) :: [map()]

Every queue's state right now — configured queues are listed even when idle:

[%{queue: "default", paused: false, running: 8, ready: 31, scheduled: 412,
   oldest_ready_at: ~U[...], done_last_5m: 910, wait_p95_ms: 1_300}]

wait_p95_ms is scheduled → started over the last hour.

@spec step_performance(keyword()) :: [map()]

Per-step performance over attempts that started in the window, in the order steps first ran. Durations are active time — sleeps and waits are not attempts. Compensations are included with compensation: true.

failed_runs counts runs whose last attempt at the step failed — runs that gave up there, as opposed to failed_attempts that a retry recovered.

[%{workflow: "fulfill_order", step: "ship_order", compensation: false,
   runs: 660, attempts: 771, failed_attempts: 117, failed_runs: 41,
   retried_runs: 121, p50_ms: 640, p95_ms: 2_140, max_ms: 6_040}]

@spec timeline(keyword()) :: [map()]

The window cut into :bucket_seconds (default: ~48 buckets), oldest first, zero-filled:

[%{at: ~U[...], started: 260, completed: 251, failed: 6, cancelled: 1,
   failure_rate: 0.023, p50_ms: 1_900, p95_ms: 4_200}, ...]

failed includes rolled-back runs; p50_ms/p95_ms are over runs completed in the bucket (nil when none).

@spec workflows(keyword()) :: [map()]

Every workflow that has run, with its numbers for the window — one grouped query, for a catalog page. Busiest first.

[%{workflow: "fulfill_order", module: "Elixir.App.Orders", started: 660,
   finished: 655, failed: 41, failure_rate: 0.063, retried: 121,
   p50_ms: 1_920, p95_ms: 4_210, running: 4, waiting: 0,
   last_run_at: ~U[...]}]

retried counts runs where some step needed a second attempt. running, waiting and last_run_at are as of now, not limited to the window.