Durable.Telemetry (Durable v1.0.0)

View Source

The :telemetry events Durable emits. This list is a contract: names, measurements and metadata keys only change in a major version.

Spans follow :telemetry.span/3: a :start event, then exactly one of :stop or :exception, with duration in native time units — the same shape as Phoenix, Ecto and Oban, so existing tooling reads them as-is.

Run, step, compensation and schedule events include :durable (the instance name) in their metadata.

Runs

A run executes in segments: it starts, runs until it finishes or suspends (sleep, wait for an event or input, parallel fan-out), and later resumes as a new segment, possibly on another node. Each segment is one span.

  • [:durable, :workflow, :start] — %{system_time, monotonic_time}; metadata :workflow_id, :workflow, :workflow_module, :queue, :trigger, :resumed (false on a run's first segment)
  • [:durable, :workflow, :stop] — %{duration, monotonic_time}; adds :status: :completed, :failed, :compensated, :compensation_failed, :waiting (suspended) or :superseded (cancelled or reclaimed by another node mid-segment)
  • [:durable, :workflow, :exception] — the executor itself crashed; adds :kind, :reason, :stacktrace

Steps

One span per attempt. A step that raises or returns an error is a normal :stop with result: :error — :exception means Durable itself failed.

  • [:durable, :step, :start] — metadata :workflow_id, :workflow, :step, :step_type, :attempt, :max_attempts
  • [:durable, :step, :stop] — adds :result (:ok, :error or :suspended) and, for errors, :error (%{type, message})
  • [:durable, :step, :exception]
  • [:durable, :step, :retry] — %{backoff_ms}; the step metadata plus :error, emitted before sleeping for the next attempt

Compensations

  • [:durable, :compensation, :start | :stop | :exception] — metadata :workflow_id, :workflow, :compensation, :step (the step it undoes); :stop adds :result (:ok or :error)

Queue, scheduler and waits

  • [:durable, :queue, :job_claimed] — %{count}; :job_id, :queue, :node_id
  • [:durable, :queue, :job_completed] — %{duration}; :job_id, :queue, :workflow_module, :workflow_name, :status
  • [:durable, :queue, :poll] — %{jobs_fetched}; :queue, :available_slots
  • [:durable, :queue, :heartbeat] — %{count}; :job_id, :queue
  • [:durable, :queue, :stale_recovered], [:durable, :queue, :zombie_recovered] — %{count}
  • [:durable, :queue, :worker_fenced] — %{count, duration}; :job_id, :queue
  • [:durable, :queue, :ack_failed] — %{count, attempts}; :job_id, :reason
  • [:durable, :schedule, :trigger] — %{count}; :schedule, :workflow_id, :workflow_module, :workflow, :queue
  • [:durable, :wait, :sleep_woken] — %{count}
  • [:durable, :pruner, :pruned] — %{count}; :group (:completed, :failed or :cancelled), once per deleted batch

Reporting

metrics/0 returns Telemetry.Metrics definitions for any reporter (Prometheus, StatsD, LiveDashboard) when :telemetry_metrics is installed:

def metrics, do: my_metrics() ++ Durable.Telemetry.metrics()

attach_default_logger/1 logs one line per finished run and per failed step.

Summary

Functions

Logs one line per finished run (level, default :info) and one warning per failed step attempt. Detach with :telemetry.detach("durable-default-logger").

Every event name Durable emits, for :telemetry.attach_many/4.

Telemetry.Metrics definitions for Durable's events. Tags are bounded — workflow, step, queue and schedule names, never run ids.

Functions

attach_default_logger(opts \\ [])

@spec attach_default_logger(keyword()) :: :ok | {:error, :already_exists}

Logs one line per finished run (level, default :info) and one warning per failed step attempt. Detach with :telemetry.detach("durable-default-logger").

events()

@spec events() :: [[atom()]]

Every event name Durable emits, for :telemetry.attach_many/4.

metrics()

@spec metrics() :: [Telemetry.Metrics.t()]

Telemetry.Metrics definitions for Durable's events. Tags are bounded — workflow, step, queue and schedule names, never run ids.