Skip to main content

Coordinator Components

Describes the major internal responsibilities of the coordinator runtime.

Notes​

  • Run-creation recovery runs first. It claims expired creating runs, replays idempotent placement seeds, and opens dispatch cursors only after every placement is recorded as seeded.
  • Chunk lease recovery then runs before dispatch and finalization: expired worker chunk leases are either requeued with recovery-scoped chunk-ready events or marked failed after the recovery limit.
  • Dispatch claims open run_id + run_shard cursors for pending or running runs, starts runs when needed, and enqueues bounded chunk-ready windows from one shard partition at a time.
  • Already-running run dispatch is cursor-owned and takes only a compatible shared runs lock, allowing multiple coordinators to dispatch different shard cursors for the same run while cancellation and finalization retain exclusive lifecycle guards.
  • Finalization is coordinator-owned and lease guarded; workers persist execution outcomes but do not complete the run row.
  • Outbox publication is deliberately separate from state changes so committed ledger events can be retried after broker or process failures.
  • Coordinator throughput knobs can tune cycle interval, creation-recovery batch size, chunk lease recovery batch size and limit, dispatch windows, outbox batch size, publish parallelism, leases, and retry delay.
  • Coordinator structured logs expose dispatch backlog, finalization backlog, oldest finalization candidate lag, per-placement outbox backlog, route resolution timing, pool acquisition timing, and cycle-stage durations for metrics pipelines.
  • Outbox publishers claim rows from the sharded outbox_delivery_queue, assign claim tokens, and publish joined ledger payloads with bounded concurrency.
  • Confirmed publishes update outbox_events to published and delete the delivery row, keeping the hot queue limited to unfinished work.
  • RabbitMQ publishes use durable topology, persistent messages, mandatory routing, publisher confirms, and the outbox dedupe key as the AMQP message id.
  • Broker delivery remains at-least-once; exactly-once effects are enforced by database idempotency and authority checks.