Skip to main content

Chunk Lifecycle

How run_chunks.status protects worker ownership and prevents stale workers from completing or releasing chunks they no longer own.

Notes

  • Run creation inserts chunks as pending in their assigned execution placement. They cannot be dispatched while the control run is creating; dispatch cursors are created only after all placements are seeded.
  • Each chunk receives a stable run_shard in the 128-shard logical range. Child execution, attempt, aggregate, and evaluator result rows inherit that shard.
  • Coordinator dispatch marks selected pending chunks with dispatched_at as it enqueues bounded run.chunk.ready windows.
  • Coordinator recovery scans expired leased chunks for running runs, increments recovery_count, resets recoverable chunks to pending, and emits a recovery-scoped run.chunk.ready event.
  • Chunks that exceed the configured recovery limit are marked failed so run finalization can complete with a failed gate instead of waiting forever.
  • Workers can claim a chunk only when the owning run is running.
  • A chunk is completed only after all executions in its case range are terminal. If any execution is retry_scheduled, the worker releases the chunk and delays the chunk-ready message until the next retry_after window.
  • Completion and release both compare the stored lease timestamp to the worker's claimed lease.
  • Cancellation changes open chunks to cancelled.