Chunk Lifecycle
How run_chunks.status protects worker ownership and prevents stale workers from completing or releasing chunks they no longer own.
Notes
- Run creation inserts chunks as
pendingin their assigned execution placement. They cannot be dispatched while the control run iscreating; dispatch cursors are created only after all placements are seeded. - Each chunk receives a stable
run_shardin the 128-shard logical range. Child execution, attempt, aggregate, and evaluator result rows inherit that shard. - Coordinator dispatch marks selected pending chunks with
dispatched_atas it enqueues boundedrun.chunk.readywindows. - Coordinator recovery scans expired
leasedchunks for running runs, incrementsrecovery_count, resets recoverable chunks topending, and emits a recovery-scopedrun.chunk.readyevent. - Chunks that exceed the configured recovery limit are marked
failedso run finalization can complete with a failed gate instead of waiting forever. - Workers can claim a chunk only when the owning run is
running. - A chunk is completed only after all executions in its case range are terminal. If any execution is
retry_scheduled, the worker releases the chunk and delays the chunk-ready message until the nextretry_afterwindow. - Completion and release both compare the stored lease timestamp to the worker's claimed lease.
- Cancellation changes open chunks to
cancelled.