Failure Modes

Run creation crash

Run creation commits a control-owned creating run, placement ledger, and exact chunk plan before seeding remote execution databases. Each placement seed is an idempotent transaction. If the process exits after a remote commit but before its control acknowledgement, a coordinator claims the expired creation lease and safely applies the same seed again for the original run_id. Dispatch cursors are created only when every ledger row is seeded, in the same control transaction that moves the run to pending and deletes the temporary plan. Retryable failures leave progress durable; incompatible existing rows mark the run failed without making it dispatchable.

Worker crash

Chunk messages are delivered at least once. A worker holds the RabbitMQ delivery unacked while it owns the chunk lease, and it renews both the chunk lease and the running attempt leases while processing. If the worker exits before acknowledgement, RabbitMQ may redeliver the message and the next worker must win the database chunk claim by run_id + run_shard + chunk_id before doing work. The coordinator also scans expired chunk leases, marks current running attempts stale, requeues recoverable chunks with recovery-scoped run.chunk.ready events, and marks chunks failed after the recovery limit.

Duplicate delivery

Duplicate broker deliveries are allowed. Exactly-once effects are managed in the database: chunks can only be claimed from an open or expired lease, recovered chunk-ready events carry unique recovery dedupe keys, executions only accept terminal transitions from the current attempt id, attempt number, worker id, and live attempt lease, and evaluator findings are unique per run, attempt, evaluator, and finding index.

Outbox publisher crash

Outbox claims assign a claim_token and future availability window on outbox_delivery_queue. A publisher only deletes or reschedules a delivery row while that token is still current. If it crashes, the delivery row becomes available again and another coordinator can retry publication from the durable outbox_events ledger.

Broker routing failure

RabbitMQ publishes use durable topology, persistent messages, mandatory routing, and publisher confirms. Broker sessions are recreated after connection or channel loss, and publish operations retry once on a fresh session before returning failure. Unroutable or negatively acknowledged publishes leave the delivery row pending for retry instead of marking the ledger event published.

Partial persistence

Worker persistence batches are guarded by current-attempt and live-attempt-lease authority checks. If a stale worker loses authority, its final state changes are rejected rather than overwriting a newer attempt.

Execution retry waits

Failed case attempts that still have retry budget are stored as retry_scheduled with a bounded retry_after timestamp. Workers do not keep spinning on those rows: they release the chunk, delay the chunk-ready message until the next retry window, and re-check the database before creating another attempt. defaults.max_attempts closes the execution retry loop; actual worker processing failures still use the bounded RabbitMQ failure retry budget and then quarantine or fail the chunk.

Evaluator resource exhaustion

Each Wasm evaluator invocation runs in a fresh Wasmtime store with memory, table, instance, fuel, timeout, and log-message limits. A worker-wide semaphore caps active evaluator invocations so aggregate Wasm memory and CPU exposure stay bounded. If an evaluator exceeds its budget, the worker records an evaluator runtime error and continues through the normal result persistence path.