Skip to main content

Failure Modes

Run creation crash​

Run creation commits a control-owned creating run, placement ledger, and exact chunk plan before seeding remote execution databases. Each placement seed is an idempotent transaction. If the process exits after a remote commit but before its control acknowledgement, a coordinator claims the expired creation lease and safely applies the same seed again for the original run_id. Dispatch cursors are created only when every ledger row is seeded, in the same control transaction that moves the run to pending and deletes the temporary plan. Retryable failures leave progress durable; incompatible existing rows mark the run failed without making it dispatchable.

Worker crash​

Chunk-ready broker messages are delivered at least once. A worker holds the RabbitMQ delivery unacked while it owns the chunk lease, and it renews both the chunk lease and the running attempt leases while processing. If the worker exits before acknowledgement, RabbitMQ may redeliver the message and the next worker must win the database chunk claim by run_id + run_shard + chunk_id before doing work. The coordinator also scans expired chunk leases, marks current running attempts stale, requeues recoverable chunks with recovery-scoped run.chunk.ready outbox event records, and marks chunks failed after the recovery limit.

Duplicate delivery​

Duplicate broker deliveries are allowed. Exactly-once effects are managed in the database: chunks can only be claimed from an open or expired lease, recovered chunk-ready outbox event records carry unique recovery dedupe keys, executions only accept terminal transitions from the current attempt id, attempt number, worker id, and live attempt lease, and evaluator invocation results are unique per run, attempt, and stable binding id. Diagnostics are unique by invocation result and diagnostic index.

Stale shard route​

Chunk-ready messages include the execution database alias and write epoch used at dispatch. The worker validates that epoch against local_shard_admissions in the same execution-database transaction as the chunk claim. A stale or closed owner cannot accept the write. The worker invalidates its route cache, refreshes control metadata once, and retries the current owner; if cutover is in progress, it delays the message. Move recovery may temporarily leave no writable owner, but source closure always precedes target opening.

Shard-move retries reconcile the durable boundary states. A prepared target can resume while the source is still open; a locally draining source is never reopened when control still reports copying; a closed source can finish control activation; and an active target route repairs a target admission that was not opened before the mover exited. Reclaiming an expired move increments its control generation and installs that generation with the new claim token under exclusive target admission. Once installed, target transactions from the expired coordinator fail before mutation, even if that coordinator resumes after the new mover writes.

Outbox publisher crash​

Outbox claims assign a claim_token and future availability window on outbox_delivery_queue. A publisher only deletes or reschedules a delivery row while that token is still current. If it crashes, the delivery row becomes available again and another coordinator can retry publication from the durable outbox_events ledger.

Execution database outage​

Each process keeps an independent circuit breaker per execution database alias. Connection-level failures open only the affected alias after the configured threshold. Coordinator creation, recovery, dispatch, finalization-summary, and outbox operations then skip that alias without consuming its durable work. Workers delay only the affected alias's message without spending its ordinary failure retry budget. Healthy aliases continue through the coordinator cycle or worker process. One half-open probe is admitted after a bounded, jittered cooldown; success closes the circuit and a repeated availability failure increases the cooldown up to its configured maximum.

Circuit state is not routing authority and is not stored in the control database. A restart begins closed, direct administration bypasses the breaker, and application, route, contention, configuration, and local pool errors do not mark a remote database unavailable.

Broker routing failure​

RabbitMQ publishes use durable topology, persistent messages, mandatory routing, and publisher confirms. Broker sessions are recreated after connection or channel loss, and publish operations retry once on a fresh session before returning failure. Unroutable or negatively acknowledged publishes leave the delivery row pending for retry instead of marking the ledger event published.

Each coordinator and worker process also keeps one . Repeated connection, channel, I/O, or heartbeat failures from publishing, one-shot receipt, or consumer creation open the circuit. Availability classification uses preserved Lapin error types rather than formatted error messages. Broker operations then fail locally until one bounded, jittered half-open probe is admitted. Durable outbox rows remain pending, unacknowledged deliveries remain eligible for redelivery, and long-running workers pace consumer reopening with bounded exponential backoff. Broker responses that prove RabbitMQ is reachable do not open the circuit, even when the operation is unroutable or negatively acknowledged. Message processing retries and budgets remain separate from transport availability.

Partial persistence​

Worker persistence batches are guarded by current-attempt and live-attempt-lease authority checks. If a stale worker loses authority, its final state changes are rejected rather than overwriting a newer attempt.

Execution retry waits​

Failed case attempts that still have retry budget are stored as retry_scheduled with a bounded retry_after timestamp. Workers do not keep spinning on those rows: they release the chunk, delay the chunk-ready broker message until the next retry window, and re-check the database before creating another attempt. defaults.max_attempts closes the execution retry loop; actual worker processing failures still use the bounded RabbitMQ failure retry budget and then quarantine or fail the chunk.

Evaluator resource exhaustion​

Each Wasm evaluator invocation runs in a fresh Wasmtime store with memory, table, instance, fuel, timeout, and log-message limits. A worker-wide semaphore caps active evaluator invocations so aggregate Wasm memory and CPU exposure stay bounded. If an evaluator exceeds its budget, the worker records an evaluator runtime error and continues through the normal result persistence path.