Skip to main content

Worker Components

Describes the major internal responsibilities of the worker runtime.

Notes​

  • Workers consume run.chunk.ready messages carrying run_id, run_shard, and chunk_id, then claim chunk leases before loading case data.
  • vigilo worker start defaults to one in-flight chunk per worker process. --max-inflight-chunks / VIGILO_WORKER_MAX_INFLIGHT_CHUNKS can raise that limit, and RabbitMQ prefetch is set to the same value.
  • Workers attach a process worker_id, host label, internal queue message UUID, broker message id, attempt lease, and heartbeat timestamp to newly allocated attempts.
  • While a chunk is processing, the worker renews the chunk lease and current running attempt leases. Terminal execution transitions require current-attempt authority and a live attempt lease for that worker.
  • Run context and evaluator components are cached inside the worker process to avoid repeated profile parsing and WASM compilation.
  • Workers use a shared HTTP client and run chunk cases with bounded chunk-local parallelism before persisting results in batch.
  • Evaluator execution is admitted through a process-wide Wasm semaphore and each invocation runs in a fresh Wasmtime store with memory, table, instance, fuel, timeout, and log-message limits.
  • Execution writes use shared run-state guards so chunks for the same running run do not serialize on the run row.
  • Execution terminal transitions are authority guarded by the current attempt id, attempt number, worker id, and attempt lease.
  • Workers reload chunk executions from the database on each pass; retry-scheduled executions run only after retry_after is due, and exhausted executions are failed instead of retried again.
  • Queue messages are acknowledged only after invalid, stale, or completed work is handled; recoverable chunk failures release the lease and use bounded retry buckets, while planned execution retry waits use delayed redelivery without consuming the failure retry budget.
  • Duplicate broker deliveries are expected under at-least-once messaging; chunk claims, attempt authority tokens, and evaluator-result uniqueness make processing idempotent at the persistence boundary.