Skip to main content

Worker Components

Describes the major internal responsibilities of the worker runtime.

Notes​

  • Workers consume run.chunk.ready broker messages carrying run_id, run_shard, chunk_id, database_alias, and write_epoch, then validate local shard admission while claiming the chunk lease.
  • vigilo worker start defaults to one in-flight chunk per worker process. --max-inflight-chunks / VIGILO_WORKER_MAX_INFLIGHT_CHUNKS can raise that limit, and RabbitMQ prefetch is set to the same value.
  • Workers attach a process worker_id, host label, internal queue message UUID, broker message id, attempt lease, and heartbeat timestamp to newly allocated attempts.
  • While a chunk is processing, the worker renews the chunk lease and current running attempt leases. Terminal execution transitions require current-attempt authority and a live attempt lease for that worker.
  • Run context caches verify the execution plan and scoring-policy hashes from the immutable run snapshot. Evaluator component caches use the artifact hash, WIT contract hash, and adapter id, preventing collisions across ABI versions.
  • Workers fetch each evaluator by its pinned registry id and reject artifact or ABI drift before dispatching its prepared version-specific host adapter. The adapter is resolved once per cached artifact, not per case. Lifecycle changes affect new runs without changing an already-created run.
  • Workers use a shared HTTP client and run chunk cases with bounded chunk-local parallelism before persisting results in batch.
  • Evaluator execution is admitted through a process-wide Wasm semaphore and each invocation runs in a fresh Wasmtime store with memory, table, instance, fuel, timeout, and log-message limits.
  • The completeness and aggregation component verifies every required evaluator binding before publishing scores. Missing, errored, skipped, or unscored required output produces an errored aggregate with no score; optional diagnostic output cannot affect policy.
  • Execution writes use shared run-state guards so chunks for the same running run do not serialize on the run row.
  • Execution terminal transitions are authority guarded by the current attempt id, attempt number, worker id, and attempt lease.
  • Workers reload chunk executions from the database on each pass; retry-scheduled executions run only after retry_after is due, and exhausted executions are failed instead of retried again.
  • Queue messages are acknowledged only after invalid, stale, or completed work is handled; recoverable chunk failures release the lease and use bounded retry buckets, while planned execution retry waits use delayed redelivery without consuming the failure retry budget.
  • Consumer creation and retry or quarantine publishing pass through one process-local broker circuit. Consumer reopening also uses bounded backoff; message retry budgets remain independent from broker availability.
  • Duplicate broker deliveries are expected under at-least-once messaging; chunk claims, attempt authority tokens, and evaluator-result uniqueness make processing idempotent at the persistence boundary.