Skip to main content

Execution Flow & State Model

This document describes how Agent Vigilo processes an evaluation run from start to completion, including execution lifecycle, retry behavior, and event publication.

Overview​

At a high level, the system follows a fan-out / fan-in pattern:

  1. A is created
  2. Dataset cases are grouped into shard-local
  3. The coordinator dispatches chunks through and RabbitMQ
  4. Workers process case through
  5. Evaluators return one primary measurement or abstention plus optional diagnostics
  6. Required is verified
  7. Complete host-normalized binding results are aggregated
  8. The run is finalized
  9. Its completion event record is published as a

Evaluation Counts And Relationships​

The runtime uses the following :

one run
  -> many chunks
       -> many dataset cases
            -> one expected execution per case
                 -> one or more attempts over time
  -> one or more used run shards
       -> one shard placement per used shard
       -> one dispatch cursor per used shard
       -> many chunk dispatch windows over time
            -> one run.chunk.ready outbox record per selected chunk

one run.started outbox record per run
one broker message per publish attempt, with possible redelivery

Multiple run shards can share one database alias. A coordinator pass can report counts for several aliases even though each routed operation targets one database at a time. See Scaling for routing and placement details.

End-to-End Flow​

1. Run Creation​

A run is created with:

  • dataset
  • run profile
  • aggregation policy
  • agent configuration (versioned target)

At creation, every evaluator reference is resolved to an immutable . The plan pins registry id, artifact hash, evaluator ABI, host adapter, runtime version, and the complete scoring-policy hash. Each execution placement receives the same plan and workers verify its hash before use.

Reference formats:

run.status = pending
run.gate_status = unknown

Run chunks are then generated and assigned a stable run_shard from the 128 logical shard range. Executions are allocated later by workers from the chunk's dataset cases:

run_chunk.status = pending
run_chunk.run_shard = 0..127

2. Execution Dispatch​

The coordinator selects a bounded chunk dispatch window from one run shard and inserts one run.chunk.ready outbox event record per selected chunk. The outbox publisher turns those records into RabbitMQ messages carrying run_id, run_shard, and chunk_id. Workers consume the broker messages, claim the identified chunks, and allocate due executions:

run_chunk.status = leased
execution.status = running
execution_attempt.status = running

Each attempt is associated with:

  • a worker
  • an attempt lease and heartbeat
  • an internal queue message UUID plus the broker message id, when available
  • an attempt number

For each loaded case, workers resolve the evaluation plan from the run profile. An explicit dataset case_group selects the profile group with the same id. When case_group is omitted, workers use the profile group's applies_to task and tag rules.

The resolved evaluation plan is stored on the execution before the attempt runs:

  • profile_group_id: the selected profile group id, or a deterministic comma-separated id list for multiple automatic matches
  • evaluator_manifest: the resolved evaluator bindings and, when persistence.mode: full, evaluator configs
  • expected_evaluator_count: the number of evaluator invocations expected for the execution

The dataset case_group value is part of immutable case content. It is loaded from case_blobs, passed into evaluator input, and affects the case hash because changing it changes evaluator routing.

3. Agent Invocation​

The worker invokes the configured agent target:

  • may be a single model call
  • may be a multi-step workflow
  • is invoked through the run profile agent.http endpoint

The worker sends run/attempt ids, the agent identity, the case input, and non-oracle case metadata. The response is mapped into the evaluator actual envelope before evaluators run.

If the agent call fails:

attempt.status = failed_agent_call
execution.status = retry_scheduled (if retryable)

4. Evaluation Phase​

After a successful agent response:

  • evaluators are executed
  • each pinned evaluator artifact and ABI identity is verified before its version-specific host adapter is selected
  • Wasm evaluator invocations acquire a worker-local semaphore permit
  • each evaluator runs in a fresh Wasmtime store with memory, table, instance, fuel, timeout, and log-message limits
  • the profile binding normalizes a completed measurement and derives its judgment
  • one invocation row is appended to evaluator_results; diagnostics are appended to evaluator_diagnostics

After all invocations settle, the worker compares their outcomes with the resolved evaluator manifest. Every required binding must have exactly one valid measurement and must not error or abstain. Optional bindings are diagnostic and do not participate in completeness, scores, or gates.

Each evaluator invocation result records:

  • stable binding id and evaluator identity
  • execution outcome (completed / error / abstained)
  • host-derived judgment (passed / failed) for completed measurements
  • profile aggregation dimension
  • typed raw measurement (binary, numeric, or ordinal), normalized score, normalization policy hash, host threshold, weight, and blocking policy
  • raw evaluator output according to persistence.persist_raw_outputs

Each diagnostic records severity, category, reason, tags, and evidence. It has no dimension, score, judgment, weight, or blocking authority.

When persistence.mode: summary, execution-level case payload snapshots and evaluator binding config are replaced with redaction markers. Dataset case blobs remain durable for retry execution and reproducibility. When persistence.persist_evaluator_evidence: false, raw evaluator output is also redacted because raw output may contain embedded evidence.

If evaluation fails:

attempt.status = failed_evaluation
execution.status = retry_scheduled (if retryable)

5. Execution Completion​

Once all evaluator invocations settle:

  • evaluator completeness is recorded in execution_aggregates.summary
  • incomplete required output produces overall_status = error with no dimension or aggregate score
  • complete required output is aggregated with the matching run profile policy
  • invocation results, diagnostics, and the resulting aggregate are persisted together
  • execution is marked terminal
attempt.status = completed
execution.status = completed | failed | timed_out

6. Retry Flow​

If an execution fails but is retryable:

execution.status = retry_scheduled
execution.retry_after = now() + bounded exponential backoff

The worker finishes the current chunk pass, summarizes the chunk's execution state from the database, and releases the chunk back to pending when any cases are still waiting for retry. The chunk-ready broker message is delayed until the next retry window instead of being treated as a worker failure.

When the message returns, the worker reloads the chunk cases and the execution allocation query decides which cases should run:

attempt.status = pending → running

Cases whose retry_after is still in the future are skipped for that pass and keep the chunk open. Cases whose retry window is due receive the next authoritative attempt.

Older attempts may become:

attempt.status = stale

This occurs when:

  • a worker loses its lease
  • a newer attempt supersedes it

Workers renew the chunk lease and the running attempt leases while processing. Terminal execution transitions are accepted only from the current attempt owner with a live attempt lease, except for retry-budget exhaustion closures after recovery.

Retries are bounded by defaults.max_attempts. When the current attempt number has reached that limit, the worker closes the execution as failed instead of scheduling another attempt. Planned retry waits do not consume the worker-message failure retry budget; actual worker processing failures still use bounded RabbitMQ retry buckets and are quarantined or fail the chunk after exhaustion.

7. Run Finalization​

Before finalization, the coordinator recovers expired chunk leases for running runs. Recoverable chunks are reset to pending, their current running attempts are marked stale, and they receive a recovery-scoped run.chunk.ready outbox event record. Chunks that exceed the recovery limit are marked failed.

The coordinator checks a bounded number of candidates per cycle:

Are all chunks terminal?

pending or leased chunks keep a run out of finalization, including chunks waiting on execution retry windows. Once chunks are terminal, each shard publishes bounded scorecard counters for the configured run gates.

A candidate with missing or non-terminal shard summaries is recorded as checked and rotated behind unchecked candidates. The coordinator continues to later candidates in the same cycle, preventing one slow shard from blocking unrelated ready runs.

If all chunks are terminal:

run.status = finalizing

The system:

  • verifies every shard scorecard uses the frozen policy hash and gate set
  • merges counts and score sums without scanning execution rows centrally
  • applies run-level dimension or evaluator gates
  • atomically stores the final scorecard and gate_status
run.status = completed
run.gate_status = pass | fail

8. Event Publication (Outbox Pattern)​

A run.completed outbox event record is inserted into the durable outbox ledger:

outbox_events.status = pending

The database creates a matching hot delivery row in the same transaction:

outbox_delivery_queue.available_at = now()

The coordinator's outbox publisher claims delivery rows, publishes each joined event payload as a RabbitMQ message, and waits for broker confirmation:

delivery row claimed -> confirmed publish -> delete delivery row
                     -> publish failure   -> retry delivery row later

The ledger row is retained for idempotency, audit, and replay. The delivery queue only contains unfinished publish work.

State Machines​

Run Lifecycle​

pending → running → finalizing → completed
                  ↘ failed
                  ↘ cancelled

Execution Lifecycle​

pending → running → completed
                  ↘ retry_scheduled → running
                  ↘ failed
                  ↘ timed_out
                  ↘ cancelled

Attempt Lifecycle​

pending → running → completed
                  ↘ failed_agent_call
                  ↘ failed_evaluation
                  ↘ timed_out
                  ↘ cancelled
                  ↘ stale

Key Design Properties​

1. Append-only evaluator results​

Evaluator invocation results and diagnostics are never updated, only inserted. This provides:

  • auditability
  • reproducibility
  • traceability

2. Separation of state vs evidence​

  • state tables (runs, executions, attempts) are mutable
  • evaluator invocation results and diagnostics are immutable facts, with evidence/raw-output fields governed by the run profile persistence policy

3. Idempotent finalization​

Multiple coordinators may attempt to finalize a run.

The system ensures:

  • only one finalization succeeds
  • duplicate attempts are safe

4. Retry-safe execution​

Executions may have multiple attempts.

Only the most recent non-stale attempt is authoritative.

5. Reliable event delivery​

The outbox pattern ensures:

  • no lost events
  • retryable publishing
  • eventual consistency
  • a bounded hot delivery table separate from event history

Design Philosophy​

Agent Vigilo evaluates the behavior of a target system, not just a model.

An "agent" may represent:

  • a single model call
  • a prompt pipeline
  • a multi-step workflow
  • a deployed HTTP service

The evaluation system treats all targets uniformly via a versioned invocation interface.

Summary​

The system is designed to:

  • handle distributed execution safely
  • tolerate worker failure and retries
  • preserve evaluation evidence according to the configured persistence policy
  • produce deterministic, policy-driven outcomes
  • reliably signal completion to downstream systems