Skip to main content

Execution Flow & State Model

This document describes how Agent Vigilo processes an evaluation run from start to completion, including execution lifecycle, retry behavior, and event publication.

Overview​

At a high level, the system follows a fan-out / fan-in pattern:

  1. Run is created
  2. Executions are generated from a dataset cases
  3. Workers process executions via attempts
  4. Evaluators produce append-only results
  5. Execution aggregates are computed
  6. Run is finalized
  7. Completion event is published

Core Concepts​

Run​

A run represents a batch evaluation of a versioned agent target against a dataset and evaluation profile.

Execution​

An execution represents one dataset case evaluated against the target system.

Attempt​

An attempt represents a single worker’s effort to process an execution. Multiple attempts may exist due to retries or failures.

Evaluator Result​

An append-only evidence row representing one finding emitted by an evaluator invocation.

End-to-End Flow​

1. Run Creation​

A run is created with:

  • dataset
  • evaluation profile
  • aggregation policy
  • agent configuration (versioned target)

Reference formats:

run.status = pending
run.gate_status = unknown

Run chunks are then generated and assigned a stable run_shard from the 128 logical shard range. Executions are allocated later by workers from the chunk's dataset cases:

run_chunk.status = pending
run_chunk.run_shard = 0..127

2. Execution Dispatch​

The coordinator emits run.chunk.ready messages with run_id, run_shard, and chunk_id. Workers claim chunks and allocate due executions for the chunk:

run_chunk.status = leased
execution.status = running
execution_attempt.status = running

Each attempt is associated with:

  • a worker
  • an attempt lease and heartbeat
  • an internal queue message UUID plus the broker message id, when available
  • an attempt number

For each loaded case, workers resolve the evaluation plan from the run profile. An explicit dataset case_group selects the profile group with the same id. When case_group is omitted, workers use the profile group's applies_to task and tag rules.

The resolved evaluation plan is stored on the execution before the attempt runs:

  • profile_group_id: the selected profile group id, or a deterministic comma-separated id list for multiple automatic matches
  • evaluator_manifest: the resolved evaluator bindings and, when persistence.mode: full, evaluator configs
  • expected_evaluator_count: the number of evaluator invocations expected for the execution

The dataset case_group value is part of immutable case content. It is loaded from case_blobs, passed into evaluator input, and affects the case hash because changing it changes evaluator routing.

3. Agent Invocation​

The worker invokes the configured agent target:

  • may be a single model call
  • may be a multi-step workflow
  • is invoked through the run profile agent.http endpoint

The worker sends run/attempt ids, the agent identity, the case input, and non-oracle case metadata. The response is mapped into the evaluator actual envelope before evaluators run.

If the agent call fails:

attempt.status = failed_agent_call
execution.status = retry_scheduled (if retryable)

4. Evaluation Phase​

After a successful agent response:

  • evaluators are executed
  • Wasm evaluator invocations acquire a worker-local semaphore permit
  • each evaluator runs in a fresh Wasmtime store with memory, table, instance, fuel, timeout, and log-message limits
  • each normalized evaluator finding is appended to evaluator_results

Each evaluator finding records:

  • status (passed / failed / error / skipped)
  • severity
  • profile aggregation dimension
  • normalized score
  • evidence, unless persistence.persist_evaluator_evidence: false
  • raw evaluator output according to persistence.persist_raw_outputs

When persistence.mode: summary, execution-level case payload snapshots and evaluator binding config are replaced with redaction markers. Dataset case blobs remain durable for retry execution and reproducibility. When persistence.persist_evaluator_evidence: false, raw evaluator output is also redacted because raw output may contain embedded evidence.

If evaluation fails:

attempt.status = failed_evaluation
execution.status = retry_scheduled (if retryable)

5. Execution Completion​

Once all evaluator results are persisted:

  • execution aggregate is computed from the matching run profile aggregation policy
  • execution is marked terminal
attempt.status = completed
execution.status = completed | failed | timed_out

6. Retry Flow​

If an execution fails but is retryable:

execution.status = retry_scheduled
execution.retry_after = now() + bounded exponential backoff

The worker finishes the current chunk pass, summarizes the chunk's execution state from the database, and releases the chunk back to pending when any cases are still waiting for retry. The chunk-ready message is delayed until the next retry window instead of being treated as a worker failure.

When the message returns, the worker reloads the chunk cases and the execution allocation query decides which cases should run:

attempt.status = pending → running

Cases whose retry_after is still in the future are skipped for that pass and keep the chunk open. Cases whose retry window is due receive the next authoritative attempt.

Older attempts may become:

attempt.status = stale

This occurs when:

  • a worker loses its lease
  • a newer attempt supersedes it

Workers renew the chunk lease and the running attempt leases while processing. Terminal execution transitions are accepted only from the current attempt owner with a live attempt lease, except for retry-budget exhaustion closures after recovery.

Retries are bounded by defaults.max_attempts. When the current attempt number has reached that limit, the worker closes the execution as failed instead of scheduling another attempt. Planned retry waits do not consume the worker-message failure retry budget; actual worker processing failures still use bounded RabbitMQ retry buckets and are quarantined or fail the chunk after exhaustion.

7. Run Finalization​

Before finalization, the coordinator recovers expired chunk leases for running runs. Recoverable chunks are reset to pending, their current running attempts are marked stale, and they receive a recovery-scoped run.chunk.ready event. Chunks that exceed the recovery limit are marked failed.

The coordinator checks:

Are all chunks terminal?

pending or leased chunks keep a run out of finalization, including chunks waiting on execution retry windows. Once chunks are terminal, finalization rolls up terminal executions and treats missing execution coverage or failed chunks as a failed gate.

If all chunks are terminal:

run.status = finalizing

The system:

  • aggregates execution results
  • computes run summary
  • determines gate_status
run.status = completed
run.gate_status = pass | fail

8. Event Publication (Outbox Pattern)​

A run-completed event is inserted into the durable outbox ledger:

outbox_events.status = pending

The database creates a matching hot delivery row in the same transaction:

outbox_delivery_queue.available_at = now()

A publisher process claims delivery rows, publishes the joined event payload to RabbitMQ, and waits for broker confirmation:

delivery row claimed -> confirmed publish -> delete delivery row
                     -> publish failure   -> retry delivery row later

The ledger row is retained for idempotency, audit, and replay. The delivery queue only contains unfinished publish work.

State Machines​

Run Lifecycle​

pending → running → finalizing → completed
                  ↘ failed
                  ↘ cancelled

Execution Lifecycle​

pending → running → completed
                  ↘ retry_scheduled → running
                  ↘ failed
                  ↘ timed_out
                  ↘ cancelled

Attempt Lifecycle​

pending → running → completed
                  ↘ failed_agent_call
                  ↘ failed_evaluation
                  ↘ timed_out
                  ↘ cancelled
                  ↘ stale

Key Design Properties​

1. Append-only evaluator results​

Evaluator findings are never updated, only inserted. This provides:

  • auditability
  • reproducibility
  • traceability

2. Separation of state vs evidence​

  • state tables (runs, executions, attempts) are mutable
  • evaluator results are immutable facts, with evidence/raw-output fields governed by the run profile persistence policy

3. Idempotent finalization​

Multiple workers may attempt to finalize a run.

The system ensures:

  • only one finalization succeeds
  • duplicate attempts are safe

4. Retry-safe execution​

Executions may have multiple attempts.

Only the most recent non-stale attempt is authoritative.

5. Reliable event delivery​

The outbox pattern ensures:

  • no lost events
  • retryable publishing
  • eventual consistency
  • a bounded hot delivery table separate from event history

Design Philosophy​

Agent Vigilo evaluates the behavior of a target system, not just a model.

An "agent" may represent:

  • a single model call
  • a prompt pipeline
  • a multi-step workflow
  • a deployed HTTP service

The evaluation system treats all targets uniformly via a versioned invocation interface.

Summary​

The system is designed to:

  • handle distributed execution safely
  • tolerate worker failure and retries
  • preserve evaluation evidence according to the configured persistence policy
  • produce deterministic, policy-driven outcomes
  • reliably signal completion to downstream systems