Skip to main content

Getting Started

Agent Vigilo turns generative AI evaluation into release infrastructure.

Instead of a one-off script that prints a score, Vigilo gives you a system for publishing versioned WASM evaluators, creating durable evaluation runs, processing work through coordinator and worker services, and using pass/fail gates before an agent change ships.

Why It Is Different​

  • Evaluators are versioned artifacts. Publish reusable WASM evaluators and refer to them by stable identifiers like vigilo/sentiment-basic-en:0.1.0.
  • Runs are durable. Profiles, datasets, chunks, attempts, evaluator results, summaries, and gate outcomes are stored in Postgres.
  • Execution is distributed. Coordinators dispatch work through RabbitMQ; workers claim chunks, call the target agent, run evaluators, and persist normalized results.
  • The CLI is automation-friendly. Commands return structured output, support quiet mode, and expose watch/status/results/export flows for CI and agentic clients.

The Mental Model​

profile + dataset
      |
      v
vigilo run create
      |
      v
coordinator -> queue -> worker -> target agent -> WASM evaluators
      |
      v
run outcome: completed/pass, completed/fail, running, cancelled, or failed

A profile describes the agent, request format, evaluator set, aggregation, and gate rules. A dataset supplies the cases. Vigilo turns those files into a durable run that can be watched, queried, exported, and used as a deployment gate.

How Results Are Calculated​

Vigilo calculates results in this order:

finding score -> dimension_scores -> aggregate_score -> execution status -> run gate_status
  • Evaluators emit findings with a status, score, optional blocking flag, and source dimension.
  • The run profile decides which dimension each evaluator contributes to.
  • dimension_scores are per-dimension scores.
  • aggregate_score is one total score for the execution, calculated across scored dimensions.
  • defaults.min_execution_score is an overall score gate for aggregate_score, not a per-dimension threshold.
  • An execution passes when aggregate_score >= defaults.min_execution_score and no hard blocking finding fails or errors.
  • A run passes when every expected execution has an aggregate, no chunk failed or was cancelled, and no execution failed or errored.

A finding is blocking if the evaluator finding, evaluator binding, or dimension policy says blocking: true. Use blocking for hard requirements that should not be averaged away. defaults.fail_on_any_blocking_failure controls whether blocking failed findings hard-fail the execution; blocking errors always error the execution.

Where Failure Happens​

case attempt -> execution -> run -> gate
  • Attempts can fail because the agent call, evaluator runtime, persistence, or retry budget failed.
  • Executions can fail because the total aggregate_score is below the overall score gate or a hard blocking finding failed.
  • Runs can fail operationally because chunks failed, coverage is incomplete, aggregates are missing, or executions errored.
  • Gates fail when the run completes but gate_status is fail.

Prerequisites​

You need:

  • Rust and Cargo
  • The wasm32-wasip2 Rust target
  • Docker Compose
  • A local checkout of the Agent Vigilo repository
  • A local model file if you want to run the bundled llama.cpp example end to end

The bundled example expects this model path:

models/qwen2.5-0.5b-instruct-q4_k_m.gguf

Install the WASM target if you do not already have it:

rustup target add wasm32-wasip2

Start The Local Stack​

From the repository root:

docker compose --env-file infra/dev/.env.single \
  -f infra/dev/docker-compose.single.yml \
  up -d postgres rabbitmq

Set the local connection strings:

export DATABASE_URL='postgresql://postgres:password@localhost:5432/agent_vigilo'
export MESSAGING_URL='amqp://guest:guest@localhost:5672'

On PowerShell:

$env:DATABASE_URL='postgresql://postgres:password@localhost:5432/agent_vigilo'
$env:MESSAGING_URL='amqp://guest:guest@localhost:5672'

For sharding tests, start the overlay instead:

docker compose --env-file infra/dev/.env.sharded \
  -f infra/dev/docker-compose.single.yml \
  -f infra/dev/docker-compose.sharded.yml \
  up -d postgres postgres-shard-001 rabbitmq

Host-side tests use the published ports:

export DATABASE_URL='postgresql://postgres:password@localhost:5432/agent_vigilo'
export VIGILO_TEST_SHARD_001_DATABASE_URL='postgresql://postgres:password@localhost:5433/agent_vigilo'
export MESSAGING_URL='amqp://guest:guest@localhost:5672'

Inside Compose containers, services use Docker DNS names such as postgres, postgres-shard-001, and rabbitmq; see infra/dev/README.md.

Apply migrations and publish the bundled evaluator:

cargo run -p vigilo -- setup

The setup command is safe to rerun. It applies pending migrations, builds bundled evaluators for release, and publishes them into the local registry when they are not already present.

Validate The Example​

Before creating durable work, verify the example profile and dataset are executable:

cargo run -p vigilo -- run test \
  --profile-file example/profile.yaml \
  --dataset-file example/dataset.yaml

This checks that the profile, dataset, evaluator references, and runnable evaluator set line up.

Run The System​

Start the full dev stack, including the local agent, coordinator, and worker:

docker compose --env-file infra/dev/.env.single \
  -f infra/dev/docker-compose.single.yml \
  up -d

For a full sharded runtime stack:

docker compose --env-file infra/dev/.env.sharded \
  -f infra/dev/docker-compose.single.yml \
  -f infra/dev/docker-compose.sharded.yml \
  up -d

Create a run:

cargo run -p vigilo -- run create \
  --profile-file example/profile.yaml \
  --dataset-file example/dataset.yaml

Copy the returned run_id, then watch it:

cargo run -p vigilo -- run watch <run_id>

When the run reaches a terminal status, inspect the summary:

cargo run -p vigilo -- run results <run_id>

Export detailed artifacts when you need evidence for debugging, release notes, or CI archives:

cargo run -p vigilo -- run export <run_id> --format jsonl

Use It As A Gate​

For CI and release automation, the important command is run watch.

By default, it exits successfully only when the run reaches an accepted terminal outcome. A failed run or failed gate becomes a failed command, which makes it natural to wire into deployment jobs:

cargo run -p vigilo -- run create \
  --profile-file example/profile.yaml \
  --dataset-file example/dataset.yaml

cargo run -p vigilo -- run watch <run_id> --timeout-seconds 900

Use structured output for automation:

cargo run -p vigilo -- -q -f json run status <run_id>

Build Your Own Evaluator​

The fastest path to custom evaluation is:

  1. Start from the bundled evaluator shape in evaluators/sentiment-basic-en.
  2. Implement the WIT input to output contract.
  3. Build to wasm32-wasip2.
  4. Publish the evaluator.
  5. Reference it from a run profile.

Next guide: Creating Evaluators.

Explore The Architecture​

If you want to understand how the system works before extending it: