Skip to main content

Getting Started

Agent Vigilo turns generative AI evaluation into release infrastructure.

Instead of a one-off script that prints a score, Vigilo gives you a system for publishing versioned Wasm evaluators, creating durable evaluation runs, processing work through coordinator and worker services, and using pass/fail gates before an agent change ships.

Why It Is Different​

  • Evaluators are versioned artifacts. Publish reusable and refer to them by stable identifiers like vigilo/sentiment-basic-en:0.1.0.
  • Runs are durable. Profiles, datasets, chunks, attempts, evaluator results, summaries, and gate outcomes are stored in Postgres.
  • Execution is distributed. Coordinators create chunk-ready outbox event records that are published through RabbitMQ; workers claim the identified chunks, call the agent target, run evaluators, and persist normalized results.
  • The CLI is automation-friendly. Commands return structured output, support quiet mode, and expose watch/status/results/export flows for CI and agentic clients.

The Mental Model​

profile + dataset
      |
      v
vigilo run create
      |
      v
coordinator -> queue -> worker -> agent target -> Wasm evaluators
      |
      v
run outcome: completed/pass, completed/fail, running, cancelled, or failed

A describes the agent, request format, evaluator set, aggregation, and gate rules. A supplies the cases. Vigilo turns those files into a durable that can be watched, queried, exported, and used as a deployment gate.

How Results Are Calculated​

Vigilo calculates results in this order:

evaluator measurement -> host binding policy -> dimension_scores -> execution status -> shard scorecards -> run scorecard -> gate_status
  • Evaluators return one or abstention plus zero or more non-authoritative diagnostics.
  • The run profile owns normalization, pass threshold, dimension, requiredness, weight, and blocking behavior for each stable binding.
  • Every required binding must produce exactly one valid measurement before any dimension or aggregate score is published.
  • dimension_scores are per-dimension scores.
  • aggregate_score is one total score for the execution, calculated across scored dimensions.
  • defaults.min_execution_score is an overall score gate for aggregate_score, not a per-dimension threshold.
  • An passes when evaluator completeness is satisfied, aggregate_score >= defaults.min_execution_score, and no hard blocking host-derived blocking result fails.
  • A run passes when execution coverage is complete, no chunk or execution failed, and every configured run scorecard gate passes.

A result is blocking only when its evaluator binding or dimension policy says blocking: true. Evaluator diagnostics cannot block. Missing, errored, abstained, duplicated, or invalid required output fails evaluator completeness regardless of blocking policy.

Where Failure Happens​

case attempt -> execution -> run -> gate
  • can fail because the agent call, evaluator runtime, persistence, or retry budget failed.
  • Executions can fail because evaluator completeness is not satisfied, the total aggregate_score is below the overall score gate, or a host-derived blocking result failed.
  • Runs can fail operationally because chunks failed, coverage is incomplete, aggregates are missing, or executions errored.
  • Gates fail when the run completes but its is fail.

Prerequisites​

You need:

  • Rust and Cargo
  • The wasm32-wasip2 Rust target
  • Docker Compose
  • A local checkout of the Agent Vigilo repository
  • A local model file if you want to run the bundled llama.cpp example end to end

The bundled example expects this model path:

models/qwen2.5-0.5b-instruct-q4_k_m.gguf

Install the Wasm target if you do not already have it:

rustup target add wasm32-wasip2

Start The Local Stack​

From the repository root:

docker compose --env-file infra/dev/.env.single \
  -f infra/dev/docker-compose.single.yml \
  up -d postgres rabbitmq

Set the local connection strings:

export DATABASE_URL='postgresql://postgres:password@localhost:5432/agent_vigilo'
export MESSAGING_URL='amqp://guest:guest@localhost:5672'

On PowerShell:

$env:DATABASE_URL='postgresql://postgres:password@localhost:5432/agent_vigilo'
$env:MESSAGING_URL='amqp://guest:guest@localhost:5672'

For sharding tests, start the overlay instead:

docker compose --env-file infra/dev/.env.sharded \
  -f infra/dev/docker-compose.single.yml \
  -f infra/dev/docker-compose.sharded.yml \
  up -d postgres postgres-shard-001 rabbitmq

Host-side tests use the published ports:

export DATABASE_URL='postgresql://postgres:password@localhost:5432/agent_vigilo'
export VIGILO_TEST_SHARD_001_DATABASE_URL='postgresql://postgres:password@localhost:5433/agent_vigilo'
export MESSAGING_URL='amqp://guest:guest@localhost:5672'

Inside Compose containers, services use Docker DNS names such as postgres, postgres-shard-001, and rabbitmq; see infra/dev/README.md.

Apply migrations and publish the bundled evaluator:

cargo run -p vigilo -- setup

The setup command is safe to rerun. It applies pending migrations, builds bundled evaluators for release, and publishes them into the local registry when they are not already present.

Validate The Example​

Before creating durable work, verify the example profile and dataset are executable:

cargo run -p vigilo -- run validate \
  --profile-file example/profile.yaml \
  --dataset-file example/dataset.yaml

This checks that the profile, dataset, evaluator references, and runnable evaluator set line up.

Run The System​

Start the full dev stack, including the local agent, coordinator, and worker:

docker compose --env-file infra/dev/.env.single \
  -f infra/dev/docker-compose.single.yml \
  up -d

For a full sharded runtime stack:

docker compose --env-file infra/dev/.env.sharded \
  -f infra/dev/docker-compose.single.yml \
  -f infra/dev/docker-compose.sharded.yml \
  up -d

Create a run:

cargo run -p vigilo -- run create \
  --profile-file example/profile.yaml \
  --dataset-file example/dataset.yaml

Copy the returned run_id, then watch it:

cargo run -p vigilo -- run watch <run_id>

When the run reaches a terminal status, inspect the summary:

cargo run -p vigilo -- run results <run_id>

Export detailed artifacts when you need evidence for debugging, release notes, or CI archives:

cargo run -p vigilo -- run export <run_id> --format jsonl

Use It As A Gate​

For CI and release automation, the important command is run watch.

By default, it exits successfully only when the run reaches an accepted terminal outcome. A failed run or failed gate becomes a failed command, which makes it natural to wire into deployment jobs:

cargo run -p vigilo -- run create \
  --profile-file example/profile.yaml \
  --dataset-file example/dataset.yaml

cargo run -p vigilo -- run watch <run_id> --timeout-seconds 900

Use structured output for automation:

cargo run -p vigilo -- -q -f json run status <run_id>

Build Your Own Evaluator​

The fastest path to custom evaluation is:

  1. Start from the bundled evaluator shape in evaluators/sentiment-basic-en.
  2. Implement the WIT input to output contract.
  3. Build to wasm32-wasip2.
  4. Publish the evaluator.
  5. Reference it from a run profile.

Next guide: Creating Evaluators.

Explore The Architecture​

If you want to understand how the system works before extending it: