Skip to main content

Creating Evaluators

This guide is the recommended path for creating a single crate that can be built to Wasm and executed by Agent Vigilo.

What You Build​

A minimal evaluator crate should:

  • implement the evaluator from the canonical contract
  • expose one evaluator entrypoint (evaluate)
  • accept the canonical evaluator input
  • return the canonical evaluator output
  • build as a cdylib for

Use evaluators/sentiment-basic-en as the concrete reference implementation.

A practical layout for a single evaluator crate:

my-evaluator/
  Cargo.toml
  Vigilo.toml
  src/
    lib.rs
  example-input.json

Key files:

  • Cargo.toml: crate identity (name, version), cdylib output, and dependencies
  • Vigilo.toml: build artifact profile paths, WIT contract reference, and publish metadata
  • src/lib.rs: evaluator implementation and mapping logic
  • example-input.json: canonical input payload for local test runs

The reference evaluator uses wit-bindgen, imports evaluator-world, implements Guest for Evaluator, and exports the implementation with export!(Evaluator).

Contract-First Implementation​

Implement the current contract at wit/evaluator/v1.0.0/evaluator.wit.

  • Read fields from the , including run IDs, the test case, captured , and evaluator config.
  • Return exactly one primary in the as output.outcome.completed, or explicitly return output.outcome.abstained.
  • Emit zero or more in output.diagnostics. Diagnostics provide evidence but cannot choose dimensions, thresholds, weights, blocking behavior, or pass/fail policy.
  • Return evaluator-error when execution fails. Errors and abstentions from a required binding make the execution incomplete and withhold authoritative scores.
  • Use evaluator metadata in output.evaluator
  • Keep invocation metadata in output.metadata_json

The first-release contract is vigilo:evaluator@1.0.0. Rebuild and republish evaluators when their implementation changes. Once released, a breaking WIT change receives a new ABI version; existing evaluator binaries remain assigned to their original host adapter. See Evaluator Compatibility.

Evaluators can return raw binary, numeric, or ordinal measurements. See Measurement Normalization for copyable evaluator returns and the matching profile policy for every type.

Build and Test Loop​

Build your evaluator for WASI 0.2 (Preview 2):

cargo build --manifest-path evaluators/my-evaluator/Cargo.toml --target wasm32-wasip2 --release

Publish the release artifact:

vigilo evaluator publish ./evaluators/my-evaluator --release

Run a focused evaluator test against the published version using the sample input:

vigilo evaluator test 'vigilo/my-evaluator:0.1.0' --input-file evaluators/my-evaluator/example-input.json

The quoted value is the published .

If publication reports an ABI error, confirm the manifest points to the same frozen WIT version used to build the component.

Publish​

After validating behavior locally:

vigilo evaluator publish ./evaluators/my-evaluator --release

For complete publish workflow details, see Publishing Evaluators.

Using Codex​

Codex works best when you keep prompts constrained to the project contract and crate shape.

Recommended prompt constraints:

  • "Create one Rust evaluator crate with one evaluator entrypoint."
  • "Follow wit/evaluator/v1.0.0/evaluator.wit input/output contract exactly."
  • "Use evaluators/sentiment-basic-en as the style reference."
  • "Add example-input.json and include build/test commands."

Recommended validation checklist:

  1. Confirm the generated evaluator builds for wasm32-wasip2.
  2. Confirm vigilo evaluator publish ... --release publishes or skips the expected version.
  3. Confirm vigilo evaluator test ... --input-file ... succeeds.
  4. Confirm identifiers and output fields use the canonical naming (input/output).
  5. Bump evaluator version and republish after any contract-shape or behavior change.

Troubleshooting​

  • Unsupported evaluator ABI: choose a version listed in Evaluator Compatibility.
  • WIT metadata mismatch: the manifest declaration and referenced WIT file disagree.
  • configured wasm artifact was not found: build the selected profile first, or fix the profile path in Vigilo.toml.
  • evaluation manifest was modified after wasm build: rebuild the evaluator after changing Cargo.toml.
  • Existing evaluator with a different content hash: bump the evaluator version before publishing changed content.
  • No evaluator trace/debug output: run with appropriate log level filters so debug logs are visible.
  • Evaluator not found: use a fully qualified identifier: <namespace>/<name>:<version>.