Skip to main content

Run Profile Configuration

A run profile defines how evaluation should be performed across task types and case groups.

Use this profile with:

vigilo run test --profile-file <profile.yaml> --dataset-file <dataset.yaml>

Result Calculation​

finding score -> dimension_scores -> aggregate_score -> execution status -> run gate_status

dimension_scores are per-dimension scores. aggregate_score is one total score for a single execution across all scored dimensions.

The dimension on an evaluator binding is the profile-owned aggregation bucket. Evaluator-emitted dimensions are preserved as source metadata, but they do not choose the scoring bucket.

Dimension scores are calculated with:

  • weighted_mean: weighted average of scoreable findings in that profile dimension.
  • min_score: lowest scoreable finding in that profile dimension.

aggregate_score is the weighted average of dimension_scores using dimension weights. defaults.min_execution_score is an overall score gate for that total; it is not a per-dimension threshold. Dimensions with weight: 0.0 do not affect aggregate_score, but can still block when blocking: true.

An execution passes when:

  • aggregate_score >= defaults.min_execution_score
  • no hard blocking finding fails or errors
  • at least one scoreable finding exists

A run passes when every expected execution has an aggregate, no chunk failed or was cancelled, and no execution failed or errored. Check both runs.status for operational lifecycle and runs.gate_status for policy outcome; a terminal run is not necessarily a passing run.

File Format​

The CLI accepts YAML or JSON.

Recommended filename:

  • profile.yaml

Top-level shape:

profile_id: mixed_agent_release
profile_version: 1.0.0
description: Release-grade evaluation profile for mixed generative AI agent tasks.
defaults: {}
persistence: {}
agent:
  provider: llama.cpp
  name: qwen2.5-0.5b-instruct
  model: qwen2.5-0.5b-instruct-q4_k_m.gguf
  http:
    url: http://agent_vigilo_agent:8080/v1/chat/completions
  config:
    request_format: openai_compatible_chat_completions
case_groups: []

Top-Level Fields​

  • profile_id (string): stable identifier for this profile.
  • profile_version (string): profile version, independent of evaluator versions.
  • description (string): human-readable profile summary.
  • defaults (object): run-level behavior defaults.
  • persistence (object): persistence policy controls.
  • agent (object): required worker-side target invocation configuration.
  • case_groups (array): per-task evaluator and aggregation definitions.

defaults​

defaults:
  max_attempts: 2
  request_timeout_secs: 60
  fail_on_any_blocking_failure: true
  min_execution_score: 0.85
  • max_attempts (u32): hard limit for case execution attempts. Failed attempts below this limit are retried with bounded backoff; once the limit is reached, the execution is failed.
  • request_timeout_secs (u32)
  • fail_on_any_blocking_failure (bool): when true, any blocking failed finding fails the execution even if the total aggregate_score is above the threshold. When false, blocking failures are still recorded in blocking_failures, but the overall score gate determines pass/fail. Blocking errors always error the execution.
  • min_execution_score (f64): finite value from 0.0 to 1.0. This is the overall score gate for the total execution aggregate_score; it does not set per-dimension thresholds.

persistence​

persistence:
  mode: full
  persist_raw_outputs: failures_only
  persist_evaluator_evidence: true
  • mode (enum): full | summary. full stores execution-level case snapshots and evaluator binding config on each execution. summary stores only summary-safe execution identity/routing fields and redacts execution input_payload, expected_output, case_metadata, tags, and evaluator binding config in the execution snapshot.
  • persist_raw_outputs (enum): all | failures_only | none. Controls evaluator_results.raw_evaluator_output. all retains every evaluator raw output, failures_only retains raw output only for failed or errored findings, and none stores a redaction marker for every finding.
  • persist_evaluator_evidence (bool): controls evaluator_results.evidence. When false, finding evidence is replaced with a redaction marker. Raw evaluator output is also redacted because it can contain the same evidence.

These settings affect execution/result artifact retention. Dataset case blobs and dataset-version membership remain durable so workers can execute retries and runs remain reproducible by content hash. The run profile snapshot is also retained because workers load it to invoke the configured agent and evaluators.

Do not put long-lived secrets directly in agent.http.headers for shared databases. Header values are part of the run profile snapshot used at runtime. Prefer local development values, short-lived tokens, or a future secret-reference integration.

agent​

The agent block identifies the target under evaluation and provides the HTTP endpoint workers call before running evaluators.

agent:
  provider: llama.cpp
  name: qwen2.5-0.5b-instruct
  version: 1.0.0
  model: qwen2.5-0.5b-instruct-q4_k_m.gguf
  prompt_config_id: qwen-sentiment-json
  prompt_config_version: 1.0.0
  http:
    method: POST
    url: http://agent_vigilo_agent:8080/v1/chat/completions
    timeout_secs: 120
  config:
    request_format: openai_compatible_chat_completions
    temperature: 0.0
    max_tokens: 96
    response_format:
      type: json_object
    system_prompt: |
      Return only JSON with this exact shape: {"label":"<label>"}.
  • provider (string): provider or platform for the evaluated target.
  • name (string): logical agent name.
  • version (string, optional): deployment or release version.
  • model (string, optional): provider-specific model identifier.
  • prompt_config_id (string, optional): prompt/config identity persisted with the run.
  • prompt_config_version (string, optional): prompt/config version persisted with the run.
  • http (object): required worker invocation endpoint.
  • config (object, optional): unstructured agent configuration included in invocation metadata. Set request_format: openai_compatible_chat_completions for OpenAI-compatible servers such as llama.cpp.

By default, workers send a Vigilo case request containing run_id, execution_id, attempt_id, agent, input, and non-oracle case metadata. With request_format: openai_compatible_chat_completions, workers send an OpenAI-compatible model and messages payload.

The response may be either plain text or JSON. JSON responses can return the evaluator-ready output directly, or under actual/output:

{
  "actual": {
    "text": "This seems positive.",
    "structured": { "label": "positive" },
    "tool_calls": [],
    "trace": [],
    "metadata": { "latency_ms": 42 }
  }
}

case_groups​

Each case group controls evaluator selection and aggregation for matching cases.

case_groups:
  - id: classification
    description: Evaluates classification-style cases.
    applies_to:
      task_type: classification
      tags_any: [safety]
      tags_all: []
    evaluators: []
    aggregation:
      dimensions: {}

applies_to​

  • task_type (string, required)
  • tags_any (string[], optional)
  • tags_all (string[], optional)

Selection behavior:

  • If a dataset case supplies case_group, Vigilo selects the case group whose id exactly matches that value. The applies_to task/tag rules are bypassed for that case.
  • If a dataset case omits case_group, Vigilo evaluates applies_to: task_type must match, tags_any requires at least one matching tag when present, and tags_all requires every listed tag.
  • An unknown explicit case_group is an error; it does not fall back to automatic matching.

evaluators[]​

evaluators:
  - ref: core/json-schema:1.0.0
    dimension: format
    blocking: true
    weight: 1.0
    config:
      schema:
        type: object
  • ref (string): fully qualified evaluator identifier <namespace>/<name>:<version>.
  • dimension (string): profile-owned aggregation bucket for this evaluator binding. This value controls the persisted result dimension and the aggregation.dimensions policy used for scoring. Evaluator-emitted dimensions remain available in raw evaluator output for audit, but they do not override the profile bucket.
  • blocking (bool): marks failed/error findings from this binding as blocking. The dimension policy and the evaluator finding may also declare blocking behavior; any of those sources makes the normalized finding blocking.
  • weight (f64): finite value greater than or equal to 0.0. For weighted_mean, this is the binding's total contribution inside its dimension. When an evaluator emits multiple scoreable findings, the binding weight is divided across those findings so multi-finding evaluators do not become overweighted.
  • config (object, optional)

aggregation.dimensions​

aggregation:
  dimensions:
    format:
      method: min_score
      blocking: true
      weight: 0.0
    correctness:
      method: weighted_mean
      blocking: false
      weight: 1.0
  • method (enum): min_score | weighted_mean
  • blocking (bool): marks failed/error findings in this dimension as blocking, even when the individual evaluator binding is non-blocking.
  • weight (f64): finite value greater than or equal to 0.0. Dimension weights combine dimension_scores into the total execution aggregate_score; 0.0 means the dimension does not affect the total score, but blocking findings in that dimension can still fail the execution.

A common pattern is a hard format gate plus a scored quality dimension:

aggregation:
  dimensions:
    format:
      method: min_score
      blocking: true
      weight: 0.0
    quality:
      method: weighted_mean
      blocking: false
      weight: 1.0

Here, format can fail the execution but does not change aggregate_score; quality determines the total score in this minimal profile.

Aggregation behavior:

  • Findings are grouped by the profile binding dimension, not by the evaluator-emitted finding dimension.
  • weighted_mean computes a weighted arithmetic mean inside each dimension using evaluator binding weights.
  • min_score uses the lowest normalized score among scoreable findings in that dimension.
  • dimension_scores are per-dimension scores.
  • aggregate_score is the total score for one execution across all scored dimensions.
  • aggregate_score combines dimension scores using aggregation.dimensions.<dimension>.weight.
  • Missing dimension policies default to weighted_mean, non-blocking, weight 1.0.
  • passed and failed findings with normalized scores are scoreable. skipped and informational findings do not affect score.
  • If an execution has no scoreable findings, its aggregate status is error.
  • If multiple case groups match the same case, their dimension policies and duplicate evaluator bindings must be identical or non-overlapping; conflicting policies are rejected.

Evaluation Plan Resolution​

For every dataset case, Vigilo resolves an evaluation plan before invoking the target agent. The plan determines:

  • the profile case group or groups applied to the case
  • the evaluator bindings to run
  • the evaluator-specific config passed into each evaluator
  • the profile-owned dimensions used for persisted findings
  • the aggregation policy used for the execution aggregate

Resolution order:

  1. If cases[].case_group is supplied, Vigilo selects case_groups[].id with that exact value.
  2. If the supplied case_group does not exist, the case is invalid for this profile and Vigilo does not fall back to applies_to.
  3. If cases[].case_group is omitted, Vigilo evaluates every profile case_groups[].applies_to block.
  4. Automatic matching requires task_type equality.
  5. When tags_any is present, at least one listed tag must appear on the case.
  6. When tags_all is present, every listed tag must appear on the case.
  7. All automatically matching groups contribute evaluator bindings and aggregation settings.
  8. Duplicate evaluator refs and duplicate dimension policies across matching groups must be identical; conflicting policies are rejected.

The resolved plan is snapshotted on the execution through evaluator_manifest, expected_evaluator_count, and profile_group_id. Dataset case_group is also retained in evaluator input so evaluators can inspect the original routing label.

Full Example​

See the repository example profile:

  • example/profile.yaml
  • Dataset format: web/docs/configuration/dataset-format.mdx
  • Example project: example/README.md