Run Profile Configuration

A defines how evaluation is performed across task types and .

Use this profile with:

vigilo run validate --profile-file <profile.yaml> --dataset-file <dataset.yaml>

Result Calculation

evaluator outcome -> binding normalization and judgment -> evaluator completeness -> dimension_scores -> aggregate_score -> execution status -> run gate_status

are per-dimension scores. is one total score for a single execution across all scored dimensions.

The owns the dimension, normalization method, pass threshold, requiredness, weight, and blocking behavior. Evaluators return measurements and diagnostics; they do not participate in release policy.

Dimension scores are calculated with:

weighted_mean: weighted average of completed binding scores in that profile dimension.
min_score: lowest completed binding score in that profile dimension.

Before calculating either score, Vigilo checks . Every required evaluator binding must return exactly one valid measurement that the host can normalize. Missing, duplicate, errored, abstained, or invalid output is incomplete. Incomplete required output produces overall_status: error and withholds both dimension_scores and aggregate_score, preventing the remaining weights from being renormalized into a misleading score.

aggregate_score is the weighted average of dimension_scores using dimension weights. defaults.min_execution_score is an overall score gate for that total; it is not a per-dimension threshold. Dimensions with weight: 0.0 do not affect aggregate_score, but can still block when blocking: true.

An execution passes when:

aggregate_score >= defaults.min_execution_score
evaluator completeness is satisfied
no host-derived blocking result fails
at least one scoreable binding result exists

A run passes when every expected execution has an aggregate, no chunk failed or was cancelled, and no execution failed or errored. Check both runs.status for operational lifecycle and for policy outcome; a terminal run is not necessarily a passing run.

File Format

The CLI accepts YAML or JSON.

Recommended filename:

profile.yaml

Top-level shape:

profile_id: mixed_agent_release
profile_version: 1.0.0
description: Release-grade run profile for mixed generative AI agent tasks.
defaults: {}
persistence: {}
agent:
  provider: llama.cpp
  name: qwen2.5-0.5b-instruct
  model: qwen2.5-0.5b-instruct-q4_k_m.gguf
  http:
    url: http://agent_vigilo_agent:8080/v1/chat/completions
  config:
    request_format: openai_compatible_chat_completions
case_groups: []

Top-Level Fields

profile_id (string): stable identifier for this profile.
profile_version (string): profile version, independent of evaluator versions.
description (string): human-readable profile summary.
defaults (object): run-level behavior defaults.
persistence (object): persistence policy controls.
scorecard (object): run-level dimension and evaluator gates.
agent (object): required worker-side target invocation configuration.
case_groups (array): per-task evaluator and aggregation definitions.

`defaults`

defaults:
  max_attempts: 2
  request_timeout_secs: 60
  fail_on_any_blocking_failure: true
  min_execution_score: 0.85

max_attempts (u32): hard limit for case execution attempts. Failed attempts below this limit are retried with bounded backoff; once the limit is reached, the execution is failed.
request_timeout_secs (u32)
fail_on_any_blocking_failure (bool): when true, any host-derived blocking failure fails the execution even if the total aggregate_score is above the threshold. Errors and abstentions from required bindings always make evaluator completeness fail.
min_execution_score (f64): finite value from 0.0 to 1.0. This is the overall score gate for the total execution aggregate_score; it does not set per-dimension thresholds.

`persistence`

persistence:
  mode: full
  persist_raw_outputs: failures_only
  persist_evaluator_evidence: true

mode (enum): full | summary. full stores execution-level case snapshots and evaluator binding config on each execution. summary redacts payloads, metadata, and evaluator config but retains tags referenced by scorecard gates.
persist_raw_outputs (enum): all | failures_only | none. Controls evaluator_results.raw_evaluator_output; failures_only retains host-failed and errored invocation output.
persist_evaluator_evidence (bool): controls evaluator_diagnostics.evidence. When false, diagnostic evidence and raw evaluator output are redacted.

These settings affect execution/result artifact retention. Dataset case blobs and dataset-version membership remain durable so workers can execute retries and runs remain reproducible by content hash. The run profile snapshot is also retained because workers load it to invoke the configured agent and evaluators.

Do not put long-lived secrets directly in agent.http.headers for shared databases. Header values are part of the run profile snapshot used at runtime. Prefer local development values, short-lived tokens, or a future secret-reference integration.

`agent`

The agent block identifies the target under evaluation and provides the HTTP endpoint workers call before running evaluators.

agent:
  provider: llama.cpp
  name: qwen2.5-0.5b-instruct
  version: 1.0.0
  model: qwen2.5-0.5b-instruct-q4_k_m.gguf
  prompt_config_id: qwen-sentiment-json
  prompt_config_version: 1.0.0
  http:
    method: POST
    url: http://agent_vigilo_agent:8080/v1/chat/completions
    timeout_secs: 120
  config:
    request_format: openai_compatible_chat_completions
    temperature: 0.0
    max_tokens: 96
    response_format:
      type: json_object
    system_prompt: |
      Return only JSON with this exact shape: {"label":"<label>"}.

provider (string): provider or platform for the evaluated target.
name (string): logical agent name.
version (string, optional): deployment or release version.
model (string, optional): provider-specific model identifier.
prompt_config_id (string, optional): prompt/config identity persisted with the run.
prompt_config_version (string, optional): prompt/config version persisted with the run.
http (object): required worker invocation endpoint.
config (object, optional): unstructured agent configuration included in invocation metadata. Set request_format: openai_compatible_chat_completions for OpenAI-compatible servers such as llama.cpp.

By default, workers send a Vigilo case request containing run_id, execution_id, attempt_id, agent, input, and non-oracle case metadata. With request_format: openai_compatible_chat_completions, workers send an OpenAI-compatible model and messages payload.

The response may be either plain text or JSON. JSON responses can return the evaluator-ready output directly, or under actual/output:

{
  "actual": {
    "text": "This seems positive.",
    "structured": { "label": "positive" },
    "tool_calls": [],
    "trace": [],
    "metadata": { "latency_ms": 42 }
  }
}

`case_groups`

Each case group controls evaluator selection and aggregation for matching cases.

case_groups:
  - id: classification
    description: Evaluates classification-style cases.
    applies_to:
      task_type: classification
      tags_any: [safety]
      tags_all: []
    evaluators: []
    aggregation:
      dimensions: {}

`applies_to`

task_type (string, required)
tags_any (string[], optional)
tags_all (string[], optional)

Selection behavior:

If a dataset case supplies case_group, Vigilo selects the case group whose id exactly matches that value. The applies_to task/tag rules are bypassed for that case.
If a dataset case omits case_group, Vigilo evaluates applies_to: task_type must match, tags_any requires at least one matching tag when present, and tags_all requires every listed tag.
An unknown explicit case_group is an error; it does not fall back to automatic matching.

`evaluators[]`

evaluators:
  - id: response_schema
    ref: core/json-schema:1.0.0
    required: true
    dimension: format
    blocking: true
    weight: 1.0
    normalization:
      method: binary
      false_score: 0.0
      true_score: 1.0
    pass_threshold: 1.0
    config:
      schema:
        type: object

id (string): stable binding identifier using ASCII letters, numbers, ., _, or -. It is the invocation idempotency key and permits the same evaluator artifact to be bound more than once with different policy or config.
ref (string): fully qualified evaluator identifier <namespace>/<name>:<version>.
required (bool, default true): requires this binding to complete with a valid measurement. false declares a diagnostic-only binding and requires blocking: false with weight: 0.0.
dimension (string): profile-owned aggregation bucket.
blocking (bool): makes a host-derived failed judgment a hard blocking result.
weight (f64): finite value greater than or equal to 0.0; the binding's contribution inside its dimension.
normalization (object): required host interpretation for a raw binary, numeric, or ordinal measurement. Numeric policies support linear, piecewise-linear, and threshold mappings. See Measurement Normalization for every supported shape and copyable examples.
pass_threshold (f64): required finite value in 0.0..=1.0. The host derives passed when normalized_score >= pass_threshold; otherwise it derives failed.
config (object, optional)

`aggregation.dimensions`

aggregation:
  dimensions:
    format:
      method: min_score
      blocking: true
      weight: 0.0
    correctness:
      method: weighted_mean
      blocking: false
      weight: 1.0

method (enum): min_score | weighted_mean
blocking (bool): makes failed binding judgments in this dimension blocking.
weight (f64): finite value greater than or equal to 0.0. Dimension weights combine dimension_scores into the total execution aggregate_score; 0.0 means the dimension does not affect the total score, but blocking results in that dimension can still fail the execution.

A common pattern is a hard format gate plus a scored quality dimension:

aggregation:
  dimensions:
    format:
      method: min_score
      blocking: true
      weight: 0.0
    quality:
      method: weighted_mean
      blocking: false
      weight: 1.0

Here, format can fail the execution but does not change aggregate_score; quality determines the total score in this minimal profile.

Aggregation behavior:

Completed binding results are grouped by the profile binding dimension.
weighted_mean computes a weighted arithmetic mean inside each dimension using evaluator binding weights.
min_score uses the lowest normalized binding score in that dimension.
dimension_scores are per-dimension scores.
aggregate_score is the total score for one execution across all scored dimensions.
aggregate_score combines dimension scores using aggregation.dimensions.<dimension>.weight.
Missing dimension policies default to weighted_mean, non-blocking, weight 1.0.
Completed measurements receive a host-derived passed or failed judgment and are scoreable.
A required binding that is missing, duplicated, errored, abstained, or invalid makes evaluator completeness fail. Vigilo records the reason and publishes no dimension or aggregate scores for that execution.
Optional bindings are diagnostic only. Their results and diagnostics remain available for reporting but never affect scores or gates.
If an execution has no complete required score set, its aggregate status is error.
If multiple case groups match the same case, their dimension policies and duplicate evaluator bindings must be identical or non-overlapping; conflicting policies are rejected.

`scorecard.gates`

apply after all executions finish. They enforce run-wide rules without loading every execution into the control database.

scorecard:
  gates:
    - id: safety_release
      dimension: safety
      tags_all: [jailbreak]
      min_mean_score: 0.99
      score_threshold: 1.0
      min_pass_rate: 1.0
      min_coverage: 1.0
      max_error_rate: 0.0
      max_abstention_rate: 0.0

id: unique stable gate id. A profile supports at most 64 gates.
dimension: required dimension to score.
binding_id: optional required evaluator binding target. Without it, the gate uses the execution's dimension score.
case_group: optional exact case-group filter.
tags_all: optional list of tags every matching case must have.
min_mean_score: minimum mean score across scored matching executions.
score_threshold and min_pass_rate: configured together; require a fraction of scored executions to meet the threshold.
min_coverage: minimum fraction of matching executions with a score.
max_error_rate and max_abstention_rate: maximum fractions over matching executions.

A gate with no matching executions fails. Any failed scorecard gate fails the run even when every execution passed its overall score gate. vigilo run results and run exports include the final scorecard and failed rules.

Evaluation Plan Resolution

For every dataset case, Vigilo resolves an before invoking the agent target. The plan determines:

the profile case group or groups applied to the case
the evaluator bindings to run
the evaluator-specific config passed into each evaluator
the profile-owned dimensions and binding policies used for persisted results
the aggregation policy used for the execution aggregate

Resolution order:

If cases[].case_group is supplied, Vigilo selects case_groups[].id with that exact value.
If the supplied case_group does not exist, the case is invalid for this profile and Vigilo does not fall back to applies_to.
If cases[].case_group is omitted, Vigilo evaluates every profile case_groups[].applies_to block.
Automatic matching requires task_type equality.
When tags_any is present, at least one listed tag must appear on the case.
When tags_all is present, every listed tag must appear on the case.
All automatically matching groups contribute evaluator bindings and aggregation settings.
Duplicate evaluator refs and duplicate dimension policies across matching groups must be identical; conflicting policies are rejected.

The resolved plan is snapshotted on the execution through evaluator_manifest, expected_evaluator_count, and profile_group_id. The manifest retains each binding's required policy even in summary persistence mode. Dataset case_group is also retained in evaluator input so evaluators can inspect the original routing label.

Full Example

See the repository example profile:

example/profile.yaml

Dataset format
Glossary
Repository example: example/README.md

Result Calculation​

File Format​

Top-Level Fields​

defaults​

persistence​

agent​

case_groups​

applies_to​

evaluators[]​

aggregation.dimensions​

scorecard.gates​

Evaluation Plan Resolution​

Full Example​

Related​