Skip to main content

Scaling

Agent Vigilo starts with one PostgreSQL database and a routing model that can grow beyond it. The default deployment stores control data and execution data in the same placement. As evaluation volume grows, execution-owned rows can move to additional PostgreSQL placements without changing coordinator or worker code.

The design principle is simple: control metadata decides where work lives; workers and coordinators follow the stored route.

Default Posture​

Most teams can start with one database:

DATABASE_URL=postgres://...
VIGILO_CONTROL_DATABASE_ALIAS=primary
VIGILO_DEFAULT_SHARD_DATABASE_ALIAS=primary

vigilo setup applies migrations and seeds:

alias = primary
database_url_env = DATABASE_URL
role = control_and_shard
status = active

All logical shards route to primary until another shard-capable placement is added and selected for new or moved run shards.

Control And Execution Storage​

Agent Vigilo separates database responsibility from database location.

  • Control storage owns evaluator registry rows, global run metadata, placement catalog rows, recoverable run-creation plans, dispatch cursors, coordinator leases, and global run lifecycle updates.
  • Execution storage owns high-volume run work: chunks, run snapshots, executions, attempts, aggregates, evaluator results, shard summaries, and chunk-ready outbox events.
  • Database placements name PostgreSQL targets such as primary or shard_001.
  • Shard placements map one run_id + run_shard pair to a database placement.

There is one active control-capable placement. Additional placements can have role shard and store execution data. Environment variables hold connection secrets; the placement catalog in PostgreSQL is the routing authority.

Placement status and role are checked in control storage before Vigilo admits new database work. Disabling a placement therefore blocks new pool lookups even when that process already has a pool for the alias. Work that already acquired the pool may finish.

Connection parameters are process-lifetime configuration. Adding an alias, changing database_url_env, changing its URL value, or rotating credentials requires the catalog row and secret to be present before the relevant Vigilo processes start or restart. Runtime metadata checks do not distribute or reload database secrets.

Logical Shards​

The system uses 128 stable logical shards:

run_shard = 0..127

Run chunks receive a run_shard during run creation. Execution-owned rows keep that same key:

run_chunks(run_id, run_shard, id)
executions(run_id, run_shard, id)
execution_attempts(run_id, run_shard, id)
execution_aggregates(run_id, run_shard, execution_id)
evaluator_results(run_id, run_shard, id)
run_snapshots(run_id, run_shard)
run_shard_summaries(run_id, run_shard)

High-volume execution tables are list partitioned by run_shard. The key is stable for the lifetime of a chunk and its child rows. Adding database capacity does not recalculate run_shard; it changes stored placement rows.

Routing Lifecycle​

Run creation persists a recoverable control plan before it writes to any remote execution placement:

  • Control storage receives the canonical dataset metadata, a non-dispatchable creating run, shard placements, one creation-ledger row per selected database, and the exact temporary chunk plan.
  • The shard assignment policy chooses the execution placement for each used run_shard and persists the result in shard_placements.
  • Each selected execution placement receives local prerequisites and only its pending chunks in one idempotent transaction. A committed seed can therefore be applied again safely if the process fails before control storage records that placement as seeded.
  • After every placement is seeded, one control transaction creates dispatch cursors, changes the run to pending, and removes the temporary chunk plan. No coordinator or worker can dispatch a partially seeded run.
  • Before normal lease recovery and dispatch, coordinators claim expired creating runs and continue their remaining placement work. Retryable failures remain recoverable; incompatible persisted data fails the run without creating dispatch cursors.
  • The default policy is explicit and conservative: VIGILO_SHARD_ASSIGNMENT_POLICY=single-default assigns all new run shards to VIGILO_DEFAULT_SHARD_DATABASE_ALIAS.
  • VIGILO_SHARD_ASSIGNMENT_POLICY=spread-active spreads one run's used shards across active shard-capable placements.

Coordinator dispatch is route-constrained:

  • Control storage selects one open run_id + run_shard dispatch cursor.
  • The database service resolves that shard placement to an execution pool.
  • The execution placement upserts run_snapshots, marks pending chunks dispatched, and writes run.chunk.ready outbox events.
  • The control cursor is released or marked drained.
  • If an execution placement operation fails, the cursor remains open and that database alias is excluded from the rest of the current dispatch pass. Healthy aliases continue within the same cycle.

Workers stay chunk-local:

  • Worker messages carry run_id, run_shard, and chunk_id.
  • The worker resolves the execution route once, then claims chunks, loads the local run snapshot and case data, persists attempts/results/aggregates, and refreshes the shard summary in that execution placement.
  • Evaluator registry metadata remains a control-plane read.

Finalization fans in:

  • Each execution placement maintains run_shard_summaries.
  • The coordinator reads routed summaries for the run.
  • Control storage writes the global run outcome and emits run.completed.
  • A failed placement read defers that run without discarding summaries already read from healthy placements. The run cannot finalize until every routed summary is available and terminal.

Cancellation fans out:

  • Control storage marks the run cancelled and drains dispatch cursors first.
  • The workflow resolves the run's execution placements and cancels local chunks, executions, and attempts in each placement with bounded fanout.
  • run.cancelled remains a control-plane event.

Status and watch reads fan in lightweight progress:

  • The control run row remains the authoritative lifecycle record.
  • While a run is creating, reads report placement counts, attempts, and the latest creation error from control storage without contacting unseeded execution databases.
  • Live progress is read from routed run_shard_summaries when available.
  • Payloads include data.live_progress and progress-source metadata so callers can distinguish control-only state from execution-shard progress.

Outbox publication is local to the database where the state changed. Control events publish from the control outbox. Execution events publish from the execution placement that owns the changed rows. Lease recovery and outbox publication iterate placements independently; a failed alias is logged and skipped while healthy aliases finish the pass.

Control-Plane Scale​

The control database stays singular and authoritative. It is expected to scale vertically first, while execution rows spread across shard-capable placements. Coordinator hot paths emit structured log fields that can be treated as metrics: dispatch cursor backlog, finalization candidate backlog, per-placement outbox backlog, oldest finalization candidate lag, route resolution timing, pool acquisition timing, and stage timings for recovery, dispatch, finalization, and outbox publication. Each completed cycle also reports unique skipped placements, failed placement operations, and retryable versus terminal placement errors. Contained failure logs include the database alias, operation, error kind, and retryability.

The hot outbox publisher scan is indexed by availability time. Worker runtime metadata reads remain centralized for now; evaluator metadata replication is an explicit future contract, not hidden routing behavior.

Operating Placements​

Add a shard-capable placement by deploying the secret env var and registering the placement:

VIGILO_SHARD_001_DATABASE_URL=postgres://...
vigilo shard databases add shard_001 --database-url-env VIGILO_SHARD_001_DATABASE_URL

Set the default for future run shards:

VIGILO_DEFAULT_SHARD_DATABASE_ALIAS=shard_001

Choose how new run shards are assigned:

VIGILO_SHARD_ASSIGNMENT_POLICY=single-default
VIGILO_SHARD_ASSIGNMENT_POLICY=spread-active

single-default is the local/small-deployment default. spread-active deterministically assigns used logical shards across active shard-capable placements and stores each assignment in shard_placements.

Inspect routing metadata:

vigilo shard databases list
vigilo shard placements list <run_id>
vigilo shard placements show <run_id> <run_shard>
vigilo shard route <run_id> <run_shard>

vigilo shard route reports the database alias, placement statuses, route_version, database_url_env, env-var availability, and whether the route is dispatchable, read_only, or blocked. It never prints the database URL.

Moving A Run Shard​

Use shard move for a specific run_id + run_shard:

vigilo shard move <run_id> <run_shard> --alias <alias> --dry-run
vigilo shard move <run_id> <run_shard> --alias <alias>
vigilo shard move <run_id> <run_shard> --alias <alias> --verify-only

Routes are immutable while their run is creating; move and direct placement changes are rejected until creation reaches pending or failed. Rebalance plans omit creating runs for the same reason.

The normal move marks the shard placement moving, rejects active leased or running work unless --force is passed, copies shard-owned rows, verifies row counts and checksums, then switches the route to the target alias with status active.

Every route alias or status change increments shard_placements.route_version. Execution route caches are accepted only while alias, status, and route_version still match control metadata, so stale cached active routes are fenced before new dispatch or worker claims continue.

Moved shard-owned tables:

run_chunks
run_snapshots
executions
execution_attempts
execution_aggregates
evaluator_results
run_shard_summaries

The move also copies local FK prerequisites needed by the target placement. Those prerequisites are:

case_blobs
dataset_versions
dataset_version_cases
runs

run_shard_dispatch_cursors stays in control storage and is never copied between execution placements. Source cleanup is intentionally outside the move command so teams can apply their own backup and retention policy.

Bulk Rebalance​

Use rebalance operations when adding capacity or draining a placement:

vigilo shard rebalance plan --to shard_001 --max-items 100
vigilo shard rebalance plan --from primary --to shard_001 --max-items 100
vigilo shard rebalance apply <operation_id> --max-items 25
vigilo shard rebalance verify <operation_id> --max-items 25
vigilo shard rebalance cancel <operation_id>

A rebalance plan records one item per run_id + run_shard in control storage. Apply is resumable and bounded; each item still uses the single-shard move workflow, including route fencing, copy verification, and active-work guards. Plans verify the stored route_version before moving an item so stale plans fail narrowly instead of rewriting an unexpected route.

Current Scope​

Implemented today:

  • Single-database default with a seeded primary placement.
  • 128 logical shards and partitioned execution tables.
  • Stored database and shard placement catalogs.
  • Recoverable routed run creation, dispatch, worker processing, lease recovery, outbox publication, shard summaries, finalization, cancellation, live status/watch progress, results, export, route inspection, route fencing, placement assignment policy, explicit shard move, and persisted bulk rebalance operations.
  • Control-plane scale hardening through outbox queue indexing and structured backlog/timing logs for coordinator and routing hot paths.
  • Live placement admission checks that reject disabled targets and role changes without discarding reusable connection pools.
  • Placement-isolated coordinator recovery, dispatch, finalization reads, and outbox publication with per-cycle failure classification and counters.
  • CI coverage for the black-box and runtime multi-database paths: run creation to primary and shard_001, recovery after a remote seed commit, route inspection, control-owned cursors, coordinator dispatch, worker execution, finalization, routed cancellation, spread-active assignment for one run, shard move, rebalance plan/apply resume/verify, route-version inspection, and prerequisite-copy verification. The runtime harness also verifies that an unavailable shard does not stop healthy dispatch, lease recovery, or outbox publication.

Intentionally deferred:

  • Evaluator metadata replication to execution placements.
  • Broader canonical case-data routing and replication.
  • Runtime database-secret and connection-parameter refresh.
  • Leased rebalance apply claims and fair multi-candidate finalization scanning.

These are product and operations contracts, not hidden formulas. Until they are designed explicitly, routing remains stored, inspectable, and conservative.