Runtime Limits

These settings bound database, messaging, and evaluator resource use. They can be supplied as CLI flags or environment variables. Vigilo parses subsystem settings only for commands that consume that subsystem; unrelated configuration cannot block a database-only command.

Command Scope

DATABASE_URL, database-pool settings, VIGILO_CONTROL_DATABASE_ALIAS, and output settings are root options because every current command uses the control database. MESSAGING_URL is accepted only by coordinator and worker. Wasm limits are accepted only by worker, setup, evaluator publish, and evaluator test. Command-scoped CLI flags follow their owning command, while the existing environment-variable names remain compatible.

vigilo coordinator --messaging-url amqp://broker start
vigilo worker --messaging-url amqp://broker --wasm-timeout-ms 5000 start
vigilo run create --shard-assignment-policy spread-active \
  --profile-file profile.yaml --dataset-file dataset.yaml

DATABASE_MAX_CONNECTIONS is a limit for each lazily initialized database pool, not an aggregate process limit. A process that contacts several database aliases can own one pool per alias; size the setting against the database connection budget and number of processes.

Database Failure Isolation

Coordinators and workers use one process-local circuit breaker per database alias. Repeated connection-level failures open only that alias's circuit. Durable coordinator work remains pending, and worker messages for that alias are delayed, while healthy aliases continue in the same process. After the cooldown, one half-open probe tests recovery. Explicit database and shard administration bypasses the breaker.

Runtime pools also apply bounded PostgreSQL statement_timeout and lock_timeout settings. Coordinator placement operations have an outer wall-clock deadline, so a connected target that stops making progress cannot hold the placement sweep indefinitely. Worker claim, load, heartbeat, persistence, and settlement operations use the same deadline; agent calls, Wasm execution, and broker operations do not. The statement timeout is 80% of the configured operation timeout; lock timeout is one quarter of the statement timeout, capped at 5 seconds. A single setting keeps these limits ordered.

Flag	Environment	Scope	Default	Purpose
--database-acquire-timeout-seconds	VIGILO_DATABASE_ACQUIRE_TIMEOUT_SECONDS	All commands	10	Maximum wait to acquire a PostgreSQL connection.
--database-operation-timeout-seconds	VIGILO_DATABASE_OPERATION_TIMEOUT_SECONDS	coordinator, worker, run create	30	Maximum wall-clock time for one retryable runtime database operation; derives the server statement and lock limits.
--database-circuit-breaker-enabled	VIGILO_DATABASE_CIRCUIT_BREAKER_ENABLED	coordinator, worker, run create	true	Enables process-local execution-database circuit breakers.
--database-circuit-failure-threshold	VIGILO_DATABASE_CIRCUIT_FAILURE_THRESHOLD	coordinator, worker, run create	3	Consecutive availability failures required to open an alias's circuit.
--database-circuit-initial-open-seconds	VIGILO_DATABASE_CIRCUIT_INITIAL_OPEN_SECONDS	coordinator, worker, run create	10	Initial open interval before one recovery probe is admitted.
--database-circuit-max-open-seconds	VIGILO_DATABASE_CIRCUIT_MAX_OPEN_SECONDS	coordinator, worker, run create	120	Maximum open interval after repeated failed probes.

I/O, TLS, protocol, connection-class, shutdown, target resource, pool acquisition timeout, statement timeout, and outer operation timeout failures affect the breaker. Route state, constraints, serialization failures, deadlocks, lock contention, and missing secrets do not. Cooldowns use process-local jitter, so processes do not all probe at the same instant.

Outbox claim and settlement SQL uses the server limits. The outer database deadline deliberately excludes broker publication, so slow RabbitMQ delivery does not open a database circuit. Cancellation drops owned transactions; uncommitted dispatch cursors and creation work remain durable for retry.

A worker checks the alias circuit before claiming a message. Availability timeouts and PostgreSQL contention delay only that message; they do not consume the bounded worker-failure retry budget or terminate the process. If the chunk was already claimed, redelivery is delayed past the processing lease so another worker cannot race the still-live claim. The lease token and attempt guards make replay idempotent. Failure to publish the delayed copy remains process-visible because acknowledging the original without a durable retry would lose work.

Messaging Failure Isolation

Each coordinator or worker process owns one circuit for its configured message broker. Publish, one-shot receive, and consumer-creation attempts acquire admission before contacting RabbitMQ. Consecutive connection, channel, I/O, or heartbeat failures open the circuit. Calls then fail locally until one half-open probe is admitted after the bounded, jittered cooldown.

Outbox events remain durable while the coordinator cannot publish them, and unacked worker deliveries remain available for redelivery. Long-running workers also retain their bounded 250 ms to 30 second consumer reconnect backoff; it paces the service loop while the circuit controls process-wide broker admission. Per-message retry queues and retry budgets remain independent because they govern processing outcomes, not broker availability.

Flag	Environment	Scope	Default	Purpose
--messaging-circuit-breaker-enabled	VIGILO_MESSAGING_CIRCUIT_BREAKER_ENABLED	coordinator, worker	true	Enables the process-local message-broker circuit breaker.
--messaging-circuit-failure-threshold	VIGILO_MESSAGING_CIRCUIT_FAILURE_THRESHOLD	coordinator, worker	3	Consecutive broker availability failures required to open the circuit.
--messaging-circuit-initial-open-seconds	VIGILO_MESSAGING_CIRCUIT_INITIAL_OPEN_SECONDS	coordinator, worker	10	Initial open interval before one recovery probe is admitted.
--messaging-circuit-max-open-seconds	VIGILO_MESSAGING_CIRCUIT_MAX_OPEN_SECONDS	coordinator, worker	120	Maximum open interval after repeated failed probes.

Returned or negatively acknowledged publishes, malformed messages, and other application-level failures do not mark RabbitMQ unavailable. Circuit state is process-local and resets on restart; durable PostgreSQL state and RabbitMQ delivery state remain authoritative.

Worker Chunk Concurrency

vigilo worker start processes chunk-ready broker messages with bounded per-process concurrency. The default keeps one per worker process, which matches the safest local development posture and scales horizontally by running more workers. RabbitMQ matches that limit so a worker does not reserve messages it cannot process. Larger deployments can raise the limit when chunks are uneven, agent calls are I/O-bound, or retry waits create tail latency.

Flag	Environment	Default	Purpose
--max-inflight-chunks	VIGILO_WORKER_MAX_INFLIGHT_CHUNKS	1	Maximum unacked chunk messages processed concurrently by one worker process. RabbitMQ stream prefetch is set to the same value so a worker does not reserve more chunk messages than it can actively process.

Run Creation Paging

Immediate creation and coordinator recovery use the same explicitly validated paging policy.

Flag	Environment	Scope	Default	Purpose
--run-creation-case-batch-size	VIGILO_RUN_CREATION_CASE_BATCH_SIZE	run create, coordinator	1000	Maximum cases written in one creation page.
--run-creation-case-page-budget	VIGILO_RUN_CREATION_CASE_PAGE_BUDGET	run create, coordinator	64	Maximum pages written for one placement attempt.
--default-shard-database-alias	VIGILO_DEFAULT_SHARD_DATABASE_ALIAS	setup, run create	primary	Default target for single-default assignment.
--shard-assignment-policy	VIGILO_SHARD_ASSIGNMENT_POLICY	setup, run create	single-default	Placement policy used for newly created runs.

When increasing this value, size the database pool, agent endpoint capacity, , and Wasm evaluator concurrency together. A practical upper-bound check is:

max_inflight_chunks * case_parallelism_per_chunk <= agent/db concurrency budget
wasm_max_concurrent_evaluations <= CPU and memory budget

Wasm Evaluator Isolation

Each evaluator invocation runs in a fresh with CPU, memory, table, instance, timeout, and logging limits. bounds deterministic instruction use. Workers also use a process-wide so valid evaluators cannot exhaust host resources. These flags are scoped to worker, setup, evaluator publish, and evaluator test.

Flag	Environment	Default	Purpose
--wasm-max-memory-bytes	VIGILO_WASM_MAX_MEMORY_BYTES	67108864	Maximum linear memory bytes per evaluator invocation.
--wasm-max-table-elements	VIGILO_WASM_MAX_TABLE_ELEMENTS	10000	Maximum table elements per evaluator invocation.
--wasm-max-instances	VIGILO_WASM_MAX_INSTANCES	3	Maximum component instances per evaluator invocation.
--wasm-max-memories	VIGILO_WASM_MAX_MEMORIES	1	Maximum linear memories per evaluator invocation.
--wasm-max-tables	VIGILO_WASM_MAX_TABLES	2	Maximum tables per evaluator invocation.
--wasm-fuel-per-evaluation	VIGILO_WASM_FUEL_PER_EVALUATION	50000000	Wasmtime fuel budget for deterministic CPU interruption.
--wasm-timeout-ms	VIGILO_WASM_TIMEOUT_MS	5000	Wall-clock timeout enforced by epoch interruption.
--wasm-epoch-tick-interval-ms	VIGILO_WASM_EPOCH_TICK_INTERVAL_MS	10	Epoch ticker granularity used for timeout checks.
--wasm-max-concurrent-evaluations	VIGILO_WASM_MAX_CONCURRENT_EVALUATIONS	8	Hard process-local cap on active Wasm evaluator invocations.
--wasm-max-log-message-bytes	VIGILO_WASM_MAX_LOG_MESSAGE_BYTES	4096	Maximum bytes retained from each evaluator host log message.
--wasm-max-log-messages	VIGILO_WASM_MAX_LOG_MESSAGES	128	Maximum evaluator host log messages per invocation.

Sizing

Set VIGILO_WASM_MAX_CONCURRENT_EVALUATIONS from available CPU and memory, not from queue depth. A worker's evaluator memory exposure is approximately:

wasm_max_concurrent_evaluations * wasm_max_memory_bytes

plus Wasmtime/runtime overhead and non-evaluator worker memory.

For example, the defaults allow up to 8 active evaluator invocations at 64 MiB each, or about 512 MiB of evaluator linear memory exposure per worker process.

Failure Behavior

If an evaluator exceeds its sandbox budget, the invocation is recorded as an evaluator runtime error. The worker continues processing the rest of the batch and persists the error result through the normal evaluator-result path.

Outbound evaluator HTTP remains denied unless a future policy explicitly enables it.