The Ceiling That Resets When a Sandbox Dies: Qualifying Agent Hypervisors Under Load

An execution ceiling that holds on an idle hypervisor proves very little. The failure that matters only shows up when a sandbox dies while its siblings are saturating the scheduler. When that happens the orchestrator re-dispatches the interrupted step, and if the budget counter lived with the dead worker, the agent comes back with a fresh allowance. Qualification has to provoke exactly that sequence at production concurrency, then check what an independent ledger says actually happened.

Where the budget actually lives

A deterministic multi-agent hypervisor meters each agent's tool calls, tokens and wall time against a ceiling, and on paper that meter belongs to the control plane while the WebAssembly sandbox only holds a lease on it. To get throughput, real implementations often cache the remaining budget inside the worker and reconcile it asynchronously. That works until the worker is killed between a spend and the reconcile, and the replacement reads the last reconciled value, which is stale and too generous. At light load the reconcile window is so short that a kill almost never lands in it. At saturation, reconciliation queues behind the same starved scheduler, the window widens, and kills start landing inside it routinely. The Model Context Protocol tool server supplies the other half of the failure: the call that was in flight may already have committed its side effect, so the retry commits it a second time.

The test: kill on spend, at saturation

Run the hypervisor at its rated agent concurrency with a synthetic workload. Give every agent a deliberately tight ceiling and have it call an instrumented MCP tool server that writes each invocation, with its idempotency key, to an append-only ledger outside the system under test. Once the scheduler queue reaches steady state, a fault injector subscribes to the hypervisor's trace stream and kills the sandbox host after a tool call has been dispatched but before its budget acknowledgement returns. The kill has to be timed this way: random kills mostly land in idle gaps, and the run passes without ever testing the race. Repeat the campaign with three different kill targets, namely the sandbox, the MCP transport, and the control plane replica holding the lease, because each one exposes a different owner of the same counter. The verdict comes only from the external ledger: no agent's committed count exceeds its ceiling, no idempotency key shows two committed effects, and every killed agent either resumes inside its lease or is terminated with a recorded reason, with throughput and latency reported alongside but never deciding the outcome.

The parameter that decides it

Make the sandbox lease heartbeat timeout the axis of the sweep, starting from a typical 500 ms. A timeout shorter than the control plane's reconcile latency at saturation produces split-brain ownership of an agent, a longer one stalls every agent stranded on a dead host, and the only defensible value is the one your saturated run measures, not the one an idle benchmark suggests.

Reading the verdict

Treat the ceiling as a safety function in the sense of the functional safety standards descended from IEC 61508. Those standards accept fault injection as evidence of diagnostic coverage and do not accept a low observed rate as evidence of absence. The decision rule follows from that: if any idempotency key commits twice, or any ledger count crosses its ceiling in any run, the build fails qualification no matter how rarely it happened, and a rare event is a reason to run more campaigns. As an estimate, a few thousand targeted kills per configuration is roughly where a reconcile race that really exists starts showing up reliably. Fixing it by moving the counter entirely into the control plane closes the race but adds a synchronous round trip to every tool call. The alternative is to redefine the ceiling formally as the limit plus one in-flight call and qualify against that weaker promise in writing, and choosing between the two is a product decision that test results alone can't settle.

Can deterministic replay replace the saturated run?

No. Replay is excellent for reproducing a race once you have found it, but it serializes scheduling, and that removes the widened reconcile window the saturated run exists to expose. Use replay as the regression test for a known fix, and keep the saturated kill-on-spend campaign for finding races in the first place.

Back to AgenticHypervisor · Domains

← Back to AgenticHypervisor
Engineering & Architecture Brief

The Ceiling That Resets When a Sandbox Dies: Qualifying Agent Hypervisors Under Load

An execution ceiling that holds on an idle hypervisor proves very little. The failure that matters only shows up when a sandbox dies while its siblings are saturating the scheduler. When that happens the orchestrator re-dispatches the interrupted step, and if the budget counter lived with the dead worker, the agent comes back with a fresh allowance. Qualification has to provoke exactly that sequence at production concurrency, then check what an independent ledger says actually happened.

Where the budget actually lives

A deterministic multi-agent hypervisor meters each agent's tool calls, tokens and wall time against a ceiling, and on paper that meter belongs to the control plane while the WebAssembly sandbox only holds a lease on it. To get throughput, real implementations often cache the remaining budget inside the worker and reconcile it asynchronously. That works until the worker is killed between a spend and the reconcile, and the replacement reads the last reconciled value, which is stale and too generous. At light load the reconcile window is so short that a kill almost never lands in it. At saturation, reconciliation queues behind the same starved scheduler, the window widens, and kills start landing inside it routinely. The Model Context Protocol tool server supplies the other half of the failure: the call that was in flight may already have committed its side effect, so the retry commits it a second time.

The test: kill on spend, at saturation

Run the hypervisor at its rated agent concurrency with a synthetic workload. Give every agent a deliberately tight ceiling and have it call an instrumented MCP tool server that writes each invocation, with its idempotency key, to an append-only ledger outside the system under test. Once the scheduler queue reaches steady state, a fault injector subscribes to the hypervisor's trace stream and kills the sandbox host after a tool call has been dispatched but before its budget acknowledgement returns. The kill has to be timed this way: random kills mostly land in idle gaps, and the run passes without ever testing the race. Repeat the campaign with three different kill targets, namely the sandbox, the MCP transport, and the control plane replica holding the lease, because each one exposes a different owner of the same counter. The verdict comes only from the external ledger: no agent's committed count exceeds its ceiling, no idempotency key shows two committed effects, and every killed agent either resumes inside its lease or is terminated with a recorded reason, with throughput and latency reported alongside but never deciding the outcome.

The parameter that decides it

Make the sandbox lease heartbeat timeout the axis of the sweep, starting from a typical 500 ms. A timeout shorter than the control plane's reconcile latency at saturation produces split-brain ownership of an agent, a longer one stalls every agent stranded on a dead host, and the only defensible value is the one your saturated run measures, not the one an idle benchmark suggests.

Reading the verdict

Treat the ceiling as a safety function in the sense of the functional safety standards descended from IEC 61508. Those standards accept fault injection as evidence of diagnostic coverage and do not accept a low observed rate as evidence of absence. The decision rule follows from that: if any idempotency key commits twice, or any ledger count crosses its ceiling in any run, the build fails qualification no matter how rarely it happened, and a rare event is a reason to run more campaigns. As an estimate, a few thousand targeted kills per configuration is roughly where a reconcile race that really exists starts showing up reliably. Fixing it by moving the counter entirely into the control plane closes the race but adds a synchronous round trip to every tool call. The alternative is to redefine the ceiling formally as the limit plus one in-flight call and qualify against that weaker promise in writing, and choosing between the two is a product decision that test results alone can't settle.

Technical FAQ: Can deterministic replay replace the saturated run?

No. Replay is excellent for reproducing a race once you have found it, but it serializes scheduling, and that removes the widened reconcile window the saturated run exists to expose. Use replay as the regression test for a known fix, and keep the saturated kill-on-spend campaign for finding races in the first place.

Related Architecture Specifications (AGENTIC): orchestrationagentic.com · agenticsupervisory.com · maintenanceagentic.com

More Technical Articles on agentichypervisor.com

Benchmarking WebAssembly vs MicroVM Execution Latency in Agentic Hypervisors →Formal Verification of Constitutional AI Guardrail Invariants under Adversarial Prompts →