poimen/tasks/T6.5-sandboxed-replay.md

T6.5 — Sandboxed replay

Field	Value
Phase	P6 — Learning loop
Size	L — over 3 days
Status	Not started
Flags	—
Spec	inlined below
Blocks	—

Goal

Re-execute a task under a new workflow version with Unsafe effects denied. Depends on T2.2's effect classes.

Facts (inlined — no spec read needed)

• "Replay against recorded episodes" means re-running the task under a new workflow version, not pushing a recorded trajectory through new logic. Trajectory replay tells you only where behaviour would first diverge, and everything after divergence is unknown — near-worthless for grading.
• Replay is not free. "0% live traffic" means no user sees the result, not that it costs nothing. N variants × M tasks is N·M full agent runs plus judge calls. Shadow is the most expensive rung, not the cheapest.
• Replay executes real tools. Re-running a workflow that pushes commits pushes commits. Shadow execution runs in a sandbox with Unsafe effects denied, and a workflow whose steps cannot run sandboxed is ineligible for shadow evaluation and must say so at load time rather than at 3am.
• Sandbox is layer 3 of the effect-class defence: declaration (T2.2), keyed capability (T8.3), sandbox here.
• Replay needs its own spend ceiling, separate from judging.

Steps

1. Implement the sandbox as a capability restriction over the tool registry: Unsafe tools are not resolvable inside a replay context. Denial is structural, not a runtime if.
2. Add the load-time eligibility check: walk the workflow's steps, resolve each ToolId, and reject the workflow for shadow evaluation if any resolves to Unsafe. The diagnostic names the tool.
3. Replay execution path: fresh runs against the recorded task input (TaskId), under the challenger version, inside the sandbox.
4. Record replay runs with a Purpose distinguishing them from live work, so metering (T8.1) separates them and their own ceiling applies.
5. Feed the resulting episodes to the tournament strategy — the group exists because you paid for it, so pairwise grading would waste it.
6. Wire the replay ceiling: exceeding it stops replay and reports, rather than silently truncating the variant set.

Acceptance

• A workflow calling an unsafe tool is rejected for shadow evaluation at load time, with a diagnostic naming the tool.

Verify

Harness: the external side-effect ledger from T2.2, plus a workflow that calls an Unsafe tool and one that does not.

Integration test — tests/it_sandboxed_replay.rs:

1. Submit the Unsafe-calling workflow for shadow evaluation. Assert it is rejected at load time, with a diagnostic naming the tool.
2. Assert the rejection happens before any run is spawned — check the run counter is 0 and the ledger is empty.
3. Positive control: the safe workflow is accepted and replays successfully.
4. Defence in depth: bypass the load check in a test-only path and attempt an Unsafe call inside the sandbox anyway. Assert it is denied at call time too. The load check is the good error; the sandbox is the guarantee.
5. Assert replay re-executes against the recorded task input, not a recorded trajectory — plant a divergence and assert execution continues past it and produces a complete episode.
6. Assert replay runs are recorded with a distinct Purpose, and that they count against the replay ceiling, not the judging one.
7. Cost visibility: assert N variants × M tasks produces N·M runs, and that the count is reported — shadow is the most expensive rung and must not look free.

Command: cargo test -p loop sandboxed_replay

False pass:

• Step 1 alone. A load-time check with no runtime enforcement passes it, and any code path that skips validation then executes real effects. Step 4 is the guarantee.
• Step 5 omitted: trajectory replay produces an episode that looks complete and is meaningless after the first divergence, and no assertion about tool denial catches it.
• Charging replay to the judging ceiling — step 6. It passes every functional test and consumes the entire grading budget in production.

Traps

• Trajectory replay. It is cheaper, it looks like replay, and its output cannot be graded past the first divergence.
• Denying Unsafe at call time instead of load time. The rejection then arrives mid-run, at 3am, after real work has been done.
• Charging replay against the judging ceiling. Replay dominates and will consume the whole budget.

---

Background (not required to do this task): rust-agentic-sys.md §12.4, §13.2, §14.1 · rust-agentic-task.md