104 lines
4.7 KiB
Markdown
104 lines
4.7 KiB
Markdown
# T6.5 — Sandboxed replay
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | P6 — Learning loop |
|
||
| Size | L — over 3 days |
|
||
| Status | Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | — |
|
||
|
||
## Goal
|
||
|
||
Re-execute a task under a new workflow version with `Unsafe` effects denied.
|
||
Depends on T2.2's effect classes.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
- **"Replay against recorded episodes" means re-running the task under a new
|
||
workflow version**, not pushing a recorded trajectory through new logic.
|
||
Trajectory replay tells you only where behaviour would first diverge, and
|
||
everything after divergence is unknown — near-worthless for grading.
|
||
- **Replay is not free.** "0% live traffic" means no user sees the result, not
|
||
that it costs nothing. N variants × M tasks is N·M full agent runs plus judge
|
||
calls. **Shadow is the most expensive rung, not the cheapest.**
|
||
- **Replay executes real tools.** Re-running a workflow that pushes commits
|
||
pushes commits. Shadow execution runs in a sandbox with `Unsafe` effects
|
||
denied, and a workflow whose steps cannot run sandboxed is **ineligible for
|
||
shadow evaluation and must say so at load time rather than at 3am**.
|
||
- Sandbox is layer 3 of the effect-class defence: declaration (T2.2), keyed
|
||
capability (T8.3), sandbox here.
|
||
- Replay needs its own spend ceiling, separate from judging.
|
||
|
||
## Steps
|
||
|
||
1. Implement the sandbox as a capability restriction over the tool registry:
|
||
`Unsafe` tools are not resolvable inside a replay context. Denial is
|
||
structural, not a runtime `if`.
|
||
2. Add the **load-time** eligibility check: walk the workflow's steps, resolve
|
||
each `ToolId`, and reject the workflow for shadow evaluation if any resolves
|
||
to `Unsafe`. The diagnostic **names the tool**.
|
||
3. Replay execution path: fresh runs against the recorded task input (`TaskId`),
|
||
under the challenger version, inside the sandbox.
|
||
4. Record replay runs with a `Purpose` distinguishing them from live work, so
|
||
metering (T8.1) separates them and their own ceiling applies.
|
||
5. Feed the resulting episodes to the tournament strategy — the group exists
|
||
because you paid for it, so pairwise grading would waste it.
|
||
6. Wire the replay ceiling: exceeding it stops replay and reports, rather than
|
||
silently truncating the variant set.
|
||
|
||
## Acceptance
|
||
|
||
- A workflow calling an unsafe tool is **rejected for shadow evaluation at load
|
||
time**, with a diagnostic naming the tool.
|
||
|
||
## Verify
|
||
|
||
**Harness:** the external side-effect ledger from T2.2, plus a workflow that
|
||
calls an `Unsafe` tool and one that does not.
|
||
|
||
**Integration test** — `tests/it_sandboxed_replay.rs`:
|
||
1. Submit the `Unsafe`-calling workflow for shadow evaluation. Assert it is
|
||
**rejected at load time**, with a diagnostic **naming the tool**.
|
||
2. Assert the rejection happens **before** any run is spawned — check the run
|
||
counter is 0 and the ledger is empty.
|
||
3. Positive control: the safe workflow is accepted and replays successfully.
|
||
4. **Defence in depth:** bypass the load check in a test-only path and attempt an
|
||
`Unsafe` call inside the sandbox anyway. Assert it is denied at call time too.
|
||
The load check is the good error; the sandbox is the guarantee.
|
||
5. Assert replay re-executes against the recorded **task input**, not a recorded
|
||
trajectory — plant a divergence and assert execution continues past it and
|
||
produces a complete episode.
|
||
6. Assert replay runs are recorded with a distinct `Purpose`, and that they count
|
||
against the **replay** ceiling, not the judging one.
|
||
7. Cost visibility: assert N variants × M tasks produces N·M runs, and that the
|
||
count is reported — shadow is the most expensive rung and must not look free.
|
||
|
||
**Command:** `cargo test -p loop sandboxed_replay`
|
||
|
||
**False pass:**
|
||
- Step 1 alone. A load-time check with no runtime enforcement passes it, and any
|
||
code path that skips validation then executes real effects. Step 4 is the
|
||
guarantee.
|
||
- Step 5 omitted: trajectory replay produces an episode that looks complete and
|
||
is meaningless after the first divergence, and no assertion about tool denial
|
||
catches it.
|
||
- Charging replay to the judging ceiling — step 6. It passes every functional
|
||
test and consumes the entire grading budget in production.
|
||
|
||
## Traps
|
||
|
||
- Trajectory replay. It is cheaper, it looks like replay, and its output cannot
|
||
be graded past the first divergence.
|
||
- Denying `Unsafe` at call time instead of load time. The rejection then arrives
|
||
mid-run, at 3am, after real work has been done.
|
||
- Charging replay against the judging ceiling. Replay dominates and will consume
|
||
the whole budget.
|
||
|
||
---
|
||
|
||
Background (not required to do this task):
|
||
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §12.4, §13.2, §14.1 ·
|
||
[rust-agentic-task.md](../../../rust-agentic-task.md)
|