104 lines
4.7 KiB
Markdown
104 lines
4.7 KiB
Markdown
|
# T6.5 — Sandboxed replay
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | P6 — Learning loop |
|
|||
|
|
| Size | L — over 3 days |
|
|||
|
|
| Status | Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | — |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Re-execute a task under a new workflow version with `Unsafe` effects denied.
|
|||
|
|
Depends on T2.2's effect classes.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
- **"Replay against recorded episodes" means re-running the task under a new
|
|||
|
|
workflow version**, not pushing a recorded trajectory through new logic.
|
|||
|
|
Trajectory replay tells you only where behaviour would first diverge, and
|
|||
|
|
everything after divergence is unknown — near-worthless for grading.
|
|||
|
|
- **Replay is not free.** "0% live traffic" means no user sees the result, not
|
|||
|
|
that it costs nothing. N variants × M tasks is N·M full agent runs plus judge
|
|||
|
|
calls. **Shadow is the most expensive rung, not the cheapest.**
|
|||
|
|
- **Replay executes real tools.** Re-running a workflow that pushes commits
|
|||
|
|
pushes commits. Shadow execution runs in a sandbox with `Unsafe` effects
|
|||
|
|
denied, and a workflow whose steps cannot run sandboxed is **ineligible for
|
|||
|
|
shadow evaluation and must say so at load time rather than at 3am**.
|
|||
|
|
- Sandbox is layer 3 of the effect-class defence: declaration (T2.2), keyed
|
|||
|
|
capability (T8.3), sandbox here.
|
|||
|
|
- Replay needs its own spend ceiling, separate from judging.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Implement the sandbox as a capability restriction over the tool registry:
|
|||
|
|
`Unsafe` tools are not resolvable inside a replay context. Denial is
|
|||
|
|
structural, not a runtime `if`.
|
|||
|
|
2. Add the **load-time** eligibility check: walk the workflow's steps, resolve
|
|||
|
|
each `ToolId`, and reject the workflow for shadow evaluation if any resolves
|
|||
|
|
to `Unsafe`. The diagnostic **names the tool**.
|
|||
|
|
3. Replay execution path: fresh runs against the recorded task input (`TaskId`),
|
|||
|
|
under the challenger version, inside the sandbox.
|
|||
|
|
4. Record replay runs with a `Purpose` distinguishing them from live work, so
|
|||
|
|
metering (T8.1) separates them and their own ceiling applies.
|
|||
|
|
5. Feed the resulting episodes to the tournament strategy — the group exists
|
|||
|
|
because you paid for it, so pairwise grading would waste it.
|
|||
|
|
6. Wire the replay ceiling: exceeding it stops replay and reports, rather than
|
|||
|
|
silently truncating the variant set.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- A workflow calling an unsafe tool is **rejected for shadow evaluation at load
|
|||
|
|
time**, with a diagnostic naming the tool.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** the external side-effect ledger from T2.2, plus a workflow that
|
|||
|
|
calls an `Unsafe` tool and one that does not.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_sandboxed_replay.rs`:
|
|||
|
|
1. Submit the `Unsafe`-calling workflow for shadow evaluation. Assert it is
|
|||
|
|
**rejected at load time**, with a diagnostic **naming the tool**.
|
|||
|
|
2. Assert the rejection happens **before** any run is spawned — check the run
|
|||
|
|
counter is 0 and the ledger is empty.
|
|||
|
|
3. Positive control: the safe workflow is accepted and replays successfully.
|
|||
|
|
4. **Defence in depth:** bypass the load check in a test-only path and attempt an
|
|||
|
|
`Unsafe` call inside the sandbox anyway. Assert it is denied at call time too.
|
|||
|
|
The load check is the good error; the sandbox is the guarantee.
|
|||
|
|
5. Assert replay re-executes against the recorded **task input**, not a recorded
|
|||
|
|
trajectory — plant a divergence and assert execution continues past it and
|
|||
|
|
produces a complete episode.
|
|||
|
|
6. Assert replay runs are recorded with a distinct `Purpose`, and that they count
|
|||
|
|
against the **replay** ceiling, not the judging one.
|
|||
|
|
7. Cost visibility: assert N variants × M tasks produces N·M runs, and that the
|
|||
|
|
count is reported — shadow is the most expensive rung and must not look free.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p loop sandboxed_replay`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Step 1 alone. A load-time check with no runtime enforcement passes it, and any
|
|||
|
|
code path that skips validation then executes real effects. Step 4 is the
|
|||
|
|
guarantee.
|
|||
|
|
- Step 5 omitted: trajectory replay produces an episode that looks complete and
|
|||
|
|
is meaningless after the first divergence, and no assertion about tool denial
|
|||
|
|
catches it.
|
|||
|
|
- Charging replay to the judging ceiling — step 6. It passes every functional
|
|||
|
|
test and consumes the entire grading budget in production.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Trajectory replay. It is cheaper, it looks like replay, and its output cannot
|
|||
|
|
be graded past the first divergence.
|
|||
|
|
- Denying `Unsafe` at call time instead of load time. The rejection then arrives
|
|||
|
|
mid-run, at 3am, after real work has been done.
|
|||
|
|
- Charging replay against the judging ceiling. Replay dominates and will consume
|
|||
|
|
the whole budget.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background (not required to do this task):
|
|||
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §12.4, §13.2, §14.1 ·
|
|||
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|