106 lines
4.7 KiB
Markdown
106 lines
4.7 KiB
Markdown
|
# T6.6 — Held-out split
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | P6 — Learning loop |
|
|||
|
|
| Size | M — 1 to 3 days |
|
|||
|
|
| Status | Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | — |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Partition tasks into selection and held-out sets. Promote on selection, report
|
|||
|
|
held-out without optimizing against it.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
- Selecting on a fixed set of recorded tasks overfits to those tasks, **silently**:
|
|||
|
|
shadow scores improve while live performance does not.
|
|||
|
|
- **No promotion gate takes held-out as an entry criterion.** A gate that reads
|
|||
|
|
held-out has converted it into a second selection set and left nothing
|
|||
|
|
measuring generalization.
|
|||
|
|
- **The proposer reads the selection set only.** If the slow loop consumes
|
|||
|
|
held-out failures to generate candidates, the held-out set is contaminated
|
|||
|
|
through the generator instead of the selector. **This leak is easy to introduce
|
|||
|
|
and invisible once present** — which is why the audit test is the control, not
|
|||
|
|
a nicety.
|
|||
|
|
- A widening selection-versus-held-out gap is the overfitting alarm.
|
|||
|
|
- Under one-state convergence there is no competing variant whose divergence
|
|||
|
|
would reveal a stalled loop, so **the held-out report is also the only stall
|
|||
|
|
detector there is**. It must be produced and surfaced even when no promotion is
|
|||
|
|
pending.
|
|||
|
|
- Held-out catches overfitting to *episodes*. It does not catch drift in the
|
|||
|
|
*task mix* — that is T6.7.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Assign each `TaskId` to selection or held-out by a deterministic hash of the
|
|||
|
|
id, so the partition is stable and needs no stored membership list.
|
|||
|
|
2. Expose two separate query surfaces — `selection_tasks()` and
|
|||
|
|
`heldout_tasks()` — rather than one with a flag. A flag defaults wrong.
|
|||
|
|
3. Give the proposer access to the selection surface only, at the type level if
|
|||
|
|
possible.
|
|||
|
|
4. Compute the held-out report on a **schedule**, independent of promotion
|
|||
|
|
activity, and surface the selection-versus-held-out gap as a metric.
|
|||
|
|
5. Write the audit test: enumerate every proposer input and every gate
|
|||
|
|
evaluation, assert no held-out `TaskId` appears in either.
|
|||
|
|
6. Keep the audit test running in CI permanently. It is the only thing that will
|
|||
|
|
notice the leak.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- An audit test asserting **no held-out `TaskId` appears in proposer input**, and
|
|||
|
|
none in any promotion-gate evaluation. This leak is invisible once present, so
|
|||
|
|
the test is the control.
|
|||
|
|
- The held-out report is **emitted on a schedule, not only at a gate** — asserted,
|
|||
|
|
since under one-state convergence it is the only stall detector there is.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** an interception layer recording **every** `TaskId` that reaches the
|
|||
|
|
proposer and every `TaskId` read during a gate evaluation. The audit is the
|
|||
|
|
deliverable — it is the only thing that will ever notice this leak.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_heldout_audit.rs`:
|
|||
|
|
1. Partition a corpus of 1000 `TaskId`s. Assert the split is deterministic:
|
|||
|
|
recompute it in a second process and assert identical membership.
|
|||
|
|
2. Run a full generate-and-promote cycle with interception on.
|
|||
|
|
3. Assert **no held-out `TaskId`** appears in the recorded proposer input set —
|
|||
|
|
intersection with the held-out set is empty.
|
|||
|
|
4. Assert **no held-out `TaskId`** appears in any gate evaluation.
|
|||
|
|
5. Assert the two query surfaces are distinct types or distinct functions —
|
|||
|
|
`trybuild` a call that tries to pass held-out tasks into the proposer.
|
|||
|
|
6. **Schedule:** advance simulated time with **no promotion pending**; assert the
|
|||
|
|
held-out report is still emitted. Under one-state convergence it is the only
|
|||
|
|
stall detector there is.
|
|||
|
|
7. Assert the selection-versus-held-out gap is emitted as a metric.
|
|||
|
|
8. Negative control: deliberately wire a held-out task into the proposer in a
|
|||
|
|
test-only build; assert the audit **fails**. An audit that never fails is not
|
|||
|
|
a control.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p loop heldout_audit`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Steps 3–4 passing because the interception layer only sees one of several code
|
|||
|
|
paths into the proposer. Assert the interception is at the single chokepoint,
|
|||
|
|
or the audit measures nothing.
|
|||
|
|
- Step 8 omitted. This is the most important one: an audit test that has never
|
|||
|
|
been observed failing may simply be looking in the wrong place, and the leak it
|
|||
|
|
guards is invisible once present.
|
|||
|
|
- Step 6 omitted, so the report is computed only at gates — exactly when a
|
|||
|
|
stalled loop produces none, which is when it is needed.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- A single task query with an `include_heldout: bool`. Someone passes `true`.
|
|||
|
|
- Computing the report only when a promotion is pending, which is exactly when a
|
|||
|
|
stalled loop produces none.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background (not required to do this task):
|
|||
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §12.1, §12.3, §12.5, §15 ·
|
|||
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|