(tasks) add tasks for harness

tasks/T6.6-held-out-split.md

@@ -0,0 +1,105 @@
+# T6.6 — Held-out split
+
+| Field | Value |
+|---|---|
+| Phase | P6 — Learning loop |
+| Size | M — 1 to 3 days |
+| Status | Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | — |
+
+## Goal
+
+Partition tasks into selection and held-out sets. Promote on selection, report
+held-out without optimizing against it.
+
+## Facts (inlined — no spec read needed)
+
+- Selecting on a fixed set of recorded tasks overfits to those tasks, **silently**:
+  shadow scores improve while live performance does not.
+- **No promotion gate takes held-out as an entry criterion.** A gate that reads
+  held-out has converted it into a second selection set and left nothing
+  measuring generalization.
+- **The proposer reads the selection set only.** If the slow loop consumes
+  held-out failures to generate candidates, the held-out set is contaminated
+  through the generator instead of the selector. **This leak is easy to introduce
+  and invisible once present** — which is why the audit test is the control, not
+  a nicety.
+- A widening selection-versus-held-out gap is the overfitting alarm.
+- Under one-state convergence there is no competing variant whose divergence
+  would reveal a stalled loop, so **the held-out report is also the only stall
+  detector there is**. It must be produced and surfaced even when no promotion is
+  pending.
+- Held-out catches overfitting to *episodes*. It does not catch drift in the
+  *task mix* — that is T6.7.
+
+## Steps
+
+1. Assign each `TaskId` to selection or held-out by a deterministic hash of the
+   id, so the partition is stable and needs no stored membership list.
+2. Expose two separate query surfaces — `selection_tasks()` and
+   `heldout_tasks()` — rather than one with a flag. A flag defaults wrong.
+3. Give the proposer access to the selection surface only, at the type level if
+   possible.
+4. Compute the held-out report on a **schedule**, independent of promotion
+   activity, and surface the selection-versus-held-out gap as a metric.
+5. Write the audit test: enumerate every proposer input and every gate
+   evaluation, assert no held-out `TaskId` appears in either.
+6. Keep the audit test running in CI permanently. It is the only thing that will
+   notice the leak.
+
+## Acceptance
+
+- An audit test asserting **no held-out `TaskId` appears in proposer input**, and
+  none in any promotion-gate evaluation. This leak is invisible once present, so
+  the test is the control.
+- The held-out report is **emitted on a schedule, not only at a gate** — asserted,
+  since under one-state convergence it is the only stall detector there is.
+
+## Verify
+
+**Harness:** an interception layer recording **every** `TaskId` that reaches the
+proposer and every `TaskId` read during a gate evaluation. The audit is the
+deliverable — it is the only thing that will ever notice this leak.
+
+**Integration test** — `tests/it_heldout_audit.rs`:
+1. Partition a corpus of 1000 `TaskId`s. Assert the split is deterministic:
+   recompute it in a second process and assert identical membership.
+2. Run a full generate-and-promote cycle with interception on.
+3. Assert **no held-out `TaskId`** appears in the recorded proposer input set —
+   intersection with the held-out set is empty.
+4. Assert **no held-out `TaskId`** appears in any gate evaluation.
+5. Assert the two query surfaces are distinct types or distinct functions —
+   `trybuild` a call that tries to pass held-out tasks into the proposer.
+6. **Schedule:** advance simulated time with **no promotion pending**; assert the
+   held-out report is still emitted. Under one-state convergence it is the only
+   stall detector there is.
+7. Assert the selection-versus-held-out gap is emitted as a metric.
+8. Negative control: deliberately wire a held-out task into the proposer in a
+   test-only build; assert the audit **fails**. An audit that never fails is not
+   a control.
+
+**Command:** `cargo test -p loop heldout_audit`
+
+**False pass:**
+- Steps 3–4 passing because the interception layer only sees one of several code
+  paths into the proposer. Assert the interception is at the single chokepoint,
+  or the audit measures nothing.
+- Step 8 omitted. This is the most important one: an audit test that has never
+  been observed failing may simply be looking in the wrong place, and the leak it
+  guards is invisible once present.
+- Step 6 omitted, so the report is computed only at gates — exactly when a
+  stalled loop produces none, which is when it is needed.
+
+## Traps
+
+- A single task query with an `include_heldout: bool`. Someone passes `true`.
+- Computing the report only when a promotion is pending, which is exactly when a
+  stalled loop produces none.
+
+---
+
+Background (not required to do this task):
+[rust-agentic-sys.md](../../../rust-agentic-sys.md) §12.1, §12.3, §12.5, §15 ·
+[rust-agentic-task.md](../../../rust-agentic-task.md)