Files

114 lines
5.3 KiB
Markdown
Raw Permalink Normal View History

2026-08-17 23:05:20 -07:00
# T6.8 — P6 composition gate
| Field | Value |
|---|---|
| Phase | P6 — Learning loop |
| Size | L — over 3 days |
| Status | Not started |
| Flags | gate |
| Spec | inlined below |
| Blocks | P7 |
## Goal
Prove the loop closes **unattended** end to end, and that the default
(one-challenger, pairwise) and resourced (N-challenger, tournament) paths drive
the same gate machinery.
**Phase gate criterion:** an automatic challenger accept and an automatic
rollback, both unattended, on one resident model.
## Facts (inlined — no spec read needed)
- The loop: allocation → runs → verify → grade → sequential test → promote or
reject; rollback on a single `Score::Capped`. **Unattended** means no human
action anywhere in that path.
- The properties no single P6 task owns:
- **calibration gates promotion.** Under one resident model the judge *is* the
agent's model, and promotion without calibration is the self-critique failure
mode. The blocking relationship must be enforced, not documented.
- **held-out is never read by a gate or the proposer**, across the whole loop —
a leak introduced anywhere is invisible once present.
- **the two ladders share machinery.** Default is three rungs (shadow, trial,
current); resourced is four (shadow, canary, ramp, current). Both must drive
the same state machine, or the opt-in path is a fork.
- the slow loop fires on a **trigger** — a rejection with no replacement — not
a timer.
- The cost of one-state convergence is stated rather than discovered: it is
hill-climbing, it can sit in a local optimum, and the held-out report is the
only instrument that will notice. So the gate must assert that report is
produced even when nothing is being promoted.
## Steps
1. Drive a full unattended cycle with a challenger that is genuinely better;
assert promotion.
2. Drive a second cycle with a challenger that trips one `CoreViolation`; assert
automatic rollback.
3. Repeat both under the resourced ladder with the tournament enabled and N
challengers.
4. Run the held-out audit and the calibration interlock across the whole cycle.
5. Assert zero human-intervention hooks were invoked in any path.
6. Make this the required CI job gating P7.
## Acceptance
- A challenger is **accepted automatically** and a challenger is **rolled back
automatically**, both unattended, on one resident model with zero swaps.
- The same gate state machine drives both the default and the resourced ladder.
- No held-out `TaskId` reaches the proposer or any gate.
- Promotion is blocked when calibration is missing or stale.
## Verify
**Harness:** mock judges from T5.6; a `screen` stub that can emit one
`CoreViolation` on demand; the held-out interception layer from T6.6; a
human-intervention hook that records if it is ever called.
**Integration test**`tests/it_p6_composition.rs`:
1. **Unattended accept:** full cycle with the 70% judge. Assert promotion, and
assert the intervention hook recorded **zero** calls.
2. **Unattended rollback:** fire one `CoreViolation` post-promotion. Assert
rollback without human action, the prior version restored with **no gap**
where neither is live, and the failed version still in the DAG with results.
3. **Calibration interlock:** clear the calibration result, attempt promotion,
assert it is **blocked**. Then stale it past its window; assert still blocked.
Then refresh; assert it proceeds.
4. **Held-out audit across the whole loop:** assert zero held-out `TaskId`s in
proposer input and in every gate evaluation, measured over the complete cycle
rather than a single gate call. Include T6.6's negative control so the audit
is known to be capable of failing.
5. **Ladder parity:** run steps 12 under the resourced ladder. Assert the same
state machine and the same rollback trigger — assert on the recorded rung
transitions, so a forked implementation is visible.
6. **Trigger, not timer:** advance simulated time with no rejection; assert zero
generate-loop triggers. Then reject without replacement; assert exactly one.
7. **Stall detector:** with no promotion pending, advance the schedule; assert the
held-out report is still emitted.
8. **Zero swaps** across the entire cycle.
9. **Regression:** re-run P0P5 gates and all P6 suites.
**Command:** `cargo test -p loop --test it_p6_composition -- --test-threads=1 --nocapture`
**False pass:**
- A cycle driven by test code that nudges the loop between rungs. That is the
human action the word "unattended" excludes — step 1's intervention counter is
the guard, and it must wrap every manual entry point, not just one.
- Step 3 omitted: calibration then exists as a dashboard number gating nothing,
which is precisely the failure mode it was introduced to prevent.
- Step 5 omitted: the resourced ladder quietly becoming a second implementation
means every fix to the default path has to be made twice, and will not be.
- Step 4 evaluated at one gate call rather than across the cycle — the proposer
leak is the invisible one.
## Traps
- Declaring the gate green with the tournament enabled. The default path is what
ships; assert the default cycle first.
---
Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §12, §15, §16 ·
[rust-agentic-task.md](../../../rust-agentic-task.md)