114 lines
5.3 KiB
Markdown
114 lines
5.3 KiB
Markdown
# T6.8 — P6 composition gate
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | P6 — Learning loop |
|
||
| Size | L — over 3 days |
|
||
| Status | Not started |
|
||
| Flags | gate |
|
||
| Spec | inlined below |
|
||
| Blocks | P7 |
|
||
|
||
## Goal
|
||
|
||
Prove the loop closes **unattended** end to end, and that the default
|
||
(one-challenger, pairwise) and resourced (N-challenger, tournament) paths drive
|
||
the same gate machinery.
|
||
|
||
**Phase gate criterion:** an automatic challenger accept and an automatic
|
||
rollback, both unattended, on one resident model.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
- The loop: allocation → runs → verify → grade → sequential test → promote or
|
||
reject; rollback on a single `Score::Capped`. **Unattended** means no human
|
||
action anywhere in that path.
|
||
- The properties no single P6 task owns:
|
||
- **calibration gates promotion.** Under one resident model the judge *is* the
|
||
agent's model, and promotion without calibration is the self-critique failure
|
||
mode. The blocking relationship must be enforced, not documented.
|
||
- **held-out is never read by a gate or the proposer**, across the whole loop —
|
||
a leak introduced anywhere is invisible once present.
|
||
- **the two ladders share machinery.** Default is three rungs (shadow, trial,
|
||
current); resourced is four (shadow, canary, ramp, current). Both must drive
|
||
the same state machine, or the opt-in path is a fork.
|
||
- the slow loop fires on a **trigger** — a rejection with no replacement — not
|
||
a timer.
|
||
- The cost of one-state convergence is stated rather than discovered: it is
|
||
hill-climbing, it can sit in a local optimum, and the held-out report is the
|
||
only instrument that will notice. So the gate must assert that report is
|
||
produced even when nothing is being promoted.
|
||
|
||
## Steps
|
||
|
||
1. Drive a full unattended cycle with a challenger that is genuinely better;
|
||
assert promotion.
|
||
2. Drive a second cycle with a challenger that trips one `CoreViolation`; assert
|
||
automatic rollback.
|
||
3. Repeat both under the resourced ladder with the tournament enabled and N
|
||
challengers.
|
||
4. Run the held-out audit and the calibration interlock across the whole cycle.
|
||
5. Assert zero human-intervention hooks were invoked in any path.
|
||
6. Make this the required CI job gating P7.
|
||
|
||
## Acceptance
|
||
|
||
- A challenger is **accepted automatically** and a challenger is **rolled back
|
||
automatically**, both unattended, on one resident model with zero swaps.
|
||
- The same gate state machine drives both the default and the resourced ladder.
|
||
- No held-out `TaskId` reaches the proposer or any gate.
|
||
- Promotion is blocked when calibration is missing or stale.
|
||
|
||
## Verify
|
||
|
||
**Harness:** mock judges from T5.6; a `screen` stub that can emit one
|
||
`CoreViolation` on demand; the held-out interception layer from T6.6; a
|
||
human-intervention hook that records if it is ever called.
|
||
|
||
**Integration test** — `tests/it_p6_composition.rs`:
|
||
1. **Unattended accept:** full cycle with the 70% judge. Assert promotion, and
|
||
assert the intervention hook recorded **zero** calls.
|
||
2. **Unattended rollback:** fire one `CoreViolation` post-promotion. Assert
|
||
rollback without human action, the prior version restored with **no gap**
|
||
where neither is live, and the failed version still in the DAG with results.
|
||
3. **Calibration interlock:** clear the calibration result, attempt promotion,
|
||
assert it is **blocked**. Then stale it past its window; assert still blocked.
|
||
Then refresh; assert it proceeds.
|
||
4. **Held-out audit across the whole loop:** assert zero held-out `TaskId`s in
|
||
proposer input and in every gate evaluation, measured over the complete cycle
|
||
rather than a single gate call. Include T6.6's negative control so the audit
|
||
is known to be capable of failing.
|
||
5. **Ladder parity:** run steps 1–2 under the resourced ladder. Assert the same
|
||
state machine and the same rollback trigger — assert on the recorded rung
|
||
transitions, so a forked implementation is visible.
|
||
6. **Trigger, not timer:** advance simulated time with no rejection; assert zero
|
||
generate-loop triggers. Then reject without replacement; assert exactly one.
|
||
7. **Stall detector:** with no promotion pending, advance the schedule; assert the
|
||
held-out report is still emitted.
|
||
8. **Zero swaps** across the entire cycle.
|
||
9. **Regression:** re-run P0–P5 gates and all P6 suites.
|
||
|
||
**Command:** `cargo test -p loop --test it_p6_composition -- --test-threads=1 --nocapture`
|
||
|
||
**False pass:**
|
||
- A cycle driven by test code that nudges the loop between rungs. That is the
|
||
human action the word "unattended" excludes — step 1's intervention counter is
|
||
the guard, and it must wrap every manual entry point, not just one.
|
||
- Step 3 omitted: calibration then exists as a dashboard number gating nothing,
|
||
which is precisely the failure mode it was introduced to prevent.
|
||
- Step 5 omitted: the resourced ladder quietly becoming a second implementation
|
||
means every fix to the default path has to be made twice, and will not be.
|
||
- Step 4 evaluated at one gate call rather than across the cycle — the proposer
|
||
leak is the invisible one.
|
||
|
||
## Traps
|
||
|
||
- Declaring the gate green with the tournament enabled. The default path is what
|
||
ships; assert the default cycle first.
|
||
|
||
---
|
||
|
||
Background (not required to do this task):
|
||
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §12, §15, §16 ·
|
||
[rust-agentic-task.md](../../../rust-agentic-task.md)
|