poimen/tasks/T5.12-p5-composition-gate.md

# T5.12 — P5 composition gate

| Field | Value |
|---|---|
| Phase | P5 — Grading |
| Size | L — over 3 days |
| Status | Not started |
| Flags | gate |
| Spec | inlined below |
| Blocks | P6 |

## Goal

Prove the grading strategies are **interchangeable**: the same recorded episodes,
graded under any strategy, flow through the same downstream pipeline unchanged.

**Phase gate criterion:** `PairwiseSequential` decides an accept and a reject
against a mock judge, on one resident model with zero swaps.

## Facts (inlined — no spec read needed)

- **Defaults ship working; every default is a port.** Grading is
  strategy-pluggable, and the strategy declares what hardware it needs before it
  is allowed to run. That claim is only proven by running **every** strategy
  through the same pipeline on the same input.
- The four strategies and their declared costs per episode, on a group of eight:

| Strategy | Calls / episode | Models resident | Produces | Default |
|---|---|---|---|---|
| `DeterministicGrader` | 0 | 0 | `Ranked` | — |
| **`PairwiseSequential`** | **1–2** | **1, the agent's** | `Relative` | **yes** |
| `TournamentGrader` | 3–5 | 1 | `Ranked` | opt-in |
| `ReplayTournament` | 3–5 + N agent runs | 1 | `Ranked` | opt-in |

- The properties no single P5 task owns:
  - every strategy's output flows into retention (T4.4), metering and the run
    lifecycle **identically** — `Ranked`, `Relative`, `Capped` and `Ungraded` all
    reach a terminal run state;
  - the declared `calls_per_episode` **matches measured** calls, or every
    capacity and spend projection built on it is fiction;
  - **zero model swaps** hold across the whole phase, not just per strategy;
  - the outcome-class bracketing rule has **no exemptions** — assert it once per
    strategy, since each has its own pairing path.
- `opt-in` strategies ship **disabled** and gate nothing. This gate must pass
  with them off, and separately with them on.

## Steps

1. Fix one corpus of recorded episodes covering: same-outcome groups,
   mixed-outcome groups, a novel `TaskId`, a capped episode, and a
   budget-exhausted tenant.
2. Grade the corpus under each of the four strategies.
3. Assert the downstream pipeline — score handling, run terminal state, retention
   eligibility, metering attribution — is identical in shape across all four.
4. Assert declared-versus-measured cost for each strategy.
5. Run the whole gate twice: opt-in flags **off**, then **on**.
6. Make this the required CI job gating P6.

## Acceptance

- All four strategies grade the same corpus; every episode reaches a terminal run
  state with retention eligibility correct under each.
- `PairwiseSequential` decides an **accept** and a **reject** against mock judges.
- **Zero model swaps** across the entire gate.
- Measured `compare` calls per episode fall within each strategy's declared
  `calls_per_episode`.
- The gate passes with opt-in flags off **and** on.

## Verify

**Harness:** one committed episode corpus; mock judges (70% winner, 50/50,
all-draws, position-biased, one `CoreViolation`); counters for `compare` calls,
model **loads**, and agent runs, all separate.

**Integration test** — `tests/it_p5_composition.rs`:
1. **Strategy interchange:** parameterize over all four strategies. For each,
   assert every episode ends in a terminal run state, and that
   `is_reducible` (T4.4) agrees with that state. A strategy leaving runs in
   `Grading` fails here regardless of its scores.
2. **Cost truth:** assert measured `compare` calls per episode fall inside the
   declared `calls_per_episode` band. A declaration that does not match
   measurement makes every capacity and spend projection downstream wrong.
3. **Zero swaps:** assert the swap counter is **exactly 0** across the whole
   parameterized run, not per strategy.
4. **Bracketing, per strategy:** feed the mixed-outcome group to each strategy;
   assert **zero** cross-class `compare` calls in every one. Each strategy has
   its own pairing path and the rule has no exemptions.
5. **Score-type handling:** assert `Capped` and `Ungraded` episodes are handled
   identically by every strategy's caller — same terminal state, excluded from
   aggregates, no numeric coercion anywhere (cross-check T5.3's compile-fail).
6. **Accept and reject:** with the 70% judge assert `Accept`; with the 50/50
   judge assert `Reject`. Assert both are reached in fewer comparisons than
   fixed-n.
7. **Flag matrix:** run steps 1–6 with opt-in off (tournament paths must be
   unreachable) and on.
8. **Regression:** re-run P0–P4 gates and all P5 suites.

**Command:** `cargo test -p grading --test it_p5_composition -- --nocapture`

**False pass:**
- Testing each strategy against its own tailored corpus. The claim is
  interchangeability, which only means something on **one** shared input.
- Step 2 omitted: `calls_per_episode` is a declared number nobody measures, and
  admission control, capacity planning and spend projection all consume it.
- Step 3 counted per strategy rather than across the run — a swap between two
  strategies' evaluations is exactly the failure the one-resident-model default
  exists to prevent.
- Step 1 asserting scores only. A strategy that produces correct scores and
  leaves runs resting in `Grading` strands storage forever.

## Traps

- Enabling the opt-in flags to make the gate pass. They ship disabled and gate
  nothing; the off-run is the one that must be green.

---

Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §11, §14.2 ·
[rust-agentic-task.md](../../../rust-agentic-task.md)