poimen/tasks/T5.12-p5-composition-gate.md

T5.12 — P5 composition gate

Field	Value
Phase	P5 — Grading
Size	L — over 3 days
Status	Not started
Flags	gate
Spec	inlined below
Blocks	P6

Goal

Prove the grading strategies are interchangeable: the same recorded episodes, graded under any strategy, flow through the same downstream pipeline unchanged.

Phase gate criterion: PairwiseSequential decides an accept and a reject against a mock judge, on one resident model with zero swaps.

Facts (inlined — no spec read needed)

• Defaults ship working; every default is a port. Grading is strategy-pluggable, and the strategy declares what hardware it needs before it is allowed to run. That claim is only proven by running every strategy through the same pipeline on the same input.
• The four strategies and their declared costs per episode, on a group of eight:

Strategy	Calls / episode	Models resident	Produces	Default
DeterministicGrader	0	0	Ranked	—
PairwiseSequential	1–2	1, the agent's	Relative	yes
TournamentGrader	3–5	1	Ranked	opt-in
ReplayTournament	3–5 + N agent runs	1	Ranked	opt-in

• The properties no single P5 task owns:
  • every strategy's output flows into retention (T4.4), metering and the run lifecycle identically — Ranked, Relative, Capped and Ungraded all reach a terminal run state;
  • the declared calls_per_episode matches measured calls, or every capacity and spend projection built on it is fiction;
  • zero model swaps hold across the whole phase, not just per strategy;
  • the outcome-class bracketing rule has no exemptions — assert it once per strategy, since each has its own pairing path.
• opt-in strategies ship disabled and gate nothing. This gate must pass with them off, and separately with them on.

Steps

1. Fix one corpus of recorded episodes covering: same-outcome groups, mixed-outcome groups, a novel TaskId, a capped episode, and a budget-exhausted tenant.
2. Grade the corpus under each of the four strategies.
3. Assert the downstream pipeline — score handling, run terminal state, retention eligibility, metering attribution — is identical in shape across all four.
4. Assert declared-versus-measured cost for each strategy.
5. Run the whole gate twice: opt-in flags off, then on.
6. Make this the required CI job gating P6.

Acceptance

• All four strategies grade the same corpus; every episode reaches a terminal run state with retention eligibility correct under each.
• PairwiseSequential decides an accept and a reject against mock judges.
• Zero model swaps across the entire gate.
• Measured compare calls per episode fall within each strategy's declared calls_per_episode.
• The gate passes with opt-in flags off and on.

Verify

Harness: one committed episode corpus; mock judges (70% winner, 50/50, all-draws, position-biased, one CoreViolation); counters for compare calls, model loads, and agent runs, all separate.

Integration test — tests/it_p5_composition.rs:

1. Strategy interchange: parameterize over all four strategies. For each, assert every episode ends in a terminal run state, and that is_reducible (T4.4) agrees with that state. A strategy leaving runs in Grading fails here regardless of its scores.
2. Cost truth: assert measured compare calls per episode fall inside the declared calls_per_episode band. A declaration that does not match measurement makes every capacity and spend projection downstream wrong.
3. Zero swaps: assert the swap counter is exactly 0 across the whole parameterized run, not per strategy.
4. Bracketing, per strategy: feed the mixed-outcome group to each strategy; assert zero cross-class compare calls in every one. Each strategy has its own pairing path and the rule has no exemptions.
5. Score-type handling: assert Capped and Ungraded episodes are handled identically by every strategy's caller — same terminal state, excluded from aggregates, no numeric coercion anywhere (cross-check T5.3's compile-fail).
6. Accept and reject: with the 70% judge assert Accept; with the 50/50 judge assert Reject. Assert both are reached in fewer comparisons than fixed-n.
7. Flag matrix: run steps 1–6 with opt-in off (tournament paths must be unreachable) and on.
8. Regression: re-run P0–P4 gates and all P5 suites.

Command: cargo test -p grading --test it_p5_composition -- --nocapture

False pass:

• Testing each strategy against its own tailored corpus. The claim is interchangeability, which only means something on one shared input.
• Step 2 omitted: calls_per_episode is a declared number nobody measures, and admission control, capacity planning and spend projection all consume it.
• Step 3 counted per strategy rather than across the run — a swap between two strategies' evaluations is exactly the failure the one-resident-model default exists to prevent.
• Step 1 asserting scores only. A strategy that produces correct scores and leaves runs resting in Grading strands storage forever.

Traps

• Enabling the opt-in flags to make the gate pass. They ship disabled and gate nothing; the off-run is the one that must be green.

---

Background (not required to do this task): rust-agentic-sys.md §11, §14.2 · rust-agentic-task.md