poimen/tasks/T5.5-pairwisesequential-reference-and-comparison.md

T5.5 — PairwiseSequential: reference and comparison

Field	Value
Phase	P5 — Grading
Size	M — 1 to 3 days
Status	Not started
Flags	—
Spec	inlined below
Blocks	T5.6, T5.7

Goal

The default grading strategy: one current version, at most one challenger, one Judge::compare per new episode.

Facts (inlined — no spec read needed)

 current version ──► episode ──┐
                               ├──► Judge::compare ──► verdict
 challenger      ──► episode ──┘         │
   (same TaskId)                         ▼
                              accumulate into WinRecord

• Grading answers one question: has the challenger accumulated enough evidence to replace the current version? Not "rank these eight", not "what is each episode worth".
• The reference is the current version's recorded episode on the same TaskId. It is already on disk — no re-run, no group to fill, no group timeout. A TaskId never seen before has no reference and degrades per T5.11.
• One comparison per episode. Against the tournament's (G/2)·log₂(G) pairs doubled for both orderings, that is 24 judge calls dropping to 8 at G = 8, and the saving grows with G rather than shrinking.
• ResourceProfile declares exactly one model — the agent's, already resident. That is precisely why the default grading path adds no resident model and forces no swap.
• A fixed reference is a cacheable prefix. The same reference episode leads every comparison in a decision, so it can be cached across calls. The tournament cannot do this — shuffling into brackets makes every pair a novel combination by design.
• Episodes are bracketed within a verifier outcome class, never across. A verified pass beats a verified fail by definition and that pairing is never shown to a judge.
• What this gives up, stated rather than discovered later: parallel exploration and composable strengths. One challenger at a time is hill-climbing; a Relative score answers "better than the current version on this task" and does not compose across tasks.

Steps

1. Implement EvaluationStrategy with resources() naming exactly one model — the agent's pinned model — and calls_per_episode: 1.0.
2. Reference lookup: given (TenantId, TaskId), find the current version's most recent recorded episode. Return None for a novel task and hand off to T5.11.
3. Check the verifier outcome class before pairing. Reference and candidate must share a class or no comparison is issued.
4. Build the comparison prompt with the reference half first and byte-stable across calls. Assert that stability in a test — prompt caching depends on it.
5. Issue exactly one Judge::compare per new episode and fold the verdict into a WinRecord (consumed by T5.6).
6. Instrument two counters: compare calls and model loads. Both are asserted.

Acceptance

• Grading one episode issues exactly one compare call and zero model loads — both asserted by counting.
• A second episode on the same TaskId reuses the identical reference, so the reference half of the prompt is byte-identical across calls.

Verify

Harness: a mock Judge recording every (prompt_bytes, verdict); a model load counter distinct from the call counter; two recorded episodes on one TaskId.

Integration test — tests/it_pairwise_reference.rs:

1. Grade one episode against an existing reference. Assert exactly 1 compare call and exactly 0 model loads.
2. Grade a second episode on the same TaskId. Assert the reference half of the prompt is byte-identical between the two calls — capture both prompts from the mock judge and compare the reference prefix directly. This is the property prompt caching depends on and it is easy to break invisibly.
3. Assert the reference selected is the current version's recorded episode, not the highest-scoring one — plant a better-scoring non-current episode on the same TaskId and assert it is not chosen.
4. Outcome-class guard: reference is a verified pass, candidate is a verified fail. Assert zero compare calls.
5. Novel TaskId: assert compare is not called and the path hands off to T5.11's NoReference.
6. Assert resources().models.len() == 1 and that it equals the pinned agent model.

Command: cargo test -p grading pairwise

False pass:

• Step 2 comparing whole prompts. They legitimately differ in the candidate half; comparing the whole thing either always fails or, if the test is loosened to "both non-empty", always passes. Compare the prefix.
• Step 1 counting calls but not loads. A judge running on a non-resident model returns the right verdict and costs tens of seconds per call.
• Step 3 omitted: "pick the best episode as reference" produces sensible-looking verdicts against a moving target, and no other assertion here catches it.

Traps

• Re-rendering the reference half per call with a timestamp or a fresh id in it. It still works and it silently destroys the cacheable prefix.
• Selecting the reference by "best episode" rather than the current version's episode. That makes the comparison a moving target.

---

Background (not required to do this task): rust-agentic-sys.md §11.1, §11.3, §11.8, §14.2 · rust-agentic-task.md