# T5.7 — Order alternation and sampled consistency

| Field | Value |
|---|---|
| Phase | P5 — Grading |
| Size | S — under 1 day |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |

## Goal

Cancel judge position bias across the sequence instead of within each pair, and
measure order-consistency on a sample rather than by doubling every call.

## Facts (inlined — no spec read needed)

- Pairwise judges have **position bias**. The tournament path cancels it by
  judging both orderings, paying 2× on every comparison.
- The default path does not pay that: **the challenger takes position A on
  even-numbered comparisons and position B on odd ones.** Bias cancels across the
  sequence, at no extra cost.
- Order-consistency is still measured — on a **sampled fraction** of comparisons
  — and reported as the grader's own error bar.
- That disagreement rate is a first-class grader-health metric. An inconsistent
  judge should widen the sample gate rather than silently promote.
- Total calls must stay near 1× the comparison count. A doubling implementation
  produces the right consistency number and the wrong cost, which is why the call
  count is asserted rather than the metric.

## Steps

1. Alternate position by comparison index: even → challenger is A, odd →
   challenger is B. Record the position used on each comparison.
2. Normalize the verdict back to challenger-relative terms before folding into
   the `WinRecord` (T5.6) — otherwise alternation inverts half the record.
3. Sample a configurable fraction of comparisons for a second, order-swapped
   call. Default the fraction low; it is an error bar, not the measurement.
4. Compute the disagreement rate over sampled pairs and emit it as a metric
   tagged by judge and rubric.
5. Count total `compare` calls and assert the ratio to comparison count in the
   test.
6. Build a deliberately position-biased mock judge — always picks A — as the
   detection case.

## Acceptance

- A deliberately position-biased mock judge is **detected** and its disagreement
  rate reported.
- Total `compare` calls stay within **1.0–1.2×** the comparison count. A doubling
  implementation fails this, which is the point.

## Verify

**Harness:** a mock judge that **always picks position A** regardless of content
— maximal position bias, so detection is unambiguous.

**Integration test** — `tests/it_order_alternation.rs`:
1. Run 100 comparisons against the always-A judge.
2. Assert the disagreement rate is **detected and reported** near 100% on the
   sampled subset.
3. Assert total `compare` calls fall in **1.0–1.2×** the comparison count. A
   doubling implementation lands at 2.0× and fails here, which is the point.
4. Assert position alternation: challenger in A on even indices, B on odd. Read
   the recorded position per comparison, do not infer it.
5. **Normalization:** with the always-A judge, assert the resulting `WinRecord`
   is near 50/50 rather than 100% challenger wins. A missing verdict
   normalization after the swap inverts half the record and produces exactly
   100%, which looks like a strong result.
6. Unbiased judge control: a content-driven mock judge yields a low disagreement
   rate — otherwise the detector fires on everything.
7. Assert the disagreement rate is emitted as a metric tagged by judge and
   rubric.

**Command:** `cargo test -p grading order_alternation`

**False pass:**
- Step 3 omitted. Judging both orderings on every comparison produces a perfect
  consistency measurement at double the cost — correct-looking and exactly the
  design this task rejects.
- Step 5 omitted: forgetting to normalize verdicts after alternating is the most
  likely implementation error, and it makes a biased judge look like a decisive
  one.
- Step 6 omitted, so a detector that always reports high bias passes.

## Traps

- Forgetting to normalize the verdict after swapping positions. The record then
  averages toward 50% regardless of the truth.
- Sampling "every comparison, it is cheap" — that is the doubling implementation
  with extra steps.

---

Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §11.3, §11.4, §15 ·
[rust-agentic-task.md](../../../rust-agentic-task.md)