99 lines
4.2 KiB
Markdown
99 lines
4.2 KiB
Markdown
# T5.7 — Order alternation and sampled consistency
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | P5 — Grading |
|
||
| Size | S — under 1 day |
|
||
| Status | Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | — |
|
||
|
||
## Goal
|
||
|
||
Cancel judge position bias across the sequence instead of within each pair, and
|
||
measure order-consistency on a sample rather than by doubling every call.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
- Pairwise judges have **position bias**. The tournament path cancels it by
|
||
judging both orderings, paying 2× on every comparison.
|
||
- The default path does not pay that: **the challenger takes position A on
|
||
even-numbered comparisons and position B on odd ones.** Bias cancels across the
|
||
sequence, at no extra cost.
|
||
- Order-consistency is still measured — on a **sampled fraction** of comparisons
|
||
— and reported as the grader's own error bar.
|
||
- That disagreement rate is a first-class grader-health metric. An inconsistent
|
||
judge should widen the sample gate rather than silently promote.
|
||
- Total calls must stay near 1× the comparison count. A doubling implementation
|
||
produces the right consistency number and the wrong cost, which is why the call
|
||
count is asserted rather than the metric.
|
||
|
||
## Steps
|
||
|
||
1. Alternate position by comparison index: even → challenger is A, odd →
|
||
challenger is B. Record the position used on each comparison.
|
||
2. Normalize the verdict back to challenger-relative terms before folding into
|
||
the `WinRecord` (T5.6) — otherwise alternation inverts half the record.
|
||
3. Sample a configurable fraction of comparisons for a second, order-swapped
|
||
call. Default the fraction low; it is an error bar, not the measurement.
|
||
4. Compute the disagreement rate over sampled pairs and emit it as a metric
|
||
tagged by judge and rubric.
|
||
5. Count total `compare` calls and assert the ratio to comparison count in the
|
||
test.
|
||
6. Build a deliberately position-biased mock judge — always picks A — as the
|
||
detection case.
|
||
|
||
## Acceptance
|
||
|
||
- A deliberately position-biased mock judge is **detected** and its disagreement
|
||
rate reported.
|
||
- Total `compare` calls stay within **1.0–1.2×** the comparison count. A doubling
|
||
implementation fails this, which is the point.
|
||
|
||
## Verify
|
||
|
||
**Harness:** a mock judge that **always picks position A** regardless of content
|
||
— maximal position bias, so detection is unambiguous.
|
||
|
||
**Integration test** — `tests/it_order_alternation.rs`:
|
||
1. Run 100 comparisons against the always-A judge.
|
||
2. Assert the disagreement rate is **detected and reported** near 100% on the
|
||
sampled subset.
|
||
3. Assert total `compare` calls fall in **1.0–1.2×** the comparison count. A
|
||
doubling implementation lands at 2.0× and fails here, which is the point.
|
||
4. Assert position alternation: challenger in A on even indices, B on odd. Read
|
||
the recorded position per comparison, do not infer it.
|
||
5. **Normalization:** with the always-A judge, assert the resulting `WinRecord`
|
||
is near 50/50 rather than 100% challenger wins. A missing verdict
|
||
normalization after the swap inverts half the record and produces exactly
|
||
100%, which looks like a strong result.
|
||
6. Unbiased judge control: a content-driven mock judge yields a low disagreement
|
||
rate — otherwise the detector fires on everything.
|
||
7. Assert the disagreement rate is emitted as a metric tagged by judge and
|
||
rubric.
|
||
|
||
**Command:** `cargo test -p grading order_alternation`
|
||
|
||
**False pass:**
|
||
- Step 3 omitted. Judging both orderings on every comparison produces a perfect
|
||
consistency measurement at double the cost — correct-looking and exactly the
|
||
design this task rejects.
|
||
- Step 5 omitted: forgetting to normalize verdicts after alternating is the most
|
||
likely implementation error, and it makes a biased judge look like a decisive
|
||
one.
|
||
- Step 6 omitted, so a detector that always reports high bias passes.
|
||
|
||
## Traps
|
||
|
||
- Forgetting to normalize the verdict after swapping positions. The record then
|
||
averages toward 50% regardless of the truth.
|
||
- Sampling "every comparison, it is cheap" — that is the doubling implementation
|
||
with extra steps.
|
||
|
||
---
|
||
|
||
Background (not required to do this task):
|
||
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §11.3, §11.4, §15 ·
|
||
[rust-agentic-task.md](../../../rust-agentic-task.md)
|