T5.8 — Swiss pairing

Field	Value
Phase	P5 — Grading
Size	M — 1 to 3 days
Status	Not started
Flags	opt-in
Spec	inlined below
Blocks	—

Goal

Swiss pairing over a comparison group: log₂(G) rounds, G/2 comparisons per round. Ships disabled — it gates nothing.

Facts (inlined — no spec read needed)

 [ G comparable episodes for one task, one outcome class ]
              │
              ▼
   [ shuffle into brackets ]        ◄── shuffling also cancels position bias
              │
              ▼
 ┌─────────────────────────────────┐
 │  Swiss pairing, log₂(G) rounds  │ ◄── rubric-guided Judge, relative only
 └────────────────┬────────────────┘
                  ▼
     [ Bradley-Terry fit over all comparisons ]   ◄── T5.9

Swiss rather than round-robin: O(G log G) instead of O(G²). Eight episodes is twelve comparisons rather than twenty-eight.
Swiss rather than single elimination: we want a full ranking, not a champion — eliminated candidates still carry signal.
Both orderings are judged on this path, so 12 pairs at G = 8 is 24 compare calls. (The default path instead alternates order — T5.7.)
Draws are permitted. A judge forced to separate two equivalent episodes invents a distinction, and the optimizer will chase the invention. Draws cost gradient; forced choices cost correctness.
Groups are bracketed within one verifier outcome class, never across.
Enable this where episodes are already co-present at no extra cost: attempt tournaments (T5.10) and replay (T6.5).

Steps

Take a group of G episodes sharing (TenantId, TaskId, VerifierOutcome, GroupEpoch).
Shuffle into the first round from a recorded seed — the shuffle is part of bias cancellation, and an unreproducible pairing is an unauditable result.
Run ceil(log2(G)) rounds. Each round pairs on accumulated score, avoiding rematches; carry a bye when G is odd.
Issue both orderings per pair and record order-consistency per comparison.
Track the constraint explicitly: each episode paired at most once per round, and no episode idle in more than one round.
Emit the full comparison list — the BT fit (T5.9) consumes pairs, not standings.
Keep it behind an off-by-default feature flag.

Acceptance

G = 8 produces exactly 12 pairs and 24 compare calls under the both-orderings rule.
Every episode is paired at most once per round.
No episode is idle in more than one round.

Verify

Harness: a counting mock judge; a fixed shuffle seed so pairings are reproducible.

Integration test — tests/it_swiss_pairing.rs:

G = 8: assert exactly 12 pairs and exactly 24 compare calls (both orderings).
Assert every episode is paired at most once per round.
Assert no episode is idle in more than one round.
Assert no pair repeats across rounds.
Odd G (G = 7): assert the bye is assigned, assigned to a different episode each round, and that the comparison count matches the expected formula.
G = 2 and G = 3 boundary cases: assert the pairing does not panic and produces the minimum sensible schedule.
Assert the shuffle seed is recorded with the group, and that replaying the seed reproduces the identical pairing.
Assert every pair sits within one VerifierOutcome class — feed a mixed group and assert zero cross-class pairs.

Command: cargo test -p grading swiss

False pass:

Asserting the pair count alone. Round-robin at G = 8 gives 28; a single elimination gives 7. But an implementation that pairs the same two episodes repeatedly can also hit 12 — steps 2 and 4 are what make the count meaningful.
Testing only G = 8, a clean power of two. The bye logic at odd G is where the "idle in more than one round" rule actually bites.
An unseeded shuffle, which makes a surprising standings result impossible to reproduce or audit.

Traps

Round-robin "since G is small". It is 28 pairs at G = 8 and grows quadratically.
Dropping eliminated episodes. Swiss was chosen precisely to keep their signal.
An unseeded shuffle, which makes a surprising result impossible to reproduce.

Background (not required to do this task): rust-agentic-sys.md §11.4, §11.8 · rust-agentic-task.md

4.5 KiB Raw Blame History