110 lines
4.5 KiB
Markdown
110 lines
4.5 KiB
Markdown
# T5.8 — Swiss pairing
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | P5 — Grading |
|
|
| Size | M — 1 to 3 days |
|
|
| Status | Not started |
|
|
| Flags | opt-in |
|
|
| Spec | inlined below |
|
|
| Blocks | — |
|
|
|
|
## Goal
|
|
|
|
Swiss pairing over a comparison group: log₂(G) rounds, G/2 comparisons per round.
|
|
Ships **disabled** — it gates nothing.
|
|
|
|
## Facts (inlined — no spec read needed)
|
|
|
|
```
|
|
[ G comparable episodes for one task, one outcome class ]
|
|
│
|
|
▼
|
|
[ shuffle into brackets ] ◄── shuffling also cancels position bias
|
|
│
|
|
▼
|
|
┌─────────────────────────────────┐
|
|
│ Swiss pairing, log₂(G) rounds │ ◄── rubric-guided Judge, relative only
|
|
└────────────────┬────────────────┘
|
|
▼
|
|
[ Bradley-Terry fit over all comparisons ] ◄── T5.9
|
|
```
|
|
|
|
- **Swiss rather than round-robin**: O(G log G) instead of O(G²). Eight episodes
|
|
is twelve comparisons rather than twenty-eight.
|
|
- **Swiss rather than single elimination**: we want a full ranking, not a
|
|
champion — eliminated candidates still carry signal.
|
|
- **Both orderings are judged** on this path, so 12 pairs at G = 8 is 24
|
|
`compare` calls. (The default path instead alternates order — T5.7.)
|
|
- **Draws are permitted.** A judge forced to separate two equivalent episodes
|
|
invents a distinction, and the optimizer will chase the invention. Draws cost
|
|
gradient; forced choices cost correctness.
|
|
- Groups are bracketed within **one verifier outcome class**, never across.
|
|
- Enable this where episodes are already co-present at no extra cost: attempt
|
|
tournaments (T5.10) and replay (T6.5).
|
|
|
|
## Steps
|
|
|
|
1. Take a group of G episodes sharing `(TenantId, TaskId, VerifierOutcome,
|
|
GroupEpoch)`.
|
|
2. Shuffle into the first round from a **recorded seed** — the shuffle is part of
|
|
bias cancellation, and an unreproducible pairing is an unauditable result.
|
|
3. Run `ceil(log2(G))` rounds. Each round pairs on accumulated score, avoiding
|
|
rematches; carry a bye when G is odd.
|
|
4. Issue both orderings per pair and record order-consistency per comparison.
|
|
5. Track the constraint explicitly: each episode paired at most once per round,
|
|
and no episode idle in more than one round.
|
|
6. Emit the full comparison list — the BT fit (T5.9) consumes pairs, not
|
|
standings.
|
|
7. Keep it behind an off-by-default feature flag.
|
|
|
|
## Acceptance
|
|
|
|
- G = 8 produces **exactly 12 pairs and 24 `compare` calls** under the
|
|
both-orderings rule.
|
|
- Every episode is paired at most once per round.
|
|
- No episode is idle in more than one round.
|
|
|
|
## Verify
|
|
|
|
**Harness:** a counting mock judge; a fixed shuffle seed so pairings are
|
|
reproducible.
|
|
|
|
**Integration test** — `tests/it_swiss_pairing.rs`:
|
|
1. G = 8: assert **exactly 12 pairs** and **exactly 24 `compare` calls** (both
|
|
orderings).
|
|
2. Assert every episode is paired **at most once per round**.
|
|
3. Assert **no episode is idle in more than one round**.
|
|
4. Assert no pair repeats across rounds.
|
|
5. Odd G (G = 7): assert the bye is assigned, assigned to a different episode
|
|
each round, and that the comparison count matches the expected formula.
|
|
6. G = 2 and G = 3 boundary cases: assert the pairing does not panic and produces
|
|
the minimum sensible schedule.
|
|
7. Assert the shuffle seed is **recorded** with the group, and that replaying the
|
|
seed reproduces the identical pairing.
|
|
8. Assert every pair sits within one `VerifierOutcome` class — feed a mixed group
|
|
and assert zero cross-class pairs.
|
|
|
|
**Command:** `cargo test -p grading swiss`
|
|
|
|
**False pass:**
|
|
- Asserting the pair count alone. Round-robin at G = 8 gives 28; a single
|
|
elimination gives 7. But an implementation that pairs the same two episodes
|
|
repeatedly can also hit 12 — steps 2 and 4 are what make the count meaningful.
|
|
- Testing only G = 8, a clean power of two. The bye logic at odd G is where the
|
|
"idle in more than one round" rule actually bites.
|
|
- An unseeded shuffle, which makes a surprising standings result impossible to
|
|
reproduce or audit.
|
|
|
|
## Traps
|
|
|
|
- Round-robin "since G is small". It is 28 pairs at G = 8 and grows quadratically.
|
|
- Dropping eliminated episodes. Swiss was chosen precisely to keep their signal.
|
|
- An unseeded shuffle, which makes a surprising result impossible to reproduce.
|
|
|
|
---
|
|
|
|
Background (not required to do this task):
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §11.4, §11.8 ·
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|