rock/poimen
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

(tasks) add tasks for harness

Browse Source
This commit is contained in:
Story Crater Bot
2026-08-17 23:05:20 -07:00
parent e2d678b118
commit 5a30d0ffc6
74 changed files with 8204 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tasks/T5.9-bradley-terry-fit.md
+132
View File
@@ -0,0 +1,132 @@
# T5.9 — Bradley-Terry fit
| Field | Value |
|---|---|
| Phase | P5 — Grading |
| Size | L — over 3 days |
| Status | Not started |
| Flags | opt-in |
| Spec | inlined below |
| Blocks | T6.4 |
## Goal
Strength parameter plus confidence interval over the pairwise outcomes. Ships
**disabled** — it gates nothing.
## Facts (inlined — no spec read needed)
**Bradley-Terry, not point-tally z-scores.** Accumulating tournament points and
normalizing to mean 0 / sd 1 within a group is the obvious approach and it is
statistically wrong for the aggregate: small groups produce extreme z-scores, so
a variant appearing in many small groups wins on variance rather than quality. A
BT fit yields a strength with a real confidence interval, which composes across
groups of different sizes.
Three decisions, none optional. Textbook BT does none of them, and each failure
looks like a result rather than a bug:
- **Davidson tie term.** Plain BT is binary and has no tie parameter, so the
permitted draws have nowhere to go. Dropping them discards the judge's most
confident statements; splitting each half-and-half fabricates two comparisons
that never happened and tightens the interval on invented evidence. Fit one
additional tie parameter alongside the strengths.
- **Weakly-informative prior — required, not tuning.** At G = 4..8 an episode
that wins every comparison drives the unpenalized maximum-likelihood estimate
to infinite strength. The prior (equivalently, a penalized likelihood) is what
turns "won all three of its comparisons" into a **wide** interval rather than
an unbounded one. Without it, the reassuring claim that small groups produce
wide intervals is simply false.
- **Control pinned to zero.** The fit is identified only up to an additive
constant, so an interval on a single raw strength is an interval on an
arbitrary origin. Pin control to zero and report every strength as a delta
against it. Downstream gates say "BT interval excludes zero" — zero is control,
and it is only zero because it was pinned there.
- A group containing **no control episode is not aggregatable**. It still grades
its own members and is worth reading; it just does not feed the aggregate, and
it is recorded as such rather than folded in on the assumption that scales
match.
## Steps
1. Take the comparison list from T5.8: `(episode_a, episode_b, Verdict)`.
2. Implement the Davidson model: strengths plus one tie parameter, fit by
penalized maximum likelihood.
3. Add the weakly-informative prior on strengths. Make its strength a named
constant with a comment stating it is required for convergence, not a tuning
knob — otherwise it gets removed as "unnecessary regularization".
4. Constrain control's strength to zero during the fit rather than subtracting it
afterwards.
5. Compute intervals (profile likelihood or bootstrap over comparisons) and emit
`Score::Ranked { strength, interval, group_size }`.
6. Reject a group with no control episode as non-aggregatable, with a typed
error — do not fit against an arbitrary origin.
7. Build the synthetic generator: known strengths, configurable draw rate.
## Acceptance
- On synthetic data with known strengths, recovered ordering is correct and the
delta-from-control interval covers the true delta at the stated rate.
Three targeted cases beyond that:
- A group where one episode wins every comparison **converges, with a wide
interval**. Without the prior this diverges — assert the bound, not just that
the fit returns.
- A synthetic generator producing **30% draws** recovers the true strengths; a
fit that drops draws does not, and the test asserts the gap.
- A group with **no control episode is rejected** as non-aggregatable rather than
fit against an arbitrary origin.
## Verify
**Harness:** a synthetic generator with **known** true strengths and a
configurable draw rate. Every claim here is checkable against ground truth, so
none of it needs a real judge.
**Integration test** — `tests/it_bradley_terry.rs`:
1. Generate comparisons from known strengths. Assert the recovered **ordering**
is correct.
2. **Coverage test:** repeat over 200 seeded datasets; assert the
delta-from-control interval covers the true delta at the stated rate (e.g.
~95% for a 95% interval). One dataset proves nothing about an interval.
3. **Clean sweep:** a group where one episode wins every comparison. Assert the
fit **converges** and assert a **numeric upper bound** on the interval width —
not merely that the call returned. Without the prior this diverges.
4. **Draws at 30%:** assert recovered strengths are close to truth. Then run a
draw-dropping fit on the same data and **assert the gap** — the test must show
the Davidson term is doing work, not just that the fit runs.
5. **Half-split control:** also fit with draws split half-and-half; assert its
interval is **narrower than truth warrants** (over-confident), demonstrating
why splitting is rejected.
6. **No control:** a group with no control episode is **rejected as
non-aggregatable** with a typed error — not fit against an arbitrary origin.
7. Assert control's fitted strength is exactly 0 and every other strength is
reported as a delta.
**Command:** `cargo test -p grading bradley_terry -- --nocapture`
**False pass:**
- Step 3 asserting only that the fit returns a value. An unpenalized fit returns
a very large finite number on many datasets and looks fine. Assert the bound.
- Step 4 without the comparison fit from step 5. "The fit recovers strengths"
passes on easy data whether or not draws are modelled — the **gap** is the
evidence.
- Step 2 with one dataset. Interval coverage is a frequency claim; it needs
repetition.
- Reporting raw strengths anywhere: an interval on a raw strength is an interval
on an arbitrary origin, and it will look perfectly reasonable.
## Traps
- Splitting draws half-and-half. It is the standard workaround and it invents
evidence.
- Removing the prior after seeing it barely move the estimate on well-separated
data. It exists for the clean-sweep case.
- Reporting raw strengths anywhere outside the fit.
---
Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §11.2, §11.4 ·
[rust-agentic-task.md](../../../rust-agentic-task.md)