118 lines
5.4 KiB
Markdown
118 lines
5.4 KiB
Markdown
# T5.5 — `PairwiseSequential`: reference and comparison
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | P5 — Grading |
|
|
| Size | M — 1 to 3 days |
|
|
| Status | Not started |
|
|
| Flags | — |
|
|
| Spec | inlined below |
|
|
| Blocks | T5.6, T5.7 |
|
|
|
|
## Goal
|
|
|
|
The default grading strategy: one current version, at most one challenger, one
|
|
`Judge::compare` per new episode.
|
|
|
|
## Facts (inlined — no spec read needed)
|
|
|
|
```
|
|
current version ──► episode ──┐
|
|
├──► Judge::compare ──► verdict
|
|
challenger ──► episode ──┘ │
|
|
(same TaskId) ▼
|
|
accumulate into WinRecord
|
|
```
|
|
|
|
- Grading answers **one** question: has the challenger accumulated enough
|
|
evidence to replace the current version? Not "rank these eight", not "what is
|
|
each episode worth".
|
|
- **The reference is the current version's recorded episode on the same
|
|
`TaskId`.** It is already on disk — no re-run, no group to fill, no group
|
|
timeout. A `TaskId` never seen before has no reference and degrades per T5.11.
|
|
- One comparison per episode. Against the tournament's `(G/2)·log₂(G)` pairs
|
|
doubled for both orderings, that is **24 judge calls dropping to 8 at G = 8**,
|
|
and the saving grows with G rather than shrinking.
|
|
- `ResourceProfile` declares **exactly one model — the agent's, already
|
|
resident**. That is precisely why the default grading path adds no resident
|
|
model and forces no swap.
|
|
- **A fixed reference is a cacheable prefix.** The same reference episode leads
|
|
every comparison in a decision, so it can be cached across calls. The
|
|
tournament cannot do this — shuffling into brackets makes every pair a novel
|
|
combination by design.
|
|
- Episodes are bracketed **within** a verifier outcome class, never across. A
|
|
verified pass beats a verified fail by definition and that pairing is never
|
|
shown to a judge.
|
|
- What this gives up, stated rather than discovered later: parallel exploration
|
|
and composable strengths. One challenger at a time is hill-climbing; a
|
|
`Relative` score answers "better than the current version on this task" and
|
|
does not compose across tasks.
|
|
|
|
## Steps
|
|
|
|
1. Implement `EvaluationStrategy` with `resources()` naming exactly one model —
|
|
the agent's pinned model — and `calls_per_episode: 1.0`.
|
|
2. Reference lookup: given `(TenantId, TaskId)`, find the current version's most
|
|
recent recorded episode. Return `None` for a novel task and hand off to T5.11.
|
|
3. Check the verifier outcome class before pairing. Reference and candidate must
|
|
share a class or no comparison is issued.
|
|
4. Build the comparison prompt with the **reference half first and byte-stable**
|
|
across calls. Assert that stability in a test — prompt caching depends on it.
|
|
5. Issue exactly one `Judge::compare` per new episode and fold the verdict into a
|
|
`WinRecord` (consumed by T5.6).
|
|
6. Instrument two counters: `compare` calls and model loads. Both are asserted.
|
|
|
|
## Acceptance
|
|
|
|
- Grading one episode issues **exactly one** `compare` call and **zero** model
|
|
loads — both asserted by counting.
|
|
- A second episode on the same `TaskId` reuses the identical reference, so the
|
|
reference half of the prompt is **byte-identical** across calls.
|
|
|
|
## Verify
|
|
|
|
**Harness:** a mock `Judge` recording every `(prompt_bytes, verdict)`; a model
|
|
**load** counter distinct from the call counter; two recorded episodes on one
|
|
`TaskId`.
|
|
|
|
**Integration test** — `tests/it_pairwise_reference.rs`:
|
|
1. Grade one episode against an existing reference. Assert **exactly 1**
|
|
`compare` call and **exactly 0** model loads.
|
|
2. Grade a second episode on the same `TaskId`. Assert the **reference half of
|
|
the prompt is byte-identical** between the two calls — capture both prompts
|
|
from the mock judge and compare the reference prefix directly. This is the
|
|
property prompt caching depends on and it is easy to break invisibly.
|
|
3. Assert the reference selected is the **current version's** recorded episode,
|
|
not the highest-scoring one — plant a better-scoring non-current episode on
|
|
the same `TaskId` and assert it is not chosen.
|
|
4. Outcome-class guard: reference is a verified pass, candidate is a verified
|
|
fail. Assert **zero** `compare` calls.
|
|
5. Novel `TaskId`: assert `compare` is not called and the path hands off to
|
|
T5.11's `NoReference`.
|
|
6. Assert `resources().models.len() == 1` and that it equals the pinned agent
|
|
model.
|
|
|
|
**Command:** `cargo test -p grading pairwise`
|
|
|
|
**False pass:**
|
|
- Step 2 comparing whole prompts. They legitimately differ in the candidate half;
|
|
comparing the whole thing either always fails or, if the test is loosened to
|
|
"both non-empty", always passes. Compare the **prefix**.
|
|
- Step 1 counting calls but not loads. A judge running on a non-resident model
|
|
returns the right verdict and costs tens of seconds per call.
|
|
- Step 3 omitted: "pick the best episode as reference" produces sensible-looking
|
|
verdicts against a moving target, and no other assertion here catches it.
|
|
|
|
## Traps
|
|
|
|
- Re-rendering the reference half per call with a timestamp or a fresh id in it.
|
|
It still works and it silently destroys the cacheable prefix.
|
|
- Selecting the reference by "best episode" rather than the current version's
|
|
episode. That makes the comparison a moving target.
|
|
|
|
---
|
|
|
|
Background (not required to do this task):
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §11.1, §11.3, §11.8, §14.2 ·
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|