(tasks) add tasks for harness

2026-08-17 23:05:20 -07:00
parent e2d678b118
commit 5a30d0ffc6
74 changed files with 8204 additions and 1 deletions
@@ -0,0 +1,117 @@
+# T5.5 — `PairwiseSequential`: reference and comparison
+
+| Field | Value |
+|---|---|
+| Phase | P5 — Grading |
+| Size | M — 1 to 3 days |
+| Status | Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | T5.6, T5.7 |
+
+## Goal
+
+The default grading strategy: one current version, at most one challenger, one
+`Judge::compare` per new episode.
+
+## Facts (inlined — no spec read needed)
+
+```
+ current version ──► episode ──┐
+                               ├──► Judge::compare ──► verdict
+ challenger      ──► episode ──┘         │
+   (same TaskId)                         ▼
+                              accumulate into WinRecord
+```
+
+- Grading answers **one** question: has the challenger accumulated enough
+  evidence to replace the current version? Not "rank these eight", not "what is
+  each episode worth".
+- **The reference is the current version's recorded episode on the same
+  `TaskId`.** It is already on disk — no re-run, no group to fill, no group
+  timeout. A `TaskId` never seen before has no reference and degrades per T5.11.
+- One comparison per episode. Against the tournament's `(G/2)·log₂(G)` pairs
+  doubled for both orderings, that is **24 judge calls dropping to 8 at G = 8**,
+  and the saving grows with G rather than shrinking.
+- `ResourceProfile` declares **exactly one model — the agent's, already
+  resident**. That is precisely why the default grading path adds no resident
+  model and forces no swap.
+- **A fixed reference is a cacheable prefix.** The same reference episode leads
+  every comparison in a decision, so it can be cached across calls. The
+  tournament cannot do this — shuffling into brackets makes every pair a novel
+  combination by design.
+- Episodes are bracketed **within** a verifier outcome class, never across. A
+  verified pass beats a verified fail by definition and that pairing is never
+  shown to a judge.
+- What this gives up, stated rather than discovered later: parallel exploration
+  and composable strengths. One challenger at a time is hill-climbing; a
+  `Relative` score answers "better than the current version on this task" and
+  does not compose across tasks.
+
+## Steps
+
+1. Implement `EvaluationStrategy` with `resources()` naming exactly one model —
+   the agent's pinned model — and `calls_per_episode: 1.0`.
+2. Reference lookup: given `(TenantId, TaskId)`, find the current version's most
+   recent recorded episode. Return `None` for a novel task and hand off to T5.11.
+3. Check the verifier outcome class before pairing. Reference and candidate must
+   share a class or no comparison is issued.
+4. Build the comparison prompt with the **reference half first and byte-stable**
+   across calls. Assert that stability in a test — prompt caching depends on it.
+5. Issue exactly one `Judge::compare` per new episode and fold the verdict into a
+   `WinRecord` (consumed by T5.6).
+6. Instrument two counters: `compare` calls and model loads. Both are asserted.
+
+## Acceptance
+
+- Grading one episode issues **exactly one** `compare` call and **zero** model
+  loads — both asserted by counting.
+- A second episode on the same `TaskId` reuses the identical reference, so the
+  reference half of the prompt is **byte-identical** across calls.
+
+## Verify
+
+**Harness:** a mock `Judge` recording every `(prompt_bytes, verdict)`; a model
+**load** counter distinct from the call counter; two recorded episodes on one
+`TaskId`.
+
+**Integration test** — `tests/it_pairwise_reference.rs`:
+1. Grade one episode against an existing reference. Assert **exactly 1**
+   `compare` call and **exactly 0** model loads.
+2. Grade a second episode on the same `TaskId`. Assert the **reference half of
+   the prompt is byte-identical** between the two calls — capture both prompts
+   from the mock judge and compare the reference prefix directly. This is the
+   property prompt caching depends on and it is easy to break invisibly.
+3. Assert the reference selected is the **current version's** recorded episode,
+   not the highest-scoring one — plant a better-scoring non-current episode on
+   the same `TaskId` and assert it is not chosen.
+4. Outcome-class guard: reference is a verified pass, candidate is a verified
+   fail. Assert **zero** `compare` calls.
+5. Novel `TaskId`: assert `compare` is not called and the path hands off to
+   T5.11's `NoReference`.
+6. Assert `resources().models.len() == 1` and that it equals the pinned agent
+   model.
+
+**Command:** `cargo test -p grading pairwise`
+
+**False pass:**
+- Step 2 comparing whole prompts. They legitimately differ in the candidate half;
+  comparing the whole thing either always fails or, if the test is loosened to
+  "both non-empty", always passes. Compare the **prefix**.
+- Step 1 counting calls but not loads. A judge running on a non-resident model
+  returns the right verdict and costs tens of seconds per call.
+- Step 3 omitted: "pick the best episode as reference" produces sensible-looking
+  verdicts against a moving target, and no other assertion here catches it.
+
+## Traps
+
+- Re-rendering the reference half per call with a timestamp or a fresh id in it.
+  It still works and it silently destroys the cacheable prefix.
+- Selecting the reference by "best episode" rather than the current version's
+  episode. That makes the comparison a moving target.
+
+---
+
+Background (not required to do this task):
+[rust-agentic-sys.md](../../../rust-agentic-sys.md) §11.1, §11.3, §11.8, §14.2 ·
+[rust-agentic-task.md](../../../rust-agentic-task.md)