poimen/tasks/T6.2-judge-calibration-set.md

# T6.2 — Judge calibration set

| Field | Value |
|---|---|
| Phase | P6 — Learning loop |
| Size | M — 1 to 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | T6.3. Promotion without calibration is the self-critique failure mode |

## Goal

Replay a fixed set of tasks with known verifier ground truth through the judge on
a schedule, and track judge-versus-verifier agreement.

## Facts (inlined — no spec read needed)

- **This is a precondition, not polish.** Under one resident model the judge *is*
  the agent's model. Self-critique using the same model family being graded is
  admissible **only** with this bootstrap: validation against verifiable tasks.
- Without it the grader's bias is unmeasured and the optimizer will find it. And
  under one-state convergence (T6.1) there is no competing variant whose
  divergence would make that visible.
- **Judge-versus-verifier agreement on the calibration set is the only
  instrument that can see the grader drifting toward its own bias.** It belongs
  in the metrics list beside `Indeterminate` count and judge order-inconsistency.
- **It costs zero agent runs**, since the episodes are already recorded. Re-running
  the episodes is the obvious wrong implementation, which is why the run count is
  asserted.
- Verification decides, grading explains. Agreement is measured against the
  verifier's ground truth, never the other way round.

## Steps

1. Define the calibration set: a stable list of `(TaskId, episode, known verifier
   outcome)` drawn from recorded history. Store it explicitly — a set that is
   re-derived each cycle is not a fixed set.
2. Schedule the calibration pass on an interval, decoupled from promotion.
3. For each entry, call the judge on the **recorded** episodes. No agent
   execution, no replay, no new runs.
4. Compute agreement overall and **per rubric dimension**, so a judge biased on
   one dimension is caught with the dimension named rather than showing as a
   small overall dip.
5. Emit agreement as a metric tagged by judge, model and rubric version.
6. Instrument an agent-run counter across the calibration pass and assert it is
   zero.
7. Build two judge stubs: one agreeing 95% of the time, one deliberately biased
   against a chosen rubric dimension.

## Acceptance

- A judge stub agreeing with the verifier 95% of the time reports ~95%.
- A judge deliberately biased against a rubric dimension is **caught and the
  dimension named**.
- Calibration consumes **zero agent runs** — asserted by count, since re-running
  episodes is the obvious wrong implementation.

## Verify

**Harness:** two judge stubs — one agreeing with the verifier 95% of the time,
one biased against a single named rubric dimension. Plus an **agent-run counter**
at the executor boundary.

**Integration test** — `tests/it_judge_calibration.rs`:
1. Run the calibration pass with the 95% stub over ≥200 calibration entries.
   Assert reported agreement is ~95% within sampling error. State the tolerance
   from the sample size rather than guessing.
2. Run with the dimension-biased stub. Assert the report **names the dimension**,
   not just a lower overall figure.
3. **Assert the agent-run counter is exactly 0** across the whole pass. The
   episodes are already recorded; re-running them is the obvious wrong
   implementation and it is expensive rather than incorrect-looking.
4. Assert the calibration set is **stored and stable** — run the pass twice and
   assert the same entries were used both times. A set re-derived each cycle is
   not a fixed set and its trend is meaningless.
5. Assert agreement is emitted as a metric tagged by judge, model and rubric
   version.
6. Assert T6.3 refuses to promote when no calibration result exists or the
   latest is stale — the blocking relationship must be enforced, not documented.

**Command:** `cargo test -p loop calibration`

**False pass:**
- Step 2 asserting only that overall agreement dropped. A judge biased on one
  dimension of five moves the aggregate by a few points, which is
  indistinguishable from noise — the per-dimension breakdown is the instrument.
- Step 4 omitted: a set rebuilt from "recent episodes" each cycle drifts with the
  workload, so the metric measures the workload rather than the judge.
- Step 6 omitted, leaving calibration as a dashboard number that gates nothing —
  which is precisely the self-critique failure mode this exists to prevent.

## Traps

- Re-executing the calibration tasks to "get fresh episodes". It costs agent runs
  and measures the wrong thing.
- Reporting only an aggregate agreement number, which hides single-dimension bias
  — the exact failure the loop will exploit.

---

Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §1, §14.2, §15, §16 ·
[rust-agentic-task.md](../../../rust-agentic-task.md)