poimen-memory/tasks/M5.2-labeler-calibration.md

M5.2 — Labeler calibration

Field	Value
Phase	M5 — Post-training
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M5.1

Goal

Measure how good the proxy is, before training a policy to imitate it.

Facts (inlined — no spec read needed)

M5.1's labels are distant supervision. Training on them without measuring agreement means the policy learns the 32B model's bias and reports it as improvement — and under a single-model deployment there is no competing variant whose divergence would make that visible.

Method: hand-label a stratified holdout, compare, and report more than accuracy.

• Sample size: 100 chunks is enough to distinguish 0.7 from 0.9 agreement.
• Stratify by the labeler's own answer — 50 it called evidence, 50 it did not. Random sampling from a corpus where ~5% is evidence gives ~5 positives, and the positive class is the one that matters.
• Report Cohen's κ, not raw agreement. With a 95/5 class balance, a labeler that always says "no" scores 95% agreement and is useless. κ corrects for chance.
• Also report precision and recall on the positive class separately. They fail differently: low recall silently starves memory, low precision pollutes it.

Disagreements are the artifact. Read them; they are usually either a genuinely ambiguous chunk or a question that is too vague — and the second is fixable at M1.2 and worth much more than a better labeler.

Steps

1. mem label sample --project P --n 100 --stratified writes a blind worksheet — chunk text and question, no labeler answer visible.
2. Hand-label it. Record the human labels separately.
3. mem label calibrate joins them; reports agreement, κ, precision, recall, and the confusion matrix.
4. Dump all disagreements with both justifications side by side.
5. Commit the holdout and the human labels; they are reusable for every future labeler change.
6. Gate: κ ≥ 0.6 before the labels are used for training.

Acceptance

• Worksheet hides the labeler's answer.
• κ, precision, recall and the confusion matrix are all reported.
• Disagreements are dumped with justifications.
• The holdout is committed and reusable.

Verify

Harness: a synthetic labeled set with known agreement, so the statistics themselves are testable.

Integration test — tests/it_calibration.rs:

1. a1_worksheet_is_blind — assert the labeler's answer appears nowhere in the output file.
2. a2_stratified — assert the sample is ~50/50 by labeler answer, not corpus proportional.
3. a3_kappa_correct — feed a set with hand-computed κ; assert the reported value matches to 3 decimals.
4. a4_kappa_vs_accuracy — a synthetic all-negative labeler on a 95/5 set: assert accuracy > 0.9 and κ ≈ 0. This is the assertion that justifies reporting κ at all.
5. a5_confusion_matrix — all four cells match hand counts.
6. a6_disagreements_dumped — count equals off-diagonal total; each carries both justifications.
7. a7_holdout_stable — rerunning the sampler with the same seed reproduces the same chunks.

Command: cargo test -p mem-cli calibration

False pass:

• Reporting accuracy only. On this class balance it is nearly meaningless, and it will look excellent right up until the trained policy learns to always answer "no".
• Hand-labeling with the labeler's answer visible. Anchoring makes agreement look high and the whole exercise decorative — assertion 1 is a real safeguard, not hygiene.

Traps

• Sampling proportionally. A 5%-positive corpus yields five positives in a hundred, and precision on the positive class — the number that decides whether memory gets polluted — is estimated from five examples.
• Treating low κ as "the labeler needs a better prompt". Check the questions first: an ambiguous standing question makes evidence genuinely undecidable, and no labeler can fix that.

---

Background: DESIGN.md — Risks