poimen-memory/tasks/M5.1-evidence-labeler.md

M5.1 — mem label — evidence labeler

Field	Value
Phase	M5 — Post-training
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M1.6

Goal

Produce the per-chunk ground truth U_t that r_update needs, since this corpus does not come with evidence labels.

Facts (inlined — no spec read needed)

Paper r_update:

r_update_t = +1  if U_t is correct
             -1  if U_t is incorrect

"Correct" means: for chunks containing evidence for Q, the agent should emit <check>yes</check>; for chunks without, <check>no</check>. That requires knowing which chunks contain evidence.

The paper had it for free — synthetic NIAH tasks place the needle deliberately, and HotpotQA ships supporting facts. We have neither. Agent transcripts have no annotation of which turn contained the answer.

Cheapest honest substitute: distant supervision from the 32B model. Ask reasoning (DeepSeek-R1-Distill-Qwen-32B, vLLM) per (question, chunk) whether the chunk contains evidence. It is ~10× the controller's size and sees each chunk independently, without the memory state that might bias the 3B's decision.

This inherits the labeler's bias, which is why M5.2 exists and must run before anyone trains on these labels.

Constraint: reasoning has a 16384 total context and vLLM rejects input + max_tokens > 16384. A 5000-token chunk plus question plus instructions fits with room; keep max_tokens small (labels are one token of signal) and do not batch chunks into one prompt.

The labeler emits a binary label plus a short justification. Keep the justification — it is what makes M5.2's disagreement analysis possible.

Steps

1. mem label --project P --query Q reads chunks from the log.
2. Per chunk, prompt reasoning: question, chunk, "does this contain evidence for the question? Answer yes or no, then one sentence why."
3. Send no tools — the reasoning route rejects any request carrying them.
4. Write label/<project>/<query-id>.jsonl: {"chunk_sha":"...","t":7,"label":true,"why":"...","model":"reasoning","ts":"..."}.
5. Resumable: skip chunks already labelled.
6. Report the label rate — the fraction of chunks the labeler calls evidence. Compare it to the controller's update-rate from M1.7; a large gap is the finding, not a bug.

Acceptance

• Every chunk in the log gets exactly one label.
• Labels key on chunk_sha, so they survive re-chunking only if content is unchanged.
• Resume skips completed work.
• Label rate is reported alongside the controller's update-rate.

Verify

Harness: scripted client offline; one #[ignore] live run.

Integration test — tests/it_label.rs:

1. a1_one_label_per_chunk — no duplicates, no gaps against the log's chunks.
2. a2_keyed_by_sha — labels reference chunk_sha, not t, so reordering the log does not corrupt them.
3. a3_no_tools_sent — assert the request body has no tools key.
4. a4_context_budget — assert every labeling prompt is under 16384 − max_tokens.
5. a5_resume — label, rerun, assert zero new calls.
6. a6_justification_kept — every label has non-empty why.
7. a7_rate_reported — the summary prints both label rate and the controller's update-rate.
8. a8_live — #[ignore]; 20 real chunks through reasoning; print the labels and justifications for a human to sanity-check.

Command: cargo test -p mem-cli label (add -- --ignored for a8)

False pass:

• Keying labels by t. A re-chunk shifts every t, the labels silently misalign, and the training set is quietly wrong in a way nothing downstream can detect.
• Dropping the justification to save space. M5.2 then has nothing to analyse and the calibration step degenerates into a single agreement number with no way to understand it.

Traps

• Batching several chunks into one labeling prompt to save calls. The labels become order-dependent and the 16K context is exceeded on the third chunk.
• Treating the 32B's labels as ground truth. They are a proxy, and M5.2 is the task that measures how good a proxy.

---

Background: DESIGN.md — P6 · paper §3.2.1