109 lines
4.2 KiB
Markdown
109 lines
4.2 KiB
Markdown
|
# M5.1 — `mem label` — evidence labeler
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M5 — Post-training |
|
|||
|
|
| Size | M — 1–3 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M1.6 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Produce the per-chunk ground truth `U_t` that `r_update` needs, since this corpus
|
|||
|
|
does not come with evidence labels.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
Paper `r_update`:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
r_update_t = +1 if U_t is correct
|
|||
|
|
-1 if U_t is incorrect
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
"Correct" means: for chunks containing evidence for `Q`, the agent should emit
|
|||
|
|
`<check>yes</check>`; for chunks without, `<check>no</check>`. That requires
|
|||
|
|
knowing which chunks contain evidence.
|
|||
|
|
|
|||
|
|
The paper had it for free — synthetic NIAH tasks place the needle deliberately,
|
|||
|
|
and HotpotQA ships supporting facts. **We have neither.** Agent transcripts have
|
|||
|
|
no annotation of which turn contained the answer.
|
|||
|
|
|
|||
|
|
Cheapest honest substitute: **distant supervision from the 32B model.** Ask
|
|||
|
|
`reasoning` (DeepSeek-R1-Distill-Qwen-32B, vLLM) per `(question, chunk)` whether
|
|||
|
|
the chunk contains evidence. It is ~10× the controller's size and sees each chunk
|
|||
|
|
independently, without the memory state that might bias the 3B's decision.
|
|||
|
|
|
|||
|
|
This inherits the labeler's bias, which is why M5.2 exists and must run before
|
|||
|
|
anyone trains on these labels.
|
|||
|
|
|
|||
|
|
Constraint: `reasoning` has a **16384 total context** and vLLM rejects
|
|||
|
|
`input + max_tokens > 16384`. A 5000-token chunk plus question plus instructions
|
|||
|
|
fits with room; keep `max_tokens` small (labels are one token of signal) and do
|
|||
|
|
not batch chunks into one prompt.
|
|||
|
|
|
|||
|
|
The labeler emits a binary label plus a short justification. Keep the
|
|||
|
|
justification — it is what makes M5.2's disagreement analysis possible.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. `mem label --project P --query Q` reads chunks from the log.
|
|||
|
|
2. Per chunk, prompt `reasoning`: question, chunk, "does this contain evidence for
|
|||
|
|
the question? Answer yes or no, then one sentence why."
|
|||
|
|
3. Send **no tools** — the reasoning route rejects any request carrying them.
|
|||
|
|
4. Write `label/<project>/<query-id>.jsonl`:
|
|||
|
|
`{"chunk_sha":"...","t":7,"label":true,"why":"...","model":"reasoning","ts":"..."}`.
|
|||
|
|
5. Resumable: skip chunks already labelled.
|
|||
|
|
6. Report the label rate — the fraction of chunks the labeler calls evidence.
|
|||
|
|
Compare it to the controller's update-rate from M1.7; a large gap is the
|
|||
|
|
finding, not a bug.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- Every chunk in the log gets exactly one label.
|
|||
|
|
- Labels key on `chunk_sha`, so they survive re-chunking only if content is
|
|||
|
|
unchanged.
|
|||
|
|
- Resume skips completed work.
|
|||
|
|
- Label rate is reported alongside the controller's update-rate.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** scripted client offline; one `#[ignore]` live run.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_label.rs`:
|
|||
|
|
1. `a1_one_label_per_chunk` — no duplicates, no gaps against the log's chunks.
|
|||
|
|
2. `a2_keyed_by_sha` — labels reference `chunk_sha`, not `t`, so reordering the
|
|||
|
|
log does not corrupt them.
|
|||
|
|
3. `a3_no_tools_sent` — assert the request body has no `tools` key.
|
|||
|
|
4. `a4_context_budget` — assert every labeling prompt is under
|
|||
|
|
16384 − max_tokens.
|
|||
|
|
5. `a5_resume` — label, rerun, assert zero new calls.
|
|||
|
|
6. `a6_justification_kept` — every label has non-empty `why`.
|
|||
|
|
7. `a7_rate_reported` — the summary prints both label rate and the controller's
|
|||
|
|
update-rate.
|
|||
|
|
8. `a8_live` — `#[ignore]`; 20 real chunks through `reasoning`; print the labels
|
|||
|
|
and justifications for a human to sanity-check.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p mem-cli label` (add `-- --ignored` for a8)
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Keying labels by `t`. A re-chunk shifts every `t`, the labels silently
|
|||
|
|
misalign, and the training set is quietly wrong in a way nothing downstream can
|
|||
|
|
detect.
|
|||
|
|
- Dropping the justification to save space. M5.2 then has nothing to analyse and
|
|||
|
|
the calibration step degenerates into a single agreement number with no way to
|
|||
|
|
understand it.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Batching several chunks into one labeling prompt to save calls. The labels
|
|||
|
|
become order-dependent and the 16K context is exceeded on the third chunk.
|
|||
|
|
- Treating the 32B's labels as ground truth. They are a *proxy*, and M5.2 is the
|
|||
|
|
task that measures how good a proxy.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — P6 · paper §3.2.1
|