115 lines
4.7 KiB
Markdown
115 lines
4.7 KiB
Markdown
# M5.3 — Training corpus export
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M5 — Post-training |
|
||
| Size | M — 1–3 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M5.1 |
|
||
|
||
## Goal
|
||
|
||
Turn the log plus labels into trajectories verl can train on — the boundary
|
||
between the Rust side and the Python side.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
**The JSONL log is the boundary.** Rust produces it; Python consumes it. Nothing
|
||
else crosses, which is what keeps the two halves independent.
|
||
|
||
A training example is a **trajectory**, not a turn: the paper's advantage mixes a
|
||
trajectory-level term with a turn-level one (`Â = α·Â_traj + (1−α)·Â_turn`,
|
||
α=0.9), so turns must stay grouped by run.
|
||
|
||
Per turn, verl needs: the exact prompt sent, the exact response, and the rewards.
|
||
|
||
```jsonl
|
||
{"trajectory_id":"01HXYZ","turns":[
|
||
{"t":1,"prompt":"<full assembled prompt>","response":"<think>...</next>",
|
||
"r_update":-1,"parsed":true},
|
||
...],
|
||
"r_exit":-0.5,"r_format":1,"r_outcome":null}
|
||
```
|
||
|
||
Reward assembly, from the paper:
|
||
|
||
- `r_update_t` = +1 if the recorded `U_t` matches M5.1's label, −1 otherwise.
|
||
- `r_exit` — one value per trajectory: `0` if `t_exit == t_last_evidence`,
|
||
`−0.75` if earlier, `−0.5` if later. `t_last_evidence` is the largest `t` whose
|
||
label is true. **Note L1 runs never exit** (exit gate off), so every L1
|
||
trajectory is a "late" exit at −0.5 unless the exit signal is taken from the
|
||
recorded `E_t` rather than the loop's behaviour — take it from the record.
|
||
- `r_format` = 1 only if **every** turn in the trajectory parsed, 0 otherwise.
|
||
Strict, because a malformed turn may be caused by the previous one.
|
||
- `r_outcome` is null. We have no answer-correctness signal; the paper's
|
||
`is_equiv(A, Â)` has no analogue in extraction. Say so explicitly rather than
|
||
fabricating one.
|
||
|
||
The prompt must be the **exact bytes sent**, not re-assembled. Re-assembly drifts
|
||
from what the model actually saw the moment M1.3's template changes.
|
||
|
||
## Steps
|
||
|
||
1. `mem export --project P --format verl --out corpus/`.
|
||
2. Join log turns to labels by `chunk_sha`.
|
||
3. Reconstruct each turn's prompt from the recorded request if `MEM_LLM_RECORD`
|
||
captured it; otherwise fail loudly rather than re-assembling.
|
||
4. Compute rewards as above; carry `r_outcome: null` through.
|
||
5. Group by run into trajectories, ordered by `t`.
|
||
6. Emit a summary: trajectories, turns, positive/negative `r_update` split,
|
||
`r_format` pass rate, `t_last_evidence` distribution.
|
||
7. Refuse to export if M5.2's κ is below the threshold or absent.
|
||
|
||
## Acceptance
|
||
|
||
- Turns are grouped into trajectories, ordered.
|
||
- Prompts are byte-exact recordings, never re-assembled.
|
||
- `r_format` is 0 for a trajectory with any unparsed turn.
|
||
- Export is refused without calibration.
|
||
|
||
## Verify
|
||
|
||
**Harness:** log fixture with known labels and a deliberately unparsed turn.
|
||
|
||
**Integration test** — `tests/it_export.rs`:
|
||
1. `a1_trajectory_grouping` — turns grouped by run, `t` ascending, none lost.
|
||
2. `a2_r_update_signs` — matching label → +1, mismatching → −1, checked per turn.
|
||
3. `a3_r_format_strict` — a trajectory with one unparsed turn scores 0 overall,
|
||
not per turn.
|
||
4. `a4_r_exit_from_record` — assert `r_exit` derives from the recorded `E_t`, not
|
||
from whether the loop stopped. A fixture where the gate said `end` at t=5 but
|
||
the loop continued must score as an exit at 5.
|
||
5. `a5_prompt_is_recorded_bytes` — assert the exported prompt equals the recorded
|
||
request body; corrupt the recording and assert export fails rather than
|
||
silently re-assembling.
|
||
6. `a6_r_outcome_null` — assert the field is present and null, not omitted and not
|
||
zero.
|
||
7. `a7_refuses_without_calibration` — no κ file → non-zero exit naming M5.2.
|
||
8. `a8_summary_counts` — reported splits match a hand count on the fixture.
|
||
|
||
**Command:** `cargo test -p mem-cli export`
|
||
|
||
**False pass:**
|
||
- Re-assembling prompts at export time. Every assertion except 5 passes, and the
|
||
policy is trained on prompts the model never saw — which shows up as a training
|
||
run that will not converge, with no obvious cause.
|
||
- Applying `r_format` per turn. It looks more granular and it is wrong: the paper
|
||
is explicit that the strictness exists because a bad turn may be caused by the
|
||
previous one.
|
||
- Emitting `r_outcome: 0` instead of null. Zero is a real reward value and the
|
||
trainer will use it as signal.
|
||
|
||
## Traps
|
||
|
||
- Deriving `r_exit` from loop behaviour at L1. The gate is switched off there, so
|
||
every trajectory scores as a late exit and the exit signal becomes constant
|
||
noise the policy cannot learn from.
|
||
- Joining labels by `t`. Same failure as M5.1's trap — a re-chunk misaligns
|
||
everything silently.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — P6 · paper §3.2
|