poimen-memory/tasks/M1.4-gate-response-parser.md

# M1.4 — Gate-response parser

| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.3 |

## Goal

Turn the model's tagged output into `(U_t, M̂_t, E_t)`, strictly — because a
lenient parser silently fabricates gate decisions.

## Files

| Action | Path |
|---|---|
| Create | `crates/mem-core/src/gate_parser.rs` — `parse_gate_response()`, `GateResponse`, `ParseError` |
| Modify | `crates/mem-core/src/lib.rs` — add `pub mod gate_parser;` and re-exports |
| Create | `fixtures/gate-response-valid.txt` — well-formed model output |
| Create | `fixtures/gate-response-nested-think.txt` — nested `<think>` tags |
| Create | `tests/it_gate_parser.rs` — integration tests (workspace root) |

## Dependencies

**None new.** Use `str::find()` and `str::rfind()` for tag extraction. No regex
crate needed — the tags are simple XML-like delimiters, not a grammar.

## Existing code to reuse

- Pattern reference: `lesson.rs` uses similar string scanning for error markers
  (`markers()`, `GENERIC_MARKERS`). Same technique, different tags.
- `thiserror` already in `mem-core` deps for error enum derivation.

## Expected input/output

```
Input:
  <think>
  This chunk shows a kubectl error. The user fixed it by adding --namespace.
  </think>
  <check>yes</check>
  <update>kubectl get pods fails without --namespace; fixed by adding --namespace=kube-system</update>
  <next>continue</next>

Output:
  GateResponse {
    think: "This chunk shows a kubectl error. The user fixed it by adding --namespace.",
    update_gate: true,
    candidate: "kubectl get pods fails without --namespace; fixed by adding --namespace=kube-system",
    exit_gate: false,
  }
```

## Facts (inlined — no spec read needed)

Expected response shape:

```
<think>...</think>
<check>yes|no</check>
<update>candidate memory, or the previous memory verbatim</update>
<next>continue|end</next>
```

Semantics, from the paper:

| tag | value | meaning |
|---|---|---|
| `<check>` | `yes` | `U_t = true` — memory becomes `M̂_t` |
| `<check>` | `no` | `U_t = false` — memory stays `M_{t-1}`, chunk discarded |
| `<next>` | `continue` | `E_t = false` |
| `<next>` | `end` | `E_t = true` |

**Strict parsing is the design, matching the paper's `r_format`:** it awards 1
only when *every* turn in the trajectory parses, 0 otherwise, "because we can not
infer whether the incorrect format is caused by the previous erroneous parsing".

So: exactly one of each tag, properly closed, `<check>` content exactly `yes` or
`no` after trimming, `<next>` exactly `continue` or `end`. Anything else is a
`ParseError` naming which tag failed and carrying the raw text.

**A parse failure must not default.** Defaulting `U_t` to `false` silently drops
evidence; defaulting to `true` pollutes memory. The loop (M1.5) decides the
retry policy; the parser only reports.

Reasoning models emit `<think>` natively, which can nest or repeat. Extract by
locating the *last* `</think>` before the first `<check>`, not by regex over the
whole body.

## Steps

1. `parse_gate_response(&str) -> Result<GateResponse, ParseError>` in `mem-core`.
2. `GateResponse { think: String, update_gate: bool, candidate: String, exit_gate: bool }`.
3. Reject duplicates of any tag, a missing tag, an unclosed tag, and any
   `<check>`/`<next>` value outside the allowed set.
4. `ParseError` variants name the tag and include the raw response, truncated.
5. Trim surrounding whitespace inside tags; do not otherwise normalise — memory
   text is preserved verbatim.
6. When `U_t = false`, still capture `candidate` so the log can show what the
   model *would* have written. The loop ignores it; the record is diagnostic.

## Acceptance

- All four tags parse from a well-formed response.
- Every malformed shape errors, naming the failing tag.
- No input produces a default `U_t` or `E_t`.

## Verify

**Harness:** table-driven over recorded real responses plus hand-built malformed
cases. Capture real ones with `MEM_LLM_RECORD` from M1.1.

**Integration test** — `tests/it_gate_parser.rs`:
1. `a1_wellformed_yes_continue` — `U=true`, `E=false`, candidate matches.
2. `a2_wellformed_no_end` — `U=false`, `E=true`.
3. `a3_missing_check_errors` — error names `check`.
4. `a4_duplicate_update_errors` — two `<update>` blocks error.
5. `a5_bad_check_value_errors` — `<check>maybe</check>` errors, message shows the
   value.
6. `a6_unclosed_tag_errors` — `<update>` never closed.
7. `a7_nested_think` — a response with `<think>` inside `<think>` still finds the
   right boundary.
8. `a8_no_defaults` — property test over 1000 random mutations of a valid
   response: every result is either an exact parse or an error, never a
   silently-defaulted `GateResponse`.
9. `a9_real_responses` — every recorded real response parses.

**Command:** `cargo test --test it_gate_parser`

**False pass:**
- A regex that finds the first `<check>` and stops. It passes 1–2 and silently
  accepts duplicates, which is assertion 4's job.
- Testing only hand-written responses. Real 3B output has whitespace,
  markdown fences and stray prose the author would not think to write —
  assertion 9 is the only one that sees it.
- Omitting assertion 8. A parser with `unwrap_or(false)` anywhere passes every
  positive test and quietly halves the update rate.

## Traps

- Defaulting on parse failure. The one thing that must not happen: a fabricated
  gate decision is indistinguishable from a real one in the log, and it poisons
  the M5 training data at the source.
- Normalising the candidate memory (collapsing whitespace, stripping markdown).
  Memory text is content; the hash and every downstream projection depend on it
  being verbatim.

---

Background: [DESIGN.md](../DESIGN.md) — Architecture · paper §3.2.1 `r_format`