(plan) system review and break down plans

2026-08-19 09:52:07 -07:00
commit 9d28b63ff2
46 changed files with 5620 additions and 0 deletions
@@ -0,0 +1,99 @@
+# M1.8 — M1 composition gate
+
+| Field | Value |
+|---|---|
+| Phase | M1 — Gated loop at L1 |
+| Size | M — 1–3 days |
+| Status | ⬜ Not started |
+| Flags | gate |
+| Spec | inlined below |
+| Blocks | all of M1 |
+
+## Goal
+
+Answer the only question that matters at this stage: **did we build a gate, or an
+expensive summarizer?**
+
+## Facts (inlined — no spec read needed)
+
+This gate runs against the **live gateway on a real project** and asserts
+properties of the resulting log. It is the first phase that costs money in wall
+clock, and the first that can fail for reasons no unit test can see.
+
+Two thresholds, both from the paper:
+
+**Update-rate < 30%.** Agent transcripts are ~43% tool results, mostly
+evidence-free. Paper Figure 6 shows the failure mode directly: the ungated
+MemAgent's memory climbs to its 1024-token ceiling and saturates, after which
+"the accumulated noise can further impede subsequent updates". A high update-rate
+is that curve starting.
+
+**Memory size flat, not climbing.** Plot `memory.tokens` against `t`. GRU-Mem's
+curve is low and roughly flat; the ungated curve rises to the cap and stays
+pinned. A monotonic climb means the gate is open too often even if the rate
+looks acceptable.
+
+Third property, cheap and load-bearing: **the same run twice produces the same
+chunk hashes**. Temperature affects the model's text, not the chunking; if chunk
+shas differ between runs, identity includes something it should not (M0.2) and
+every projection will churn.
+
+## Steps
+
+1. Run `mem ingest --project poimen` for all standing queries against the live
+   gateway.
+2. Compute per query: update-rate, memory-token series, parse-failure count,
+   elapsed.
+3. Assert the thresholds below.
+4. Emit `expected/m1-gate.txt` with the summary table; commit it. Subsequent runs
+   diff against it, and a changed expectation is a reviewable claim.
+5. Sample 20 gate decisions and have the 32B `reasoning` model audit them; report
+   agreement. Advisory at this gate, and the seed of M5.2's calibration.
+
+## Acceptance
+
+- Update-rate < 30% on every standing query.
+- Memory tokens ≤ 1024 and not monotonically increasing.
+- Parse-failure rate < 5%.
+- Chunk shas stable across two runs.
+
+## Verify
+
+**Harness:** live gateway. Long-running; a nightly or on-demand job, not
+per-push.
+
+**Integration test** — `tests/it_m1_gate.rs`, all `#[ignore]` by default:
+1. `a1_update_rate_under_threshold` — per query, assert < 0.30, print actual.
+2. `a2_memory_bounded` — every `memory.tokens` ≤ 1024.
+3. `a3_memory_not_climbing` — fit a line to tokens vs `t`; assert the slope is
+   below a small positive bound. Do not assert non-increasing — a legitimately
+   growing memory rises early then plateaus.
+4. `a4_parse_failure_rate` — `parse_failed` / turns < 0.05.
+5. `a5_chunk_sha_stable` — two runs, same chunk shas in the same order.
+6. `a6_evidence_traceable` — every `evidence` sha appears as a parent of some
+   `memory` record.
+7. `a7_judge_audit` — sample 20 decisions, ask the 32B model, print agreement.
+   Advisory; does not fail the gate.
+
+**Command:** `cargo test --workspace m1_gate -- --ignored --nocapture`
+
+**False pass:**
+- Running the gate on a tiny `--limit`. Update-rate on the first 20 chunks is
+  noise; the failure mode is cumulative and needs a full project.
+- Asserting the rate without printing it. The number is the artifact — a run at
+  29% passes and is telling you something a boolean hides.
+- Asserting memory is non-increasing rather than bounded-slope. Real memory does
+  grow early, and a strict assertion here fails on correct behaviour, which
+  trains everyone to skip the gate.
+
+## Traps
+
+- Treating a high update-rate as a model-quality problem first. Check the
+  prompt (M1.3 golden file) and the parser (M1.4 defaults) before blaming the
+  3B model — a parser with `unwrap_or(true)` produces exactly this symptom.
+- Tuning the threshold to whatever the first run produced. 30% comes from the
+  corpus composition; moving it to accommodate a bad result deletes the gate.
+
+---
+
+Background: [DESIGN.md](../DESIGN.md) — Verification · paper Fig 6, §4.2