135 lines
6.7 KiB
Markdown
135 lines
6.7 KiB
Markdown
# M3.6.6 — M3.6 composition gate
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M3.6 — Reference corpora |
|
||
| Size | M — 1–3 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | gate |
|
||
| Spec | inlined below |
|
||
| Blocks | all of M3.6 |
|
||
| Depends | M3.6.1, M3.6.2, M3.6.4, M3.6.5 |
|
||
|
||
## Goal
|
||
|
||
Answer the question no single task in this phase can: **did adding documentation
|
||
change the memory system?** It must not have.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
Every task here was verified alone. What none of them own is the property that
|
||
makes the phase safe: a corpus is additive to *retrieval* and invisible to
|
||
*everything else*. Three ways that can silently fail, and this gate exists for
|
||
them.
|
||
|
||
**1. The M1.8 metric can be gamed by accident.** Update-rate is
|
||
`chunks_used / chunks_seen`, and M1 fails above 30%. Documentation is
|
||
evidence-free against almost any standing question, so a corpus routed through
|
||
the controller would push the ratio *down* and make M1.8 easier to pass while the
|
||
memory got worse. Any implementation that improves a quality metric by adding
|
||
unrelated text has inverted it. The gate re-runs M1.8 and asserts the numbers are
|
||
**unchanged**, not merely still-passing.
|
||
|
||
**2. Retrieval quality can degrade without any test noticing.** Each task asserts
|
||
its own behaviour on its own fixture. The composite risk is a corpus that
|
||
outcompetes real project memory in the candidate pool — invisible to M3.6.5's
|
||
unit fixture, obvious on the real poimen log with a real corpus loaded.
|
||
|
||
**3. The cycle guard has two producers now.** M4.2 writes skills to the manifest,
|
||
M3.6.4 writes reference sections. They share a normaliser. Skills exclusion
|
||
regressing when a corpus is added is the failure that no test in either phase
|
||
catches, because each tests only its own kind.
|
||
|
||
**Swappable parts.** The phase claims two seams are real: `DocCorpusSource` is
|
||
just another `RecordSource`, and the corpus is just another projection input.
|
||
Prove both — swap the doc tree for a differently-shaped one and re-run, and
|
||
rebuild the whole store from the log with the corpus present.
|
||
|
||
## Steps
|
||
|
||
1. Establish the baseline: on a clean store, run `mem ingest --project poimen`
|
||
for all standing queries; record the M1.8 summary table.
|
||
2. `mem ref add --project poimen --corpus homelab-knowledge <tree>` against a
|
||
real corpus of at least 200 chunks.
|
||
3. Re-run the full ingest. Diff the M1.8 summary against the baseline.
|
||
4. Run the assertions below.
|
||
5. Emit `expected/m3.6-gate.txt` with the summary; commit it. Later runs diff
|
||
against it and a changed expectation is a reviewable claim, same rule as M1.8.
|
||
6. Sample 10 abstentions and 10 R hits; eyeball whether the floor is set sanely.
|
||
Advisory, as M1.8's judge audit is.
|
||
|
||
## Acceptance
|
||
|
||
- M1.8's numbers are identical before and after the corpus exists.
|
||
- Default query output is byte-identical before and after.
|
||
- No L1 or L2 node has an R parent.
|
||
- Skills exclusion behaviour is unchanged with a corpus loaded.
|
||
- Drop and rebuild reproduces the mixed store byte-identically.
|
||
|
||
## Verify
|
||
|
||
**Harness:** live gateway, real corpus, real poimen log. Long-running; nightly or
|
||
on-demand, `#[ignore]` by default, same posture as M1.8.
|
||
|
||
**Integration test** — `tests/it_m3_6_gate.rs`:
|
||
1. `a1_update_rate_identical` — per standing query, assert update-rate before and
|
||
after the corpus is added is equal, not merely both under 0.30. Equality is
|
||
the assertion; a threshold check here would pass the exact failure described
|
||
above.
|
||
2. `a2_chunks_seen_identical` — `chunks_seen` per run is unchanged, proving no R
|
||
chunk entered the recurrence.
|
||
3. `a3_no_controller_calls_during_ref_ingest` — run `mem ref add` under a chat
|
||
transport that panics on request; assert it completes. Embeddings are allowed,
|
||
controller calls are not, so the fake must distinguish the two endpoints.
|
||
4. `a4_default_query_byte_identical` — snapshot default `mem query` output for 10
|
||
fixed questions before and after; assert byte-identical.
|
||
5. `a5_no_r_parents` — `SELECT count(*) FROM memory_edge WHERE parent_sha IN
|
||
(SELECT sha256 FROM memory_node WHERE level='R')` is 0 on the live store.
|
||
6. `a6_l2_stream_excludes_r` — re-run L2 synthesis; assert its input stream
|
||
contained only L1 nodes and the resulting L2 memories have no R ancestor.
|
||
7. `a7_skill_exclusion_unregressed` — re-run M4.2's fixtures against the store
|
||
with the corpus loaded; assert identical exclusion decisions.
|
||
8. `a8_rebuild_mixed_store` — drop database and vault, `mem rebuild --from-log`,
|
||
assert byte-identical across all four levels.
|
||
9. `a9_source_seam_swappable` — point `mem ref add` at a structurally different
|
||
tree (deep nesting, no headings in one file, one non-UTF8 file) and assert it
|
||
ingests or fails cleanly, never partially.
|
||
10. `a10_corpus_does_not_starve_recall` — for 10 project questions, assert the
|
||
top-5 default hits are the same nodes as the pre-corpus baseline, with the
|
||
corpus present in the table.
|
||
11. `a11_m5_export_excludes_r` — run the M5.3 training-corpus export shape; assert
|
||
zero R records appear. R carries no gate decision, so its presence would
|
||
poison `r_update` labels with rows that have no ground truth.
|
||
|
||
**Command:** `cargo test --workspace m3_6_gate -- --ignored --nocapture`
|
||
|
||
**False pass:**
|
||
- Asserting update-rate is still below 30% instead of unchanged. That is the
|
||
precise shape that goes green while the gate is being fed documentation —
|
||
assertion 1 must be equality.
|
||
- Running the gate with a corpus small enough not to matter. 200 chunks is a
|
||
floor, not a suggestion; below it, assertions 4 and 10 pass because the corpus
|
||
never reaches the candidate pool.
|
||
- Allowing `a3`'s fake transport to reject all HTTP. Reference ingest legitimately
|
||
calls the embeddings endpoint; a blanket panic passes the assertion for the
|
||
wrong reason and would also pass if ingest did nothing at all.
|
||
- Rebuilding into a fresh database rather than dropping the live one. A rebuild
|
||
that never exercises deletion has not proved the projections are droppable.
|
||
|
||
## Traps
|
||
|
||
- Comparing M1.8 summaries by eye. The numbers move in the third decimal when the
|
||
gateway is under load; the committed `expected/` file plus an explicit
|
||
tolerance is the only version of this that stays honest over months.
|
||
- Treating a changed baseline as a corpus problem. If update-rate shifts, first
|
||
confirm the gateway model has not changed underneath — `reasoning` and the 3B
|
||
controller are both moving targets, and misattributing that to this phase burns
|
||
a day.
|
||
- Skipping assertion 11 because M5 is not built. The export *shape* is checkable
|
||
now, and discovering R in the training corpus during M5.3 means re-running an
|
||
expensive labelling pass.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — reference corpora, the tier model · [M1.8](M1.8-m1-gate.md) · [M4.2](M4.2-derived-filter.md)
|