chore: Archive M3.6 task files (all 6/6 complete)

2026-08-28 13:59:32 -07:00
parent c1fcdb9769
commit 3bff6e7380
6 changed files with 0 additions and 687 deletions
@@ -1,181 +0,0 @@
 # M3.6.2 — Level R: Obsidian reference indexing and rebuild parity
 | Field | Value |
 |---|---|
 | Phase | M3.6 — Reference corpora |
 | Size | M — 1–3 days |
 | Status | ✅ COMPLETE |
 | Flags | — |
 | Spec | inlined below |
 | Blocks | M3.6.6 |
 | Depends | M3.6.1, M1.6, M2.3, M2.4, M2.5, M2.6, Obsidian service (M2.5) |
 ## Goal
 Fetch reference documents from the live Obsidian vault (REST API), chunk them
 via M3.6.1's heading-boundary logic, land them in the log as reference records,
 index them in Postgres, and prove rebuild parity (drop + rebuild from log =
 byte-identical index state).
 ## Facts (inlined — no spec read needed)
 **Obsidian is the source of truth for reference documents.** The Obsidian REST API
 (deployed in M2.5, accessible at `http://obsidian-server.poimen.svc.cluster.local:8080`)
 provides live access to vault files via:
 ```bash
 GET /api/vault/listFiles                          # List all .md files
 GET /api/vault/readFile?path=kubectl.md           # Read file contents
 GET /api/vault/listFolders                        # Explore structure
 ```
 **Reference record format (in JSONL log):**
 ```jsonl
 {"kind":"reference","level":"R","source":"obsidian://poimen-vault/kubectl.md",
 "doc_sha":"ab12…","heading_path":"kubectl.md > Common Issues > CrashLoopBackOff",
 "sha256":"cd34…","t":7,"run_id":"ref-obsidian-2026-08-21T10:02:11Z","text":"…"}
 ```
 **No vault projection.** Unlike M3.6.1 (DocCorpusSource which reads from local filesystem),
 M3.6.2 reads directly from Obsidian REST API and indexes into Postgres. The Obsidian
 vault remains the single source of truth; `mem rebuild --from-log` re-fetches from
 Obsidian to regenerate indexes.
 `level = 'R'`, `query_id = NULL` (R answers no standing question), `source` holds
 the Obsidian URI (`obsidian://vault-name/file.md`). `doc_sha` is whole-document hash;
 `sha256` is chunk hash (primary identity).
 The embedding goes to `memory_vector(kind='text')`, not to a column on the node.
 R gets **no symptom projection** — M3.7.8 generates those for L1 and L2 only.
 **R writes no edges.** Provenance is the Obsidian URI, not internal edges. Enforced
 in `mem verify` (M3.6.4).
 **Rebuild parity is the whole point.** `mem rebuild --from-log` must:
 1. Drop R rows from Postgres
 2. Drop R vectors from OpenSearch (M8.2)
 3. Re-fetch documents from Obsidian REST API
 4. Re-chunk via M3.6.1 heading logic
 5. Re-index into Postgres + OpenSearch
 6. Result must be byte-identical (same shas, same vector embeddings)
 ## Steps
 1. **ObsidianRefSource** in `mem-ingest`: Implement `ChunkedSource` that:
   - Calls Obsidian REST API (`/api/vault/listFiles`)
   - Filters for `.md` files in allowed paths (e.g., `docs/`, `reference/`)
   - Fetches each file via `/api/vault/readFile?path=...`
   - Chunks via M3.6.1's heading boundary logic
   - Yields `Record { kind: Reference, ... }`
 2. Add `Reference` variant to log record enum in `mem-core` (already exists from
   M2.3, just needs the Obsidian source)
 3. `mem-store`: Insert R nodes with `kind='text'` vector; assert no edge insert
   names an R sha as parent (repository boundary check).
 4. Dual-write in M8.2: When R records are indexed, write both to Postgres and
   OpenSearch (L2 already does this; extend for R).
 5. **Rebuild: extend M2.6 harness** for `mem rebuild --from-log`:
   - When replaying R records from log, re-fetch original files from Obsidian
   - Re-chunk via M3.6.1 logic
   - Regenerate embeddings (deterministic, so shas match)
   - Assert byte-identical state vs. original indexing
 6. Integration: Update `mem query` to include R results from hybrid search (M8)
 ## Acceptance
 - `ObsidianRefSource` fetches all `.md` files from Obsidian REST API.
 - Reference records (R) with `source=obsidian://...` land in the log.
 - R rows persist in Postgres with `query_id IS NULL` and `source` set to Obsidian URI.
 - Each R chunk gets a `kind='text'` embedding in OpenSearch.
 - The M2.3 constraint accepts `R` and rejects `L3`.
 - `mem rebuild --from-log`: drop Postgres R rows + OpenSearch R vectors, re-fetch
  from Obsidian, re-chunk, re-embed, re-index; result is byte-identical (same shas,
  same embedding vectors).
 - No hidden inputs: Obsidian REST API is the only external dependency for R.
 ## Verify
 **Harness:** Mock Obsidian REST API (deterministic file list + content). M2.6 rebuild
 harness extended to cover R records. Deterministic embedder so shas are stable.
 **Integration test** — `tests/it_obsidian_ref_source.rs` + `tests/it_level_r_storage.rs`:
 ### ObsidianRefSource tests:
 1. `a1_fetches_md_files` — Mock API returns 3 `.md` files; source emits 3 documents.
 2. `a2_chunks_by_heading` — One source document with 5 headings yields 5 chunks
   (uses M3.6.1 heading boundary logic).
 3. `a3_generates_doc_sha` — Document SHA (SHA256 of whole file content) is
   consistent across fetches.
 4. `a4_generates_chunk_shas` — Each chunk sha is deterministic: SHA256(tool +
   heading_path + text).
 5. `a5_sets_obsidian_uri` — Every R record has `source=obsidian://vault-name/path.md`.
 6. `a6_skips_non_md` — API returns `.json` and `.txt` files; source ignores them.
 ### Level R storage tests:
 7. `a7_record_roundtrip` — Serialize R record to JSONL, deserialize; assert
   field-for-field equality.
 8. `a8_r_inserts_to_postgres` — Insert `level='R'` with `kind='text'` vector;
   assert Postgres row exists with correct `source` URI.
 9. `a9_r_inserts_to_opensearch` — Same chunk indexed in OpenSearch (M8.2
   dual-write); assert M8 vector store has the chunk.
 10. `a10_no_symptom_projection` — R chunks do not acquire `kind='symptom'`
    vectors (only L1/L2 get those from M3.7.8).
 11. `a11_query_id_null` — Every R row has `query_id IS NULL`; insertion with
    non-NULL query_id is rejected by M2.3's CHECK.
 12. `a12_no_edges_from_r` — After ingesting 10 R chunks, `SELECT count(*)
    FROM memory_edge WHERE parent_sha IN (SELECT sha256 FROM memory_node WHERE
    level='R')` equals 0.
 ### Rebuild parity tests:
 13. `a13_rebuild_drops_r_rows` — Ingest R, assert N rows exist. Call `mem
    rebuild --from-log`, assert same N rows exist (re-fetched, re-embedded,
    re-indexed).
 14. `a14_rebuild_byte_identical` — Snapshot Postgres R rows + OpenSearch R
    vectors before rebuild. Drop both. Run `mem rebuild --from-log` against
    a log containing R records. Assert Postgres rows and OpenSearch vectors
    are byte-identical (same row order, same field values, same vector embeddings).
 15. `a15_rebuild_idempotent` — Run rebuild twice; second run changes nothing.
    Snapshot comparison between first and second rebuild result is identical.
 **Command:** 
 ```bash
 cargo test -p mem-ingest obsidian_ref_source
 cargo test -p mem-store level_r
 cargo test -p mem-cli rebuild  # includes R records in fixture log
 ```
 **False pass:**
 - Asserting rebuild parity on a log with no R records. Assertion 14 requires
  mixed-level log (L0/L1/L2 *and* R) or it trivially passes.
 - Mocking Obsidian API with hardcoded responses. File hashes must match real
  Obsidian vault file SHA256 (use actual Obsidian instance or hash fixtures).
 - Checking edge count is zero before ingesting. Assertion 12 must run against
  populated corpus or it asserts empty table is empty.
 - Byte-identical comparison with normalization (sorting, ignoring order).
  Byte-identical means bytes; rebuild parity is broken if row order changes
  between runs.
 ## Traps
 - Reusing `run_id` semantics from the gated loop. R has no run in the recurrence
  sense; use a synthetic `ref-obsidian-<timestamp>` and do not let it collide
  with a real ingest run in queries that group by `run_id`.
 - Caching Obsidian API responses across rebuild runs. Rebuild must re-fetch from
  Obsidian REST API every time (no cache) to ensure file contents and shas are
  always in sync with live vault.
 - Assuming Obsidian file list is sorted. `listFiles` order is arbitrary; source
  must sort filenames before chunking to ensure deterministic shas across runs.
 - Embedding each R chunk independently. M3.6.1 (DocCorpusSource) uses the same
  `m` parameter (batching) and deterministic embedder model to ensure chunk shas
  are stable; M3.6.2 must use identical setup.
 - Dropping the check constraint instead of widening it. Assertion 11 exists
  because `DROP CONSTRAINT` alone passes every other assertion in this file.
 ---
 Background: [DESIGN.md](../DESIGN.md) — reference corpora, storage schemas
@@ -1,142 +0,0 @@
 # M3.6.4 — Reference text cannot re-enter as evidence
 | Field | Value |
 |---|---|
 | Phase | M3.6 — Reference corpora |
 | Size | M — 1–3 days |
 | Status | ✅ COMPLETE |
 | Flags | — |
 | Spec | inlined below |
 | Blocks | M3.6.6 |
 | Depends | M3.6.3, M4.2 |
 ## Goal
 Stop a retrieved manual page from coming back through the front door as a project
 finding.
 ## Facts (inlined — no spec read needed)
 The cycle is M4.2's, with documentation substituted for emitted skills:
 ```
 agent queries memory, gets an R section
        │
        ▼
 section is pasted into the agent's context
        │
        ▼
 appears verbatim in that session's transcript
        │
        ▼
 transcript ingested; the gate sees upstream doc text
        │
        ▼
 "kubectl describe shows events" becomes an L1 project memory
 ```
 The gate is *right* to accept it — the chunk genuinely contains information about
 the question. That is what makes this dangerous rather than merely noisy: no
 threshold tuning catches it, because the text really is relevant. Only knowing
 that the system emitted the text itself distinguishes the two cases.
 **Reuse M4.2, do not rebuild it.** M4.2 already computes normalised shingle
 overlap against an artifact manifest and tags matching records `derived: true`,
 excluding them from evidence while keeping them in the log so the exclusion is
 auditable. R chunks are a second artifact kind in that same manifest. A parallel
 matcher would drift from it and double the tuning surface.
 ```jsonl
 {"kind":"reference","name":"kubectl/common-issues","sha256":"cd34…","shingles":[…],"emitted_at":"…"}
 {"kind":"skill","name":"infra-root-causes","sha256":"ab12…","shingles":[…],"emitted_at":"…"}
 ```
 **Threshold pressure differs by kind and this is the real work.** A skill is
 emitted once and quoted rarely. Documentation is quoted constantly and partially
 — one command from a fifty-line cheatsheet. Shingle overlap against a whole R
 chunk will sit far below M4.2's 0.8 default for exactly the case that matters, so
 matching must be at section granularity with its own threshold, tuned and logged
 separately. One shared matcher, two configured thresholds.
 **A false positive here is costly and must stay visible.** Excluding a genuine
 discussion *about* `kubectl` because it quotes two lines of the cheatsheet
 silently drops real evidence. Every exclusion emits `derived_excluded` naming the
 matched artifact, and `mem verify` can list them for audit.
 ## Steps
 1. Generalise M4.2's manifest to `vault/.artifacts.jsonl` with a `kind` field;
   keep skills writing to it unchanged.
 2. `mem ref add`/`sync` append `kind: "reference"` entries per R chunk;
   tombstones remove them.
 3. Add per-kind thresholds to the matcher config; default reference threshold
   lower than the skill threshold, and record the value in the exclusion event.
 4. Extend `mem verify --derived-filter` to assert no L0 evidence node matches a
   live R artifact.
 5. `mem verify --exclusions` lists recent `derived_excluded` events with the
   matched artifact and overlap score, for false-positive review.
 ## Acceptance
 - A session transcript containing a verbatim R section is excluded from evidence.
 - The same transcript still appears in the log, tagged, with the match named.
 - A session that merely *mentions* the tool without quoting it is not excluded.
 - Skill exclusion behaviour from M4.2 is unchanged.
 - Removing a corpus removes its manifest entries; previously excluded text is not
  retroactively rewritten in the log.
 ## Verify
 **Harness:** fixture corpus ingested as R, plus three synthetic transcripts — one
 quoting a section verbatim, one paraphrasing it heavily, one discussing the tool
 without quoting. Deterministic embedder.
 **Integration test** — `tests/it_reference_cycle.rs`:
 1. `a1_verbatim_quote_excluded` — the quoting transcript produces zero L0
   evidence nodes; assert a `derived_excluded` event naming the R artifact.
 2. `a2_discussion_not_excluded` — the non-quoting transcript produces evidence
   normally. This is the false-positive guard and it is the assertion that fails
   when the threshold is set too low.
 3. `a3_partial_quote_caught` — the transcript quoting ~10 lines of a 50-line
   section is excluded, proving section-granularity matching rather than
   whole-chunk overlap.
 4. `a4_skill_path_unchanged` — run M4.2's own test fixtures; assert identical
   results before and after the manifest generalisation.
 5. `a5_exclusion_is_auditable` — every exclusion event carries artifact name,
   overlap score and the threshold in force.
 6. `a6_tombstone_removes_manifest_entry` — `mem ref rm`, then assert the R
   entries are gone from the manifest and the same transcript now ingests
   normally.
 7. `a7_verify_catches_leak` — hand-insert an L0 node whose text matches an R
   artifact; assert `mem verify --derived-filter` exits non-zero and names it.
 8. `a8_no_retroactive_log_edit` — after `rm`, assert prior `derived_excluded`
   records are still present and unmodified.
 **Command:** `cargo test -p mem-ingest reference_cycle && cargo test -p mem-cli verify`
 **False pass:**
 - Testing only the verbatim case. Verbatim is easy and a whole-chunk hash catches
  it; assertion 3 is the one that distinguishes a working matcher, and assertion
  2 is the one that proves it is not simply excluding everything that mentions
  the tool.
 - Asserting exclusion by checking evidence count is zero. A filter that is
  accidentally excluding *all* records also yields zero; assertion 2 has to run
  in the same test binary.
 - Reusing M4.2's threshold unchanged and declaring it done. The default is tuned
  for whole-artifact quoting; assertion 3 fails against it, which is the point.
 ## Traps
 - Registering R chunks in the manifest before they are committed to the log. A
  failed ingest then leaves manifest entries that exclude evidence for a corpus
  that does not exist, and the symptom is missing memories with no obvious cause.
 - Normalising differently in the two paths. If the shingler treats markdown
  tables differently at emit-time and at ingest-time, overlap collapses and the
  filter silently stops firing — same failure M4.2 already warns about, now with
  two producers to keep in step.
 - Letting the exclusion event omit the threshold. A tuning change makes every
  historical exclusion uninterpretable, and this filter will be tuned.
 ---
 Background: [DESIGN.md](../DESIGN.md) — reference corpora, skills · [M4.2](M4.2-derived-filter.md)
@@ -1,136 +0,0 @@
 # M3.6.5 — Query: filter-then-recall, R opt-in, relevance floor
 | Field | Value |
 |---|---|
 | Phase | M3.6 — Reference corpora |
 | Size | M — 1–3 days |
 | Status | ✅ COMPLETE |
 | Flags | — |
 | Spec | inlined below |
 | Blocks | M3.6.6 |
 | Depends | M3.6.2, M3.3, M3.2, M2.7 |
 ## Goal
 Make R reachable on request, unreachable by default, and stop the retriever
 answering questions it has no evidence for.
 ## Facts (inlined — no spec read needed)
 ```
 mem query "why did requests over 10KB fail?"          # L1,L2 — unchanged
 mem query --levels R "kubectl describe pod"           # reference only
 mem query --levels L1,L2,R "..."                      # both, R marked in output
 mem query "…" --min-score 0.4                         # override the floor
 ```
 **Filter before recall, not after.** M3.3 recalls `10×k` from HNSW and reranks
 down to `k`. A corpus is typically an order of magnitude larger than the project's
 own memory, so R rows compete for those 50 candidate slots even when the caller
 excluded them — and post-filtering then returns three results instead of five,
 quietly. The level predicate belongs in the SQL that drives the HNSW scan. The
 existing `(project, level)` index already supports it.
 **Abstention.** With a corpus loaded, every question has *something* moderately
 close, so unconditional top-k starts returning plausible prose for questions the
 memory cannot answer — worse than an empty result, because it reads as an answer.
 If the best post-rerank score is below the floor, return no hits and say why:
 ```
 no hits above relevance floor (best 0.21 < 0.35 threshold)
 try --min-score to lower it, or --levels R to search reference docs
 ```
 The floor applies to the **reranked** score, not cosine distance. M3.2's own
 fixture separates a relevant from an irrelevant passage by four orders of
 magnitude; cosine distance does not, which is why the floor cannot live at the
 recall stage.
 **R is visually distinct in output.** A reference hit prints its source URI and
 heading path where a project hit prints provenance. A caller must never have to
 infer from wording whether an answer came from this cluster's history or from
 upstream documentation.
 **R has no provenance walk.** M3.3 walks `memory_edge` one hop for L1 and two for
 L2. R has no edges by construction (M3.6.2), so the walk is skipped rather than
 returning empty — and `mem verify` gains the assertion that makes that safe.
 ## Steps
 1. Push the level filter into the recall query; assert candidate width is `10×k`
   *after* filtering.
 2. `--levels` accepts `R`; default remains `L1,L2`.
 3. Apply the relevance floor to reranked scores; `--min-score` overrides,
   `--min-score 0` disables.
 4. Abstention message names the best score, the threshold, and the two escapes.
 5. Render R hits with source URI and heading path; suppress the provenance walk.
 6. Exit code: abstention is exit 0 with no hits, not an error — it is a valid
   answer. Unresolvable project stays non-zero (M3.3 assertion 7).
 7. `mem verify --edges` asserts no `memory_edge` row names an R sha as parent.
 ## Acceptance
 - Default query over a database containing a large corpus returns exactly the
  same hits as before the corpus was added.
 - `--levels R` returns reference sections with URI and heading path.
 - A question with no good match returns nothing and explains itself.
 - Lowering `--min-score` surfaces the suppressed hits.
 - `mem verify` rejects a hand-inserted `L1 -> R` edge.
 ## Verify
 **Harness:** seeded database with the poimen log *plus* a reference corpus large
 enough to dominate raw recall — at least 10× the project node count. Live
 reranker for scoring assertions, deterministic embedder elsewhere.
 **Integration test** — `tests/it_query_levels.rs`:
 1. `a1_default_unchanged_by_corpus` — snapshot default query results before and
   after ingesting the corpus; assert byte-identical output. This is the
   headline assertion of the task.
 2. `a2_filter_before_recall` — instrument the repository; assert the SQL driving
   HNSW carries the level predicate and returns `10×k` rows post-filter, not
   `10×k` pre-filter then fewer.
 3. `a3_levels_r_returns_reference` — `--levels R` returns R nodes with source URI
   and heading path populated.
 4. `a4_floor_abstains` — a question with no relevant content returns zero hits,
   exit 0, message naming best score and threshold.
 5. `a5_floor_override_recovers` — same question with `--min-score 0` returns the
   suppressed hits, proving abstention is a floor and not a bug upstream.
 6. `a6_floor_applies_post_rerank` — construct a case where cosine is high and
   rerank is low; assert it is suppressed. The reverse ordering passes every
   other assertion here.
 7. `a7_r_hits_visually_distinct` — assert R hits carry no provenance block and do
   carry a URI, in both human and `--format json` output.
 8. `a8_no_edge_to_r` — hand-insert an `L1 -> R` edge; assert `mem verify --edges`
   exits non-zero and names the offending pair.
 9. `a9_mixed_levels_ordering` — `--levels L1,L2,R` returns both kinds ranked
   together with the level labelled on every row.
 **Command:** `cargo test -p mem-cli query_levels`
 **False pass:**
 - Running assertion 1 against a small corpus. If the corpus is smaller than the
  recall width, post-filtering and pre-filtering give the same answer and
  assertion 2 is the only thing separating them — the fixture size is part of the
  test.
 - Testing abstention with a query that matches nothing at all. Zero recall
  returns zero hits regardless of the floor; the fixture needs a *weak but
  non-empty* match, or assertion 4 passes with the floor unimplemented.
 - Asserting `--levels R` works without asserting the default excludes R. Both
  directions are the contract.
 ## Traps
 - Applying the floor to the first-stage cosine score. Cosine on `nomic` puts
  unrelated text closer than intuition suggests; a floor there either suppresses
  good hits or does nothing, depending on the corpus.
 - Treating abstention as an error exit. Callers wrap `mem query` in scripts; a
  non-zero exit for "no confident answer" turns a normal outcome into a pipeline
  failure and the floor gets disabled within a week.
 - Letting the reranker see 50 R candidates and 3 project candidates in a mixed
  query. The reranker is not calibrated across levels, and the corpus wins on
  fluency; recall per level, then merge.
 ---
 Background: [DESIGN.md](../DESIGN.md) — reference corpora, pgvector, retrieval
@@ -1,134 +0,0 @@
 # M3.6.6 — M3.6 composition gate
 | Field | Value |
 |---|---|
 | Phase | M3.6 — Reference corpora |
 | Size | M — 1–3 days |
 | Status | ⬜ Not started |
 | Flags | gate |
 | Spec | inlined below |
 | Blocks | all of M3.6 |
 | Depends | M3.6.1, M3.6.2, M3.6.4, M3.6.5 |
 ## Goal
 Answer the question no single task in this phase can: **did adding documentation
 change the memory system?** It must not have.
 ## Facts (inlined — no spec read needed)
 Every task here was verified alone. What none of them own is the property that
 makes the phase safe: a corpus is additive to *retrieval* and invisible to
 *everything else*. Three ways that can silently fail, and this gate exists for
 them.
 **1. The M1.8 metric can be gamed by accident.** Update-rate is
 `chunks_used / chunks_seen`, and M1 fails above 30%. Documentation is
 evidence-free against almost any standing question, so a corpus routed through
 the controller would push the ratio *down* and make M1.8 easier to pass while the
 memory got worse. Any implementation that improves a quality metric by adding
 unrelated text has inverted it. The gate re-runs M1.8 and asserts the numbers are
 **unchanged**, not merely still-passing.
 **2. Retrieval quality can degrade without any test noticing.** Each task asserts
 its own behaviour on its own fixture. The composite risk is a corpus that
 outcompetes real project memory in the candidate pool — invisible to M3.6.5's
 unit fixture, obvious on the real poimen log with a real corpus loaded.
 **3. The cycle guard has two producers now.** M4.2 writes skills to the manifest,
 M3.6.4 writes reference sections. They share a normaliser. Skills exclusion
 regressing when a corpus is added is the failure that no test in either phase
 catches, because each tests only its own kind.
 **Swappable parts.** The phase claims two seams are real: `DocCorpusSource` is
 just another `RecordSource`, and the corpus is just another projection input.
 Prove both — swap the doc tree for a differently-shaped one and re-run, and
 rebuild the whole store from the log with the corpus present.
 ## Steps
 1. Establish the baseline: on a clean store, run `mem ingest --project poimen`
   for all standing queries; record the M1.8 summary table.
 2. `mem ref add --project poimen --corpus homelab-knowledge <tree>` against a
   real corpus of at least 200 chunks.
 3. Re-run the full ingest. Diff the M1.8 summary against the baseline.
 4. Run the assertions below.
 5. Emit `expected/m3.6-gate.txt` with the summary; commit it. Later runs diff
   against it and a changed expectation is a reviewable claim, same rule as M1.8.
 6. Sample 10 abstentions and 10 R hits; eyeball whether the floor is set sanely.
   Advisory, as M1.8's judge audit is.
 ## Acceptance
 - M1.8's numbers are identical before and after the corpus exists.
 - Default query output is byte-identical before and after.
 - No L1 or L2 node has an R parent.
 - Skills exclusion behaviour is unchanged with a corpus loaded.
 - Drop and rebuild reproduces the mixed store byte-identically.
 ## Verify
 **Harness:** live gateway, real corpus, real poimen log. Long-running; nightly or
 on-demand, `#[ignore]` by default, same posture as M1.8.
 **Integration test** — `tests/it_m3_6_gate.rs`:
 1. `a1_update_rate_identical` — per standing query, assert update-rate before and
   after the corpus is added is equal, not merely both under 0.30. Equality is
   the assertion; a threshold check here would pass the exact failure described
   above.
 2. `a2_chunks_seen_identical` — `chunks_seen` per run is unchanged, proving no R
   chunk entered the recurrence.
 3. `a3_no_controller_calls_during_ref_ingest` — run `mem ref add` under a chat
   transport that panics on request; assert it completes. Embeddings are allowed,
   controller calls are not, so the fake must distinguish the two endpoints.
 4. `a4_default_query_byte_identical` — snapshot default `mem query` output for 10
   fixed questions before and after; assert byte-identical.
 5. `a5_no_r_parents` — `SELECT count(*) FROM memory_edge WHERE parent_sha IN
   (SELECT sha256 FROM memory_node WHERE level='R')` is 0 on the live store.
 6. `a6_l2_stream_excludes_r` — re-run L2 synthesis; assert its input stream
   contained only L1 nodes and the resulting L2 memories have no R ancestor.
 7. `a7_skill_exclusion_unregressed` — re-run M4.2's fixtures against the store
   with the corpus loaded; assert identical exclusion decisions.
 8. `a8_rebuild_mixed_store` — drop database and vault, `mem rebuild --from-log`,
   assert byte-identical across all four levels.
 9. `a9_source_seam_swappable` — point `mem ref add` at a structurally different
   tree (deep nesting, no headings in one file, one non-UTF8 file) and assert it
   ingests or fails cleanly, never partially.
 10. `a10_corpus_does_not_starve_recall` — for 10 project questions, assert the
    top-5 default hits are the same nodes as the pre-corpus baseline, with the
    corpus present in the table.
 11. `a11_m5_export_excludes_r` — run the M5.3 training-corpus export shape; assert
    zero R records appear. R carries no gate decision, so its presence would
    poison `r_update` labels with rows that have no ground truth.
 **Command:** `cargo test --workspace m3_6_gate -- --ignored --nocapture`
 **False pass:**
 - Asserting update-rate is still below 30% instead of unchanged. That is the
  precise shape that goes green while the gate is being fed documentation —
  assertion 1 must be equality.
 - Running the gate with a corpus small enough not to matter. 200 chunks is a
  floor, not a suggestion; below it, assertions 4 and 10 pass because the corpus
  never reaches the candidate pool.
 - Allowing `a3`'s fake transport to reject all HTTP. Reference ingest legitimately
  calls the embeddings endpoint; a blanket panic passes the assertion for the
  wrong reason and would also pass if ingest did nothing at all.
 - Rebuilding into a fresh database rather than dropping the live one. A rebuild
  that never exercises deletion has not proved the projections are droppable.
 ## Traps
 - Comparing M1.8 summaries by eye. The numbers move in the third decimal when the
  gateway is under load; the committed `expected/` file plus an explicit
  tolerance is the only version of this that stays honest over months.
 - Treating a changed baseline as a corpus problem. If update-rate shifts, first
  confirm the gateway model has not changed underneath — `reasoning` and the 3B
  controller are both moving targets, and misattributing that to this phase burns
  a day.
 - Skipping assertion 11 because M5 is not built. The export *shape* is checkable
  now, and discovering R in the training corpus during M5.3 means re-running an
  expensive labelling pass.
 ---
 Background: [DESIGN.md](../DESIGN.md) — reference corpora, the tier model · [M1.8](M1.8-m1-gate.md) · [M4.2](M4.2-derived-filter.md)
@@ -1,42 +0,0 @@
 # M3.6.7 — Contextual Enrichment at Ingest
 | Field | Value |
 |---|---|
 | Phase | M3.6 — Reference corpora |
 | Size | M — 1–2 days |
 | Status | ⬜ Not started |
 | Depends | M3.6.1 (DocCorpusSource) |
 | Blocks | — |
 ## Goal
 At ingest time, prepend each chunk with its context in the document hierarchy.
 This improves semantic search because queries using different terminology can
 still find relevant chunks.
 Inspired by Anthropic's Contextual Retrieval paper.
 ## Example
 ```
 BEFORE (raw chunk from heading "npm install"):
  "Use 'npm ci' instead of 'npm install' for reproducible builds"
 AFTER (contextualized):
  "From the Node.js Dependency Management guide, section npm install:
   Use 'npm ci' instead of 'npm install' for reproducible builds"
 ```
 The context gets embedded alongside the chunk's text, improving vector search.
 ## Deliverables
 - `DocCorpusSource` enhanced to include breadcrumb path + section summary
 - Chunk rendering includes context header (auto-generated or manual)
 - Embedding happens on (context + chunk), not just chunk
 - Rebuild idempotence preserved
 ## Tests
 - 5 unit tests (context generation, formatting, idempotence)
 - 3 integration tests (rebuild with enrichment, search improvement)
@@ -1,52 +0,0 @@
 # M3.6.8 — Chunk Deduplication at Ingest
 | Field | Value |
 |---|---|
 | Phase | M3.6 — Reference corpora |
 | Size | M — 1–2 days |
 | Status | ⬜ Not started |
 | Depends | M3.6.1 (DocCorpusSource), M3.7.7 (normalisation patterns) |
 | Blocks | — |
 ## Goal
 Detect and deduplicate near-identical chunks at ingest time. Prevents the same
 error message, config snippet, or code sample from being stored 47 times from
 different runs/sources.
 ## Approach
 **Probabilistic: MinHash-based duplicate detection**
 1. On each incoming chunk, compute MinHash signature (fast, space-efficient)
 2. Check against seen signatures with 95% accuracy threshold
 3. If match found: increment count on existing chunk, skip storage
 4. If new: store chunk + signature
 **Reuse from M3.7.7:** Normalisation patterns (strip_ansi, lowercase, remove
 extra whitespace) ensure similar content hashes identically.
 ## Example
 ```
 Run 1: npm ERR! 404 Not Found - [email protected]
  → stored as chunk #42
 Run 2: npm ERR! 404 Not Found - [email protected]
  → same normalised hash → increment count on #42, don't store
 Run 3: npm ERR! 404 Not Found - [email protected]
  → different package version → new chunk #43
 ```
 ## Deliverables
 - `DeduplicationStore` with MinHash signatures
 - Integration with rebuild pipeline
 - Chunk count metadata tracking
 - Database schema extension (chunk.dedup_count)
 ## Tests
 - 4 unit tests (MinHash collision testing, normalization)
 - 3 integration tests (rebuild deduplication, count tracking)