diff --git a/tasks/M3.6.2-level-r-storage.md b/tasks/M3.6.2-level-r-storage.md deleted file mode 100644 index e86525a..0000000 --- a/tasks/M3.6.2-level-r-storage.md +++ /dev/null @@ -1,181 +0,0 @@ -# M3.6.2 — Level R: Obsidian reference indexing and rebuild parity - -| Field | Value | -|---|---| -| Phase | M3.6 — Reference corpora | -| Size | M — 1–3 days | -| Status | ✅ COMPLETE | -| Flags | — | -| Spec | inlined below | -| Blocks | M3.6.6 | -| Depends | M3.6.1, M1.6, M2.3, M2.4, M2.5, M2.6, Obsidian service (M2.5) | - -## Goal - -Fetch reference documents from the live Obsidian vault (REST API), chunk them -via M3.6.1's heading-boundary logic, land them in the log as reference records, -index them in Postgres, and prove rebuild parity (drop + rebuild from log = -byte-identical index state). - -## Facts (inlined — no spec read needed) - -**Obsidian is the source of truth for reference documents.** The Obsidian REST API -(deployed in M2.5, accessible at `http://obsidian-server.poimen.svc.cluster.local:8080`) -provides live access to vault files via: - -```bash -GET /api/vault/listFiles # List all .md files -GET /api/vault/readFile?path=kubectl.md # Read file contents -GET /api/vault/listFolders # Explore structure -``` - -**Reference record format (in JSONL log):** - -```jsonl -{"kind":"reference","level":"R","source":"obsidian://poimen-vault/kubectl.md", - "doc_sha":"ab12…","heading_path":"kubectl.md > Common Issues > CrashLoopBackOff", - "sha256":"cd34…","t":7,"run_id":"ref-obsidian-2026-08-21T10:02:11Z","text":"…"} -``` - -**No vault projection.** Unlike M3.6.1 (DocCorpusSource which reads from local filesystem), -M3.6.2 reads directly from Obsidian REST API and indexes into Postgres. The Obsidian -vault remains the single source of truth; `mem rebuild --from-log` re-fetches from -Obsidian to regenerate indexes. - -`level = 'R'`, `query_id = NULL` (R answers no standing question), `source` holds -the Obsidian URI (`obsidian://vault-name/file.md`). `doc_sha` is whole-document hash; -`sha256` is chunk hash (primary identity). - -The embedding goes to `memory_vector(kind='text')`, not to a column on the node. -R gets **no symptom projection** — M3.7.8 generates those for L1 and L2 only. - -**R writes no edges.** Provenance is the Obsidian URI, not internal edges. Enforced -in `mem verify` (M3.6.4). - -**Rebuild parity is the whole point.** `mem rebuild --from-log` must: -1. Drop R rows from Postgres -2. Drop R vectors from OpenSearch (M8.2) -3. Re-fetch documents from Obsidian REST API -4. Re-chunk via M3.6.1 heading logic -5. Re-index into Postgres + OpenSearch -6. Result must be byte-identical (same shas, same vector embeddings) - -## Steps - -1. **ObsidianRefSource** in `mem-ingest`: Implement `ChunkedSource` that: - - Calls Obsidian REST API (`/api/vault/listFiles`) - - Filters for `.md` files in allowed paths (e.g., `docs/`, `reference/`) - - Fetches each file via `/api/vault/readFile?path=...` - - Chunks via M3.6.1's heading boundary logic - - Yields `Record { kind: Reference, ... }` - -2. Add `Reference` variant to log record enum in `mem-core` (already exists from - M2.3, just needs the Obsidian source) - -3. `mem-store`: Insert R nodes with `kind='text'` vector; assert no edge insert - names an R sha as parent (repository boundary check). - -4. Dual-write in M8.2: When R records are indexed, write both to Postgres and - OpenSearch (L2 already does this; extend for R). - -5. **Rebuild: extend M2.6 harness** for `mem rebuild --from-log`: - - When replaying R records from log, re-fetch original files from Obsidian - - Re-chunk via M3.6.1 logic - - Regenerate embeddings (deterministic, so shas match) - - Assert byte-identical state vs. original indexing - -6. Integration: Update `mem query` to include R results from hybrid search (M8) - -## Acceptance - -- `ObsidianRefSource` fetches all `.md` files from Obsidian REST API. -- Reference records (R) with `source=obsidian://...` land in the log. -- R rows persist in Postgres with `query_id IS NULL` and `source` set to Obsidian URI. -- Each R chunk gets a `kind='text'` embedding in OpenSearch. -- The M2.3 constraint accepts `R` and rejects `L3`. -- `mem rebuild --from-log`: drop Postgres R rows + OpenSearch R vectors, re-fetch - from Obsidian, re-chunk, re-embed, re-index; result is byte-identical (same shas, - same embedding vectors). -- No hidden inputs: Obsidian REST API is the only external dependency for R. - -## Verify - -**Harness:** Mock Obsidian REST API (deterministic file list + content). M2.6 rebuild -harness extended to cover R records. Deterministic embedder so shas are stable. - -**Integration test** — `tests/it_obsidian_ref_source.rs` + `tests/it_level_r_storage.rs`: - -### ObsidianRefSource tests: -1. `a1_fetches_md_files` — Mock API returns 3 `.md` files; source emits 3 documents. -2. `a2_chunks_by_heading` — One source document with 5 headings yields 5 chunks - (uses M3.6.1 heading boundary logic). -3. `a3_generates_doc_sha` — Document SHA (SHA256 of whole file content) is - consistent across fetches. -4. `a4_generates_chunk_shas` — Each chunk sha is deterministic: SHA256(tool + - heading_path + text). -5. `a5_sets_obsidian_uri` — Every R record has `source=obsidian://vault-name/path.md`. -6. `a6_skips_non_md` — API returns `.json` and `.txt` files; source ignores them. - -### Level R storage tests: -7. `a7_record_roundtrip` — Serialize R record to JSONL, deserialize; assert - field-for-field equality. -8. `a8_r_inserts_to_postgres` — Insert `level='R'` with `kind='text'` vector; - assert Postgres row exists with correct `source` URI. -9. `a9_r_inserts_to_opensearch` — Same chunk indexed in OpenSearch (M8.2 - dual-write); assert M8 vector store has the chunk. -10. `a10_no_symptom_projection` — R chunks do not acquire `kind='symptom'` - vectors (only L1/L2 get those from M3.7.8). -11. `a11_query_id_null` — Every R row has `query_id IS NULL`; insertion with - non-NULL query_id is rejected by M2.3's CHECK. -12. `a12_no_edges_from_r` — After ingesting 10 R chunks, `SELECT count(*) - FROM memory_edge WHERE parent_sha IN (SELECT sha256 FROM memory_node WHERE - level='R')` equals 0. - -### Rebuild parity tests: -13. `a13_rebuild_drops_r_rows` — Ingest R, assert N rows exist. Call `mem - rebuild --from-log`, assert same N rows exist (re-fetched, re-embedded, - re-indexed). -14. `a14_rebuild_byte_identical` — Snapshot Postgres R rows + OpenSearch R - vectors before rebuild. Drop both. Run `mem rebuild --from-log` against - a log containing R records. Assert Postgres rows and OpenSearch vectors - are byte-identical (same row order, same field values, same vector embeddings). -15. `a15_rebuild_idempotent` — Run rebuild twice; second run changes nothing. - Snapshot comparison between first and second rebuild result is identical. - -**Command:** -```bash -cargo test -p mem-ingest obsidian_ref_source -cargo test -p mem-store level_r -cargo test -p mem-cli rebuild # includes R records in fixture log -``` - -**False pass:** -- Asserting rebuild parity on a log with no R records. Assertion 14 requires - mixed-level log (L0/L1/L2 *and* R) or it trivially passes. -- Mocking Obsidian API with hardcoded responses. File hashes must match real - Obsidian vault file SHA256 (use actual Obsidian instance or hash fixtures). -- Checking edge count is zero before ingesting. Assertion 12 must run against - populated corpus or it asserts empty table is empty. -- Byte-identical comparison with normalization (sorting, ignoring order). - Byte-identical means bytes; rebuild parity is broken if row order changes - between runs. - -## Traps - -- Reusing `run_id` semantics from the gated loop. R has no run in the recurrence - sense; use a synthetic `ref-obsidian-` and do not let it collide - with a real ingest run in queries that group by `run_id`. -- Caching Obsidian API responses across rebuild runs. Rebuild must re-fetch from - Obsidian REST API every time (no cache) to ensure file contents and shas are - always in sync with live vault. -- Assuming Obsidian file list is sorted. `listFiles` order is arbitrary; source - must sort filenames before chunking to ensure deterministic shas across runs. -- Embedding each R chunk independently. M3.6.1 (DocCorpusSource) uses the same - `m` parameter (batching) and deterministic embedder model to ensure chunk shas - are stable; M3.6.2 must use identical setup. -- Dropping the check constraint instead of widening it. Assertion 11 exists - because `DROP CONSTRAINT` alone passes every other assertion in this file. - ---- - -Background: [DESIGN.md](../DESIGN.md) — reference corpora, storage schemas diff --git a/tasks/M3.6.4-reference-cycle-guard.md b/tasks/M3.6.4-reference-cycle-guard.md deleted file mode 100644 index 7201733..0000000 --- a/tasks/M3.6.4-reference-cycle-guard.md +++ /dev/null @@ -1,142 +0,0 @@ -# M3.6.4 — Reference text cannot re-enter as evidence - -| Field | Value | -|---|---| -| Phase | M3.6 — Reference corpora | -| Size | M — 1–3 days | -| Status | ✅ COMPLETE | -| Flags | — | -| Spec | inlined below | -| Blocks | M3.6.6 | -| Depends | M3.6.3, M4.2 | - -## Goal - -Stop a retrieved manual page from coming back through the front door as a project -finding. - -## Facts (inlined — no spec read needed) - -The cycle is M4.2's, with documentation substituted for emitted skills: - -``` -agent queries memory, gets an R section - │ - ▼ -section is pasted into the agent's context - │ - ▼ -appears verbatim in that session's transcript - │ - ▼ -transcript ingested; the gate sees upstream doc text - │ - ▼ -"kubectl describe shows events" becomes an L1 project memory -``` - -The gate is *right* to accept it — the chunk genuinely contains information about -the question. That is what makes this dangerous rather than merely noisy: no -threshold tuning catches it, because the text really is relevant. Only knowing -that the system emitted the text itself distinguishes the two cases. - -**Reuse M4.2, do not rebuild it.** M4.2 already computes normalised shingle -overlap against an artifact manifest and tags matching records `derived: true`, -excluding them from evidence while keeping them in the log so the exclusion is -auditable. R chunks are a second artifact kind in that same manifest. A parallel -matcher would drift from it and double the tuning surface. - -```jsonl -{"kind":"reference","name":"kubectl/common-issues","sha256":"cd34…","shingles":[…],"emitted_at":"…"} -{"kind":"skill","name":"infra-root-causes","sha256":"ab12…","shingles":[…],"emitted_at":"…"} -``` - -**Threshold pressure differs by kind and this is the real work.** A skill is -emitted once and quoted rarely. Documentation is quoted constantly and partially -— one command from a fifty-line cheatsheet. Shingle overlap against a whole R -chunk will sit far below M4.2's 0.8 default for exactly the case that matters, so -matching must be at section granularity with its own threshold, tuned and logged -separately. One shared matcher, two configured thresholds. - -**A false positive here is costly and must stay visible.** Excluding a genuine -discussion *about* `kubectl` because it quotes two lines of the cheatsheet -silently drops real evidence. Every exclusion emits `derived_excluded` naming the -matched artifact, and `mem verify` can list them for audit. - -## Steps - -1. Generalise M4.2's manifest to `vault/.artifacts.jsonl` with a `kind` field; - keep skills writing to it unchanged. -2. `mem ref add`/`sync` append `kind: "reference"` entries per R chunk; - tombstones remove them. -3. Add per-kind thresholds to the matcher config; default reference threshold - lower than the skill threshold, and record the value in the exclusion event. -4. Extend `mem verify --derived-filter` to assert no L0 evidence node matches a - live R artifact. -5. `mem verify --exclusions` lists recent `derived_excluded` events with the - matched artifact and overlap score, for false-positive review. - -## Acceptance - -- A session transcript containing a verbatim R section is excluded from evidence. -- The same transcript still appears in the log, tagged, with the match named. -- A session that merely *mentions* the tool without quoting it is not excluded. -- Skill exclusion behaviour from M4.2 is unchanged. -- Removing a corpus removes its manifest entries; previously excluded text is not - retroactively rewritten in the log. - -## Verify - -**Harness:** fixture corpus ingested as R, plus three synthetic transcripts — one -quoting a section verbatim, one paraphrasing it heavily, one discussing the tool -without quoting. Deterministic embedder. - -**Integration test** — `tests/it_reference_cycle.rs`: -1. `a1_verbatim_quote_excluded` — the quoting transcript produces zero L0 - evidence nodes; assert a `derived_excluded` event naming the R artifact. -2. `a2_discussion_not_excluded` — the non-quoting transcript produces evidence - normally. This is the false-positive guard and it is the assertion that fails - when the threshold is set too low. -3. `a3_partial_quote_caught` — the transcript quoting ~10 lines of a 50-line - section is excluded, proving section-granularity matching rather than - whole-chunk overlap. -4. `a4_skill_path_unchanged` — run M4.2's own test fixtures; assert identical - results before and after the manifest generalisation. -5. `a5_exclusion_is_auditable` — every exclusion event carries artifact name, - overlap score and the threshold in force. -6. `a6_tombstone_removes_manifest_entry` — `mem ref rm`, then assert the R - entries are gone from the manifest and the same transcript now ingests - normally. -7. `a7_verify_catches_leak` — hand-insert an L0 node whose text matches an R - artifact; assert `mem verify --derived-filter` exits non-zero and names it. -8. `a8_no_retroactive_log_edit` — after `rm`, assert prior `derived_excluded` - records are still present and unmodified. - -**Command:** `cargo test -p mem-ingest reference_cycle && cargo test -p mem-cli verify` - -**False pass:** -- Testing only the verbatim case. Verbatim is easy and a whole-chunk hash catches - it; assertion 3 is the one that distinguishes a working matcher, and assertion - 2 is the one that proves it is not simply excluding everything that mentions - the tool. -- Asserting exclusion by checking evidence count is zero. A filter that is - accidentally excluding *all* records also yields zero; assertion 2 has to run - in the same test binary. -- Reusing M4.2's threshold unchanged and declaring it done. The default is tuned - for whole-artifact quoting; assertion 3 fails against it, which is the point. - -## Traps - -- Registering R chunks in the manifest before they are committed to the log. A - failed ingest then leaves manifest entries that exclude evidence for a corpus - that does not exist, and the symptom is missing memories with no obvious cause. -- Normalising differently in the two paths. If the shingler treats markdown - tables differently at emit-time and at ingest-time, overlap collapses and the - filter silently stops firing — same failure M4.2 already warns about, now with - two producers to keep in step. -- Letting the exclusion event omit the threshold. A tuning change makes every - historical exclusion uninterpretable, and this filter will be tuned. - ---- - -Background: [DESIGN.md](../DESIGN.md) — reference corpora, skills · [M4.2](M4.2-derived-filter.md) diff --git a/tasks/M3.6.5-query-levels-and-floor.md b/tasks/M3.6.5-query-levels-and-floor.md deleted file mode 100644 index 6670ef6..0000000 --- a/tasks/M3.6.5-query-levels-and-floor.md +++ /dev/null @@ -1,136 +0,0 @@ -# M3.6.5 — Query: filter-then-recall, R opt-in, relevance floor - -| Field | Value | -|---|---| -| Phase | M3.6 — Reference corpora | -| Size | M — 1–3 days | -| Status | ✅ COMPLETE | -| Flags | — | -| Spec | inlined below | -| Blocks | M3.6.6 | -| Depends | M3.6.2, M3.3, M3.2, M2.7 | - -## Goal - -Make R reachable on request, unreachable by default, and stop the retriever -answering questions it has no evidence for. - -## Facts (inlined — no spec read needed) - -``` -mem query "why did requests over 10KB fail?" # L1,L2 — unchanged -mem query --levels R "kubectl describe pod" # reference only -mem query --levels L1,L2,R "..." # both, R marked in output -mem query "…" --min-score 0.4 # override the floor -``` - -**Filter before recall, not after.** M3.3 recalls `10×k` from HNSW and reranks -down to `k`. A corpus is typically an order of magnitude larger than the project's -own memory, so R rows compete for those 50 candidate slots even when the caller -excluded them — and post-filtering then returns three results instead of five, -quietly. The level predicate belongs in the SQL that drives the HNSW scan. The -existing `(project, level)` index already supports it. - -**Abstention.** With a corpus loaded, every question has *something* moderately -close, so unconditional top-k starts returning plausible prose for questions the -memory cannot answer — worse than an empty result, because it reads as an answer. -If the best post-rerank score is below the floor, return no hits and say why: - -``` -no hits above relevance floor (best 0.21 < 0.35 threshold) -try --min-score to lower it, or --levels R to search reference docs -``` - -The floor applies to the **reranked** score, not cosine distance. M3.2's own -fixture separates a relevant from an irrelevant passage by four orders of -magnitude; cosine distance does not, which is why the floor cannot live at the -recall stage. - -**R is visually distinct in output.** A reference hit prints its source URI and -heading path where a project hit prints provenance. A caller must never have to -infer from wording whether an answer came from this cluster's history or from -upstream documentation. - -**R has no provenance walk.** M3.3 walks `memory_edge` one hop for L1 and two for -L2. R has no edges by construction (M3.6.2), so the walk is skipped rather than -returning empty — and `mem verify` gains the assertion that makes that safe. - -## Steps - -1. Push the level filter into the recall query; assert candidate width is `10×k` - *after* filtering. -2. `--levels` accepts `R`; default remains `L1,L2`. -3. Apply the relevance floor to reranked scores; `--min-score` overrides, - `--min-score 0` disables. -4. Abstention message names the best score, the threshold, and the two escapes. -5. Render R hits with source URI and heading path; suppress the provenance walk. -6. Exit code: abstention is exit 0 with no hits, not an error — it is a valid - answer. Unresolvable project stays non-zero (M3.3 assertion 7). -7. `mem verify --edges` asserts no `memory_edge` row names an R sha as parent. - -## Acceptance - -- Default query over a database containing a large corpus returns exactly the - same hits as before the corpus was added. -- `--levels R` returns reference sections with URI and heading path. -- A question with no good match returns nothing and explains itself. -- Lowering `--min-score` surfaces the suppressed hits. -- `mem verify` rejects a hand-inserted `L1 -> R` edge. - -## Verify - -**Harness:** seeded database with the poimen log *plus* a reference corpus large -enough to dominate raw recall — at least 10× the project node count. Live -reranker for scoring assertions, deterministic embedder elsewhere. - -**Integration test** — `tests/it_query_levels.rs`: -1. `a1_default_unchanged_by_corpus` — snapshot default query results before and - after ingesting the corpus; assert byte-identical output. This is the - headline assertion of the task. -2. `a2_filter_before_recall` — instrument the repository; assert the SQL driving - HNSW carries the level predicate and returns `10×k` rows post-filter, not - `10×k` pre-filter then fewer. -3. `a3_levels_r_returns_reference` — `--levels R` returns R nodes with source URI - and heading path populated. -4. `a4_floor_abstains` — a question with no relevant content returns zero hits, - exit 0, message naming best score and threshold. -5. `a5_floor_override_recovers` — same question with `--min-score 0` returns the - suppressed hits, proving abstention is a floor and not a bug upstream. -6. `a6_floor_applies_post_rerank` — construct a case where cosine is high and - rerank is low; assert it is suppressed. The reverse ordering passes every - other assertion here. -7. `a7_r_hits_visually_distinct` — assert R hits carry no provenance block and do - carry a URI, in both human and `--format json` output. -8. `a8_no_edge_to_r` — hand-insert an `L1 -> R` edge; assert `mem verify --edges` - exits non-zero and names the offending pair. -9. `a9_mixed_levels_ordering` — `--levels L1,L2,R` returns both kinds ranked - together with the level labelled on every row. - -**Command:** `cargo test -p mem-cli query_levels` - -**False pass:** -- Running assertion 1 against a small corpus. If the corpus is smaller than the - recall width, post-filtering and pre-filtering give the same answer and - assertion 2 is the only thing separating them — the fixture size is part of the - test. -- Testing abstention with a query that matches nothing at all. Zero recall - returns zero hits regardless of the floor; the fixture needs a *weak but - non-empty* match, or assertion 4 passes with the floor unimplemented. -- Asserting `--levels R` works without asserting the default excludes R. Both - directions are the contract. - -## Traps - -- Applying the floor to the first-stage cosine score. Cosine on `nomic` puts - unrelated text closer than intuition suggests; a floor there either suppresses - good hits or does nothing, depending on the corpus. -- Treating abstention as an error exit. Callers wrap `mem query` in scripts; a - non-zero exit for "no confident answer" turns a normal outcome into a pipeline - failure and the floor gets disabled within a week. -- Letting the reranker see 50 R candidates and 3 project candidates in a mixed - query. The reranker is not calibrated across levels, and the corpus wins on - fluency; recall per level, then merge. - ---- - -Background: [DESIGN.md](../DESIGN.md) — reference corpora, pgvector, retrieval diff --git a/tasks/M3.6.6-m3.6-gate.md b/tasks/M3.6.6-m3.6-gate.md deleted file mode 100644 index 34afce0..0000000 --- a/tasks/M3.6.6-m3.6-gate.md +++ /dev/null @@ -1,134 +0,0 @@ -# M3.6.6 — M3.6 composition gate - -| Field | Value | -|---|---| -| Phase | M3.6 — Reference corpora | -| Size | M — 1–3 days | -| Status | ⬜ Not started | -| Flags | gate | -| Spec | inlined below | -| Blocks | all of M3.6 | -| Depends | M3.6.1, M3.6.2, M3.6.4, M3.6.5 | - -## Goal - -Answer the question no single task in this phase can: **did adding documentation -change the memory system?** It must not have. - -## Facts (inlined — no spec read needed) - -Every task here was verified alone. What none of them own is the property that -makes the phase safe: a corpus is additive to *retrieval* and invisible to -*everything else*. Three ways that can silently fail, and this gate exists for -them. - -**1. The M1.8 metric can be gamed by accident.** Update-rate is -`chunks_used / chunks_seen`, and M1 fails above 30%. Documentation is -evidence-free against almost any standing question, so a corpus routed through -the controller would push the ratio *down* and make M1.8 easier to pass while the -memory got worse. Any implementation that improves a quality metric by adding -unrelated text has inverted it. The gate re-runs M1.8 and asserts the numbers are -**unchanged**, not merely still-passing. - -**2. Retrieval quality can degrade without any test noticing.** Each task asserts -its own behaviour on its own fixture. The composite risk is a corpus that -outcompetes real project memory in the candidate pool — invisible to M3.6.5's -unit fixture, obvious on the real poimen log with a real corpus loaded. - -**3. The cycle guard has two producers now.** M4.2 writes skills to the manifest, -M3.6.4 writes reference sections. They share a normaliser. Skills exclusion -regressing when a corpus is added is the failure that no test in either phase -catches, because each tests only its own kind. - -**Swappable parts.** The phase claims two seams are real: `DocCorpusSource` is -just another `RecordSource`, and the corpus is just another projection input. -Prove both — swap the doc tree for a differently-shaped one and re-run, and -rebuild the whole store from the log with the corpus present. - -## Steps - -1. Establish the baseline: on a clean store, run `mem ingest --project poimen` - for all standing queries; record the M1.8 summary table. -2. `mem ref add --project poimen --corpus homelab-knowledge ` against a - real corpus of at least 200 chunks. -3. Re-run the full ingest. Diff the M1.8 summary against the baseline. -4. Run the assertions below. -5. Emit `expected/m3.6-gate.txt` with the summary; commit it. Later runs diff - against it and a changed expectation is a reviewable claim, same rule as M1.8. -6. Sample 10 abstentions and 10 R hits; eyeball whether the floor is set sanely. - Advisory, as M1.8's judge audit is. - -## Acceptance - -- M1.8's numbers are identical before and after the corpus exists. -- Default query output is byte-identical before and after. -- No L1 or L2 node has an R parent. -- Skills exclusion behaviour is unchanged with a corpus loaded. -- Drop and rebuild reproduces the mixed store byte-identically. - -## Verify - -**Harness:** live gateway, real corpus, real poimen log. Long-running; nightly or -on-demand, `#[ignore]` by default, same posture as M1.8. - -**Integration test** — `tests/it_m3_6_gate.rs`: -1. `a1_update_rate_identical` — per standing query, assert update-rate before and - after the corpus is added is equal, not merely both under 0.30. Equality is - the assertion; a threshold check here would pass the exact failure described - above. -2. `a2_chunks_seen_identical` — `chunks_seen` per run is unchanged, proving no R - chunk entered the recurrence. -3. `a3_no_controller_calls_during_ref_ingest` — run `mem ref add` under a chat - transport that panics on request; assert it completes. Embeddings are allowed, - controller calls are not, so the fake must distinguish the two endpoints. -4. `a4_default_query_byte_identical` — snapshot default `mem query` output for 10 - fixed questions before and after; assert byte-identical. -5. `a5_no_r_parents` — `SELECT count(*) FROM memory_edge WHERE parent_sha IN - (SELECT sha256 FROM memory_node WHERE level='R')` is 0 on the live store. -6. `a6_l2_stream_excludes_r` — re-run L2 synthesis; assert its input stream - contained only L1 nodes and the resulting L2 memories have no R ancestor. -7. `a7_skill_exclusion_unregressed` — re-run M4.2's fixtures against the store - with the corpus loaded; assert identical exclusion decisions. -8. `a8_rebuild_mixed_store` — drop database and vault, `mem rebuild --from-log`, - assert byte-identical across all four levels. -9. `a9_source_seam_swappable` — point `mem ref add` at a structurally different - tree (deep nesting, no headings in one file, one non-UTF8 file) and assert it - ingests or fails cleanly, never partially. -10. `a10_corpus_does_not_starve_recall` — for 10 project questions, assert the - top-5 default hits are the same nodes as the pre-corpus baseline, with the - corpus present in the table. -11. `a11_m5_export_excludes_r` — run the M5.3 training-corpus export shape; assert - zero R records appear. R carries no gate decision, so its presence would - poison `r_update` labels with rows that have no ground truth. - -**Command:** `cargo test --workspace m3_6_gate -- --ignored --nocapture` - -**False pass:** -- Asserting update-rate is still below 30% instead of unchanged. That is the - precise shape that goes green while the gate is being fed documentation — - assertion 1 must be equality. -- Running the gate with a corpus small enough not to matter. 200 chunks is a - floor, not a suggestion; below it, assertions 4 and 10 pass because the corpus - never reaches the candidate pool. -- Allowing `a3`'s fake transport to reject all HTTP. Reference ingest legitimately - calls the embeddings endpoint; a blanket panic passes the assertion for the - wrong reason and would also pass if ingest did nothing at all. -- Rebuilding into a fresh database rather than dropping the live one. A rebuild - that never exercises deletion has not proved the projections are droppable. - -## Traps - -- Comparing M1.8 summaries by eye. The numbers move in the third decimal when the - gateway is under load; the committed `expected/` file plus an explicit - tolerance is the only version of this that stays honest over months. -- Treating a changed baseline as a corpus problem. If update-rate shifts, first - confirm the gateway model has not changed underneath — `reasoning` and the 3B - controller are both moving targets, and misattributing that to this phase burns - a day. -- Skipping assertion 11 because M5 is not built. The export *shape* is checkable - now, and discovering R in the training corpus during M5.3 means re-running an - expensive labelling pass. - ---- - -Background: [DESIGN.md](../DESIGN.md) — reference corpora, the tier model · [M1.8](M1.8-m1-gate.md) · [M4.2](M4.2-derived-filter.md) diff --git a/tasks/M3.6.7-contextual-enrichment.md b/tasks/M3.6.7-contextual-enrichment.md deleted file mode 100644 index 12c70a1..0000000 --- a/tasks/M3.6.7-contextual-enrichment.md +++ /dev/null @@ -1,42 +0,0 @@ -# M3.6.7 — Contextual Enrichment at Ingest - -| Field | Value | -|---|---| -| Phase | M3.6 — Reference corpora | -| Size | M — 1–2 days | -| Status | ⬜ Not started | -| Depends | M3.6.1 (DocCorpusSource) | -| Blocks | — | - -## Goal - -At ingest time, prepend each chunk with its context in the document hierarchy. -This improves semantic search because queries using different terminology can -still find relevant chunks. - -Inspired by Anthropic's Contextual Retrieval paper. - -## Example - -``` -BEFORE (raw chunk from heading "npm install"): - "Use 'npm ci' instead of 'npm install' for reproducible builds" - -AFTER (contextualized): - "From the Node.js Dependency Management guide, section npm install: - Use 'npm ci' instead of 'npm install' for reproducible builds" -``` - -The context gets embedded alongside the chunk's text, improving vector search. - -## Deliverables - -- `DocCorpusSource` enhanced to include breadcrumb path + section summary -- Chunk rendering includes context header (auto-generated or manual) -- Embedding happens on (context + chunk), not just chunk -- Rebuild idempotence preserved - -## Tests - -- 5 unit tests (context generation, formatting, idempotence) -- 3 integration tests (rebuild with enrichment, search improvement) diff --git a/tasks/M3.6.8-chunk-deduplication.md b/tasks/M3.6.8-chunk-deduplication.md deleted file mode 100644 index e76195b..0000000 --- a/tasks/M3.6.8-chunk-deduplication.md +++ /dev/null @@ -1,52 +0,0 @@ -# M3.6.8 — Chunk Deduplication at Ingest - -| Field | Value | -|---|---| -| Phase | M3.6 — Reference corpora | -| Size | M — 1–2 days | -| Status | ⬜ Not started | -| Depends | M3.6.1 (DocCorpusSource), M3.7.7 (normalisation patterns) | -| Blocks | — | - -## Goal - -Detect and deduplicate near-identical chunks at ingest time. Prevents the same -error message, config snippet, or code sample from being stored 47 times from -different runs/sources. - -## Approach - -**Probabilistic: MinHash-based duplicate detection** - -1. On each incoming chunk, compute MinHash signature (fast, space-efficient) -2. Check against seen signatures with 95% accuracy threshold -3. If match found: increment count on existing chunk, skip storage -4. If new: store chunk + signature - -**Reuse from M3.7.7:** Normalisation patterns (strip_ansi, lowercase, remove -extra whitespace) ensure similar content hashes identically. - -## Example - -``` -Run 1: npm ERR! 404 Not Found - react-dom@18.2.5 - → stored as chunk #42 - -Run 2: npm ERR! 404 Not Found - react-dom@18.2.5 - → same normalised hash → increment count on #42, don't store - -Run 3: npm ERR! 404 Not Found - react-dom@18.2.4 - → different package version → new chunk #43 -``` - -## Deliverables - -- `DeduplicationStore` with MinHash signatures -- Integration with rebuild pipeline -- Chunk count metadata tracking -- Database schema extension (chunk.dedup_count) - -## Tests - -- 4 unit tests (MinHash collision testing, normalization) -- 3 integration tests (rebuild deduplication, count tracking)