plan: add M3.6.7 contextual enrichment + M3.6.8 deduplication

tasks/M3.6.7-contextual-enrichment.md

@@ -0,0 +1,42 @@
+# M3.6.7 — Contextual Enrichment at Ingest
+
+| Field | Value |
+|---|---|
+| Phase | M3.6 — Reference corpora |
+| Size | M — 1–2 days |
+| Status | ⬜ Not started |
+| Depends | M3.6.1 (DocCorpusSource) |
+| Blocks | — |
+
+## Goal
+
+At ingest time, prepend each chunk with its context in the document hierarchy.
+This improves semantic search because queries using different terminology can
+still find relevant chunks.
+
+Inspired by Anthropic's Contextual Retrieval paper.
+
+## Example
+
+```
+BEFORE (raw chunk from heading "npm install"):
+  "Use 'npm ci' instead of 'npm install' for reproducible builds"
+
+AFTER (contextualized):
+  "From the Node.js Dependency Management guide, section npm install:
+   Use 'npm ci' instead of 'npm install' for reproducible builds"
+```
+
+The context gets embedded alongside the chunk's text, improving vector search.
+
+## Deliverables
+
+- `DocCorpusSource` enhanced to include breadcrumb path + section summary
+- Chunk rendering includes context header (auto-generated or manual)
+- Embedding happens on (context + chunk), not just chunk
+- Rebuild idempotence preserved
+
+## Tests
+
+- 5 unit tests (context generation, formatting, idempotence)
+- 3 integration tests (rebuild with enrichment, search improvement)

tasks/M3.6.8-chunk-deduplication.md

@@ -0,0 +1,52 @@
+# M3.6.8 — Chunk Deduplication at Ingest
+
+| Field | Value |
+|---|---|
+| Phase | M3.6 — Reference corpora |
+| Size | M — 1–2 days |
+| Status | ⬜ Not started |
+| Depends | M3.6.1 (DocCorpusSource), M3.7.7 (normalisation patterns) |
+| Blocks | — |
+
+## Goal
+
+Detect and deduplicate near-identical chunks at ingest time. Prevents the same
+error message, config snippet, or code sample from being stored 47 times from
+different runs/sources.
+
+## Approach
+
+**Probabilistic: MinHash-based duplicate detection**
+
+1. On each incoming chunk, compute MinHash signature (fast, space-efficient)
+2. Check against seen signatures with 95% accuracy threshold
+3. If match found: increment count on existing chunk, skip storage
+4. If new: store chunk + signature
+
+**Reuse from M3.7.7:** Normalisation patterns (strip_ansi, lowercase, remove
+extra whitespace) ensure similar content hashes identically.
+
+## Example
+
+```
+Run 1: npm ERR! 404 Not Found - [email protected]
+  → stored as chunk #42
+  
+Run 2: npm ERR! 404 Not Found - [email protected]
+  → same normalised hash → increment count on #42, don't store
+
+Run 3: npm ERR! 404 Not Found - [email protected]
+  → different package version → new chunk #43
+```
+
+## Deliverables
+
+- `DeduplicationStore` with MinHash signatures
+- Integration with rebuild pipeline
+- Chunk count metadata tracking
+- Database schema extension (chunk.dedup_count)
+
+## Tests
+
+- 4 unit tests (MinHash collision testing, normalization)
+- 3 integration tests (rebuild deduplication, count tracking)