plan: add M3.6.7 contextual enrichment + M3.6.8 deduplication

2026-08-28 09:28:10 -07:00
parent c20f8f9a9f
commit 1f9b30b1ec
2 changed files with 94 additions and 0 deletions
@@ -0,0 +1,52 @@
+# M3.6.8 — Chunk Deduplication at Ingest
+
+| Field | Value |
+|---|---|
+| Phase | M3.6 — Reference corpora |
+| Size | M — 1–2 days |
+| Status | ⬜ Not started |
+| Depends | M3.6.1 (DocCorpusSource), M3.7.7 (normalisation patterns) |
+| Blocks | — |
+
+## Goal
+
+Detect and deduplicate near-identical chunks at ingest time. Prevents the same
+error message, config snippet, or code sample from being stored 47 times from
+different runs/sources.
+
+## Approach
+
+**Probabilistic: MinHash-based duplicate detection**
+
+1. On each incoming chunk, compute MinHash signature (fast, space-efficient)
+2. Check against seen signatures with 95% accuracy threshold
+3. If match found: increment count on existing chunk, skip storage
+4. If new: store chunk + signature
+
+**Reuse from M3.7.7:** Normalisation patterns (strip_ansi, lowercase, remove
+extra whitespace) ensure similar content hashes identically.
+
+## Example
+
+```
+Run 1: npm ERR! 404 Not Found - [email protected]
+  → stored as chunk #42
+  
+Run 2: npm ERR! 404 Not Found - [email protected]
+  → same normalised hash → increment count on #42, don't store
+
+Run 3: npm ERR! 404 Not Found - [email protected]
+  → different package version → new chunk #43
+```
+
+## Deliverables
+
+- `DeduplicationStore` with MinHash signatures
+- Integration with rebuild pipeline
+- Chunk count metadata tracking
+- Database schema extension (chunk.dedup_count)
+
+## Tests
+
+- 4 unit tests (MinHash collision testing, normalization)
+- 3 integration tests (rebuild deduplication, count tracking)