From 1f9b30b1ec28b112c484b1fd169c548fc3f6fc73 Mon Sep 17 00:00:00 2001
From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com>
Date: Fri, 28 Aug 2026 09:28:10 -0700
Subject: [PATCH] plan: add M3.6.7 contextual enrichment + M3.6.8 deduplication

---
 tasks/M3.6.7-contextual-enrichment.md | 42 ++++++++++++++++++++++
 tasks/M3.6.8-chunk-deduplication.md   | 52 +++++++++++++++++++++++++++
 2 files changed, 94 insertions(+)
 create mode 100644 tasks/M3.6.7-contextual-enrichment.md
 create mode 100644 tasks/M3.6.8-chunk-deduplication.md

diff --git a/tasks/M3.6.7-contextual-enrichment.md b/tasks/M3.6.7-contextual-enrichment.md
new file mode 100644
index 0000000..12c70a1
--- /dev/null
+++ b/tasks/M3.6.7-contextual-enrichment.md
@@ -0,0 +1,42 @@
+# M3.6.7 — Contextual Enrichment at Ingest
+
+| Field | Value |
+|---|---|
+| Phase | M3.6 — Reference corpora |
+| Size | M — 1–2 days |
+| Status | ⬜ Not started |
+| Depends | M3.6.1 (DocCorpusSource) |
+| Blocks | — |
+
+## Goal
+
+At ingest time, prepend each chunk with its context in the document hierarchy.
+This improves semantic search because queries using different terminology can
+still find relevant chunks.
+
+Inspired by Anthropic's Contextual Retrieval paper.
+
+## Example
+
+```
+BEFORE (raw chunk from heading "npm install"):
+  "Use 'npm ci' instead of 'npm install' for reproducible builds"
+
+AFTER (contextualized):
+  "From the Node.js Dependency Management guide, section npm install:
+   Use 'npm ci' instead of 'npm install' for reproducible builds"
+```
+
+The context gets embedded alongside the chunk's text, improving vector search.
+
+## Deliverables
+
+- `DocCorpusSource` enhanced to include breadcrumb path + section summary
+- Chunk rendering includes context header (auto-generated or manual)
+- Embedding happens on (context + chunk), not just chunk
+- Rebuild idempotence preserved
+
+## Tests
+
+- 5 unit tests (context generation, formatting, idempotence)
+- 3 integration tests (rebuild with enrichment, search improvement)
diff --git a/tasks/M3.6.8-chunk-deduplication.md b/tasks/M3.6.8-chunk-deduplication.md
new file mode 100644
index 0000000..e76195b
--- /dev/null
+++ b/tasks/M3.6.8-chunk-deduplication.md
@@ -0,0 +1,52 @@
+# M3.6.8 — Chunk Deduplication at Ingest
+
+| Field | Value |
+|---|---|
+| Phase | M3.6 — Reference corpora |
+| Size | M — 1–2 days |
+| Status | ⬜ Not started |
+| Depends | M3.6.1 (DocCorpusSource), M3.7.7 (normalisation patterns) |
+| Blocks | — |
+
+## Goal
+
+Detect and deduplicate near-identical chunks at ingest time. Prevents the same
+error message, config snippet, or code sample from being stored 47 times from
+different runs/sources.
+
+## Approach
+
+**Probabilistic: MinHash-based duplicate detection**
+
+1. On each incoming chunk, compute MinHash signature (fast, space-efficient)
+2. Check against seen signatures with 95% accuracy threshold
+3. If match found: increment count on existing chunk, skip storage
+4. If new: store chunk + signature
+
+**Reuse from M3.7.7:** Normalisation patterns (strip_ansi, lowercase, remove
+extra whitespace) ensure similar content hashes identically.
+
+## Example
+
+```
+Run 1: npm ERR! 404 Not Found - react-dom@18.2.5
+  → stored as chunk #42
+  
+Run 2: npm ERR! 404 Not Found - react-dom@18.2.5
+  → same normalised hash → increment count on #42, don't store
+
+Run 3: npm ERR! 404 Not Found - react-dom@18.2.4
+  → different package version → new chunk #43
+```
+
+## Deliverables
+
+- `DeduplicationStore` with MinHash signatures
+- Integration with rebuild pipeline
+- Chunk count metadata tracking
+- Database schema extension (chunk.dedup_count)
+
+## Tests
+
+- 4 unit tests (MinHash collision testing, normalization)
+- 3 integration tests (rebuild deduplication, count tracking)