1.5 KiB
1.5 KiB
M3.6.8 — Chunk Deduplication at Ingest
| Field | Value |
|---|---|
| Phase | M3.6 — Reference corpora |
| Size | M — 1–2 days |
| Status | ⬜ Not started |
| Depends | M3.6.1 (DocCorpusSource), M3.7.7 (normalisation patterns) |
| Blocks | — |
Goal
Detect and deduplicate near-identical chunks at ingest time. Prevents the same error message, config snippet, or code sample from being stored 47 times from different runs/sources.
Approach
Probabilistic: MinHash-based duplicate detection
- On each incoming chunk, compute MinHash signature (fast, space-efficient)
- Check against seen signatures with 95% accuracy threshold
- If match found: increment count on existing chunk, skip storage
- If new: store chunk + signature
Reuse from M3.7.7: Normalisation patterns (strip_ansi, lowercase, remove extra whitespace) ensure similar content hashes identically.
Example
Run 1: npm ERR! 404 Not Found - [email protected]
→ stored as chunk #42
Run 2: npm ERR! 404 Not Found - [email protected]
→ same normalised hash → increment count on #42, don't store
Run 3: npm ERR! 404 Not Found - [email protected]
→ different package version → new chunk #43
Deliverables
DeduplicationStorewith MinHash signatures- Integration with rebuild pipeline
- Chunk count metadata tracking
- Database schema extension (chunk.dedup_count)
Tests
- 4 unit tests (MinHash collision testing, normalization)
- 3 integration tests (rebuild deduplication, count tracking)