53 lines
1.5 KiB
Markdown
53 lines
1.5 KiB
Markdown
|
# M3.6.8 — Chunk Deduplication at Ingest
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M3.6 — Reference corpora |
|
|||
|
|
| Size | M — 1–2 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Depends | M3.6.1 (DocCorpusSource), M3.7.7 (normalisation patterns) |
|
|||
|
|
| Blocks | — |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Detect and deduplicate near-identical chunks at ingest time. Prevents the same
|
|||
|
|
error message, config snippet, or code sample from being stored 47 times from
|
|||
|
|
different runs/sources.
|
|||
|
|
|
|||
|
|
## Approach
|
|||
|
|
|
|||
|
|
**Probabilistic: MinHash-based duplicate detection**
|
|||
|
|
|
|||
|
|
1. On each incoming chunk, compute MinHash signature (fast, space-efficient)
|
|||
|
|
2. Check against seen signatures with 95% accuracy threshold
|
|||
|
|
3. If match found: increment count on existing chunk, skip storage
|
|||
|
|
4. If new: store chunk + signature
|
|||
|
|
|
|||
|
|
**Reuse from M3.7.7:** Normalisation patterns (strip_ansi, lowercase, remove
|
|||
|
|
extra whitespace) ensure similar content hashes identically.
|
|||
|
|
|
|||
|
|
## Example
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
Run 1: npm ERR! 404 Not Found - [email protected]
|
|||
|
|
→ stored as chunk #42
|
|||
|
|
|
|||
|
|
Run 2: npm ERR! 404 Not Found - [email protected]
|
|||
|
|
→ same normalised hash → increment count on #42, don't store
|
|||
|
|
|
|||
|
|
Run 3: npm ERR! 404 Not Found - [email protected]
|
|||
|
|
→ different package version → new chunk #43
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Deliverables
|
|||
|
|
|
|||
|
|
- `DeduplicationStore` with MinHash signatures
|
|||
|
|
- Integration with rebuild pipeline
|
|||
|
|
- Chunk count metadata tracking
|
|||
|
|
- Database schema extension (chunk.dedup_count)
|
|||
|
|
|
|||
|
|
## Tests
|
|||
|
|
|
|||
|
|
- 4 unit tests (MinHash collision testing, normalization)
|
|||
|
|
- 3 integration tests (rebuild deduplication, count tracking)
|