53 lines
1.5 KiB
Markdown
53 lines
1.5 KiB
Markdown
# M3.6.8 — Chunk Deduplication at Ingest
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M3.6 — Reference corpora |
|
||
| Size | M — 1–2 days |
|
||
| Status | ⬜ Not started |
|
||
| Depends | M3.6.1 (DocCorpusSource), M3.7.7 (normalisation patterns) |
|
||
| Blocks | — |
|
||
|
||
## Goal
|
||
|
||
Detect and deduplicate near-identical chunks at ingest time. Prevents the same
|
||
error message, config snippet, or code sample from being stored 47 times from
|
||
different runs/sources.
|
||
|
||
## Approach
|
||
|
||
**Probabilistic: MinHash-based duplicate detection**
|
||
|
||
1. On each incoming chunk, compute MinHash signature (fast, space-efficient)
|
||
2. Check against seen signatures with 95% accuracy threshold
|
||
3. If match found: increment count on existing chunk, skip storage
|
||
4. If new: store chunk + signature
|
||
|
||
**Reuse from M3.7.7:** Normalisation patterns (strip_ansi, lowercase, remove
|
||
extra whitespace) ensure similar content hashes identically.
|
||
|
||
## Example
|
||
|
||
```
|
||
Run 1: npm ERR! 404 Not Found - [email protected]
|
||
→ stored as chunk #42
|
||
|
||
Run 2: npm ERR! 404 Not Found - [email protected]
|
||
→ same normalised hash → increment count on #42, don't store
|
||
|
||
Run 3: npm ERR! 404 Not Found - [email protected]
|
||
→ different package version → new chunk #43
|
||
```
|
||
|
||
## Deliverables
|
||
|
||
- `DeduplicationStore` with MinHash signatures
|
||
- Integration with rebuild pipeline
|
||
- Chunk count metadata tracking
|
||
- Database schema extension (chunk.dedup_count)
|
||
|
||
## Tests
|
||
|
||
- 4 unit tests (MinHash collision testing, normalization)
|
||
- 3 integration tests (rebuild deduplication, count tracking)
|