# M3.6.8 — Chunk Deduplication at Ingest

| Field | Value |
|---|---|
| Phase | M3.6 — Reference corpora |
| Size | M — 1–2 days |
| Status | ⬜ Not started |
| Depends | M3.6.1 (DocCorpusSource), M3.7.7 (normalisation patterns) |
| Blocks | — |

## Goal

Detect and deduplicate near-identical chunks at ingest time. Prevents the same
error message, config snippet, or code sample from being stored 47 times from
different runs/sources.

## Approach

**Probabilistic: MinHash-based duplicate detection**

1. On each incoming chunk, compute MinHash signature (fast, space-efficient)
2. Check against seen signatures with 95% accuracy threshold
3. If match found: increment count on existing chunk, skip storage
4. If new: store chunk + signature

**Reuse from M3.7.7:** Normalisation patterns (strip_ansi, lowercase, remove
extra whitespace) ensure similar content hashes identically.

## Example

```
Run 1: npm ERR! 404 Not Found - react-dom@18.2.5
  → stored as chunk #42
  
Run 2: npm ERR! 404 Not Found - react-dom@18.2.5
  → same normalised hash → increment count on #42, don't store

Run 3: npm ERR! 404 Not Found - react-dom@18.2.4
  → different package version → new chunk #43
```

## Deliverables

- `DeduplicationStore` with MinHash signatures
- Integration with rebuild pipeline
- Chunk count metadata tracking
- Database schema extension (chunk.dedup_count)

## Tests

- 4 unit tests (MinHash collision testing, normalization)
- 3 integration tests (rebuild deduplication, count tracking)