43 lines
1.2 KiB
Markdown
43 lines
1.2 KiB
Markdown
# M3.6.7 — Contextual Enrichment at Ingest
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M3.6 — Reference corpora |
|
||
| Size | M — 1–2 days |
|
||
| Status | ⬜ Not started |
|
||
| Depends | M3.6.1 (DocCorpusSource) |
|
||
| Blocks | — |
|
||
|
||
## Goal
|
||
|
||
At ingest time, prepend each chunk with its context in the document hierarchy.
|
||
This improves semantic search because queries using different terminology can
|
||
still find relevant chunks.
|
||
|
||
Inspired by Anthropic's Contextual Retrieval paper.
|
||
|
||
## Example
|
||
|
||
```
|
||
BEFORE (raw chunk from heading "npm install"):
|
||
"Use 'npm ci' instead of 'npm install' for reproducible builds"
|
||
|
||
AFTER (contextualized):
|
||
"From the Node.js Dependency Management guide, section npm install:
|
||
Use 'npm ci' instead of 'npm install' for reproducible builds"
|
||
```
|
||
|
||
The context gets embedded alongside the chunk's text, improving vector search.
|
||
|
||
## Deliverables
|
||
|
||
- `DocCorpusSource` enhanced to include breadcrumb path + section summary
|
||
- Chunk rendering includes context header (auto-generated or manual)
|
||
- Embedding happens on (context + chunk), not just chunk
|
||
- Rebuild idempotence preserved
|
||
|
||
## Tests
|
||
|
||
- 5 unit tests (context generation, formatting, idempotence)
|
||
- 3 integration tests (rebuild with enrichment, search improvement)
|