43 lines
1.2 KiB
Markdown
43 lines
1.2 KiB
Markdown
|
# M3.6.7 — Contextual Enrichment at Ingest
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M3.6 — Reference corpora |
|
|||
|
|
| Size | M — 1–2 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Depends | M3.6.1 (DocCorpusSource) |
|
|||
|
|
| Blocks | — |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
At ingest time, prepend each chunk with its context in the document hierarchy.
|
|||
|
|
This improves semantic search because queries using different terminology can
|
|||
|
|
still find relevant chunks.
|
|||
|
|
|
|||
|
|
Inspired by Anthropic's Contextual Retrieval paper.
|
|||
|
|
|
|||
|
|
## Example
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
BEFORE (raw chunk from heading "npm install"):
|
|||
|
|
"Use 'npm ci' instead of 'npm install' for reproducible builds"
|
|||
|
|
|
|||
|
|
AFTER (contextualized):
|
|||
|
|
"From the Node.js Dependency Management guide, section npm install:
|
|||
|
|
Use 'npm ci' instead of 'npm install' for reproducible builds"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The context gets embedded alongside the chunk's text, improving vector search.
|
|||
|
|
|
|||
|
|
## Deliverables
|
|||
|
|
|
|||
|
|
- `DocCorpusSource` enhanced to include breadcrumb path + section summary
|
|||
|
|
- Chunk rendering includes context header (auto-generated or manual)
|
|||
|
|
- Embedding happens on (context + chunk), not just chunk
|
|||
|
|
- Rebuild idempotence preserved
|
|||
|
|
|
|||
|
|
## Tests
|
|||
|
|
|
|||
|
|
- 5 unit tests (context generation, formatting, idempotence)
|
|||
|
|
- 3 integration tests (rebuild with enrichment, search improvement)
|