134 lines
6.2 KiB
Markdown
134 lines
6.2 KiB
Markdown
# M3.6.1 — `DocCorpusSource` + heading-boundary chunking
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M3.6 — Reference corpora |
|
||
| Size | M — 1–3 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M3.6.6 |
|
||
| Depends | M0.3, M0.4 |
|
||
|
||
## Goal
|
||
|
||
Read a tree of documentation into the same stream shape sessions use, split on
|
||
headings instead of messages, without the gate ever seeing it.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
```rust
|
||
pub enum Boundary {
|
||
Record, // existing — never split mid-Record (sessions)
|
||
Heading, // new — split on markdown ATX headings, never mid-section
|
||
}
|
||
```
|
||
|
||
`DocCorpusSource` is a third `RecordSource` alongside the pi and Claude adapters
|
||
(M0.5, M0.6). It walks a directory, reads `*.md` and `*.txt`, and emits one
|
||
`Record` per document section. The chunker never learns it came from a file tree
|
||
rather than a socket — that is the whole point of the trait.
|
||
|
||
**Heading boundary, not record boundary.** A session Record is a natural unit; a
|
||
markdown file is one Record of 8000 tokens with internal structure. Splitting a
|
||
cheatsheet mid-table produces two chunks that are each individually useless.
|
||
Split at `^#{1,6} ` and carry the heading path (`kubectl.md > Common Issues >
|
||
CrashLoopBackOff`) onto every chunk as breadcrumb text.
|
||
|
||
A section longer than `max_tokens` still has to split. Fall back to
|
||
`Boundary::Record` semantics within that section — paragraph boundaries, then
|
||
hard split — and mark the continuation chunks so the projector can rejoin them
|
||
for display.
|
||
|
||
**This task ends at the chunk stream.** No gate call, no embedding, no write. It
|
||
is the M0.7 `--dry-run` shape applied to a doc tree: `mem ref add --dry-run`
|
||
prints the plan and makes zero model calls.
|
||
|
||
**The divergence from the gated path is structural and belongs here.** `run_loop`
|
||
(M1.5) takes a `Query`, and M1.2 makes an empty question a load error because the
|
||
update gate is defined relative to `Q`. A corpus has no standing question, so the
|
||
reference path must be unable to call the recurrence — not merely choose not to.
|
||
Emit a distinct chunk type for this source so `run_loop` does not typecheck
|
||
against it. A `skip_gate: bool` threaded through the shared path is the wrong
|
||
shape: it defaults, and the default is one refactor away from feeding
|
||
documentation to the controller.
|
||
|
||
Source URI is the identity anchor for everything downstream: an absolute path or
|
||
`https://` URL, recorded per chunk, stable across re-ingest.
|
||
|
||
## Steps
|
||
|
||
1. Add `Boundary::Heading` to `ChunkPolicy` in `mem-chunk`.
|
||
2. Implement the heading splitter: parse ATX headings, build the heading path
|
||
stack, emit sections with breadcrumb prefix.
|
||
3. Implement over-long section fallback — paragraph split, then hard split, with
|
||
a `continuation: true` marker on chunks 2..n.
|
||
4. Implement `DocCorpusSource` in `mem-ingest`: walk dir, filter extensions, skip
|
||
dotfiles and anything over a size ceiling, emit `Record` per section.
|
||
5. Record `source_uri` and per-document `sha256` on every emitted record.
|
||
6. Wire `mem ref add --dry-run <path>` to print the chunk plan: file, heading
|
||
path, token count, chunk count.
|
||
|
||
## Acceptance
|
||
|
||
- A doc tree yields one chunk per heading section, breadcrumbs attached.
|
||
- No chunk crosses a heading boundary unless the section exceeded `max_tokens`.
|
||
- An 8000-token section splits and every piece after the first is marked as a
|
||
continuation.
|
||
- `--dry-run` makes zero HTTP calls.
|
||
- `DocCorpusSource` compiles against `RecordSource` with no trait change.
|
||
|
||
## Verify
|
||
|
||
**Harness:** a fixture doc tree under `fixtures/refcorpus/` — one small file, one
|
||
file with nested headings, one file with a single 8000-token section, one
|
||
non-markdown file that must be skipped.
|
||
|
||
**Integration test** — `tests/it_doc_corpus.rs`:
|
||
1. `a1_section_per_heading` — nested-heading fixture yields exactly one chunk per
|
||
ATX heading; assert count and order.
|
||
2. `a2_breadcrumb_path` — a chunk under `## Common Issues > ### CrashLoopBackOff`
|
||
carries the full heading path, not just the leaf.
|
||
3. `a3_no_mid_section_split` — for every chunk, assert it contains at most one
|
||
heading line and that heading is its first line.
|
||
4. `a4_oversize_section_splits` — the 8000-token fixture yields >1 chunk, all
|
||
under `max_tokens`, with `continuation: true` on all but the first.
|
||
5. `a5_extension_filter` — the non-markdown file produces no chunks.
|
||
6. `a6_source_uri_stable` — running the walk twice yields identical
|
||
`(source_uri, sha256)` pairs.
|
||
7. `a7_dry_run_no_network` — run under a transport that panics on any request;
|
||
assert `--dry-run` completes.
|
||
8. `a8_trait_object_safe` — `DocCorpusSource` is usable everywhere the pi adapter
|
||
is, via the same `RecordSource` bound.
|
||
9. `a9_reference_chunks_reject_the_loop` — a compile-fail test (`trybuild`)
|
||
asserting `run_loop` cannot be called with this source's chunk type. The
|
||
guarantee is "impossible", so the test has to be a compile error; a runtime
|
||
assertion proves only that today's caller happens not to do it.
|
||
|
||
**Command:** `cargo test -p mem-ingest doc_corpus`
|
||
|
||
**False pass:**
|
||
- Asserting chunk count only. A splitter that emits the right number of chunks
|
||
by hard-splitting on token count hits the count and fails assertion 3, which
|
||
is the one that proves headings were used at all.
|
||
- Testing the walk on a single flat file. Nested heading paths are where the
|
||
breadcrumb logic breaks, and a flat fixture never exercises the stack.
|
||
|
||
## Traps
|
||
|
||
- Emitting the breadcrumb as metadata only. The embedding is computed over chunk
|
||
text; a heading path that is not *in* the text does not reach the vector, and
|
||
"CrashLoopBackOff" stops being findable from the section body alone.
|
||
- Treating setext headings (`===` underlines) as prose. They are rarer in
|
||
generated docs but they exist, and a file that uses them degrades silently to
|
||
one enormous chunk.
|
||
- Walking symlinks. A docs tree with a self-referential link makes the walk hang
|
||
with no output, which reads as a slow embed rather than a loop.
|
||
- Adding the corpus to `sources:` in a standing-query YAML. That list names the
|
||
*evidence* sources for a question; a corpus listed there is documentation
|
||
entering the gate, which is the one outcome this phase exists to prevent.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — reference corpora, `mem-chunk`
|