poimen-memory/tasks/M3.6.1-doc-corpus-source.md

M3.6.1 — DocCorpusSource + heading-boundary chunking

Field	Value
Phase	M3.6 — Reference corpora
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M3.6.6
Depends	M0.3, M0.4

Goal

Read a tree of documentation into the same stream shape sessions use, split on headings instead of messages, without the gate ever seeing it.

Facts (inlined — no spec read needed)

pub enum Boundary {
    Record,     // existing — never split mid-Record (sessions)
    Heading,    // new — split on markdown ATX headings, never mid-section
}

DocCorpusSource is a third RecordSource alongside the pi and Claude adapters (M0.5, M0.6). It walks a directory, reads *.md and *.txt, and emits one Record per document section. The chunker never learns it came from a file tree rather than a socket — that is the whole point of the trait.

Heading boundary, not record boundary. A session Record is a natural unit; a markdown file is one Record of 8000 tokens with internal structure. Splitting a cheatsheet mid-table produces two chunks that are each individually useless. Split at ^#{1,6} and carry the heading path (kubectl.md > Common Issues > CrashLoopBackOff) onto every chunk as breadcrumb text.

A section longer than max_tokens still has to split. Fall back to Boundary::Record semantics within that section — paragraph boundaries, then hard split — and mark the continuation chunks so the projector can rejoin them for display.

This task ends at the chunk stream. No gate call, no embedding, no write. It is the M0.7 --dry-run shape applied to a doc tree: mem ref add --dry-run prints the plan and makes zero model calls.

The divergence from the gated path is structural and belongs here. run_loop (M1.5) takes a Query, and M1.2 makes an empty question a load error because the update gate is defined relative to Q. A corpus has no standing question, so the reference path must be unable to call the recurrence — not merely choose not to. Emit a distinct chunk type for this source so run_loop does not typecheck against it. A skip_gate: bool threaded through the shared path is the wrong shape: it defaults, and the default is one refactor away from feeding documentation to the controller.

Source URI is the identity anchor for everything downstream: an absolute path or https:// URL, recorded per chunk, stable across re-ingest.

Steps

1. Add Boundary::Heading to ChunkPolicy in mem-chunk.
2. Implement the heading splitter: parse ATX headings, build the heading path stack, emit sections with breadcrumb prefix.
3. Implement over-long section fallback — paragraph split, then hard split, with a continuation: true marker on chunks 2..n.
4. Implement DocCorpusSource in mem-ingest: walk dir, filter extensions, skip dotfiles and anything over a size ceiling, emit Record per section.
5. Record source_uri and per-document sha256 on every emitted record.
6. Wire mem ref add --dry-run <path> to print the chunk plan: file, heading path, token count, chunk count.

Acceptance

• A doc tree yields one chunk per heading section, breadcrumbs attached.
• No chunk crosses a heading boundary unless the section exceeded max_tokens.
• An 8000-token section splits and every piece after the first is marked as a continuation.
• --dry-run makes zero HTTP calls.
• DocCorpusSource compiles against RecordSource with no trait change.

Verify

Harness: a fixture doc tree under fixtures/refcorpus/ — one small file, one file with nested headings, one file with a single 8000-token section, one non-markdown file that must be skipped.

Integration test — tests/it_doc_corpus.rs:

1. a1_section_per_heading — nested-heading fixture yields exactly one chunk per ATX heading; assert count and order.
2. a2_breadcrumb_path — a chunk under ## Common Issues > ### CrashLoopBackOff carries the full heading path, not just the leaf.
3. a3_no_mid_section_split — for every chunk, assert it contains at most one heading line and that heading is its first line.
4. a4_oversize_section_splits — the 8000-token fixture yields >1 chunk, all under max_tokens, with continuation: true on all but the first.
5. a5_extension_filter — the non-markdown file produces no chunks.
6. a6_source_uri_stable — running the walk twice yields identical (source_uri, sha256) pairs.
7. a7_dry_run_no_network — run under a transport that panics on any request; assert --dry-run completes.
8. a8_trait_object_safe — DocCorpusSource is usable everywhere the pi adapter is, via the same RecordSource bound.
9. a9_reference_chunks_reject_the_loop — a compile-fail test (trybuild) asserting run_loop cannot be called with this source's chunk type. The guarantee is "impossible", so the test has to be a compile error; a runtime assertion proves only that today's caller happens not to do it.

Command: cargo test -p mem-ingest doc_corpus

False pass:

• Asserting chunk count only. A splitter that emits the right number of chunks by hard-splitting on token count hits the count and fails assertion 3, which is the one that proves headings were used at all.
• Testing the walk on a single flat file. Nested heading paths are where the breadcrumb logic breaks, and a flat fixture never exercises the stack.

Traps

• Emitting the breadcrumb as metadata only. The embedding is computed over chunk text; a heading path that is not in the text does not reach the vector, and "CrashLoopBackOff" stops being findable from the section body alone.
• Treating setext headings (=== underlines) as prose. They are rarer in generated docs but they exist, and a file that uses them degrades silently to one enormous chunk.
• Walking symlinks. A docs tree with a self-referential link makes the walk hang with no output, which reads as a slow embed rather than a loop.
• Adding the corpus to sources: in a standing-query YAML. That list names the evidence sources for a question; a corpus listed there is documentation entering the gate, which is the one outcome this phase exists to prevent.

---

Background: DESIGN.md — reference corpora, mem-chunk