poimen-memory/tasks/M3.6.1-doc-corpus-source.md

# M3.6.1 — `DocCorpusSource` + heading-boundary chunking

| Field | Value |
|---|---|
| Phase | M3.6 — Reference corpora |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.6.6 |
| Depends | M0.3, M0.4 |

## Goal

Read a tree of documentation into the same stream shape sessions use, split on
headings instead of messages, without the gate ever seeing it.

## Facts (inlined — no spec read needed)

```rust
pub enum Boundary {
    Record,     // existing — never split mid-Record (sessions)
    Heading,    // new — split on markdown ATX headings, never mid-section
}
```

`DocCorpusSource` is a third `RecordSource` alongside the pi and Claude adapters
(M0.5, M0.6). It walks a directory, reads `*.md` and `*.txt`, and emits one
`Record` per document section. The chunker never learns it came from a file tree
rather than a socket — that is the whole point of the trait.

**Heading boundary, not record boundary.** A session Record is a natural unit; a
markdown file is one Record of 8000 tokens with internal structure. Splitting a
cheatsheet mid-table produces two chunks that are each individually useless.
Split at `^#{1,6} ` and carry the heading path (`kubectl.md > Common Issues >
CrashLoopBackOff`) onto every chunk as breadcrumb text.

A section longer than `max_tokens` still has to split. Fall back to
`Boundary::Record` semantics within that section — paragraph boundaries, then
hard split — and mark the continuation chunks so the projector can rejoin them
for display.

**This task ends at the chunk stream.** No gate call, no embedding, no write. It
is the M0.7 `--dry-run` shape applied to a doc tree: `mem ref add --dry-run`
prints the plan and makes zero model calls.

**The divergence from the gated path is structural and belongs here.** `run_loop`
(M1.5) takes a `Query`, and M1.2 makes an empty question a load error because the
update gate is defined relative to `Q`. A corpus has no standing question, so the
reference path must be unable to call the recurrence — not merely choose not to.
Emit a distinct chunk type for this source so `run_loop` does not typecheck
against it. A `skip_gate: bool` threaded through the shared path is the wrong
shape: it defaults, and the default is one refactor away from feeding
documentation to the controller.

Source URI is the identity anchor for everything downstream: an absolute path or
`https://` URL, recorded per chunk, stable across re-ingest.

## Steps

1. Add `Boundary::Heading` to `ChunkPolicy` in `mem-chunk`.
2. Implement the heading splitter: parse ATX headings, build the heading path
   stack, emit sections with breadcrumb prefix.
3. Implement over-long section fallback — paragraph split, then hard split, with
   a `continuation: true` marker on chunks 2..n.
4. Implement `DocCorpusSource` in `mem-ingest`: walk dir, filter extensions, skip
   dotfiles and anything over a size ceiling, emit `Record` per section.
5. Record `source_uri` and per-document `sha256` on every emitted record.
6. Wire `mem ref add --dry-run <path>` to print the chunk plan: file, heading
   path, token count, chunk count.

## Acceptance

- A doc tree yields one chunk per heading section, breadcrumbs attached.
- No chunk crosses a heading boundary unless the section exceeded `max_tokens`.
- An 8000-token section splits and every piece after the first is marked as a
  continuation.
- `--dry-run` makes zero HTTP calls.
- `DocCorpusSource` compiles against `RecordSource` with no trait change.

## Verify

**Harness:** a fixture doc tree under `fixtures/refcorpus/` — one small file, one
file with nested headings, one file with a single 8000-token section, one
non-markdown file that must be skipped.

**Integration test** — `tests/it_doc_corpus.rs`:
1. `a1_section_per_heading` — nested-heading fixture yields exactly one chunk per
   ATX heading; assert count and order.
2. `a2_breadcrumb_path` — a chunk under `## Common Issues > ### CrashLoopBackOff`
   carries the full heading path, not just the leaf.
3. `a3_no_mid_section_split` — for every chunk, assert it contains at most one
   heading line and that heading is its first line.
4. `a4_oversize_section_splits` — the 8000-token fixture yields >1 chunk, all
   under `max_tokens`, with `continuation: true` on all but the first.
5. `a5_extension_filter` — the non-markdown file produces no chunks.
6. `a6_source_uri_stable` — running the walk twice yields identical
   `(source_uri, sha256)` pairs.
7. `a7_dry_run_no_network` — run under a transport that panics on any request;
   assert `--dry-run` completes.
8. `a8_trait_object_safe` — `DocCorpusSource` is usable everywhere the pi adapter
   is, via the same `RecordSource` bound.
9. `a9_reference_chunks_reject_the_loop` — a compile-fail test (`trybuild`)
   asserting `run_loop` cannot be called with this source's chunk type. The
   guarantee is "impossible", so the test has to be a compile error; a runtime
   assertion proves only that today's caller happens not to do it.

**Command:** `cargo test -p mem-ingest doc_corpus`

**False pass:**
- Asserting chunk count only. A splitter that emits the right number of chunks
  by hard-splitting on token count hits the count and fails assertion 3, which
  is the one that proves headings were used at all.
- Testing the walk on a single flat file. Nested heading paths are where the
  breadcrumb logic breaks, and a flat fixture never exercises the stack.

## Traps

- Emitting the breadcrumb as metadata only. The embedding is computed over chunk
  text; a heading path that is not *in* the text does not reach the vector, and
  "CrashLoopBackOff" stops being findable from the section body alone.
- Treating setext headings (`===` underlines) as prose. They are rarer in
  generated docs but they exist, and a file that uses them degrades silently to
  one enormous chunk.
- Walking symlinks. A docs tree with a self-referential link makes the walk hang
  with no output, which reads as a slow embed rather than a loop.
- Adding the corpus to `sources:` in a standing-query YAML. That list names the
  *evidence* sources for a question; a corpus listed there is documentation
  entering the gate, which is the one outcome this phase exists to prevent.

---

Background: [DESIGN.md](../DESIGN.md) — reference corpora, `mem-chunk`