Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)

tasks/M3.6.1-doc-corpus-source.md

@@ -0,0 +1,133 @@
+# M3.6.1 — `DocCorpusSource` + heading-boundary chunking
+
+| Field | Value |
+|---|---|
+| Phase | M3.6 — Reference corpora |
+| Size | M — 1–3 days |
+| Status | ⬜ Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | M3.6.6 |
+| Depends | M0.3, M0.4 |
+
+## Goal
+
+Read a tree of documentation into the same stream shape sessions use, split on
+headings instead of messages, without the gate ever seeing it.
+
+## Facts (inlined — no spec read needed)
+
+```rust
+pub enum Boundary {
+    Record,     // existing — never split mid-Record (sessions)
+    Heading,    // new — split on markdown ATX headings, never mid-section
+}
+```
+
+`DocCorpusSource` is a third `RecordSource` alongside the pi and Claude adapters
+(M0.5, M0.6). It walks a directory, reads `*.md` and `*.txt`, and emits one
+`Record` per document section. The chunker never learns it came from a file tree
+rather than a socket — that is the whole point of the trait.
+
+**Heading boundary, not record boundary.** A session Record is a natural unit; a
+markdown file is one Record of 8000 tokens with internal structure. Splitting a
+cheatsheet mid-table produces two chunks that are each individually useless.
+Split at `^#{1,6} ` and carry the heading path (`kubectl.md > Common Issues >
+CrashLoopBackOff`) onto every chunk as breadcrumb text.
+
+A section longer than `max_tokens` still has to split. Fall back to
+`Boundary::Record` semantics within that section — paragraph boundaries, then
+hard split — and mark the continuation chunks so the projector can rejoin them
+for display.
+
+**This task ends at the chunk stream.** No gate call, no embedding, no write. It
+is the M0.7 `--dry-run` shape applied to a doc tree: `mem ref add --dry-run`
+prints the plan and makes zero model calls.
+
+**The divergence from the gated path is structural and belongs here.** `run_loop`
+(M1.5) takes a `Query`, and M1.2 makes an empty question a load error because the
+update gate is defined relative to `Q`. A corpus has no standing question, so the
+reference path must be unable to call the recurrence — not merely choose not to.
+Emit a distinct chunk type for this source so `run_loop` does not typecheck
+against it. A `skip_gate: bool` threaded through the shared path is the wrong
+shape: it defaults, and the default is one refactor away from feeding
+documentation to the controller.
+
+Source URI is the identity anchor for everything downstream: an absolute path or
+`https://` URL, recorded per chunk, stable across re-ingest.
+
+## Steps
+
+1. Add `Boundary::Heading` to `ChunkPolicy` in `mem-chunk`.
+2. Implement the heading splitter: parse ATX headings, build the heading path
+   stack, emit sections with breadcrumb prefix.
+3. Implement over-long section fallback — paragraph split, then hard split, with
+   a `continuation: true` marker on chunks 2..n.
+4. Implement `DocCorpusSource` in `mem-ingest`: walk dir, filter extensions, skip
+   dotfiles and anything over a size ceiling, emit `Record` per section.
+5. Record `source_uri` and per-document `sha256` on every emitted record.
+6. Wire `mem ref add --dry-run <path>` to print the chunk plan: file, heading
+   path, token count, chunk count.
+
+## Acceptance
+
+- A doc tree yields one chunk per heading section, breadcrumbs attached.
+- No chunk crosses a heading boundary unless the section exceeded `max_tokens`.
+- An 8000-token section splits and every piece after the first is marked as a
+  continuation.
+- `--dry-run` makes zero HTTP calls.
+- `DocCorpusSource` compiles against `RecordSource` with no trait change.
+
+## Verify
+
+**Harness:** a fixture doc tree under `fixtures/refcorpus/` — one small file, one
+file with nested headings, one file with a single 8000-token section, one
+non-markdown file that must be skipped.
+
+**Integration test** — `tests/it_doc_corpus.rs`:
+1. `a1_section_per_heading` — nested-heading fixture yields exactly one chunk per
+   ATX heading; assert count and order.
+2. `a2_breadcrumb_path` — a chunk under `## Common Issues > ### CrashLoopBackOff`
+   carries the full heading path, not just the leaf.
+3. `a3_no_mid_section_split` — for every chunk, assert it contains at most one
+   heading line and that heading is its first line.
+4. `a4_oversize_section_splits` — the 8000-token fixture yields >1 chunk, all
+   under `max_tokens`, with `continuation: true` on all but the first.
+5. `a5_extension_filter` — the non-markdown file produces no chunks.
+6. `a6_source_uri_stable` — running the walk twice yields identical
+   `(source_uri, sha256)` pairs.
+7. `a7_dry_run_no_network` — run under a transport that panics on any request;
+   assert `--dry-run` completes.
+8. `a8_trait_object_safe` — `DocCorpusSource` is usable everywhere the pi adapter
+   is, via the same `RecordSource` bound.
+9. `a9_reference_chunks_reject_the_loop` — a compile-fail test (`trybuild`)
+   asserting `run_loop` cannot be called with this source's chunk type. The
+   guarantee is "impossible", so the test has to be a compile error; a runtime
+   assertion proves only that today's caller happens not to do it.
+
+**Command:** `cargo test -p mem-ingest doc_corpus`
+
+**False pass:**
+- Asserting chunk count only. A splitter that emits the right number of chunks
+  by hard-splitting on token count hits the count and fails assertion 3, which
+  is the one that proves headings were used at all.
+- Testing the walk on a single flat file. Nested heading paths are where the
+  breadcrumb logic breaks, and a flat fixture never exercises the stack.
+
+## Traps
+
+- Emitting the breadcrumb as metadata only. The embedding is computed over chunk
+  text; a heading path that is not *in* the text does not reach the vector, and
+  "CrashLoopBackOff" stops being findable from the section body alone.
+- Treating setext headings (`===` underlines) as prose. They are rarer in
+  generated docs but they exist, and a file that uses them degrades silently to
+  one enormous chunk.
+- Walking symlinks. A docs tree with a self-referential link makes the walk hang
+  with no output, which reads as a slow embed rather than a loop.
+- Adding the corpus to `sources:` in a standing-query YAML. That list names the
+  *evidence* sources for a question; a corpus listed there is documentation
+  entering the gate, which is the one outcome this phase exists to prevent.
+
+---
+
+Background: [DESIGN.md](../DESIGN.md) — reference corpora, `mem-chunk`