(plan) system review and break down plans

tasks/M0.3-recordsource-and-chunkpolicy.md

@@ -0,0 +1,106 @@
+# M0.3 — `RecordSource` trait + `ChunkPolicy`
+
+| Field | Value |
+|---|---|
+| Phase | M0 — Read-only spine |
+| Size | M — 1–3 days |
+| Status | ⬜ Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | M0.2 |
+
+## Goal
+
+The seam where new input kinds arrive, shaped as a stream from the first commit
+so a future streaming source implements a trait instead of forcing a rewrite.
+
+## Facts (inlined — no spec read needed)
+
+```rust
+pub trait RecordSource {
+    /// Sources decide how to produce records; the chunker never learns
+    /// whether they came from pi, claude, or a socket.
+    fn records(self) -> impl Stream<Item = Result<Record>>;
+}
+
+pub struct ChunkPolicy {
+    pub max_tokens: usize,     // 5000 — GRU-Mem paper default
+    pub split_on: Boundary,    // Boundary::Record — never mid-Record
+    pub flush: FlushTrigger,
+}
+
+pub enum FlushTrigger {
+    Tokens(usize),
+    // OrIdle(Duration) lands with the first streaming source. Carrying the
+    // enum now means that change is one variant, not a signature change
+    // threaded through the loop.
+}
+
+pub fn chunks<S: RecordSource>(src: S, p: ChunkPolicy) -> impl Stream<Item = Chunk>;
+```
+
+Why a stream when both current sources are files: sources today have an EOF;
+telemetry, a live session tail, or a broker will not. Batch sources become
+streams for free via `futures::stream::iter`, so this costs nothing today and
+removes a rewrite later. The rest of the stack is already tokio.
+
+**A single `Record` larger than `max_tokens` is not an error.** Tool results can
+be enormous. It becomes a chunk of one, over budget, and the chunker records that
+it did — silently truncating would destroy evidence, and silently dropping would
+lose it.
+
+## Steps
+
+1. Define `RecordSource`, `ChunkPolicy`, `Boundary`, `FlushTrigger` in `mem-chunk`.
+2. Implement `chunks()` as a `Stream` adapter that accumulates until the next
+   record would exceed `max_tokens`, then yields.
+3. Never split a `Record`. An oversized single record yields alone, with
+   `Chunk::over_budget = true`.
+4. `t` is 1-based and contiguous across the whole stream.
+5. Provide `VecSource(Vec<Record>)` implementing `RecordSource` for tests, so
+   chunking is testable with no I/O and no model.
+6. Token counting is behind a `TokenCounter` trait — M0.4 supplies the real one;
+   a `CharsOverFour` stub is enough here.
+
+## Acceptance
+
+- Chunk boundaries never fall inside a `Record`.
+- Chunk `t` values are 1-based, contiguous, no gaps.
+- An oversized single record yields one chunk flagged `over_budget`.
+- Concatenating all chunks' records reproduces the input sequence exactly.
+
+## Verify
+
+**Harness:** `VecSource` + the stub counter. No files, no network.
+
+**Integration test** — `tests/it_chunking.rs`:
+1. `a1_no_record_is_split` — for every chunk, every record equals some input
+   record byte for byte.
+2. `a2_lossless` — flatten all chunk records; assert the sequence equals the
+   input sequence, same order, same length.
+3. `a3_t_is_contiguous` — assert `t` values are exactly `1..=n`.
+4. `a4_respects_budget` — every chunk is either under `max_tokens` or has exactly
+   one record and `over_budget = true`.
+5. `a5_oversized_record_survives` — feed one record of 3× budget; assert it
+   appears whole in the output, not truncated and not dropped.
+6. `a6_empty_source` — zero records yields zero chunks, no panic.
+
+**Command:** `cargo test -p mem-chunk chunking`
+
+**False pass:**
+- Testing only with uniformly small records. The budget logic is never exercised
+  and the oversized path never runs. Assertion 5 is the guard.
+- Asserting chunk count rather than losslessness. A chunker that drops the final
+  partial chunk produces a plausible count and loses the tail — assertion 2 is
+  what catches it.
+
+## Traps
+
+- Truncating an oversized record to fit the budget. That is silent evidence
+  destruction, and the update gate will later be blamed for missing it.
+- Materializing the stream into a `Vec` inside `chunks()`. It compiles, passes
+  every test here, and defeats the entire reason this crate is separate.
+
+---
+
+Background: [DESIGN.md](../DESIGN.md) — `mem-chunk`, stream-shaped from day one