poimen-memory/tasks/M0.3-recordsource-and-chunkpolicy.md

# M0.3 — `RecordSource` trait + `ChunkPolicy`

| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.2 |

## Goal

The seam where new input kinds arrive, shaped as a stream from the first commit
so a future streaming source implements a trait instead of forcing a rewrite.

## Facts (inlined — no spec read needed)

```rust
pub trait RecordSource {
    /// Sources decide how to produce records; the chunker never learns
    /// whether they came from pi, claude, or a socket.
    fn records(self) -> impl Stream<Item = Result<Record>>;
}

pub struct ChunkPolicy {
    pub max_tokens: usize,     // 5000 — GRU-Mem paper default
    pub split_on: Boundary,    // Boundary::Record — never mid-Record
    pub flush: FlushTrigger,
}

pub enum FlushTrigger {
    Tokens(usize),
    // OrIdle(Duration) lands with the first streaming source. Carrying the
    // enum now means that change is one variant, not a signature change
    // threaded through the loop.
}

pub fn chunks<S: RecordSource>(src: S, p: ChunkPolicy) -> impl Stream<Item = Chunk>;
```

Why a stream when both current sources are files: sources today have an EOF;
telemetry, a live session tail, or a broker will not. Batch sources become
streams for free via `futures::stream::iter`, so this costs nothing today and
removes a rewrite later. The rest of the stack is already tokio.

**A single `Record` larger than `max_tokens` is not an error.** Tool results can
be enormous. It becomes a chunk of one, over budget, and the chunker records that
it did — silently truncating would destroy evidence, and silently dropping would
lose it.

## Steps

1. Define `RecordSource`, `ChunkPolicy`, `Boundary`, `FlushTrigger` in `mem-chunk`.
2. Implement `chunks()` as a `Stream` adapter that accumulates until the next
   record would exceed `max_tokens`, then yields.
3. Never split a `Record`. An oversized single record yields alone, with
   `Chunk::over_budget = true`.
4. `t` is 1-based and contiguous across the whole stream.
5. Provide `VecSource(Vec<Record>)` implementing `RecordSource` for tests, so
   chunking is testable with no I/O and no model.
6. Token counting is behind a `TokenCounter` trait — M0.4 supplies the real one;
   a `CharsOverFour` stub is enough here.

## Acceptance

- Chunk boundaries never fall inside a `Record`.
- Chunk `t` values are 1-based, contiguous, no gaps.
- An oversized single record yields one chunk flagged `over_budget`.
- Concatenating all chunks' records reproduces the input sequence exactly.

## Verify

**Harness:** `VecSource` + the stub counter. No files, no network.

**Integration test** — `tests/it_chunking.rs`:
1. `a1_no_record_is_split` — for every chunk, every record equals some input
   record byte for byte.
2. `a2_lossless` — flatten all chunk records; assert the sequence equals the
   input sequence, same order, same length.
3. `a3_t_is_contiguous` — assert `t` values are exactly `1..=n`.
4. `a4_respects_budget` — every chunk is either under `max_tokens` or has exactly
   one record and `over_budget = true`.
5. `a5_oversized_record_survives` — feed one record of 3× budget; assert it
   appears whole in the output, not truncated and not dropped.
6. `a6_empty_source` — zero records yields zero chunks, no panic.

**Command:** `cargo test -p mem-chunk chunking`

**False pass:**
- Testing only with uniformly small records. The budget logic is never exercised
  and the oversized path never runs. Assertion 5 is the guard.
- Asserting chunk count rather than losslessness. A chunker that drops the final
  partial chunk produces a plausible count and loses the tail — assertion 2 is
  what catches it.

## Traps

- Truncating an oversized record to fit the budget. That is silent evidence
  destruction, and the update gate will later be blamed for missing it.
- Materializing the stream into a `Vec` inside `chunks()`. It compiles, passes
  every test here, and defeats the entire reason this crate is separate.

---

Background: [DESIGN.md](../DESIGN.md) — `mem-chunk`, stream-shaped from day one