poimen-memory/tasks/M0.3-recordsource-and-chunkpolicy.md

M0.3 — RecordSource trait + ChunkPolicy

Field	Value
Phase	M0 — Read-only spine
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M0.2

Goal

The seam where new input kinds arrive, shaped as a stream from the first commit so a future streaming source implements a trait instead of forcing a rewrite.

Facts (inlined — no spec read needed)

pub trait RecordSource {
    /// Sources decide how to produce records; the chunker never learns
    /// whether they came from pi, claude, or a socket.
    fn records(self) -> impl Stream<Item = Result<Record>>;
}

pub struct ChunkPolicy {
    pub max_tokens: usize,     // 5000 — GRU-Mem paper default
    pub split_on: Boundary,    // Boundary::Record — never mid-Record
    pub flush: FlushTrigger,
}

pub enum FlushTrigger {
    Tokens(usize),
    // OrIdle(Duration) lands with the first streaming source. Carrying the
    // enum now means that change is one variant, not a signature change
    // threaded through the loop.
}

pub fn chunks<S: RecordSource>(src: S, p: ChunkPolicy) -> impl Stream<Item = Chunk>;

Why a stream when both current sources are files: sources today have an EOF; telemetry, a live session tail, or a broker will not. Batch sources become streams for free via futures::stream::iter, so this costs nothing today and removes a rewrite later. The rest of the stack is already tokio.

A single Record larger than max_tokens is not an error. Tool results can be enormous. It becomes a chunk of one, over budget, and the chunker records that it did — silently truncating would destroy evidence, and silently dropping would lose it.

Steps

1. Define RecordSource, ChunkPolicy, Boundary, FlushTrigger in mem-chunk.
2. Implement chunks() as a Stream adapter that accumulates until the next record would exceed max_tokens, then yields.
3. Never split a Record. An oversized single record yields alone, with Chunk::over_budget = true.
4. t is 1-based and contiguous across the whole stream.
5. Provide VecSource(Vec<Record>) implementing RecordSource for tests, so chunking is testable with no I/O and no model.
6. Token counting is behind a TokenCounter trait — M0.4 supplies the real one; a CharsOverFour stub is enough here.

Acceptance

• Chunk boundaries never fall inside a Record.
• Chunk t values are 1-based, contiguous, no gaps.
• An oversized single record yields one chunk flagged over_budget.
• Concatenating all chunks' records reproduces the input sequence exactly.

Verify

Harness: VecSource + the stub counter. No files, no network.

Integration test — tests/it_chunking.rs:

1. a1_no_record_is_split — for every chunk, every record equals some input record byte for byte.
2. a2_lossless — flatten all chunk records; assert the sequence equals the input sequence, same order, same length.
3. a3_t_is_contiguous — assert t values are exactly 1..=n.
4. a4_respects_budget — every chunk is either under max_tokens or has exactly one record and over_budget = true.
5. a5_oversized_record_survives — feed one record of 3× budget; assert it appears whole in the output, not truncated and not dropped.
6. a6_empty_source — zero records yields zero chunks, no panic.

Command: cargo test -p mem-chunk chunking

False pass:

• Testing only with uniformly small records. The budget logic is never exercised and the oversized path never runs. Assertion 5 is the guard.
• Asserting chunk count rather than losslessness. A chunker that drops the final partial chunk produces a plausible count and loses the tail — assertion 2 is what catches it.

Traps

• Truncating an oversized record to fit the budget. That is silent evidence destruction, and the update gate will later be blamed for missing it.
• Materializing the stream into a Vec inside chunks(). It compiles, passes every test here, and defeats the entire reason this crate is separate.

---

Background: DESIGN.md — mem-chunk, stream-shaped from day one