107 lines
4.1 KiB
Markdown
107 lines
4.1 KiB
Markdown
# M0.3 — `RecordSource` trait + `ChunkPolicy`
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M0 — Read-only spine |
|
||
| Size | M — 1–3 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M0.2 |
|
||
|
||
## Goal
|
||
|
||
The seam where new input kinds arrive, shaped as a stream from the first commit
|
||
so a future streaming source implements a trait instead of forcing a rewrite.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
```rust
|
||
pub trait RecordSource {
|
||
/// Sources decide how to produce records; the chunker never learns
|
||
/// whether they came from pi, claude, or a socket.
|
||
fn records(self) -> impl Stream<Item = Result<Record>>;
|
||
}
|
||
|
||
pub struct ChunkPolicy {
|
||
pub max_tokens: usize, // 5000 — GRU-Mem paper default
|
||
pub split_on: Boundary, // Boundary::Record — never mid-Record
|
||
pub flush: FlushTrigger,
|
||
}
|
||
|
||
pub enum FlushTrigger {
|
||
Tokens(usize),
|
||
// OrIdle(Duration) lands with the first streaming source. Carrying the
|
||
// enum now means that change is one variant, not a signature change
|
||
// threaded through the loop.
|
||
}
|
||
|
||
pub fn chunks<S: RecordSource>(src: S, p: ChunkPolicy) -> impl Stream<Item = Chunk>;
|
||
```
|
||
|
||
Why a stream when both current sources are files: sources today have an EOF;
|
||
telemetry, a live session tail, or a broker will not. Batch sources become
|
||
streams for free via `futures::stream::iter`, so this costs nothing today and
|
||
removes a rewrite later. The rest of the stack is already tokio.
|
||
|
||
**A single `Record` larger than `max_tokens` is not an error.** Tool results can
|
||
be enormous. It becomes a chunk of one, over budget, and the chunker records that
|
||
it did — silently truncating would destroy evidence, and silently dropping would
|
||
lose it.
|
||
|
||
## Steps
|
||
|
||
1. Define `RecordSource`, `ChunkPolicy`, `Boundary`, `FlushTrigger` in `mem-chunk`.
|
||
2. Implement `chunks()` as a `Stream` adapter that accumulates until the next
|
||
record would exceed `max_tokens`, then yields.
|
||
3. Never split a `Record`. An oversized single record yields alone, with
|
||
`Chunk::over_budget = true`.
|
||
4. `t` is 1-based and contiguous across the whole stream.
|
||
5. Provide `VecSource(Vec<Record>)` implementing `RecordSource` for tests, so
|
||
chunking is testable with no I/O and no model.
|
||
6. Token counting is behind a `TokenCounter` trait — M0.4 supplies the real one;
|
||
a `CharsOverFour` stub is enough here.
|
||
|
||
## Acceptance
|
||
|
||
- Chunk boundaries never fall inside a `Record`.
|
||
- Chunk `t` values are 1-based, contiguous, no gaps.
|
||
- An oversized single record yields one chunk flagged `over_budget`.
|
||
- Concatenating all chunks' records reproduces the input sequence exactly.
|
||
|
||
## Verify
|
||
|
||
**Harness:** `VecSource` + the stub counter. No files, no network.
|
||
|
||
**Integration test** — `tests/it_chunking.rs`:
|
||
1. `a1_no_record_is_split` — for every chunk, every record equals some input
|
||
record byte for byte.
|
||
2. `a2_lossless` — flatten all chunk records; assert the sequence equals the
|
||
input sequence, same order, same length.
|
||
3. `a3_t_is_contiguous` — assert `t` values are exactly `1..=n`.
|
||
4. `a4_respects_budget` — every chunk is either under `max_tokens` or has exactly
|
||
one record and `over_budget = true`.
|
||
5. `a5_oversized_record_survives` — feed one record of 3× budget; assert it
|
||
appears whole in the output, not truncated and not dropped.
|
||
6. `a6_empty_source` — zero records yields zero chunks, no panic.
|
||
|
||
**Command:** `cargo test -p mem-chunk chunking`
|
||
|
||
**False pass:**
|
||
- Testing only with uniformly small records. The budget logic is never exercised
|
||
and the oversized path never runs. Assertion 5 is the guard.
|
||
- Asserting chunk count rather than losslessness. A chunker that drops the final
|
||
partial chunk produces a plausible count and loses the tail — assertion 2 is
|
||
what catches it.
|
||
|
||
## Traps
|
||
|
||
- Truncating an oversized record to fit the budget. That is silent evidence
|
||
destruction, and the update gate will later be blamed for missing it.
|
||
- Materializing the stream into a `Vec` inside `chunks()`. It compiles, passes
|
||
every test here, and defeats the entire reason this crate is separate.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — `mem-chunk`, stream-shaped from day one
|