poimen-memory/tasks/M0.7-ingest-dry-run.md

M0.7 — mem ingest --dry-run

Field	Value
Phase	M0 — Read-only spine
Size	S — under 1 day
Status	✅ Done
Flags	—
Spec	inlined below
Blocks	M0.4, M0.6

Goal

See the chunk plan for a real project before spending a single model call on it.

Facts (inlined — no spec read needed)

--dry-run makes zero network calls. That is the property, not a side effect: this is the last checkpoint before inference, and its value is telling you the corpus is sane while feedback is still free.

Output shape:

project  /Users/rockliang/workplace/Poimen/agent-rust
sources  pi:4 files  claude:2 files
records  3118   (assistant 1445  toolResult 1261  user 196  system 216)
chunks   412    over-budget 3
tokens   min 84  p50 4870  p95 5000  max 11204

over-budget counts single records exceeding the chunk budget (M0.3). A nonzero count is expected — large tool results — and is worth surfacing because it predicts requests the gateway may reject.

Numbers to sanity-check against: a real pi session in this project has 2902 messages with the role split above, and toolResult being ~43% is the signal that the update gate has something to discriminate.

Steps

1. mem ingest --project <path-or-key> --dry-run.
2. Resolve sources: scan both source roots for directories whose cwd matches the project. Report which files matched.
3. Stream records through mem-chunk with the real tokenizer; accumulate stats without retaining chunk bodies.
4. Print the table above. Machine-readable variant behind --format json.
5. --limit <n> to stop after n chunks, for iterating on a large project.
6. Exit non-zero if zero sources matched — a silent empty plan reads like success.

Acceptance

• No network syscall occurs during --dry-run.
• Stats are computed streaming; memory does not scale with corpus size.
• Zero matched sources exits non-zero with a message naming the project key.

Verify

Harness: the fixtures from M0.5/M0.6, plus a network guard.

Integration test — tests/it_dry_run.rs:

1. a1_no_network — run the command with outbound TCP blocked (inject a reqwest client that panics on use, or set an unroutable proxy); assert exit 0.
2. a2_counts_match_sources — record and role counts equal the sum of what the two adapters yield independently.
3. a3_chunk_count_matches_chunker — the reported chunk count equals chunks(...).count() computed separately.
4. a4_over_budget_reported — fixture with one oversized record; assert over-budget 1.
5. a5_empty_project_fails — unknown project key exits non-zero, message names the key.
6. a6_constant_memory — run against a 50 MB fixture; assert peak RSS stays under a bound well below file size.

Command: cargo test -p mem-cli dry_run

False pass:

• Asserting the command exits 0 and printing looks right. A dry run that matched no sources also exits 0 and prints a tidy table of zeros — assertion 5 is what separates them.
• Computing stats by collecting chunks into a Vec first. Every assertion here passes and assertion 6 is the only one that fails, which is why it is present.

Traps

• Making --dry-run construct the LLM client "but not call it". Construction reads credentials and can fail; the guarantee is no network, and the cheapest way to keep it is to not build the client at all.
• Reporting p50 chunk size only. The max is the interesting number — it predicts which requests will be rejected downstream.

---

Background: DESIGN.md — Verification, P1