2026-08-19 09:52:07 -07:00
# M1.7 — `mem ingest` end to end
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–
2026-08-22 21:43:23 -07:00
| Status | ✅ Done |
2026-08-19 09:52:07 -07:00
| Flags | — |
| Spec | inlined below |
| Blocks | M1.6 |
## Goal
One command that reads a real project and produces a real log — and reports the
number that says whether the gate works.
2026-08-22 21:43:23 -07:00
## Files
| Action | Path |
|---|---|
| Modify | `crates/mem-cli/src/main.rs` — extend `Commands::Ingest` with `--query` and `--resume` flags; replace stub `cmd_ingest()` body with real pipeline |
| Create | `tests/it_ingest.rs` — integration tests (workspace root, 7 assertions) |
## Dependencies
**None new.** All crates already depend on what they need. This task wires existing
pieces together.
## Existing code to reuse
- `cmd_ingest()` in `main.rs` — **replace the stub body** , keep the CLI struct
- `PiSessionSource` from `mem-ingest/src/pi_session.rs` — already works (M0.5)
- `ClaudeTranscriptSource` from `mem-ingest/src/claude_transcript.rs` — already works (M0.6)
- `chunks()` from `mem-chunk/src/chunker.rs` — already works (M0.3)
- `ChunkPolicy` from `mem-chunk/src/chunk_policy.rs` — already works
- `QuerySet::load()` from `mem-core/src/query.rs` — from M1.2
- `run_loop()` from `mem-core/src/gated_loop.rs` — from M1.5
- `LogWriter` from `mem-store/src/event_log.rs` — from M1.6
- `ChatClient` from `mem-llm/src/chat.rs` — from M1.1
## Wiring diagram
```
CLI: mem ingest --project poimen --query tool-failures
│
├─ QuerySet::load("queries/poimen.yaml") ← M1.2
│ └─ query = set.by_id("tool-failures")
│
├─ PiSessionSource::new(session_files) ← M0.5 (existing)
│ └─ source.records() → Stream<Record>
│
├─ chunks(source, policy) ← M0.3 (existing)
│ └─ Stream<Chunk>
│
├─ ChatClient::new(base_url, api_key, model) ← M1.1
│
├─ run_loop(L1, query, chunks, client, config) ← M1.5
│ └─ RunOutcome { events, chunks_seen, chunks_used, ... }
│
├─ LogWriter::open(project, query, run_id) ← M1.6
│ └─ write events to log/poimen/tool-failures/<ulid>.jsonl
│
└─ Print summary:
chunks 412 used 17 update-rate 4.1% memory 142tok elapsed 6m12s
```
2026-08-19 09:52:07 -07:00
## Facts (inlined — no spec read needed)
```
mem ingest --project poimen --query infra-root-causes
mem ingest --project poimen # all queries in the set
mem ingest --project poimen --limit 50 # first 50 chunks, for iterating
mem ingest --project poimen --resume # skip chunks already in the log
```
Progress output, because a run that prints nothing cannot be distinguished from
one that has hung:
```
[ 17/412] t=17 update=yes mem=142tok 1.9s
[ 18/412] t=18 update=no mem=142tok 0.8s
...
run 01HXYZ chunks 412 used 17 update-rate 4.1% memory 142tok elapsed 6m12s
```
**Update-rate is the headline number.** Tool results are ~43% of records and
mostly evidence-free; a correct gate rejects the large majority of chunks. A rate
above ~30% means the gate is not discriminating and the run is an expensive
summarizer — that is the paper's memory-explosion failure and it is what M1.8
gates on.
Runs are long. 412 chunks at ~1– –
chunk costs a model call, so `--resume` is not a nicety.
## Steps
1. Wire adapters (M0.5/M0.6) → chunker (M0.3) → loop (M1.5) → log (M1.6).
2. Per-chunk progress line to stderr; summary to stdout so it pipes cleanly.
3. Report update-rate in the summary and as `--format json` .
4. `--resume` reads the existing log, finds the highest `t` with a `gate` record,
and restarts from `t+1` with that turn's memory.
5. `--limit` caps chunks processed.
6. Exit non-zero if the run did not reach `run_end` .
7. Ctrl-C finishes the in-flight turn, writes `run_end` , exits — no half-turn.
## Acceptance
- A real project produces a complete log with `run_end` .
- Reported update-rate equals the value computed independently from the log.
- `--resume` on a complete log is a no-op; on a partial one it continues.
- Interrupt produces a valid log.
## Verify
**Harness:** scripted client for determinism, plus one live `#[ignore]` run.
**Integration test** — `tests/it_ingest.rs` :
1. `a1_produces_complete_log` — scripted run; assert `run_end` present and event
counts match the script.
2. `a2_update_rate_matches_log` — compare the reported rate to
`LogWriter::stats()` recomputed from the file.
3. `a3_resume_is_noop_when_complete` — run, resume, assert zero additional model
calls.
4. `a4_resume_continues_partial` — truncate a log after t=10, resume, assert the
next call is t=11 and memory at t=11 equals the replayed memory at t=10.
5. `a5_interrupt_is_clean` — send SIGINT mid-run; assert the log parses, has
`run_end` , and the last `gate` has a matching `memory` -or-not decision.
6. `a6_limit_respected` — `--limit 5` produces exactly 5 gate records.
7. `a7_live_smoke` — `#[ignore]` ; real gateway, `--limit 20` on a real project;
assert `run_end` and **print** the update-rate for a human to read.
2026-08-22 21:43:23 -07:00
**Command:** `cargo test --test it_ingest` (add `-- --ignored` for a7)
2026-08-19 09:52:07 -07:00
**False pass:**
- Asserting only that the command exits 0. A run whose gate always answers `no`
exits 0, writes a valid log, and has learned nothing — the update-rate is the
only thing that distinguishes it, which is why a7 prints it rather than
merely asserting a run happened.
- Resuming by counting lines rather than reading the highest `t` with a `gate`
record. Line counts break the moment an `evidence` record is present, i.e. as
soon as the gate ever opened.
## Traps
- No progress output. A 14-minute run that prints nothing is indistinguishable
from a hang, and the first instinct will be to kill it.
- Resume that replays from `t=1` with the old memory. It costs a full run and
produces a log with duplicate turns that `mem verify` will reject.
---
Background: [DESIGN.md ](../DESIGN.md ) — Verification, P2
