M1.7 — `mem ingest` end to end

Field	Value
Phase	M1 — Gated loop at L1
Size	M — 1–3 days
Status	✅ Done
Flags	—
Spec	inlined below
Blocks	M1.6

Goal

One command that reads a real project and produces a real log — and reports the number that says whether the gate works.

Files

Action	Path
Modify	`crates/mem-cli/src/main.rs` — extend `Commands::Ingest` with `--query` and `--resume` flags; replace stub `cmd_ingest()` body with real pipeline
Create	`tests/it_ingest.rs` — integration tests (workspace root, 7 assertions)

Dependencies

None new. All crates already depend on what they need. This task wires existing pieces together.

Existing code to reuse

cmd_ingest() in main.rs — replace the stub body, keep the CLI struct
PiSessionSource from mem-ingest/src/pi_session.rs — already works (M0.5)
ClaudeTranscriptSource from mem-ingest/src/claude_transcript.rs — already works (M0.6)
chunks() from mem-chunk/src/chunker.rs — already works (M0.3)
ChunkPolicy from mem-chunk/src/chunk_policy.rs — already works
QuerySet::load() from mem-core/src/query.rs — from M1.2
run_loop() from mem-core/src/gated_loop.rs — from M1.5
LogWriter from mem-store/src/event_log.rs — from M1.6
ChatClient from mem-llm/src/chat.rs — from M1.1

Wiring diagram

CLI: mem ingest --project poimen --query tool-failures
  │
  ├─ QuerySet::load("queries/poimen.yaml")        ← M1.2
  │    └─ query = set.by_id("tool-failures")
  │
  ├─ PiSessionSource::new(session_files)           ← M0.5 (existing)
  │    └─ source.records()  →  Stream<Record>
  │
  ├─ chunks(source, policy)                        ← M0.3 (existing)
  │    └─ Stream<Chunk>
  │
  ├─ ChatClient::new(base_url, api_key, model)     ← M1.1
  │
  ├─ run_loop(L1, query, chunks, client, config)   ← M1.5
  │    └─ RunOutcome { events, chunks_seen, chunks_used, ... }
  │
  ├─ LogWriter::open(project, query, run_id)       ← M1.6
  │    └─ write events to log/poimen/tool-failures/<ulid>.jsonl
  │
  └─ Print summary:
       chunks 412  used 17  update-rate 4.1%  memory 142tok  elapsed 6m12s

Facts (inlined — no spec read needed)

mem ingest --project poimen --query infra-root-causes
mem ingest --project poimen                 # all queries in the set
mem ingest --project poimen --limit 50      # first 50 chunks, for iterating
mem ingest --project poimen --resume        # skip chunks already in the log

Progress output, because a run that prints nothing cannot be distinguished from one that has hung:

[  17/412] t=17 update=yes  mem=142tok  1.9s
[  18/412] t=18 update=no   mem=142tok  0.8s
...
run 01HXYZ  chunks 412  used 17  update-rate 4.1%  memory 142tok  elapsed 6m12s

Update-rate is the headline number. Tool results are ~43% of records and mostly evidence-free; a correct gate rejects the large majority of chunks. A rate above ~30% means the gate is not discriminating and the run is an expensive summarizer — that is the paper's memory-explosion failure and it is what M1.8 gates on.

Runs are long. 412 chunks at ~1–2s each is 6–14 minutes per query, and every chunk costs a model call, so --resume is not a nicety.

Steps

Wire adapters (M0.5/M0.6) → chunker (M0.3) → loop (M1.5) → log (M1.6).
Per-chunk progress line to stderr; summary to stdout so it pipes cleanly.
Report update-rate in the summary and as --format json.
--resume reads the existing log, finds the highest t with a gate record, and restarts from t+1 with that turn's memory.
--limit caps chunks processed.
Exit non-zero if the run did not reach run_end.
Ctrl-C finishes the in-flight turn, writes run_end, exits — no half-turn.

Acceptance

A real project produces a complete log with run_end.
Reported update-rate equals the value computed independently from the log.
--resume on a complete log is a no-op; on a partial one it continues.
Interrupt produces a valid log.

Verify

Harness: scripted client for determinism, plus one live #[ignore] run.

Integration test — tests/it_ingest.rs:

a1_produces_complete_log — scripted run; assert run_end present and event counts match the script.
a2_update_rate_matches_log — compare the reported rate to LogWriter::stats() recomputed from the file.
a3_resume_is_noop_when_complete — run, resume, assert zero additional model calls.
a4_resume_continues_partial — truncate a log after t=10, resume, assert the next call is t=11 and memory at t=11 equals the replayed memory at t=10.
a5_interrupt_is_clean — send SIGINT mid-run; assert the log parses, has run_end, and the last gate has a matching memory-or-not decision.
a6_limit_respected — --limit 5 produces exactly 5 gate records.
a7_live_smoke — #[ignore]; real gateway, --limit 20 on a real project; assert run_end and print the update-rate for a human to read.

Command: cargo test --test it_ingest (add -- --ignored for a7)

False pass:

Asserting only that the command exits 0. A run whose gate always answers no exits 0, writes a valid log, and has learned nothing — the update-rate is the only thing that distinguishes it, which is why a7 prints it rather than merely asserting a run happened.
Resuming by counting lines rather than reading the highest t with a gate record. Line counts break the moment an evidence record is present, i.e. as soon as the gate ever opened.

Traps

No progress output. A 14-minute run that prints nothing is indistinguishable from a hang, and the first instinct will be to kill it.
Resume that replays from t=1 with the old memory. It costs a full run and produces a log with duplicate turns that mem verify will reject.

Background: DESIGN.md — Verification, P2

6.0 KiB Raw Permalink Blame History Unescape Escape

M1.7 — mem ingest end to end