poimen-memory/tasks/M1.7-ingest-end-to-end.md

M1.7 — mem ingest end to end

Field	Value
Phase	M1 — Gated loop at L1
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M1.6

Goal

One command that reads a real project and produces a real log — and reports the number that says whether the gate works.

Facts (inlined — no spec read needed)

mem ingest --project poimen --query infra-root-causes
mem ingest --project poimen                 # all queries in the set
mem ingest --project poimen --limit 50      # first 50 chunks, for iterating
mem ingest --project poimen --resume        # skip chunks already in the log

Progress output, because a run that prints nothing cannot be distinguished from one that has hung:

[  17/412] t=17 update=yes  mem=142tok  1.9s
[  18/412] t=18 update=no   mem=142tok  0.8s
...
run 01HXYZ  chunks 412  used 17  update-rate 4.1%  memory 142tok  elapsed 6m12s

Update-rate is the headline number. Tool results are ~43% of records and mostly evidence-free; a correct gate rejects the large majority of chunks. A rate above ~30% means the gate is not discriminating and the run is an expensive summarizer — that is the paper's memory-explosion failure and it is what M1.8 gates on.

Runs are long. 412 chunks at ~1–2s each is 6–14 minutes per query, and every chunk costs a model call, so --resume is not a nicety.

Steps

1. Wire adapters (M0.5/M0.6) → chunker (M0.3) → loop (M1.5) → log (M1.6).
2. Per-chunk progress line to stderr; summary to stdout so it pipes cleanly.
3. Report update-rate in the summary and as --format json.
4. --resume reads the existing log, finds the highest t with a gate record, and restarts from t+1 with that turn's memory.
5. --limit caps chunks processed.
6. Exit non-zero if the run did not reach run_end.
7. Ctrl-C finishes the in-flight turn, writes run_end, exits — no half-turn.

Acceptance

• A real project produces a complete log with run_end.
• Reported update-rate equals the value computed independently from the log.
• --resume on a complete log is a no-op; on a partial one it continues.
• Interrupt produces a valid log.

Verify

Harness: scripted client for determinism, plus one live #[ignore] run.

Integration test — tests/it_ingest.rs:

1. a1_produces_complete_log — scripted run; assert run_end present and event counts match the script.
2. a2_update_rate_matches_log — compare the reported rate to LogWriter::stats() recomputed from the file.
3. a3_resume_is_noop_when_complete — run, resume, assert zero additional model calls.
4. a4_resume_continues_partial — truncate a log after t=10, resume, assert the next call is t=11 and memory at t=11 equals the replayed memory at t=10.
5. a5_interrupt_is_clean — send SIGINT mid-run; assert the log parses, has run_end, and the last gate has a matching memory-or-not decision.
6. a6_limit_respected — --limit 5 produces exactly 5 gate records.
7. a7_live_smoke — #[ignore]; real gateway, --limit 20 on a real project; assert run_end and print the update-rate for a human to read.

Command: cargo test -p mem-cli ingest (add -- --ignored for a7)

False pass:

• Asserting only that the command exits 0. A run whose gate always answers no exits 0, writes a valid log, and has learned nothing — the update-rate is the only thing that distinguishes it, which is why a7 prints it rather than merely asserting a run happened.
• Resuming by counting lines rather than reading the highest t with a gate record. Line counts break the moment an evidence record is present, i.e. as soon as the gate ever opened.

Traps

• No progress output. A 14-minute run that prints nothing is indistinguishable from a hang, and the first instinct will be to kill it.
• Resume that replays from t=1 with the old memory. It costs a full run and produces a log with duplicate turns that mem verify will reject.

---

Background: DESIGN.md — Verification, P2