poimen-memory/tasks/M1.7-ingest-end-to-end.md

# M1.7 — `mem ingest` end to end

| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.6 |

## Goal

One command that reads a real project and produces a real log — and reports the
number that says whether the gate works.

## Files

| Action | Path |
|---|---|
| Modify | `crates/mem-cli/src/main.rs` — extend `Commands::Ingest` with `--query` and `--resume` flags; replace stub `cmd_ingest()` body with real pipeline |
| Create | `tests/it_ingest.rs` — integration tests (workspace root, 7 assertions) |

## Dependencies

**None new.** All crates already depend on what they need. This task wires existing
pieces together.

## Existing code to reuse

- `cmd_ingest()` in `main.rs` — **replace the stub body**, keep the CLI struct
- `PiSessionSource` from `mem-ingest/src/pi_session.rs` — already works (M0.5)
- `ClaudeTranscriptSource` from `mem-ingest/src/claude_transcript.rs` — already works (M0.6)
- `chunks()` from `mem-chunk/src/chunker.rs` — already works (M0.3)
- `ChunkPolicy` from `mem-chunk/src/chunk_policy.rs` — already works
- `QuerySet::load()` from `mem-core/src/query.rs` — from M1.2
- `run_loop()` from `mem-core/src/gated_loop.rs` — from M1.5
- `LogWriter` from `mem-store/src/event_log.rs` — from M1.6
- `ChatClient` from `mem-llm/src/chat.rs` — from M1.1

## Wiring diagram

```
CLI: mem ingest --project poimen --query tool-failures
  │
  ├─ QuerySet::load("queries/poimen.yaml")        ← M1.2
  │    └─ query = set.by_id("tool-failures")
  │
  ├─ PiSessionSource::new(session_files)           ← M0.5 (existing)
  │    └─ source.records()  →  Stream<Record>
  │
  ├─ chunks(source, policy)                        ← M0.3 (existing)
  │    └─ Stream<Chunk>
  │
  ├─ ChatClient::new(base_url, api_key, model)     ← M1.1
  │
  ├─ run_loop(L1, query, chunks, client, config)   ← M1.5
  │    └─ RunOutcome { events, chunks_seen, chunks_used, ... }
  │
  ├─ LogWriter::open(project, query, run_id)       ← M1.6
  │    └─ write events to log/poimen/tool-failures/<ulid>.jsonl
  │
  └─ Print summary:
       chunks 412  used 17  update-rate 4.1%  memory 142tok  elapsed 6m12s
```

## Facts (inlined — no spec read needed)

```
mem ingest --project poimen --query infra-root-causes
mem ingest --project poimen                 # all queries in the set
mem ingest --project poimen --limit 50      # first 50 chunks, for iterating
mem ingest --project poimen --resume        # skip chunks already in the log
```

Progress output, because a run that prints nothing cannot be distinguished from
one that has hung:

```
[  17/412] t=17 update=yes  mem=142tok  1.9s
[  18/412] t=18 update=no   mem=142tok  0.8s
...
run 01HXYZ  chunks 412  used 17  update-rate 4.1%  memory 142tok  elapsed 6m12s
```

**Update-rate is the headline number.** Tool results are ~43% of records and
mostly evidence-free; a correct gate rejects the large majority of chunks. A rate
above ~30% means the gate is not discriminating and the run is an expensive
summarizer — that is the paper's memory-explosion failure and it is what M1.8
gates on.

Runs are long. 412 chunks at ~1–2s each is 6–14 minutes per query, and every
chunk costs a model call, so `--resume` is not a nicety.

## Steps

1. Wire adapters (M0.5/M0.6) → chunker (M0.3) → loop (M1.5) → log (M1.6).
2. Per-chunk progress line to stderr; summary to stdout so it pipes cleanly.
3. Report update-rate in the summary and as `--format json`.
4. `--resume` reads the existing log, finds the highest `t` with a `gate` record,
   and restarts from `t+1` with that turn's memory.
5. `--limit` caps chunks processed.
6. Exit non-zero if the run did not reach `run_end`.
7. Ctrl-C finishes the in-flight turn, writes `run_end`, exits — no half-turn.

## Acceptance

- A real project produces a complete log with `run_end`.
- Reported update-rate equals the value computed independently from the log.
- `--resume` on a complete log is a no-op; on a partial one it continues.
- Interrupt produces a valid log.

## Verify

**Harness:** scripted client for determinism, plus one live `#[ignore]` run.

**Integration test** — `tests/it_ingest.rs`:
1. `a1_produces_complete_log` — scripted run; assert `run_end` present and event
   counts match the script.
2. `a2_update_rate_matches_log` — compare the reported rate to
   `LogWriter::stats()` recomputed from the file.
3. `a3_resume_is_noop_when_complete` — run, resume, assert zero additional model
   calls.
4. `a4_resume_continues_partial` — truncate a log after t=10, resume, assert the
   next call is t=11 and memory at t=11 equals the replayed memory at t=10.
5. `a5_interrupt_is_clean` — send SIGINT mid-run; assert the log parses, has
   `run_end`, and the last `gate` has a matching `memory`-or-not decision.
6. `a6_limit_respected` — `--limit 5` produces exactly 5 gate records.
7. `a7_live_smoke` — `#[ignore]`; real gateway, `--limit 20` on a real project;
   assert `run_end` and **print** the update-rate for a human to read.

**Command:** `cargo test --test it_ingest` (add `-- --ignored` for a7)

**False pass:**
- Asserting only that the command exits 0. A run whose gate always answers `no`
  exits 0, writes a valid log, and has learned nothing — the update-rate is the
  only thing that distinguishes it, which is why a7 prints it rather than
  merely asserting a run happened.
- Resuming by counting lines rather than reading the highest `t` with a `gate`
  record. Line counts break the moment an `evidence` record is present, i.e. as
  soon as the gate ever opened.

## Traps

- No progress output. A 14-minute run that prints nothing is indistinguishable
  from a hang, and the first instinct will be to kill it.
- Resume that replays from `t=1` with the old memory. It costs a full run and
  produces a log with duplicate turns that `mem verify` will reject.

---

Background: [DESIGN.md](../DESIGN.md) — Verification, P2