poimen-memory/tasks/M1.6-jsonl-event-log.md

# M1.6 — JSONL event log writer

| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.5 |

## Goal

Write the authoritative record — the one artifact everything else is derived
from, and the one that must survive a crash mid-run.

## Files

| Action | Path |
|---|---|
| Create | `crates/mem-store/src/event_log.rs` — `LogWriter`, `LogReader`, `RunStatus` |
| Replace | `crates/mem-store/src/lib.rs` — replace `pub mod placeholder {}` with `pub mod event_log;` + re-exports |
| Create | `tests/it_event_log.rs` — integration tests (workspace root, 8 assertions) |

## Dependencies

| Crate | Where | Already present? |
|---|---|---|
| `ulid` or `rusty_ulid` | `crates/mem-store/Cargo.toml` | ❌ add — for sortable run IDs |
| `tokio` (fs feature) | `crates/mem-store/Cargo.toml` | ✅ yes |
| `serde`, `serde_json` | `crates/mem-store/Cargo.toml` | ✅ yes |

## Existing code to reuse

- `LoopEvent` from `gated_loop.rs` (M1.5) — the events to serialize
- `Level` from `domain.rs` — carried in every record
- Pattern reference: `lessons_cmd.rs` has a JSONL writer for `Event` types (`events.jsonl`).
  Same concept but **different event schema** and **different storage location**:
  - `lessons_cmd.rs` writes `~/.mem/events.jsonl` (command execution events)
  - M1.6 writes `log/<project>/<query>/<run-id>.jsonl` (gate decision events)
  - Do not unify them. They serve different purposes.

## Output path convention

```
log/
  poimen/
    tool-failures/
      01HXYZ....jsonl     ← ULID, lexicographically sortable by time
    architecture-decisions/
      01HXYZ....jsonl
```

## Facts (inlined — no spec read needed)

Path: `log/<project>/<query-id>/<run-id>.jsonl`. Append-only, one object per line.
**Every record carries `level`.**

```jsonl
{"type":"run","level":"L1","project":"poimen","query_id":"infra-root-causes","input_level":"chunk","model":"qwen2.5:3b-instruct","chunk_tokens":5000,"memory_budget":1024,"exit_gate":false,"ts":"..."}
{"type":"chunk","level":"L0","t":1,"source":"pi:...","span":[0,42],"sha256":"..."}
{"type":"gate","level":"L1","t":1,"update":false,"exit":false,"think":"...","latency_ms":812}
{"type":"evidence","level":"L0","t":7,"source":"pi:...","text":"...","sha256":"..."}
{"type":"memory","level":"L1","t":7,"text":"...","tokens":142,"parents":["<sha>"],"sha256":"..."}
{"type":"run_end","level":"L1","chunks_seen":412,"chunks_used":17,"final_memory_sha":"..."}
```

`evidence` appears **only** when the update gate opened. That is what makes
update-rate directly measurable from the log — `gate` records give the
denominator, `evidence` records the numerator.

This log is authoritative: the vault and pgvector are projections rebuilt from
it. Two consequences — it is tracked in git, and it is never rewritten in place.

A run that crashes leaves a file with no `run_end`. That is a valid, readable
state meaning "incomplete", not corruption. Readers must handle it.

## Steps

1. `LogWriter::open(project, query_id, run_id)` in `mem-store`, creating parents.
2. `append(event)` serializes one line and **flushes**. An unflushed buffer loses
   the last turns of exactly the run you want to debug.
3. `run_id` is a ULID — lexicographically sortable by creation time, so listing
   runs in order is a directory sort.
4. `LogReader` streams events back, tolerating a truncated final line.
5. `replay_memory_at(t)` reconstructs `M_t` from the events alone, proving the log
   is sufficient.
6. `stats()` computes chunks seen/used and update-rate from a log file.
7. Never open in truncate mode. Append only.

## Acceptance

- Every emitted record has a `level`.
- `evidence` count equals the count of `gate` records with `update: true`.
- A file with no `run_end` reads cleanly and reports `incomplete`.
- `replay_memory_at(t)` matches the memory the loop held at `t`.

## Verify

**Harness:** the scripted loop from M1.5 writing to a temp dir, plus a corrupted
fixture.

**Integration test** — `tests/it_event_log.rs`:
1. `a1_every_record_has_level` — parse every line, assert `level` present and in
   `{L0,L1,L2}`.
2. `a2_evidence_matches_update_gates` — count `gate.update==true`, assert equal to
   the `evidence` count.
3. `a3_replay_equals_live` — for every `t`, `replay_memory_at(t)` equals the
   memory the loop held. This is the assertion that proves the authority model.
4. `a4_truncated_tail_reads` — chop the last line mid-object; assert all prior
   events parse and the run reports `incomplete`.
5. `a5_no_run_end_is_incomplete` — a log ending after a `memory` event reports
   incomplete, not an error.
6. `a6_append_only` — write, reopen, write again; assert the first events survive.
7. `a7_flush_per_event` — kill the process (or drop without close) after 3
   appends; assert 3 lines on disk.
8. `a8_run_id_sorts_by_time` — three runs, assert lexicographic order equals
   chronological order.

**Command:** `cargo test --test it_event_log`

**False pass:**
- Asserting the file parses. A writer that omits `evidence` events entirely
  produces a perfectly parseable log with an update-rate of zero — assertion 2 is
  what catches it.
- Testing replay only at the final `t`. A writer that records only the final
  memory passes that and fails assertion 3 at every intermediate turn.
- Omitting assertion 7. Buffered writes pass every test that closes the file
  properly, and lose data in precisely the crash case the log exists for.

## Traps

- Opening with truncate. One accidental re-run erases the authoritative record,
  and the projections are the only surviving copy — inverted authority.
- Gitignoring `log/`. Makes the whole "JSONL is authoritative" claim a fiction.
  `agent-rust/.gitignore` has a bare `tasks` entry that untracks its whole board;
  do not repeat it here.

---

Background: [DESIGN.md](../DESIGN.md) — Storage schemas, authority model