poimen-memory/tasks/M1.6-jsonl-event-log.md

M1.6 — JSONL event log writer

Field	Value
Phase	M1 — Gated loop at L1
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M1.5

Goal

Write the authoritative record — the one artifact everything else is derived from, and the one that must survive a crash mid-run.

Facts (inlined — no spec read needed)

Path: log/<project>/<query-id>/<run-id>.jsonl. Append-only, one object per line. Every record carries level.

{"type":"run","level":"L1","project":"poimen","query_id":"infra-root-causes","input_level":"chunk","model":"qwen2.5:3b-instruct","chunk_tokens":5000,"memory_budget":1024,"exit_gate":false,"ts":"..."}
{"type":"chunk","level":"L0","t":1,"source":"pi:...","span":[0,42],"sha256":"..."}
{"type":"gate","level":"L1","t":1,"update":false,"exit":false,"think":"...","latency_ms":812}
{"type":"evidence","level":"L0","t":7,"source":"pi:...","text":"...","sha256":"..."}
{"type":"memory","level":"L1","t":7,"text":"...","tokens":142,"parents":["<sha>"],"sha256":"..."}
{"type":"run_end","level":"L1","chunks_seen":412,"chunks_used":17,"final_memory_sha":"..."}

evidence appears only when the update gate opened. That is what makes update-rate directly measurable from the log — gate records give the denominator, evidence records the numerator.

This log is authoritative: the vault and pgvector are projections rebuilt from it. Two consequences — it is tracked in git, and it is never rewritten in place.

A run that crashes leaves a file with no run_end. That is a valid, readable state meaning "incomplete", not corruption. Readers must handle it.

Steps

1. LogWriter::open(project, query_id, run_id) in mem-store, creating parents.
2. append(event) serializes one line and flushes. An unflushed buffer loses the last turns of exactly the run you want to debug.
3. run_id is a ULID — lexicographically sortable by creation time, so listing runs in order is a directory sort.
4. LogReader streams events back, tolerating a truncated final line.
5. replay_memory_at(t) reconstructs M_t from the events alone, proving the log is sufficient.
6. stats() computes chunks seen/used and update-rate from a log file.
7. Never open in truncate mode. Append only.

Acceptance

• Every emitted record has a level.
• evidence count equals the count of gate records with update: true.
• A file with no run_end reads cleanly and reports incomplete.
• replay_memory_at(t) matches the memory the loop held at t.

Verify

Harness: the scripted loop from M1.5 writing to a temp dir, plus a corrupted fixture.

Integration test — tests/it_event_log.rs:

1. a1_every_record_has_level — parse every line, assert level present and in {L0,L1,L2}.
2. a2_evidence_matches_update_gates — count gate.update==true, assert equal to the evidence count.
3. a3_replay_equals_live — for every t, replay_memory_at(t) equals the memory the loop held. This is the assertion that proves the authority model.
4. a4_truncated_tail_reads — chop the last line mid-object; assert all prior events parse and the run reports incomplete.
5. a5_no_run_end_is_incomplete — a log ending after a memory event reports incomplete, not an error.
6. a6_append_only — write, reopen, write again; assert the first events survive.
7. a7_flush_per_event — kill the process (or drop without close) after 3 appends; assert 3 lines on disk.
8. a8_run_id_sorts_by_time — three runs, assert lexicographic order equals chronological order.

Command: cargo test -p mem-store event_log

False pass:

• Asserting the file parses. A writer that omits evidence events entirely produces a perfectly parseable log with an update-rate of zero — assertion 2 is what catches it.
• Testing replay only at the final t. A writer that records only the final memory passes that and fails assertion 3 at every intermediate turn.
• Omitting assertion 7. Buffered writes pass every test that closes the file properly, and lose data in precisely the crash case the log exists for.

Traps

• Opening with truncate. One accidental re-run erases the authoritative record, and the projections are the only surviving copy — inverted authority.
• Gitignoring log/. Makes the whole "JSONL is authoritative" claim a fiction. agent-rust/.gitignore has a bare tasks entry that untracks its whole board; do not repeat it here.

---

Background: DESIGN.md — Storage schemas, authority model