poimen-memory/tasks/M0.5-pi-session-adapter.md

M0.5 — pi session adapter

Field	Value
Phase	M0 — Read-only spine
Size	M — 1–3 days
Status	✅ Done
Flags	—
Spec	inlined below
Blocks	M0.3

Goal

Turn pi's session JSONL into normalised Records, and resolve the project key from the path without guessing.

Facts (inlined — no spec read needed)

Layout, verified on this machine:

~/.pi/agent/sessions/--Users-rockliang-workplace-Poimen-agent-rust--/<ts>_<uuid>.jsonl
                     └─ cwd with / replaced by -, wrapped in leading and trailing --

Record types observed in a real 2902-message session:

message                2902
model_change            112
thinking_level_change    16
compaction                8
session                   1     <- always first line

Shapes:

// first line
{"type":"session","version":..,"id":"..","timestamp":"..","cwd":"/Users/.../Poimen"}
// message
{"type":"message","id":"..","parentId":"..","timestamp":"..",
 "message":{"role":"assistant|user|toolResult","content":..,"timestamp":".."}}

Role distribution in that same session — this is the whole reason the update gate exists:

assistant   1445
toolResult  1261     43%, mostly evidence-free
user         196

cwd in the session header is authoritative for the project key. The directory name is a lossy encoding (a real - in a path is indistinguishable from a separator) — parse cwd, do not decode the directory name.

content is not always a string. Assistant messages carry content blocks; tool results carry structured payloads. Normalise to text, and keep the block type in Provenance so a later filter can act on it.

Steps

1. Implement PiSessionSource in mem-ingest, implementing RecordSource.
2. Read the first line, require type == "session", take cwd as the project key. A file whose first line is not a session header is an error naming the file, not a skip.
3. Stream subsequent lines; emit a Record per type == "message".
4. Map roles: user -> Role::User, assistant -> Role::Assistant, toolResult -> Role::ToolResult.
5. Flatten content to text for all shapes; preserve the original block type in Provenance.
6. Ignore model_change, thinking_level_change. Do not ignore compaction — emit it as Role::System with the marker text, because a compaction boundary is where context was lost and that is worth seeing in the log.
7. Provenance = pi:<session-file-stem> plus the record's id and line offset.
8. A malformed line is a counted, reported skip — never a panic. These files are appended to by a live process and the last line may be a partial write.

Acceptance

• Project key comes from cwd, matching for a path containing a literal -.
• All three roles are emitted with correct counts on a real session.
• A truncated final line is skipped with a warning, not a panic.
• Compaction events appear in the record stream.

Verify

Harness: two committed fixtures — one small hand-built session covering every record type and content shape, one real session copied verbatim (secrets scrubbed) for volume.

Integration test — tests/it_pi_source.rs:

1. a1_project_from_cwd — fixture whose cwd is /tmp/my-project; assert the key is /tmp/my-project, proving the directory name was not decoded.
2. a2_role_counts — on the real fixture, assert exact counts per role.
3. a3_content_shapes — string content, block-array content, and structured tool result all flatten to non-empty text.
4. a4_truncated_tail — append half a JSON object; assert the source yields all prior records and reports exactly one skip.
5. a5_missing_header — file whose first line is a message; assert an error naming the path.
6. a6_compaction_emitted — assert compaction events appear as Role::System.
7. a7_stream_is_lazy — a source over a 50 MB fixture must yield its first record before reading the whole file (assert peak allocation, or instrument reads).

Command: cargo test -p mem-ingest pi_source

False pass:

• Testing only against the hand-built fixture. It will contain the content shapes you thought of, which is the set you already handle. Assertion 2 against a real session is what finds the rest.
• Asserting "no panic" on malformed input without asserting the count of skips. A source that silently drops every line panics never and ingests nothing.

Traps

• Decoding the directory name to get the project. --Users-rockliang-workplace-my-proj-- is ambiguous the moment a path component contains -, which is common.
• Treating content as String. It parses for user messages and fails on assistant blocks, so the bug appears to be about assistants specifically and wastes an afternoon.
• Materializing the file. These reach tens of MB and the trait is a stream for a reason.

---

Background: DESIGN.md — Context, mem-chunk