132 lines
4.9 KiB
Markdown
132 lines
4.9 KiB
Markdown
# M0.5 — pi session adapter
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M0 — Read-only spine |
|
||
| Size | M — 1–3 days |
|
||
| Status | ✅ Done |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M0.3 |
|
||
|
||
## Goal
|
||
|
||
Turn pi's session JSONL into normalised `Record`s, and resolve the project key
|
||
from the path without guessing.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
Layout, verified on this machine:
|
||
|
||
```
|
||
~/.pi/agent/sessions/--Users-rockliang-workplace-Poimen-agent-rust--/<ts>_<uuid>.jsonl
|
||
└─ cwd with / replaced by -, wrapped in leading and trailing --
|
||
```
|
||
|
||
Record types observed in a real 2902-message session:
|
||
|
||
```
|
||
message 2902
|
||
model_change 112
|
||
thinking_level_change 16
|
||
compaction 8
|
||
session 1 <- always first line
|
||
```
|
||
|
||
Shapes:
|
||
|
||
```jsonc
|
||
// first line
|
||
{"type":"session","version":..,"id":"..","timestamp":"..","cwd":"/Users/.../Poimen"}
|
||
// message
|
||
{"type":"message","id":"..","parentId":"..","timestamp":"..",
|
||
"message":{"role":"assistant|user|toolResult","content":..,"timestamp":".."}}
|
||
```
|
||
|
||
Role distribution in that same session — this is the whole reason the update gate
|
||
exists:
|
||
|
||
```
|
||
assistant 1445
|
||
toolResult 1261 43%, mostly evidence-free
|
||
user 196
|
||
```
|
||
|
||
`cwd` in the `session` header is authoritative for the project key. The directory
|
||
name is a lossy encoding (a real `-` in a path is indistinguishable from a
|
||
separator) — **parse `cwd`, do not decode the directory name.**
|
||
|
||
`content` is not always a string. Assistant messages carry content blocks; tool
|
||
results carry structured payloads. Normalise to text, and keep the block type in
|
||
`Provenance` so a later filter can act on it.
|
||
|
||
## Steps
|
||
|
||
1. Implement `PiSessionSource` in `mem-ingest`, implementing `RecordSource`.
|
||
2. Read the first line, require `type == "session"`, take `cwd` as the project
|
||
key. A file whose first line is not a session header is an error naming the
|
||
file, not a skip.
|
||
3. Stream subsequent lines; emit a `Record` per `type == "message"`.
|
||
4. Map roles: `user -> Role::User`, `assistant -> Role::Assistant`,
|
||
`toolResult -> Role::ToolResult`.
|
||
5. Flatten `content` to text for all shapes; preserve the original block type in
|
||
`Provenance`.
|
||
6. Ignore `model_change`, `thinking_level_change`. **Do not ignore `compaction`** —
|
||
emit it as `Role::System` with the marker text, because a compaction boundary
|
||
is where context was lost and that is worth seeing in the log.
|
||
7. `Provenance` = `pi:<session-file-stem>` plus the record's `id` and line offset.
|
||
8. A malformed line is a counted, reported skip — never a panic. These files are
|
||
appended to by a live process and the last line may be a partial write.
|
||
|
||
## Acceptance
|
||
|
||
- Project key comes from `cwd`, matching for a path containing a literal `-`.
|
||
- All three roles are emitted with correct counts on a real session.
|
||
- A truncated final line is skipped with a warning, not a panic.
|
||
- Compaction events appear in the record stream.
|
||
|
||
## Verify
|
||
|
||
**Harness:** two committed fixtures — one small hand-built session covering every
|
||
record type and content shape, one real session copied verbatim (secrets
|
||
scrubbed) for volume.
|
||
|
||
**Integration test** — `tests/it_pi_source.rs`:
|
||
1. `a1_project_from_cwd` — fixture whose `cwd` is `/tmp/my-project`; assert the
|
||
key is `/tmp/my-project`, proving the directory name was not decoded.
|
||
2. `a2_role_counts` — on the real fixture, assert exact counts per role.
|
||
3. `a3_content_shapes` — string content, block-array content, and structured tool
|
||
result all flatten to non-empty text.
|
||
4. `a4_truncated_tail` — append half a JSON object; assert the source yields all
|
||
prior records and reports exactly one skip.
|
||
5. `a5_missing_header` — file whose first line is a `message`; assert an error
|
||
naming the path.
|
||
6. `a6_compaction_emitted` — assert compaction events appear as `Role::System`.
|
||
7. `a7_stream_is_lazy` — a source over a 50 MB fixture must yield its first
|
||
record before reading the whole file (assert peak allocation, or instrument
|
||
reads).
|
||
|
||
**Command:** `cargo test -p mem-ingest pi_source`
|
||
|
||
**False pass:**
|
||
- Testing only against the hand-built fixture. It will contain the content shapes
|
||
you thought of, which is the set you already handle. Assertion 2 against a real
|
||
session is what finds the rest.
|
||
- Asserting "no panic" on malformed input without asserting the *count* of
|
||
skips. A source that silently drops every line panics never and ingests
|
||
nothing.
|
||
|
||
## Traps
|
||
|
||
- Decoding the directory name to get the project. `--Users-rockliang-workplace-my-proj--`
|
||
is ambiguous the moment a path component contains `-`, which is common.
|
||
- Treating `content` as `String`. It parses for user messages and fails on
|
||
assistant blocks, so the bug appears to be about assistants specifically and
|
||
wastes an afternoon.
|
||
- Materializing the file. These reach tens of MB and the trait is a stream for a
|
||
reason.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — Context, `mem-chunk`
|