132 lines
4.9 KiB
Markdown
132 lines
4.9 KiB
Markdown
|
# M0.5 — pi session adapter
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M0 — Read-only spine |
|
|||
|
|
| Size | M — 1–3 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M0.3 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Turn pi's session JSONL into normalised `Record`s, and resolve the project key
|
|||
|
|
from the path without guessing.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
Layout, verified on this machine:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
~/.pi/agent/sessions/--Users-rockliang-workplace-Poimen-agent-rust--/<ts>_<uuid>.jsonl
|
|||
|
|
└─ cwd with / replaced by -, wrapped in leading and trailing --
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Record types observed in a real 2902-message session:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
message 2902
|
|||
|
|
model_change 112
|
|||
|
|
thinking_level_change 16
|
|||
|
|
compaction 8
|
|||
|
|
session 1 <- always first line
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Shapes:
|
|||
|
|
|
|||
|
|
```jsonc
|
|||
|
|
// first line
|
|||
|
|
{"type":"session","version":..,"id":"..","timestamp":"..","cwd":"/Users/.../Poimen"}
|
|||
|
|
// message
|
|||
|
|
{"type":"message","id":"..","parentId":"..","timestamp":"..",
|
|||
|
|
"message":{"role":"assistant|user|toolResult","content":..,"timestamp":".."}}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Role distribution in that same session — this is the whole reason the update gate
|
|||
|
|
exists:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
assistant 1445
|
|||
|
|
toolResult 1261 43%, mostly evidence-free
|
|||
|
|
user 196
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`cwd` in the `session` header is authoritative for the project key. The directory
|
|||
|
|
name is a lossy encoding (a real `-` in a path is indistinguishable from a
|
|||
|
|
separator) — **parse `cwd`, do not decode the directory name.**
|
|||
|
|
|
|||
|
|
`content` is not always a string. Assistant messages carry content blocks; tool
|
|||
|
|
results carry structured payloads. Normalise to text, and keep the block type in
|
|||
|
|
`Provenance` so a later filter can act on it.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Implement `PiSessionSource` in `mem-ingest`, implementing `RecordSource`.
|
|||
|
|
2. Read the first line, require `type == "session"`, take `cwd` as the project
|
|||
|
|
key. A file whose first line is not a session header is an error naming the
|
|||
|
|
file, not a skip.
|
|||
|
|
3. Stream subsequent lines; emit a `Record` per `type == "message"`.
|
|||
|
|
4. Map roles: `user -> Role::User`, `assistant -> Role::Assistant`,
|
|||
|
|
`toolResult -> Role::ToolResult`.
|
|||
|
|
5. Flatten `content` to text for all shapes; preserve the original block type in
|
|||
|
|
`Provenance`.
|
|||
|
|
6. Ignore `model_change`, `thinking_level_change`. **Do not ignore `compaction`** —
|
|||
|
|
emit it as `Role::System` with the marker text, because a compaction boundary
|
|||
|
|
is where context was lost and that is worth seeing in the log.
|
|||
|
|
7. `Provenance` = `pi:<session-file-stem>` plus the record's `id` and line offset.
|
|||
|
|
8. A malformed line is a counted, reported skip — never a panic. These files are
|
|||
|
|
appended to by a live process and the last line may be a partial write.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- Project key comes from `cwd`, matching for a path containing a literal `-`.
|
|||
|
|
- All three roles are emitted with correct counts on a real session.
|
|||
|
|
- A truncated final line is skipped with a warning, not a panic.
|
|||
|
|
- Compaction events appear in the record stream.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** two committed fixtures — one small hand-built session covering every
|
|||
|
|
record type and content shape, one real session copied verbatim (secrets
|
|||
|
|
scrubbed) for volume.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_pi_source.rs`:
|
|||
|
|
1. `a1_project_from_cwd` — fixture whose `cwd` is `/tmp/my-project`; assert the
|
|||
|
|
key is `/tmp/my-project`, proving the directory name was not decoded.
|
|||
|
|
2. `a2_role_counts` — on the real fixture, assert exact counts per role.
|
|||
|
|
3. `a3_content_shapes` — string content, block-array content, and structured tool
|
|||
|
|
result all flatten to non-empty text.
|
|||
|
|
4. `a4_truncated_tail` — append half a JSON object; assert the source yields all
|
|||
|
|
prior records and reports exactly one skip.
|
|||
|
|
5. `a5_missing_header` — file whose first line is a `message`; assert an error
|
|||
|
|
naming the path.
|
|||
|
|
6. `a6_compaction_emitted` — assert compaction events appear as `Role::System`.
|
|||
|
|
7. `a7_stream_is_lazy` — a source over a 50 MB fixture must yield its first
|
|||
|
|
record before reading the whole file (assert peak allocation, or instrument
|
|||
|
|
reads).
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p mem-ingest pi_source`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Testing only against the hand-built fixture. It will contain the content shapes
|
|||
|
|
you thought of, which is the set you already handle. Assertion 2 against a real
|
|||
|
|
session is what finds the rest.
|
|||
|
|
- Asserting "no panic" on malformed input without asserting the *count* of
|
|||
|
|
skips. A source that silently drops every line panics never and ingests
|
|||
|
|
nothing.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Decoding the directory name to get the project. `--Users-rockliang-workplace-my-proj--`
|
|||
|
|
is ambiguous the moment a path component contains `-`, which is common.
|
|||
|
|
- Treating `content` as `String`. It parses for user messages and fails on
|
|||
|
|
assistant blocks, so the bug appears to be about assistants specifically and
|
|||
|
|
wastes an afternoon.
|
|||
|
|
- Materializing the file. These reach tens of MB and the trait is a stream for a
|
|||
|
|
reason.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — Context, `mem-chunk`
|