(plan) system review and break down plans
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,131 @@
|
||||
# M0.5 — pi session adapter
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M0 — Read-only spine |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M0.3 |
|
||||
|
||||
## Goal
|
||||
|
||||
Turn pi's session JSONL into normalised `Record`s, and resolve the project key
|
||||
from the path without guessing.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Layout, verified on this machine:
|
||||
|
||||
```
|
||||
~/.pi/agent/sessions/--Users-rockliang-workplace-Poimen-agent-rust--/<ts>_<uuid>.jsonl
|
||||
└─ cwd with / replaced by -, wrapped in leading and trailing --
|
||||
```
|
||||
|
||||
Record types observed in a real 2902-message session:
|
||||
|
||||
```
|
||||
message 2902
|
||||
model_change 112
|
||||
thinking_level_change 16
|
||||
compaction 8
|
||||
session 1 <- always first line
|
||||
```
|
||||
|
||||
Shapes:
|
||||
|
||||
```jsonc
|
||||
// first line
|
||||
{"type":"session","version":..,"id":"..","timestamp":"..","cwd":"/Users/.../Poimen"}
|
||||
// message
|
||||
{"type":"message","id":"..","parentId":"..","timestamp":"..",
|
||||
"message":{"role":"assistant|user|toolResult","content":..,"timestamp":".."}}
|
||||
```
|
||||
|
||||
Role distribution in that same session — this is the whole reason the update gate
|
||||
exists:
|
||||
|
||||
```
|
||||
assistant 1445
|
||||
toolResult 1261 43%, mostly evidence-free
|
||||
user 196
|
||||
```
|
||||
|
||||
`cwd` in the `session` header is authoritative for the project key. The directory
|
||||
name is a lossy encoding (a real `-` in a path is indistinguishable from a
|
||||
separator) — **parse `cwd`, do not decode the directory name.**
|
||||
|
||||
`content` is not always a string. Assistant messages carry content blocks; tool
|
||||
results carry structured payloads. Normalise to text, and keep the block type in
|
||||
`Provenance` so a later filter can act on it.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Implement `PiSessionSource` in `mem-ingest`, implementing `RecordSource`.
|
||||
2. Read the first line, require `type == "session"`, take `cwd` as the project
|
||||
key. A file whose first line is not a session header is an error naming the
|
||||
file, not a skip.
|
||||
3. Stream subsequent lines; emit a `Record` per `type == "message"`.
|
||||
4. Map roles: `user -> Role::User`, `assistant -> Role::Assistant`,
|
||||
`toolResult -> Role::ToolResult`.
|
||||
5. Flatten `content` to text for all shapes; preserve the original block type in
|
||||
`Provenance`.
|
||||
6. Ignore `model_change`, `thinking_level_change`. **Do not ignore `compaction`** —
|
||||
emit it as `Role::System` with the marker text, because a compaction boundary
|
||||
is where context was lost and that is worth seeing in the log.
|
||||
7. `Provenance` = `pi:<session-file-stem>` plus the record's `id` and line offset.
|
||||
8. A malformed line is a counted, reported skip — never a panic. These files are
|
||||
appended to by a live process and the last line may be a partial write.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Project key comes from `cwd`, matching for a path containing a literal `-`.
|
||||
- All three roles are emitted with correct counts on a real session.
|
||||
- A truncated final line is skipped with a warning, not a panic.
|
||||
- Compaction events appear in the record stream.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** two committed fixtures — one small hand-built session covering every
|
||||
record type and content shape, one real session copied verbatim (secrets
|
||||
scrubbed) for volume.
|
||||
|
||||
**Integration test** — `tests/it_pi_source.rs`:
|
||||
1. `a1_project_from_cwd` — fixture whose `cwd` is `/tmp/my-project`; assert the
|
||||
key is `/tmp/my-project`, proving the directory name was not decoded.
|
||||
2. `a2_role_counts` — on the real fixture, assert exact counts per role.
|
||||
3. `a3_content_shapes` — string content, block-array content, and structured tool
|
||||
result all flatten to non-empty text.
|
||||
4. `a4_truncated_tail` — append half a JSON object; assert the source yields all
|
||||
prior records and reports exactly one skip.
|
||||
5. `a5_missing_header` — file whose first line is a `message`; assert an error
|
||||
naming the path.
|
||||
6. `a6_compaction_emitted` — assert compaction events appear as `Role::System`.
|
||||
7. `a7_stream_is_lazy` — a source over a 50 MB fixture must yield its first
|
||||
record before reading the whole file (assert peak allocation, or instrument
|
||||
reads).
|
||||
|
||||
**Command:** `cargo test -p mem-ingest pi_source`
|
||||
|
||||
**False pass:**
|
||||
- Testing only against the hand-built fixture. It will contain the content shapes
|
||||
you thought of, which is the set you already handle. Assertion 2 against a real
|
||||
session is what finds the rest.
|
||||
- Asserting "no panic" on malformed input without asserting the *count* of
|
||||
skips. A source that silently drops every line panics never and ingests
|
||||
nothing.
|
||||
|
||||
## Traps
|
||||
|
||||
- Decoding the directory name to get the project. `--Users-rockliang-workplace-my-proj--`
|
||||
is ambiguous the moment a path component contains `-`, which is common.
|
||||
- Treating `content` as `String`. It parses for user messages and fails on
|
||||
assistant blocks, so the bug appears to be about assistants specifically and
|
||||
wastes an afternoon.
|
||||
- Materializing the file. These reach tens of MB and the trait is a stream for a
|
||||
reason.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Context, `mem-chunk`
|
||||
Reference in New Issue
Block a user