(plan) system review and break down plans

tasks/M2.6-rebuild-from-log.md

@@ -0,0 +1,98 @@
+# M2.6 — `mem rebuild --from-log`
+
+| Field | Value |
+|---|---|
+| Phase | M2 — Projections |
+| Size | M — 1–3 days |
+| Status | ⬜ Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | M2.4, M2.5 |
+
+## Goal
+
+Drop both projections and rebuild them from the log alone — the command that
+makes "the log is authoritative" a testable claim instead of a slogan.
+
+## Facts (inlined — no spec read needed)
+
+```
+mem rebuild --from-log --project poimen            # both projections
+mem rebuild --from-log --project poimen --vault-only
+mem rebuild --from-log --project poimen --db-only
+```
+
+The claim: **anything not reconstructible from the log has a hidden input, and
+that is a bug.** Rebuild is the executable form of that claim. If it needs the
+existing vault or database to produce correct output, something is being carried
+across that is not in the record.
+
+Rebuild does **no model calls except embeddings**. Gate decisions, memory text and
+provenance are all in the log already; re-running the controller would produce
+different text and defeat the purpose.
+
+Order matters: clear → insert all nodes → insert all edges → project vault. Edges
+before nodes violates the foreign key (M2.4).
+
+Embeddings are the expensive part. Cache by `sha256` so a rebuild after a vault
+template change does not re-embed unchanged nodes.
+
+## Steps
+
+1. `mem rebuild --from-log --project P`.
+2. Read every log file for the project, in run-id order.
+3. `clear_project`, then two-pass node/edge insert, batching embeddings.
+4. Project the vault (M2.5), overwriting.
+5. Embedding cache keyed by sha, on disk under `.cache/`, so it survives runs.
+6. Report counts: nodes by level, edges, embeddings computed vs cached.
+7. Refuse to run if any log file is incomplete (no `run_end`) unless `--allow-partial`
+   — rebuilding from a half-run silently produces a half-memory.
+
+## Acceptance
+
+- Rebuild from an empty database and empty vault produces the full state.
+- Rebuild twice produces identical database rows and identical vault bytes.
+- No controller model calls occur.
+- An incomplete log is refused by default.
+
+## Verify
+
+**Harness:** log fixture, disposable Postgres, temp vault. A controller client
+that panics if called.
+
+**Integration test** — `tests/it_rebuild.rs`:
+1. `a1_from_empty` — drop everything, rebuild, assert node counts per level match
+   the log's records.
+2. `a2_idempotent_db` — rebuild twice, assert row count unchanged and no
+   `created_at` churn on existing rows.
+3. `a3_idempotent_vault` — rebuild twice, assert vault bytes identical.
+4. `a4_no_controller_calls` — inject a panicking chat client; assert rebuild
+   succeeds.
+5. `a5_embedding_cache` — second rebuild computes zero embeddings.
+6. `a6_edge_order` — a log whose first memory references a later-inserted parent
+   still rebuilds, proving two-pass.
+7. `a7_incomplete_refused` — a log with no `run_end` exits non-zero; with
+   `--allow-partial` it succeeds.
+8. `a8_log_is_sufficient` — delete the vault and the database entirely, rebuild,
+   and assert the result equals a committed golden. This is the authority claim.
+
+**Command:** `cargo test -p mem-cli rebuild`
+
+**False pass:**
+- Rebuilding on top of existing state. It masks every hidden input, because the
+  missing piece is already there from the previous run. Assertions 1 and 8 must
+  start from nothing.
+- Asserting row counts only. A rebuild that inserts the right number of rows with
+  wrong `parents` passes; assertion 6 and M2.7's edge closure are what check the
+  graph.
+
+## Traps
+
+- Re-running the controller during rebuild. It produces different memory text
+  every time, the vault never stabilises, and M2.8 can never pass.
+- Caching embeddings by node id rather than content hash. Ids change between
+  rebuilds; hashes do not, which is the whole point of content identity.
+
+---
+
+Background: [DESIGN.md](../DESIGN.md) — Authority model