(plan) system review and break down plans

tasks/M2.4-pgvector-repo.md

@@ -0,0 +1,99 @@
+# M2.4 — pgvector repository
+
+| Field | Value |
+|---|---|
+| Phase | M2 — Projections |
+| Size | M — 1–3 days |
+| Status | ⬜ Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | M2.3, M2.1 |
+
+## Goal
+
+Write the projection into Postgres idempotently, so rebuild is safe to run at any
+time and produces the same rows.
+
+## Facts (inlined — no spec read needed)
+
+```rust
+async fn upsert_node(&self, node: &MemoryNode, embedding: &[f32]) -> Result<()>;
+async fn insert_edges(&self, child: &Sha256Hash, parents: &[Sha256Hash]) -> Result<()>;
+async fn search(&self, q: &[f32], levels: &[Level], project: &ProjectId, k: usize)
+    -> Result<Vec<ScoredNode>>;
+async fn parents_of(&self, sha: &Sha256Hash) -> Result<Vec<MemoryNode>>;
+async fn clear_project(&self, project: &ProjectId) -> Result<()>;
+```
+
+`upsert_node` is `ON CONFLICT (sha256) DO NOTHING`. Content identity means an
+identical node is the same node; re-running rebuild must not duplicate or churn
+rows. Same for edges on the composite key.
+
+`search` orders by `embedding <=> $1` — cosine distance, matching the
+`vector_cosine_ops` index. Any other operator silently drops to a seq scan.
+
+Edges are inserted **after** both endpoints exist, or the foreign key rejects
+them. Rebuild therefore has two passes: all nodes, then all edges. This is not an
+optimisation; a single-pass insert fails on the first forward reference.
+
+Embeddings are generated in batches of ≤32 (M2.1) and are the expensive part of
+rebuild — batch across nodes, not per node.
+
+## Steps
+
+1. `PgRepo::connect(url)` with a pool; run migrations on connect.
+2. Implement the five methods above.
+3. `upsert_many(nodes)` batching embedding calls at 32 and inserting with a
+   multi-row statement.
+4. Two-pass write: nodes, then edges.
+5. `clear_project` deletes nodes for one project; edges cascade.
+6. Return `ScoredNode { node, distance }` — keep the raw distance, do not convert
+   to a similarity score here. The reranker (M3.2) wants the ordering, and a
+   lossy conversion hides ties.
+
+## Acceptance
+
+- Upserting the same node twice leaves one row.
+- Edges referencing not-yet-inserted parents fail; two-pass write succeeds.
+- `search` returns nearest-first and respects the level filter.
+- `clear_project` removes only that project.
+
+## Verify
+
+**Harness:** disposable Postgres with the production image; a deterministic fake
+embedder (hash → fixed vector) so vector assertions are exact.
+
+**Integration test** — `tests/it_pg_repo.rs`:
+1. `a1_upsert_idempotent` — upsert twice, assert `count(*) == 1`.
+2. `a2_two_pass_required` — single-pass insert with a forward edge reference
+   fails; two-pass succeeds. Proves the ordering constraint is real.
+3. `a3_search_orders_by_distance` — insert three known vectors, assert returned
+   order matches hand-computed cosine distance.
+4. `a4_level_filter` — L0/L1/L2 present; search with `levels=[L1]` returns only
+   L1.
+5. `a5_project_isolation` — two projects with identical text; search one, assert
+   no cross-project results.
+6. `a6_clear_project_scoped` — clear one, assert the other is intact and no
+   orphan edges remain.
+7. `a7_batching` — upsert 100 nodes, assert the embedder saw exactly 4 calls.
+8. `a8_parents_of` — walk a two-level graph, assert the returned parents match.
+
+**Command:** `cargo test -p mem-store pg_repo`
+
+**False pass:**
+- Using a random embedder. Assertion 3 becomes untestable and is usually deleted,
+  which removes the only check that the distance operator matches the index.
+- Testing `search` with one project in the database. Assertion 5 is the only one
+  that catches a missing `WHERE project = $1`, and that bug leaks another
+  project's memory into every answer.
+
+## Traps
+
+- Converting distance to similarity in the repo. It loses precision, and the
+  reranker downstream wants candidates in order rather than scores.
+- Per-node embedding calls. 412 chunks becomes 412 HTTP round trips where 13
+  would do, and rebuild goes from seconds to minutes.
+
+---
+
+Background: [DESIGN.md](../DESIGN.md) — pgvector, retrieval