# M2.4 — pgvector repository

| Field | Value |
|---|---|
| Phase | M2 — Projections |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M2.3, M2.1 |

## Goal

Write the projection into Postgres idempotently, so rebuild is safe to run at any
time and produces the same rows.

## Facts (inlined — no spec read needed)

```rust
async fn upsert_node(&self, node: &MemoryNode, embedding: &[f32]) -> Result<()>;
async fn insert_edges(&self, child: &Sha256Hash, parents: &[Sha256Hash]) -> Result<()>;
async fn search(&self, q: &[f32], levels: &[Level], project: &ProjectId, k: usize)
    -> Result<Vec<ScoredNode>>;
async fn parents_of(&self, sha: &Sha256Hash) -> Result<Vec<MemoryNode>>;
async fn clear_project(&self, project: &ProjectId) -> Result<()>;
```

`upsert_node` is `ON CONFLICT (sha256) DO NOTHING`. Content identity means an
identical node is the same node; re-running rebuild must not duplicate or churn
rows. Same for edges on the composite key.

`search` orders by `embedding <=> $1` — cosine distance, matching the
`vector_cosine_ops` index. Any other operator silently drops to a seq scan.

Edges are inserted **after** both endpoints exist, or the foreign key rejects
them. Rebuild therefore has two passes: all nodes, then all edges. This is not an
optimisation; a single-pass insert fails on the first forward reference.

Embeddings are generated in batches of ≤32 (M2.1) and are the expensive part of
rebuild — batch across nodes, not per node.

## Steps

1. `PgRepo::connect(url)` with a pool; run migrations on connect.
2. Implement the five methods above.
3. `upsert_many(nodes)` batching embedding calls at 32 and inserting with a
   multi-row statement.
4. Two-pass write: nodes, then edges.
5. `clear_project` deletes nodes for one project; edges cascade.
6. Return `ScoredNode { node, distance }` — keep the raw distance, do not convert
   to a similarity score here. The reranker (M3.2) wants the ordering, and a
   lossy conversion hides ties.

## Acceptance

- Upserting the same node twice leaves one row.
- Edges referencing not-yet-inserted parents fail; two-pass write succeeds.
- `search` returns nearest-first and respects the level filter.
- `clear_project` removes only that project.

## Verify

**Harness:** disposable Postgres with the production image; a deterministic fake
embedder (hash → fixed vector) so vector assertions are exact.

**Integration test** — `tests/it_pg_repo.rs`:
1. `a1_upsert_idempotent` — upsert twice, assert `count(*) == 1`.
2. `a2_two_pass_required` — single-pass insert with a forward edge reference
   fails; two-pass succeeds. Proves the ordering constraint is real.
3. `a3_search_orders_by_distance` — insert three known vectors, assert returned
   order matches hand-computed cosine distance.
4. `a4_level_filter` — L0/L1/L2 present; search with `levels=[L1]` returns only
   L1.
5. `a5_project_isolation` — two projects with identical text; search one, assert
   no cross-project results.
6. `a6_clear_project_scoped` — clear one, assert the other is intact and no
   orphan edges remain.
7. `a7_batching` — upsert 100 nodes, assert the embedder saw exactly 4 calls.
8. `a8_parents_of` — walk a two-level graph, assert the returned parents match.

**Command:** `cargo test -p mem-store pg_repo`

**False pass:**
- Using a random embedder. Assertion 3 becomes untestable and is usually deleted,
  which removes the only check that the distance operator matches the index.
- Testing `search` with one project in the database. Assertion 5 is the only one
  that catches a missing `WHERE project = $1`, and that bug leaks another
  project's memory into every answer.

## Traps

- Converting distance to similarity in the repo. It loses precision, and the
  reranker downstream wants candidates in order rather than scores.
- Per-node embedding calls. 412 chunks becomes 412 HTTP round trips where 13
  would do, and rebuild goes from seconds to minutes.

---

Background: [DESIGN.md](../DESIGN.md) — pgvector, retrieval