100 lines
4.0 KiB
Markdown
100 lines
4.0 KiB
Markdown
|
# M2.4 — pgvector repository
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M2 — Projections |
|
|||
|
|
| Size | M — 1–3 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M2.3, M2.1 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Write the projection into Postgres idempotently, so rebuild is safe to run at any
|
|||
|
|
time and produces the same rows.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
```rust
|
|||
|
|
async fn upsert_node(&self, node: &MemoryNode, embedding: &[f32]) -> Result<()>;
|
|||
|
|
async fn insert_edges(&self, child: &Sha256Hash, parents: &[Sha256Hash]) -> Result<()>;
|
|||
|
|
async fn search(&self, q: &[f32], levels: &[Level], project: &ProjectId, k: usize)
|
|||
|
|
-> Result<Vec<ScoredNode>>;
|
|||
|
|
async fn parents_of(&self, sha: &Sha256Hash) -> Result<Vec<MemoryNode>>;
|
|||
|
|
async fn clear_project(&self, project: &ProjectId) -> Result<()>;
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`upsert_node` is `ON CONFLICT (sha256) DO NOTHING`. Content identity means an
|
|||
|
|
identical node is the same node; re-running rebuild must not duplicate or churn
|
|||
|
|
rows. Same for edges on the composite key.
|
|||
|
|
|
|||
|
|
`search` orders by `embedding <=> $1` — cosine distance, matching the
|
|||
|
|
`vector_cosine_ops` index. Any other operator silently drops to a seq scan.
|
|||
|
|
|
|||
|
|
Edges are inserted **after** both endpoints exist, or the foreign key rejects
|
|||
|
|
them. Rebuild therefore has two passes: all nodes, then all edges. This is not an
|
|||
|
|
optimisation; a single-pass insert fails on the first forward reference.
|
|||
|
|
|
|||
|
|
Embeddings are generated in batches of ≤32 (M2.1) and are the expensive part of
|
|||
|
|
rebuild — batch across nodes, not per node.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. `PgRepo::connect(url)` with a pool; run migrations on connect.
|
|||
|
|
2. Implement the five methods above.
|
|||
|
|
3. `upsert_many(nodes)` batching embedding calls at 32 and inserting with a
|
|||
|
|
multi-row statement.
|
|||
|
|
4. Two-pass write: nodes, then edges.
|
|||
|
|
5. `clear_project` deletes nodes for one project; edges cascade.
|
|||
|
|
6. Return `ScoredNode { node, distance }` — keep the raw distance, do not convert
|
|||
|
|
to a similarity score here. The reranker (M3.2) wants the ordering, and a
|
|||
|
|
lossy conversion hides ties.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- Upserting the same node twice leaves one row.
|
|||
|
|
- Edges referencing not-yet-inserted parents fail; two-pass write succeeds.
|
|||
|
|
- `search` returns nearest-first and respects the level filter.
|
|||
|
|
- `clear_project` removes only that project.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** disposable Postgres with the production image; a deterministic fake
|
|||
|
|
embedder (hash → fixed vector) so vector assertions are exact.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_pg_repo.rs`:
|
|||
|
|
1. `a1_upsert_idempotent` — upsert twice, assert `count(*) == 1`.
|
|||
|
|
2. `a2_two_pass_required` — single-pass insert with a forward edge reference
|
|||
|
|
fails; two-pass succeeds. Proves the ordering constraint is real.
|
|||
|
|
3. `a3_search_orders_by_distance` — insert three known vectors, assert returned
|
|||
|
|
order matches hand-computed cosine distance.
|
|||
|
|
4. `a4_level_filter` — L0/L1/L2 present; search with `levels=[L1]` returns only
|
|||
|
|
L1.
|
|||
|
|
5. `a5_project_isolation` — two projects with identical text; search one, assert
|
|||
|
|
no cross-project results.
|
|||
|
|
6. `a6_clear_project_scoped` — clear one, assert the other is intact and no
|
|||
|
|
orphan edges remain.
|
|||
|
|
7. `a7_batching` — upsert 100 nodes, assert the embedder saw exactly 4 calls.
|
|||
|
|
8. `a8_parents_of` — walk a two-level graph, assert the returned parents match.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p mem-store pg_repo`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Using a random embedder. Assertion 3 becomes untestable and is usually deleted,
|
|||
|
|
which removes the only check that the distance operator matches the index.
|
|||
|
|
- Testing `search` with one project in the database. Assertion 5 is the only one
|
|||
|
|
that catches a missing `WHERE project = $1`, and that bug leaks another
|
|||
|
|
project's memory into every answer.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Converting distance to similarity in the repo. It loses precision, and the
|
|||
|
|
reranker downstream wants candidates in order rather than scores.
|
|||
|
|
- Per-node embedding calls. 412 chunks becomes 412 HTTP round trips where 13
|
|||
|
|
would do, and rebuild goes from seconds to minutes.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — pgvector, retrieval
|