98 lines
3.8 KiB
Markdown
98 lines
3.8 KiB
Markdown
|
# M3.3 — `mem query` with provenance
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M3 — L2 synthesis and retrieval |
|
|||
|
|
| Size | M — 1–3 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M3.2, M2.4 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Ask the memory a question and get an answer that can be traced back to the
|
|||
|
|
session it came from.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
mem query "why did requests over 10KB fail?"
|
|||
|
|
mem query --project poimen --levels L1,L2 --k 10 "..."
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Pipeline: embed the question (M2.1) → HNSW recall top-k over `memory_node`
|
|||
|
|
filtered by project and level (M2.4) → rerank the candidates (M3.2) → return with
|
|||
|
|
provenance walked down through `memory_edge`.
|
|||
|
|
|
|||
|
|
Default levels are **L1 and L2**, not L0. L1/L2 are synthesized answers; L0 is raw
|
|||
|
|
evidence and returning it by default buries the answer in transcript. `--levels
|
|||
|
|
L0` exists for "show me the actual source".
|
|||
|
|
|
|||
|
|
Provenance walk: for each returned node, follow `memory_edge` to its parents and
|
|||
|
|
report source and turn. One hop for L1 (to evidence), two for L2 (through L1).
|
|||
|
|
|
|||
|
|
Recall wide, rerank narrow: take 50 from HNSW, rerank, return 5. Cosine distance
|
|||
|
|
alone puts "bananas" close enough to matter; the reranker separated the same pair
|
|||
|
|
by four orders of magnitude.
|
|||
|
|
|
|||
|
|
Output is human-readable by default, `--format json` for programmatic use.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. `mem query [--project P] [--levels L] [--k N] <question>`.
|
|||
|
|
2. Embed, recall 10×k, rerank, truncate to k.
|
|||
|
|
3. Walk edges to build a provenance list per hit; deduplicate sources.
|
|||
|
|
4. Render: score, level, query id, the memory text, then provenance lines.
|
|||
|
|
5. Project defaults to the one inferred from `$PWD`; `--project` overrides. No
|
|||
|
|
project and no match is an error, not an empty result over everything.
|
|||
|
|
6. `--explain` prints the recall candidates and their pre-rerank distances, for
|
|||
|
|
debugging retrieval quality.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- A question with a known answer returns the right L1 note first.
|
|||
|
|
- Every hit carries at least one resolvable provenance entry.
|
|||
|
|
- `--levels L0` returns evidence; the default does not.
|
|||
|
|
- Reranking changes the order versus raw recall on at least one real query.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** a seeded database from a real log, deterministic fake embedder for
|
|||
|
|
the offline assertions, live for quality.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_query.rs`:
|
|||
|
|
1. `a1_known_answer` — seeded with the poimen log, query "why did requests over
|
|||
|
|
10KB fail?"; assert the top hit is the `infra-root-causes` L1 node.
|
|||
|
|
2. `a2_provenance_resolves` — every hit's provenance shas exist in
|
|||
|
|
`memory_node`.
|
|||
|
|
3. `a3_default_excludes_l0` — assert no L0 nodes in default output.
|
|||
|
|
4. `a4_levels_flag` — `--levels L0` returns evidence nodes.
|
|||
|
|
5. `a5_rerank_reorders` — capture pre- and post-rerank order; assert they differ
|
|||
|
|
on at least one fixture query, proving the reranker is wired and not a no-op.
|
|||
|
|
6. `a6_project_isolation` — two projects seeded; assert no cross-project hits.
|
|||
|
|
7. `a7_no_project_errors` — unresolvable project exits non-zero.
|
|||
|
|
8. `a8_l2_two_hop_provenance` — an L2 hit's provenance resolves through L1 to L0
|
|||
|
|
sources.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p mem-cli query`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Asserting only that results are returned. A pipeline where the reranker returns
|
|||
|
|
the input unchanged still returns results, and assertion 5 is the only check
|
|||
|
|
that it is doing anything.
|
|||
|
|
- Testing provenance existence without resolving it. A hit carrying parent shas
|
|||
|
|
that do not exist in the table looks fine in the output and is useless.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Returning L0 by default. The answer is there but buried in raw transcript, and
|
|||
|
|
the tool reads as low quality when it is a display default.
|
|||
|
|
- Recalling exactly k then reranking. Reranking cannot recover a relevant
|
|||
|
|
document that recall never returned; the width of the recall is what determines
|
|||
|
|
the ceiling.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — pgvector, retrieval
|