poimen-memory/tasks/M8.8-accuracy-benchmarks.md

M8.8 — Accuracy benchmarks: NDCG, MRR, Precision/Recall

Field	Value
Phase	M8 — Hybrid Search
Size	M — 1–2 days
Status	⬜
Flags	—
Spec	inlined below
Blocks	M8.9
Depends	M8.6 (hybrid endpoint working), M8.7 (indices tuned)

Goal

Build a repeatable benchmark harness that measures retrieval accuracy. Run it against semantic-only, lexical-only, and hybrid strategies. Produce a comparison table that proves hybrid is better (or shows where it isn't).

Design

Test fixture format:

# fixtures/search_queries.yaml
queries:
  - id: q1
    text: "How do I fix kubernetes port 8080 conflict?"
    relevant_docs: ["runbooks/networking.md"]
    category: troubleshooting

  - id: q2
    text: "What is a StatefulSet?"
    relevant_docs: ["docs/kubernetes-concepts.md", "docs/statefulsets.md"]
    category: factual

  - id: q3
    text: "#networking firewall rules"
    relevant_docs: ["docs/network-policies.md"]
    category: navigational

Metrics implemented:

• NDCG@10 — Are relevant docs ranked near the top? (0.0 worst, 1.0 perfect).
• MRR — How early is the first relevant doc? (1/rank of first hit).
• Precision@5 — What fraction of top-5 results are relevant?
• Recall@10 — What fraction of all relevant docs appear in top-10?

Benchmark runner:

pub struct BenchmarkResult {
    pub strategy: String,          // "hybrid", "semantic", "lexical"
    pub avg_ndcg: f32,
    pub avg_mrr: f32,
    pub avg_precision_at_5: f32,
    pub avg_recall_at_10: f32,
    pub avg_latency_ms: f32,
    pub per_query: Vec<QueryBenchmark>,
}

Output: Markdown table written to docs/BENCHMARK_RESULTS.md.

| Strategy | NDCG@10 | MRR | P@5 | R@10 | Latency (ms) |
|----------|---------|-----|-----|------|-------------|
| semantic | 0.72 | 0.65 | 0.60 | 0.75 | 95 |
| lexical | 0.68 | 0.70 | 0.55 | 0.70 | 62 |
| hybrid | 0.87 | 0.82 | 0.78 | 0.90 | 210 |

Steps

1. Create fixtures/search_queries.yaml with ≥ 20 queries across categories.
2. Ingest corresponding test documents into both stores.
3. Implement BenchmarkRunner that: a. Loads fixture file. b. Runs each query against each strategy. c. Computes NDCG, MRR, Precision, Recall per query. d. Averages across queries. e. Writes results to markdown.
4. Implement as CLI command: cargo run -- bench-search --queries <path> --output <path>.
5. Run benchmarks. Record results.

Acceptance

1. Benchmark runs to completion on 20+ queries × 3 strategies = 60+ query executions.
2. Output markdown table has all 5 columns populated.
3. Hybrid NDCG@10 ≥ max(semantic NDCG, lexical NDCG) — hybrid must not be worse than best single engine.
4. Per-query results show which categories benefit most from hybrid (expected: troubleshooting, procedural).
5. If hybrid is worse on any category, document why and whether it matters.

Verify

cargo run -- bench-search \
  --queries fixtures/search_queries.yaml \
  --output docs/BENCHMARK_RESULTS.md

cat docs/BENCHMARK_RESULTS.md

False pass: Benchmark uses the same documents for queries and ground truth (trivial exact match). Ensure queries use natural language and ground truth docs use technical content — the match should be semantic, not string equality.

Artifacts

• fixtures/search_queries.yaml
• crates/mem-cli/src/bench_search.rs (new)
• docs/BENCHMARK_RESULTS.md (output)