# M8.8 — Accuracy benchmarks: NDCG, MRR, Precision/Recall

| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | M — 1–2 days |
| Status | ⬜ |
| Flags | — |
| Spec | inlined below |
| Blocks | M8.9 |
| Depends | M8.6 (hybrid endpoint working), M8.7 (indices tuned) |

## Goal

Build a repeatable benchmark harness that measures retrieval accuracy. Run it against semantic-only, lexical-only, and hybrid strategies. Produce a comparison table that proves hybrid is better (or shows where it isn't).

## Design

**Test fixture format:**
```yaml
# fixtures/search_queries.yaml
queries:
  - id: q1
    text: "How do I fix kubernetes port 8080 conflict?"
    relevant_docs: ["runbooks/networking.md"]
    category: troubleshooting

  - id: q2
    text: "What is a StatefulSet?"
    relevant_docs: ["docs/kubernetes-concepts.md", "docs/statefulsets.md"]
    category: factual

  - id: q3
    text: "#networking firewall rules"
    relevant_docs: ["docs/network-policies.md"]
    category: navigational
```

**Metrics implemented:**
- **NDCG@10** — Are relevant docs ranked near the top? (0.0 worst, 1.0 perfect).
- **MRR** — How early is the first relevant doc? (1/rank of first hit).
- **Precision@5** — What fraction of top-5 results are relevant?
- **Recall@10** — What fraction of all relevant docs appear in top-10?

**Benchmark runner:**
```rust
pub struct BenchmarkResult {
    pub strategy: String,          // "hybrid", "semantic", "lexical"
    pub avg_ndcg: f32,
    pub avg_mrr: f32,
    pub avg_precision_at_5: f32,
    pub avg_recall_at_10: f32,
    pub avg_latency_ms: f32,
    pub per_query: Vec<QueryBenchmark>,
}
```

**Output:** Markdown table written to `docs/BENCHMARK_RESULTS.md`.

```markdown
| Strategy | NDCG@10 | MRR | P@5 | R@10 | Latency (ms) |
|----------|---------|-----|-----|------|-------------|
| semantic | 0.72 | 0.65 | 0.60 | 0.75 | 95 |
| lexical | 0.68 | 0.70 | 0.55 | 0.70 | 62 |
| hybrid | 0.87 | 0.82 | 0.78 | 0.90 | 210 |
```

## Steps

1. Create `fixtures/search_queries.yaml` with ≥ 20 queries across categories.
2. Ingest corresponding test documents into both stores.
3. Implement `BenchmarkRunner` that:
   a. Loads fixture file.
   b. Runs each query against each strategy.
   c. Computes NDCG, MRR, Precision, Recall per query.
   d. Averages across queries.
   e. Writes results to markdown.
4. Implement as CLI command: `cargo run -- bench-search --queries <path> --output <path>`.
5. Run benchmarks. Record results.

## Acceptance

1. Benchmark runs to completion on 20+ queries × 3 strategies = 60+ query executions.
2. Output markdown table has all 5 columns populated.
3. Hybrid NDCG@10 ≥ max(semantic NDCG, lexical NDCG) — hybrid must not be worse than best single engine.
4. Per-query results show which categories benefit most from hybrid (expected: troubleshooting, procedural).
5. If hybrid is worse on any category, document why and whether it matters.

## Verify

```bash
cargo run -- bench-search \
  --queries fixtures/search_queries.yaml \
  --output docs/BENCHMARK_RESULTS.md

cat docs/BENCHMARK_RESULTS.md
```

**False pass:** Benchmark uses the same documents for queries and ground truth (trivial exact match). Ensure queries use **natural language** and ground truth docs use **technical content** — the match should be semantic, not string equality.

## Artifacts

- `fixtures/search_queries.yaml`
- `crates/mem-cli/src/bench_search.rs` (new)
- `docs/BENCHMARK_RESULTS.md` (output)