Story Crater Bot
959c596b1d
Deleted 31 completed task files: - M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate) - M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate) - M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate) - M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate) - M3.6.1: DocCorpusSource (heading-boundary chunking) - M4.1-2: skill-draft, derived-filter Updated INDEX.md: - Removed M0 & M1 phase sections (archived in git history) - Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜) - Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done - Noted M3.5.10 JWT auth implementation complete (awaiting image rollout) - Cleaned up broken links to deleted task files Total test count: 239 passing, 2 ignored (up from 196 at M3.4) Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
106 lines
3.4 KiB
Markdown
106 lines
3.4 KiB
Markdown
# M8.8 — Accuracy benchmarks: NDCG, MRR, Precision/Recall
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M8 — Hybrid Search |
|
||
| Size | M — 1–2 days |
|
||
| Status | ⬜ |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M8.9 |
|
||
| Depends | M8.6 (hybrid endpoint working), M8.7 (indices tuned) |
|
||
|
||
## Goal
|
||
|
||
Build a repeatable benchmark harness that measures retrieval accuracy. Run it against semantic-only, lexical-only, and hybrid strategies. Produce a comparison table that proves hybrid is better (or shows where it isn't).
|
||
|
||
## Design
|
||
|
||
**Test fixture format:**
|
||
```yaml
|
||
# fixtures/search_queries.yaml
|
||
queries:
|
||
- id: q1
|
||
text: "How do I fix kubernetes port 8080 conflict?"
|
||
relevant_docs: ["runbooks/networking.md"]
|
||
category: troubleshooting
|
||
|
||
- id: q2
|
||
text: "What is a StatefulSet?"
|
||
relevant_docs: ["docs/kubernetes-concepts.md", "docs/statefulsets.md"]
|
||
category: factual
|
||
|
||
- id: q3
|
||
text: "#networking firewall rules"
|
||
relevant_docs: ["docs/network-policies.md"]
|
||
category: navigational
|
||
```
|
||
|
||
**Metrics implemented:**
|
||
- **NDCG@10** — Are relevant docs ranked near the top? (0.0 worst, 1.0 perfect).
|
||
- **MRR** — How early is the first relevant doc? (1/rank of first hit).
|
||
- **Precision@5** — What fraction of top-5 results are relevant?
|
||
- **Recall@10** — What fraction of all relevant docs appear in top-10?
|
||
|
||
**Benchmark runner:**
|
||
```rust
|
||
pub struct BenchmarkResult {
|
||
pub strategy: String, // "hybrid", "semantic", "lexical"
|
||
pub avg_ndcg: f32,
|
||
pub avg_mrr: f32,
|
||
pub avg_precision_at_5: f32,
|
||
pub avg_recall_at_10: f32,
|
||
pub avg_latency_ms: f32,
|
||
pub per_query: Vec<QueryBenchmark>,
|
||
}
|
||
```
|
||
|
||
**Output:** Markdown table written to `docs/BENCHMARK_RESULTS.md`.
|
||
|
||
```markdown
|
||
| Strategy | NDCG@10 | MRR | P@5 | R@10 | Latency (ms) |
|
||
|----------|---------|-----|-----|------|-------------|
|
||
| semantic | 0.72 | 0.65 | 0.60 | 0.75 | 95 |
|
||
| lexical | 0.68 | 0.70 | 0.55 | 0.70 | 62 |
|
||
| hybrid | 0.87 | 0.82 | 0.78 | 0.90 | 210 |
|
||
```
|
||
|
||
## Steps
|
||
|
||
1. Create `fixtures/search_queries.yaml` with ≥ 20 queries across categories.
|
||
2. Ingest corresponding test documents into both stores.
|
||
3. Implement `BenchmarkRunner` that:
|
||
a. Loads fixture file.
|
||
b. Runs each query against each strategy.
|
||
c. Computes NDCG, MRR, Precision, Recall per query.
|
||
d. Averages across queries.
|
||
e. Writes results to markdown.
|
||
4. Implement as CLI command: `cargo run -- bench-search --queries <path> --output <path>`.
|
||
5. Run benchmarks. Record results.
|
||
|
||
## Acceptance
|
||
|
||
1. Benchmark runs to completion on 20+ queries × 3 strategies = 60+ query executions.
|
||
2. Output markdown table has all 5 columns populated.
|
||
3. Hybrid NDCG@10 ≥ max(semantic NDCG, lexical NDCG) — hybrid must not be worse than best single engine.
|
||
4. Per-query results show which categories benefit most from hybrid (expected: troubleshooting, procedural).
|
||
5. If hybrid is worse on any category, document why and whether it matters.
|
||
|
||
## Verify
|
||
|
||
```bash
|
||
cargo run -- bench-search \
|
||
--queries fixtures/search_queries.yaml \
|
||
--output docs/BENCHMARK_RESULTS.md
|
||
|
||
cat docs/BENCHMARK_RESULTS.md
|
||
```
|
||
|
||
**False pass:** Benchmark uses the same documents for queries and ground truth (trivial exact match). Ensure queries use **natural language** and ground truth docs use **technical content** — the match should be semantic, not string equality.
|
||
|
||
## Artifacts
|
||
|
||
- `fixtures/search_queries.yaml`
|
||
- `crates/mem-cli/src/bench_search.rs` (new)
|
||
- `docs/BENCHMARK_RESULTS.md` (output)
|