chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1)
Deleted 31 completed task files: - M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate) - M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate) - M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate) - M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate) - M3.6.1: DocCorpusSource (heading-boundary chunking) - M4.1-2: skill-draft, derived-filter Updated INDEX.md: - Removed M0 & M1 phase sections (archived in git history) - Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜) - Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done - Noted M3.5.10 JWT auth implementation complete (awaiting image rollout) - Cleaned up broken links to deleted task files Total test count: 239 passing, 2 ignored (up from 196 at M3.4) Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
This commit is contained in:
@@ -0,0 +1,105 @@
|
||||
# M8.8 — Accuracy benchmarks: NDCG, MRR, Precision/Recall
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M8 — Hybrid Search |
|
||||
| Size | M — 1–2 days |
|
||||
| Status | ⬜ |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M8.9 |
|
||||
| Depends | M8.6 (hybrid endpoint working), M8.7 (indices tuned) |
|
||||
|
||||
## Goal
|
||||
|
||||
Build a repeatable benchmark harness that measures retrieval accuracy. Run it against semantic-only, lexical-only, and hybrid strategies. Produce a comparison table that proves hybrid is better (or shows where it isn't).
|
||||
|
||||
## Design
|
||||
|
||||
**Test fixture format:**
|
||||
```yaml
|
||||
# fixtures/search_queries.yaml
|
||||
queries:
|
||||
- id: q1
|
||||
text: "How do I fix kubernetes port 8080 conflict?"
|
||||
relevant_docs: ["runbooks/networking.md"]
|
||||
category: troubleshooting
|
||||
|
||||
- id: q2
|
||||
text: "What is a StatefulSet?"
|
||||
relevant_docs: ["docs/kubernetes-concepts.md", "docs/statefulsets.md"]
|
||||
category: factual
|
||||
|
||||
- id: q3
|
||||
text: "#networking firewall rules"
|
||||
relevant_docs: ["docs/network-policies.md"]
|
||||
category: navigational
|
||||
```
|
||||
|
||||
**Metrics implemented:**
|
||||
- **NDCG@10** — Are relevant docs ranked near the top? (0.0 worst, 1.0 perfect).
|
||||
- **MRR** — How early is the first relevant doc? (1/rank of first hit).
|
||||
- **Precision@5** — What fraction of top-5 results are relevant?
|
||||
- **Recall@10** — What fraction of all relevant docs appear in top-10?
|
||||
|
||||
**Benchmark runner:**
|
||||
```rust
|
||||
pub struct BenchmarkResult {
|
||||
pub strategy: String, // "hybrid", "semantic", "lexical"
|
||||
pub avg_ndcg: f32,
|
||||
pub avg_mrr: f32,
|
||||
pub avg_precision_at_5: f32,
|
||||
pub avg_recall_at_10: f32,
|
||||
pub avg_latency_ms: f32,
|
||||
pub per_query: Vec<QueryBenchmark>,
|
||||
}
|
||||
```
|
||||
|
||||
**Output:** Markdown table written to `docs/BENCHMARK_RESULTS.md`.
|
||||
|
||||
```markdown
|
||||
| Strategy | NDCG@10 | MRR | P@5 | R@10 | Latency (ms) |
|
||||
|----------|---------|-----|-----|------|-------------|
|
||||
| semantic | 0.72 | 0.65 | 0.60 | 0.75 | 95 |
|
||||
| lexical | 0.68 | 0.70 | 0.55 | 0.70 | 62 |
|
||||
| hybrid | 0.87 | 0.82 | 0.78 | 0.90 | 210 |
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
1. Create `fixtures/search_queries.yaml` with ≥ 20 queries across categories.
|
||||
2. Ingest corresponding test documents into both stores.
|
||||
3. Implement `BenchmarkRunner` that:
|
||||
a. Loads fixture file.
|
||||
b. Runs each query against each strategy.
|
||||
c. Computes NDCG, MRR, Precision, Recall per query.
|
||||
d. Averages across queries.
|
||||
e. Writes results to markdown.
|
||||
4. Implement as CLI command: `cargo run -- bench-search --queries <path> --output <path>`.
|
||||
5. Run benchmarks. Record results.
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. Benchmark runs to completion on 20+ queries × 3 strategies = 60+ query executions.
|
||||
2. Output markdown table has all 5 columns populated.
|
||||
3. Hybrid NDCG@10 ≥ max(semantic NDCG, lexical NDCG) — hybrid must not be worse than best single engine.
|
||||
4. Per-query results show which categories benefit most from hybrid (expected: troubleshooting, procedural).
|
||||
5. If hybrid is worse on any category, document why and whether it matters.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
cargo run -- bench-search \
|
||||
--queries fixtures/search_queries.yaml \
|
||||
--output docs/BENCHMARK_RESULTS.md
|
||||
|
||||
cat docs/BENCHMARK_RESULTS.md
|
||||
```
|
||||
|
||||
**False pass:** Benchmark uses the same documents for queries and ground truth (trivial exact match). Ensure queries use **natural language** and ground truth docs use **technical content** — the match should be semantic, not string equality.
|
||||
|
||||
## Artifacts
|
||||
|
||||
- `fixtures/search_queries.yaml`
|
||||
- `crates/mem-cli/src/bench_search.rs` (new)
|
||||
- `docs/BENCHMARK_RESULTS.md` (output)
|
||||
Reference in New Issue
Block a user