Story Crater Bot
959c596b1d
Deleted 31 completed task files: - M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate) - M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate) - M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate) - M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate) - M3.6.1: DocCorpusSource (heading-boundary chunking) - M4.1-2: skill-draft, derived-filter Updated INDEX.md: - Removed M0 & M1 phase sections (archived in git history) - Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜) - Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done - Noted M3.5.10 JWT auth implementation complete (awaiting image rollout) - Cleaned up broken links to deleted task files Total test count: 239 passing, 2 ignored (up from 196 at M3.4) Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
83 lines
3.1 KiB
Markdown
83 lines
3.1 KiB
Markdown
# M8.4 — Reciprocal Rank Fusion engine
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M8 — Hybrid Search |
|
||
| Size | S — 0.5–1 day |
|
||
| Status | ⬜ |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M8.5 |
|
||
| Depends | — (pure logic, no infra dependency) |
|
||
|
||
## Goal
|
||
|
||
Implement Reciprocal Rank Fusion (RRF) that merges two ranked lists from different scoring distributions into a single ranked list. No parameter tuning required.
|
||
|
||
## Why RRF, not weighted linear
|
||
|
||
pgvector returns cosine similarity in `[0.0, 1.0]`. OpenSearch BM25 returns unbounded scores in `[0, 50+]`. These distributions are incomparable.
|
||
|
||
**Weighted linear (0.6 * sem + 0.4 * lex)** requires min-max normalisation, which is fragile: one outlier score compresses all other scores to near-zero. It also requires choosing weights, which requires labelled data we don't have yet.
|
||
|
||
**RRF** ignores score magnitudes entirely. It uses only **rank positions**: the document that appears first in a list gets rank 1, second gets rank 2, etc. The formula is:
|
||
|
||
```
|
||
RRF_score(d) = Σ 1 / (k + rank_i(d))
|
||
lists
|
||
```
|
||
|
||
Where `k = 60` is a constant (academic standard, Cormack et al. 2009). A document in rank 1 of both lists gets `1/61 + 1/61 = 0.0328`. A document in rank 1 of only one list gets `1/61 = 0.0164`. The first always outranks the second, regardless of original score magnitudes.
|
||
|
||
## Design
|
||
|
||
```rust
|
||
pub struct RRFConfig {
|
||
pub k: f32, // 60.0 (constant, don't tune)
|
||
pub retrieve_k: usize, // 50 (top-K from each engine)
|
||
pub final_k: usize, // 10 (return top-K)
|
||
}
|
||
|
||
pub struct RRFFusion { config: RRFConfig }
|
||
|
||
impl RRFFusion {
|
||
pub fn fuse(
|
||
&self,
|
||
semantic: Vec<(String, f32)>, // (chunk_id, score) — sorted by score desc
|
||
lexical: Vec<(String, f32)>,
|
||
) -> Vec<(String, f32)>; // (chunk_id, rrf_score) — sorted desc, truncated
|
||
}
|
||
```
|
||
|
||
**Invariants:**
|
||
- Input lists must be pre-sorted by score descending (rank = position).
|
||
- Output is sorted by RRF score descending.
|
||
- Output length ≤ `final_k`.
|
||
- A document appearing in both lists always outranks one appearing in only one (given same rank positions).
|
||
|
||
## Steps
|
||
|
||
1. Implement `RRFFusion::fuse()`.
|
||
2. Implement `RRFFusion::normalize_scores()` as utility (for optional weighted-linear fallback).
|
||
3. Write tests: basic fusion, single-engine input, identical lists, disjoint lists, empty inputs.
|
||
|
||
## Acceptance
|
||
|
||
1. `fuse([(a,0.9),(b,0.8)], [(a,8.0),(c,7.0)])` → `a` is rank 1 (appears in both lists).
|
||
2. `fuse([(a,0.9)], [])` → `a` is rank 1 with score `1/61`.
|
||
3. `fuse([], [])` → empty result.
|
||
4. `fuse([(a,0.9),(b,0.8)], [(b,8.0),(a,7.0)])` → `a` and `b` have equal RRF scores (both appear in both at same combined rank sum). Either order is acceptable.
|
||
5. Output length never exceeds `final_k`.
|
||
|
||
## Verify
|
||
|
||
```bash
|
||
cargo test -p mem-cli rrf -- --nocapture
|
||
```
|
||
|
||
**False pass:** Fusion returns results but sorted by original score, not RRF score. Verify by checking that a document ranked #3 in semantic but #1 in lexical outranks a document ranked #1 in semantic but absent from lexical.
|
||
|
||
## Artifacts
|
||
|
||
- `crates/mem-cli/src/query_optimizer.rs` (RRFFusion struct, lives alongside QueryOptimizer)
|