M8.4 — Reciprocal Rank Fusion engine

Field	Value
Phase	M8 — Hybrid Search
Size	S — 0.5–1 day
Status	✅ COMPLETE
Flags	—
Spec	inlined below
Blocks	M8.5
Depends	— (pure logic, no infra dependency)

Goal

Implement Reciprocal Rank Fusion (RRF) that merges two ranked lists from different scoring distributions into a single ranked list. No parameter tuning required.

Why RRF, not weighted linear

pgvector returns cosine similarity in [0.0, 1.0]. OpenSearch BM25 returns unbounded scores in [0, 50+]. These distributions are incomparable.

Weighted linear (0.6 * sem + 0.4 * lex) requires min-max normalisation, which is fragile: one outlier score compresses all other scores to near-zero. It also requires choosing weights, which requires labelled data we don't have yet.

RRF ignores score magnitudes entirely. It uses only rank positions: the document that appears first in a list gets rank 1, second gets rank 2, etc. The formula is:

RRF_score(d) = Σ  1 / (k + rank_i(d))
              lists

Where k = 60 is a constant (academic standard, Cormack et al. 2009). A document in rank 1 of both lists gets 1/61 + 1/61 = 0.0328. A document in rank 1 of only one list gets 1/61 = 0.0164. The first always outranks the second, regardless of original score magnitudes.

Design

pub struct RRFConfig {
    pub k: f32,            // 60.0 (constant, don't tune)
    pub retrieve_k: usize, // 50 (top-K from each engine)
    pub final_k: usize,    // 10 (return top-K)
}

pub struct RRFFusion { config: RRFConfig }

impl RRFFusion {
    pub fn fuse(
        &self,
        semantic: Vec<(String, f32)>,  // (chunk_id, score) — sorted by score desc
        lexical: Vec<(String, f32)>,
    ) -> Vec<(String, f32)>;           // (chunk_id, rrf_score) — sorted desc, truncated
}

Invariants:

Input lists must be pre-sorted by score descending (rank = position).
Output is sorted by RRF score descending.
Output length ≤ final_k.
A document appearing in both lists always outranks one appearing in only one (given same rank positions).

Steps

Implement RRFFusion::fuse().
Implement RRFFusion::normalize_scores() as utility (for optional weighted-linear fallback).
Write tests: basic fusion, single-engine input, identical lists, disjoint lists, empty inputs.

Acceptance

fuse([(a,0.9),(b,0.8)], [(a,8.0),(c,7.0)]) → a is rank 1 (appears in both lists).
fuse([(a,0.9)], []) → a is rank 1 with score 1/61.
fuse([], []) → empty result.
fuse([(a,0.9),(b,0.8)], [(b,8.0),(a,7.0)]) → a and b have equal RRF scores (both appear in both at same combined rank sum). Either order is acceptable.
Output length never exceeds final_k.

Verify

cargo test -p mem-cli rrf -- --nocapture

False pass: Fusion returns results but sorted by original score, not RRF score. Verify by checking that a document ranked #3 in semantic but #1 in lexical outranks a document ranked #1 in semantic but absent from lexical.

Artifacts

crates/mem-cli/src/query_optimizer.rs (RRFFusion struct, lives alongside QueryOptimizer)

3.1 KiB Raw Blame History Unescape Escape