poimen-memory/tasks/M8.4-rrf-fusion.md

M8.4 — Reciprocal Rank Fusion engine

Field	Value
Phase	M8 — Hybrid Search
Size	S — 0.5–1 day
Status	⬜
Flags	—
Spec	inlined below
Blocks	M8.5
Depends	— (pure logic, no infra dependency)

Goal

Implement Reciprocal Rank Fusion (RRF) that merges two ranked lists from different scoring distributions into a single ranked list. No parameter tuning required.

Why RRF, not weighted linear

pgvector returns cosine similarity in [0.0, 1.0]. OpenSearch BM25 returns unbounded scores in [0, 50+]. These distributions are incomparable.

Weighted linear (0.6 * sem + 0.4 * lex) requires min-max normalisation, which is fragile: one outlier score compresses all other scores to near-zero. It also requires choosing weights, which requires labelled data we don't have yet.

RRF ignores score magnitudes entirely. It uses only rank positions: the document that appears first in a list gets rank 1, second gets rank 2, etc. The formula is:

RRF_score(d) = Σ  1 / (k + rank_i(d))
              lists

Where k = 60 is a constant (academic standard, Cormack et al. 2009). A document in rank 1 of both lists gets 1/61 + 1/61 = 0.0328. A document in rank 1 of only one list gets 1/61 = 0.0164. The first always outranks the second, regardless of original score magnitudes.

Design

pub struct RRFConfig {
    pub k: f32,            // 60.0 (constant, don't tune)
    pub retrieve_k: usize, // 50 (top-K from each engine)
    pub final_k: usize,    // 10 (return top-K)
}

pub struct RRFFusion { config: RRFConfig }

impl RRFFusion {
    pub fn fuse(
        &self,
        semantic: Vec<(String, f32)>,  // (chunk_id, score) — sorted by score desc
        lexical: Vec<(String, f32)>,
    ) -> Vec<(String, f32)>;           // (chunk_id, rrf_score) — sorted desc, truncated
}

Invariants:

• Input lists must be pre-sorted by score descending (rank = position).
• Output is sorted by RRF score descending.
• Output length ≤ final_k.
• A document appearing in both lists always outranks one appearing in only one (given same rank positions).

Steps

1. Implement RRFFusion::fuse().
2. Implement RRFFusion::normalize_scores() as utility (for optional weighted-linear fallback).
3. Write tests: basic fusion, single-engine input, identical lists, disjoint lists, empty inputs.

Acceptance

1. fuse([(a,0.9),(b,0.8)], [(a,8.0),(c,7.0)]) → a is rank 1 (appears in both lists).
2. fuse([(a,0.9)], []) → a is rank 1 with score 1/61.
3. fuse([], []) → empty result.
4. fuse([(a,0.9),(b,0.8)], [(b,8.0),(a,7.0)]) → a and b have equal RRF scores (both appear in both at same combined rank sum). Either order is acceptable.
5. Output length never exceeds final_k.

Verify

cargo test -p mem-cli rrf -- --nocapture

False pass: Fusion returns results but sorted by original score, not RRF score. Verify by checking that a document ranked #3 in semantic but #1 in lexical outranks a document ranked #1 in semantic but absent from lexical.

Artifacts

• crates/mem-cli/src/query_optimizer.rs (RRFFusion struct, lives alongside QueryOptimizer)