poimen-memory/tasks/M8.4-rrf-fusion.md

# M8.4 — Reciprocal Rank Fusion engine

| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | S — 0.5–1 day |
| Status | ✅ COMPLETE |
| Flags | — |
| Spec | inlined below |
| Blocks | M8.5 |
| Depends | — (pure logic, no infra dependency) |

## Goal

Implement Reciprocal Rank Fusion (RRF) that merges two ranked lists from different scoring distributions into a single ranked list. No parameter tuning required.

## Why RRF, not weighted linear

pgvector returns cosine similarity in `[0.0, 1.0]`. OpenSearch BM25 returns unbounded scores in `[0, 50+]`. These distributions are incomparable.

**Weighted linear (0.6 * sem + 0.4 * lex)** requires min-max normalisation, which is fragile: one outlier score compresses all other scores to near-zero. It also requires choosing weights, which requires labelled data we don't have yet.

**RRF** ignores score magnitudes entirely. It uses only **rank positions**: the document that appears first in a list gets rank 1, second gets rank 2, etc. The formula is:

```
RRF_score(d) = Σ  1 / (k + rank_i(d))
              lists
```

Where `k = 60` is a constant (academic standard, Cormack et al. 2009). A document in rank 1 of both lists gets `1/61 + 1/61 = 0.0328`. A document in rank 1 of only one list gets `1/61 = 0.0164`. The first always outranks the second, regardless of original score magnitudes.

## Design

```rust
pub struct RRFConfig {
    pub k: f32,            // 60.0 (constant, don't tune)
    pub retrieve_k: usize, // 50 (top-K from each engine)
    pub final_k: usize,    // 10 (return top-K)
}

pub struct RRFFusion { config: RRFConfig }

impl RRFFusion {
    pub fn fuse(
        &self,
        semantic: Vec<(String, f32)>,  // (chunk_id, score) — sorted by score desc
        lexical: Vec<(String, f32)>,
    ) -> Vec<(String, f32)>;           // (chunk_id, rrf_score) — sorted desc, truncated
}
```

**Invariants:**
- Input lists must be pre-sorted by score descending (rank = position).
- Output is sorted by RRF score descending.
- Output length ≤ `final_k`.
- A document appearing in both lists always outranks one appearing in only one (given same rank positions).

## Steps

1. Implement `RRFFusion::fuse()`.
2. Implement `RRFFusion::normalize_scores()` as utility (for optional weighted-linear fallback).
3. Write tests: basic fusion, single-engine input, identical lists, disjoint lists, empty inputs.

## Acceptance

1. `fuse([(a,0.9),(b,0.8)], [(a,8.0),(c,7.0)])` → `a` is rank 1 (appears in both lists).
2. `fuse([(a,0.9)], [])` → `a` is rank 1 with score `1/61`.
3. `fuse([], [])` → empty result.
4. `fuse([(a,0.9),(b,0.8)], [(b,8.0),(a,7.0)])` → `a` and `b` have equal RRF scores (both appear in both at same combined rank sum). Either order is acceptable.
5. Output length never exceeds `final_k`.

## Verify

```bash
cargo test -p mem-cli rrf -- --nocapture
```

**False pass:** Fusion returns results but sorted by original score, not RRF score. Verify by checking that a document ranked #3 in semantic but #1 in lexical outranks a document ranked #1 in semantic but absent from lexical.

## Artifacts

- `crates/mem-cli/src/query_optimizer.rs` (RRFFusion struct, lives alongside QueryOptimizer)