feat(phase3-4): Complete hybrid retrieval + LLM optimization pipeline

Phase 3: Hybrid Retrieval
- HybridRetriever: TF-IDF prefilter + semantic rerank + RRF fusion
- WikiScopedFilter: BFS wiki-graph traversal
- RetrievalRoute: Direct | WikiScoped | ReferenceOnly
- 10 unit tests

Phase 4: LLM Call Optimization
- ChunkOptimizer: unified pipeline (threshold + budget + dedup)
- ScoreThresholdFilter: configurable min_score (default 0.6)
- BudgetSelector: greedy selection within byte budget
- ShingleDeduplicator: Jaccard similarity dedup
- 8 unit tests

QueryRouter (Phase 3+4 Integration)
- Bridges WikiLinkGraph + HybridRetriever + ChunkOptimizer
- RouterConfig: max_hops, thresholds, budget, RRF weights
- WikiGraphBuilder: construct graph from markdown docs
- 11 unit tests

Integration Tests (it_phase3_phase4.rs)
- 19 end-to-end tests covering full pipeline
- Wiki-link parsing, graph traversal, route selection
- TF-IDF prefilter, RRF fusion, chunk optimization
- Edge cases (empty, no matches, config customization)

Total: 107 tests passing (was 32)
This commit is contained in:
2026-08-31 22:42:34 -07:00
parent b71831557d
commit cd76424baa
4 changed files with 1039 additions and 59 deletions
+2
View File
@@ -25,6 +25,7 @@ pub mod query_filter;
pub mod advanced_ranking;
pub mod result_compressor;
pub mod federation;
pub mod query_router;
pub use endpoints::{IngestQueue, IngestRequest, JobStatus};
pub use ingest_worker::IngestWorker;
@@ -35,3 +36,4 @@ pub use chunk_metadata::{MetadataExtractor, MetadataBooster, ChunkMetadata, Chun
pub use cache_alignment::{LruChunkCache, KvCacheAligner, CacheLocalityAnalyzer, RetrievalProfiler, CacheMetrics};
pub use query_orchestrator::{QueryOrchestrator, QueryResult, OptimizedChunk, QueryContext, MemoryProjection};
pub use query_filter::{QueryFilter, FilterableDocument, FilterEngine, FilterStatistics};
pub use query_router::{QueryRouter, RouterConfig, RoutedResult, SelectedChunk, WikiGraphBuilder};