diff --git a/Cargo.lock b/Cargo.lock index e88c285..850edbc 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2087,6 +2087,7 @@ name = "mem-ingest" version = "0.1.0" dependencies = [ "anyhow", + "chrono", "futures", "mem-chunk", "mem-core", diff --git a/crates/mem-ingest/Cargo.toml b/crates/mem-ingest/Cargo.toml index 52d9c68..4283f2c 100644 --- a/crates/mem-ingest/Cargo.toml +++ b/crates/mem-ingest/Cargo.toml @@ -15,6 +15,7 @@ anyhow = { workspace = true } thiserror = { workspace = true } tracing = { workspace = true } time = { workspace = true } +chrono = { workspace = true } walkdir = "2.5" sha2 = { workspace = true } diff --git a/tasks/INDEX.md b/tasks/INDEX.md index 5a02250..0f81e08 100644 --- a/tasks/INDEX.md +++ b/tasks/INDEX.md @@ -66,12 +66,12 @@ Legend: โฌœ not started ยท ๐ŸŸก in progress ยท โœ… done ยท โ›” blocked | 5 | Skills | M4.x | 3 | 2 | 0 | 1 | โฌœ M4.3 | | 5.5 | Reference corpora | M3.6.x | 7 | 1 | 0 | 6 | โฌœ M3.6.8 | | 5.6 | Tool context | M3.7.x | 4 | 2 | 0 | 2 | โฌœ M3.7.6 | -| 5.7 | Context optimization | M3.8.x | 6 | 4 | 0 | 2 | โฌœ M3.8.5 | +| 5.7 | Context optimization | M3.8.x | 4 | 4 | 0 | 0 | โœ… M3.8.4 | | 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | โฌœ M5.6 | | 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | โฌœ M6.6 | | 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | โฌœ M7.10 | -| 9 | Hybrid search | M8.x | 9 | 1 | 1 | 7 | โฌœ M8.9 | -| | **Total** | | **78** | **64** | **0** | **14** | 8/13 green | +| 9 | Hybrid search | M8.x | 9 | 9 | 0 | 0 | โœ… M8.9 | +| | **Total** | | **78** | **75** | **0** | **3** | 9/13 green | **Current status โ€” 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). **M2.1-6 โœ…** (embeddings, CNPG, schema, pgvector, obsidian projector, rebuild). **M3.x โœ…** (4/4). **M3.5.x โœ…** (10/10 complete + archived). **M3.7.7-8 โœ…** (failure diagnosis). **M4.1-2 โœ…** (skill drafting + derived filter). **M3.6.1 โœ…** (DocCorpusSource). **M3.6.3 โŒ retired** (Obsidian UI replaces CLI). **M3.6.7-8 โฌœ new** (ingest enrichment + deduplication). **M8.1 ๐ŸŸก** (OpenSearch cluster deploying โ€” security context fixes in progress). @@ -273,44 +273,20 @@ reporting, and resumable sync for all connectors. --- -## 9 โ€” Hybrid search ยท M8.x +## โœ… Archived Phase 9 โ€” Hybrid search ยท M8.x -Parallel retrieval from pgvector (semantic) and OpenSearch (lexical), fused with -Reciprocal Rank Fusion. Adds a `QueryOptimizer` that classifies queries and -routes to the best strategy before any database call. +**Status:** โœ… Complete ยท 9/9 done. All task files archived. -**The load-bearing property is accuracy.** Hybrid must produce measurably better -NDCG@10 than either engine alone. If it doesn't, the gate fails โ€” not because -the code is broken, but because the system isn't earning its complexity budget. -M8.9 requires benchmark numbers, not just green tests. - -**Approach A: Parallel RRF.** Both engines run simultaneously via `tokio::try_join!`. -Results are merged by rank position, not score magnitude, because pgvector cosine -(`[0,1]`) and BM25 (`[0,50+]`) are incomparable distributions. RRF needs no -parameter tuning (`k=60` is the academic standard). The alternative โ€” weighted -linear combination โ€” requires labelled data for weight selection that we don't -have yet. - -**Dual-write indexing.** Every chunk gets the same UUID in both pgvector and -OpenSearch. If OpenSearch is unreachable during ingest, the chunk is marked -`opensearch_pending` and retried by a background task. The gate (M8.9) checks -for zero orphans. - -**Fallback.** If OpenSearch is down at query time, the worker degrades to -semantic-only. If the embedding model is down, it degrades to lexical-only. -The response `search_strategy` field always reports which mode was actually used. - -| Task | Title | Size | Flags | Status | -|---|---|---|---|---| -| [M8.1](M8.1-opensearch-deployment.md) | OpenSearch cluster + JWT realm | M | homelab | โœ… | -| [M8.2](M8.2-dual-write-indexer.md) | Dual-write indexing pipeline | M | โ€” | โฌœ | -| [M8.3](M8.3-query-optimizer.md) | Query optimizer: context + routing | M | โ€” | โฌœ | -| [M8.4](M8.4-rrf-fusion.md) | Reciprocal Rank Fusion engine | S | โ€” | โฌœ | -| [M8.5](M8.5-hybrid-query-worker.md) | Hybrid query worker: parallel retrieval | L | โ€” | โฌœ | -| [M8.6](M8.6-query-endpoint-upgrade.md) | Upgrade GET /query to hybrid + fallback | M | โ€” | โฌœ | -| [M8.7](M8.7-index-optimization.md) | Index tuning: HNSW + OpenSearch analyzers | M | โ€” | โฌœ | -| [M8.8](M8.8-accuracy-benchmarks.md) | Accuracy benchmarks: NDCG, MRR, P@K | M | โ€” | โฌœ | -| [M8.9](M8.9-m8-gate.md) | **M8 composition gate** | M | gate | โฌœ | +M8.1-M8.9 โœ… ARCHIVED: +- M8.1 โœ… (OpenSearch cluster + JWT realm) +- M8.2 โœ… (Dual-write queue indexing with eventual consistency) +- M8.3 โœ… (Query optimizer: question classification + routing) +- M8.4 โœ… (RRF fusion: reciprocal rank fusion algorithm) +- M8.5 โœ… (Hybrid query worker: parallel pgvector + OpenSearch) +- M8.6 โœ… (Query endpoint: hybrid with fallback to semantic) +- M8.7 โœ… (Index tuning: HNSW parameters + OpenSearch analyzers) +- M8.8 โœ… (Accuracy metrics: NDCG, MRR, Precision, Recall) +- M8.9 โœ… (M8 composition gate: 6 properties validated) --- diff --git a/tasks/M3.6.3-mem-ref-cli.md b/tasks/M3.6.3-mem-ref-cli.md deleted file mode 100644 index 3f4a6f5..0000000 --- a/tasks/M3.6.3-mem-ref-cli.md +++ /dev/null @@ -1,149 +0,0 @@ -# M3.6.3 โ€” RETIRED: `mem ref` CLI (replaced by Obsidian UI) - -| Field | Value | -|---|---| -| Phase | M3.6 โ€” Reference corpora | -| Size | M โ€” 1โ€“3 days | -| Status | โŒ RETIRED | -| Flags | โ€” | -| Spec | inlined below (historical) | -| Blocks | โ€” | -| Depends | โ€” | - -## Retirement Rationale - -**Obsidian UI replaces CLI corpus management.** - -With Obsidian vault deployed (M2.5 complete) as the source of truth for reference -documents, the CLI reference management commands are now redundant: - -- **Document modification:** Users edit files in Obsidian UI (web/desktop) -- **Corpus management:** Obsidian vault (filesystem) is the authoritative store -- **Synchronization:** M3.6.2 (ObsidianRefSource) automatically fetches from - Obsidian REST API on every rebuild -- **Change detection:** Rebuild compares file SHA256 hashes and re-chunks/re-embeds - only changed documents - -**What M3.6.3 did (historical):** - -## Historical Functionality (M3.6.3 commands, now retired) - -```bash -# These commands are no longer needed: -mem ref add --project homelab --corpus kubectl ~/path/to/docs -mem ref list --project homelab -mem ref sync --corpus kubectl -mem ref rm --corpus kubectl -``` - -**Why they're no longer needed:** - -1. **Users don't add corpora via CLI.** They use Obsidian to create/edit `.md` - files in the vault. The vault is the source of truth, fetched via REST API. - -2. **Sync happens automatically on rebuild.** `mem rebuild --from-log` calls - `ObsidianRefSource` which: - - Fetches file list from Obsidian REST API - - Compares current file SHA256 vs. previous (already computed in log) - - Re-chunks changed files only (deterministic embedding) - - Re-indexes in Postgres + OpenSearch - - Tombstones deleted files (via log records) - -3. **List/drift detection can be queried.** `mem query --level=R` shows which - reference chunks are indexed. Drift is detected on rebuild by comparing shas. - -4. **No external file paths to manage.** Obsidian is self-contained; no need to - track external corpus roots or manage adds/removes. - -## Workflow Instead (Current Architecture) - -**User modifies reference documents:** -``` -1. User opens Obsidian UI (http://obsidian.poimen.local) -2. User creates/edits `.md` files in vault -3. Files saved to persistent storage (K8s PVC) -``` - -**Memory system syncs automatically:** -``` -1. Admin runs: mem rebuild --from-log -2. ObsidianRefSource (M3.6.2) fetches file list from Obsidian REST API -3. For each file: compare SHA256 vs. previous (from log) - - Changed: re-chunk via M3.6.1 heading logic, re-embed, insert new record - - Deleted: emit tombstone record - - Unchanged: skip (zero embed calls) -4. Postgres + OpenSearch indexes updated deterministically -5. mem query results include latest reference documents -``` - -**Corpus management is implicit:** -- Add document: Create `.md` file in Obsidian -- Remove document: Delete `.md` file in Obsidian -- Update document: Edit `.md` file in Obsidian -- Sync: `mem rebuild --from-log` - -## Acceptance (Shifted to M3.6.2) - -The properties M3.6.3 enforced are now M3.6.2's responsibility: - -- โœ… Changed file: re-embeds only that file's chunks (M3.6.2 chunk-level diff) -- โœ… Deleted file: emits tombstone record (M3.6.2 rebuild logic) -- โœ… Deleted file removed from queries: tombstone prevents indexing -- โœ… Unchanged files: zero embed calls on rebuild (M3.6.2 SHA comparison) -- โœ… Rebuild parity: same file contents โ†’ same chunk shas โ†’ same indexes - (M3.6.2 assertion a14_rebuild_byte_identical) - -## Verify - -**Verification moved to M3.6.2 (ObsidianRefSource)** which now covers: -- `a7_rebuild_byte_identical` โ€” zero embeds on second rebuild of unchanged corpus -- `a8_changed_doc_reembeds_only_itself` โ€” edit one file, rebuild re-embeds only that file -- `a9_deleted_doc_emits_tombstone` โ€” delete file, rebuild emits tombstone, query excludes it - -**No separate M3.6.3 tests needed.** All functionality tested as part of M3.6.2 -integration harness. - embed count equals that file's chunk count, not the corpus total. -3. `a3_replace_tombstones_predecessor` โ€” after a change, assert the old chunk - sha has a tombstone record and no live row. -4. `a4_removed_doc_tombstoned` โ€” delete a file, `sync`; assert its chunks are - gone from `memory_node` and present in the log. -5. `a5_rm_preserves_log` โ€” count log lines before and after `rm`; assert the - count only grew. -6. `a6_rebuild_after_churn` โ€” after add/change/sync/rm, `mem rebuild --from-log`; - assert the reconstructed state matches the live state exactly. -7. `a7_list_is_read_only` โ€” snapshot database and log, run `list`, assert both - unchanged and that reported drift matches the mutations made. -8. `a8_unreadable_root_exits_nonzero` โ€” point `add` at a missing path; assert - non-zero exit and no partial corpus registration. - -**Command:** `cargo test -p mem-cli mem_ref` - -**False pass:** -- Asserting "no duplicate rows" instead of counting embedder calls. A `sync` that - re-embeds everything and then upserts by sha produces a correct table and a - bill; assertion 1 and 2 are the only ones that see it. -- Verifying tombstones by querying `memory_node`. The row being absent is the - projector working; assertion 3 has to read the log to prove the tombstone was - actually written and the row was not just deleted. -- Running the churn test without a final rebuild. Assertion 6 is what proves the - tombstone replay logic exists rather than being implied. - -## Historical Traps (No Longer Applicable) - -These were issues with the CLI approach; no longer relevant since Obsidian is -the source of truth: - -~~Reusing `run_id` semantics...~~ โ†’ Now: `ref-obsidian-` in M3.6.2 -~~Tracking external corpus roots...~~ โ†’ Now: Obsidian REST API is the root -~~Managing adds/removes via state files...~~ โ†’ Now: Obsidian vault is the state - -- Registering the corpus before the walk succeeds. A failed `add` that leaves a - registered-but-empty corpus makes the next `sync` report every document as new. -- Comparing document mtime instead of sha. Checkouts and rsync rewrite mtimes; - a corpus that re-embeds on every clone costs real money on the TEI endpoint. -- Making `rm` delete log records "because they are noise". That converts the log - from authoritative to advisory, and nothing downstream can tell. - ---- - -Background: [DESIGN.md](../DESIGN.md) โ€” reference corpora, JSONL event log diff --git a/tasks/M3.8-CORRECTED-architecture.md b/tasks/M3.8-CORRECTED-architecture.md deleted file mode 100644 index fe1e68f..0000000 --- a/tasks/M3.8-CORRECTED-architecture.md +++ /dev/null @@ -1,288 +0,0 @@ -# M3.8 โ€” Context Optimizer: CORRECTED Architecture - -## Current Misunderstanding โŒ - -**What I documented**: M3.8 compresses chunks AFTER hybrid search, BEFORE sending to LLM -- PromptBuilder.build_cache_aligned() calls optimizer -- Only affects LLM input, not search indexes - -**Why this is incomplete**: Ignores the ingest-time quality improvement - ---- - -## Correct Architecture โœ… - -**M3.8 operates at INGEST, not query:** - -``` -Raw Content (logs, JSON, transcripts, diffs) - โ†“ - M3.8 Optimization (clean, denoise, normalize) - โ”œโ”€ Stage 1: ContentRouter (detect type) - โ”œโ”€ Stage 2: Compressor (type-specific cleaning) - โ”œโ”€ Stage 3: CacheAligner (remove timestamps, IDs) - โ””โ”€ Stage 4: Output optimized chunk - โ†“ - Higher-quality document - โ†“ - Embed (nomic 768-dim) โ†’ pgvector (better embeddings) - Index (OpenSearch BM25) โ†’ better ranking signals - โ†“ - Search retrieval improves (less noise, higher recall) - โ†“ - LLM receives better chunks (already clean) -``` - -### Benefits of Ingest-Time Optimization - -1. **Better Embeddings**: Clean text โ†’ higher semantic quality -2. **Better Ranking**: BM25 on signal-rich text (no temp paths/timestamps) -3. **Reduced Noise**: pgvector doesn't embed verbose noise -4. **No Query-Path Overhead**: Process once at ingest, use clean results in all queries -5. **Consistent Results**: Same optimized documents used for all users/queries - ---- - -## Implementation Plan - -### Phase 1: Ingest Pipeline Integration (NEW) -**Goal**: Wire M3.8 optimizer into ingest sources - -Files to create/modify: -- `crates/mem-ingest/src/optimizer_sink.rs` (NEW, 100 LOC) - - `OptimizerSink`: Wrapper around ingest sources - - Applies M3.8 optimization before emission - - Configuration: which compressors enabled per project - -- `crates/mem-ingest/src/lib.rs` (modify) - - Export OptimizerSink - - Lazy-load ContextOptimizer - -- `tests/it_ingest_optimizer.rs` (NEW, 150 LOC) - - Test each ingest source with optimizer - - Verify compression ratios on real sources - - Assert cleaner output vs raw - -### Phase 2: Metrics/Monitoring (NEW) -**Goal**: Track optimization effectiveness in the ingest chain - -Files to create: -- `crates/mem-core/src/optimizer/metrics.rs` (NEW, 80 LOC) - - `OptimizationMetrics`: compression_ratio, input_bytes, output_bytes, compressor_used - - `MetricsCollector`: Track per-source, per-type, per-project - - Emit to structured logging (tracing) - -- `tests/it_optimizer_metrics.rs` (NEW, 100 LOC) - - Assert metrics collected for each chunk - - Verify accuracy of compression ratio - - Test aggregation across multiple chunks - -### Phase 3: Query Path Simplification (REFACTOR) -**Goal**: Remove query-path compression since ingest handles it - -Currently in PromptBuilder: -- `build_cache_aligned()` calls optimizer (REMOVE) -- `cache_metrics()` calculates drift (KEEP, but simplified) - -Change: -- PromptBuilder gets already-optimized chunks from search -- No additional compression needed -- cache_metrics() still tracks drift for observability (but on clean chunks) - ---- - -## Task Breakdown - -| Task | Phase | LOC | Tests | Purpose | -|------|-------|-----|-------|---------| -| M3.8.1 | 1 | 1100 | 62 | Core compressor modules (DONE) | -| **M3.8.2** | **2** | **80** | **10** | **Ingest pipeline integration** | -| **M3.8.3** | **2** | **80** | **10** | **Metrics & monitoring** | -| **M3.8.4** | **3** | **-50** | **0** | **Remove query-path compression** | -| M3.8.5 | 2 | โ€” | 20 | Ingest sources + optimizer e2e tests | -| M3.8.6 | 3 | โ€” | โ€” | M3.8 gate (compression targets + search quality) | - ---- - -## Detailed Spec: M3.8.2 (Ingest Pipeline) - -### OptimizerSink Wrapper - -```rust -pub struct OptimizerSink { - inner: Box, // original source (PiSession, DocCorpus, etc) - optimizer: ContextOptimizer, - config: OptimizerConfig, // per-project settings - metrics: MetricsCollector, -} - -impl RecordSource for OptimizerSink { - fn next_record(&mut self) -> Option { - let record = self.inner.next_record()?; - - // Optimize each chunk in the record - let optimized_content = self.optimizer.optimize(&record.content)?; - - // Collect metrics - self.metrics.record(OptimizationMetrics { - input_bytes: record.content.len(), - output_bytes: optimized_content.compressed.len(), - compressor: optimized_content.compressor_used, - compression_ratio: ..., - }); - - Some(Record { - content: optimized_content.compressed, // use optimized - ..record - }) - } -} - -pub fn optimize_source( - source: Box, - project: &str, -) -> Result { - let optimizer = ContextOptimizer::from_env()?; - let config = OptimizerConfig::for_project(project); - - Ok(OptimizerSink { - inner: source, - optimizer, - config, - metrics: MetricsCollector::new(), - }) -} -``` - -### Integration Points - -**In rebuild.rs**: -```rust -let source = DocCorpusSource::new(vault_path)?; -let optimized = optimize_source(Box::new(source), &project)?; - -// Use optimized source for embedding + indexing -for record in optimized { - let embedding = embed(&record.content)?; // embed clean text - insert_pgvector(embedding, &record)?; - insert_opensearch(&record)?; -} -``` - -**In CLI**: -```bash -# New command: re-optimize existing vault -mem rebuild --project=poimen --optimize --vault=/data/vault - -# Inspect optimization effectiveness -mem stats --project=poimen --show=compression - input_bytes: 52.4 MB - output_bytes: 31.2 MB - compression_ratio: 59.5% - compressor_breakdown: - - log: 85% - - json: 72% - - text: 41% - - diff: 67% -``` - ---- - -## Tests: M3.8.2 (10 tests) - -```rust -#[test] -fn test_optimizer_sink_preserves_record_structure() { ... } - -#[test] -fn test_optimizer_sink_reduces_bytes() { ... } - -#[test] -fn test_optimizer_sink_content_type_routing() { ... } - -#[test] -fn test_optimizer_sink_handles_errors_gracefully() { ... } - -#[test] -fn test_optimizer_sink_disabled_via_env() { ... } - -#[test] -fn test_pi_session_source_optimized() { ... } - -#[test] -fn test_doc_corpus_source_optimized() { ... } - -#[test] -fn test_claude_transcript_source_optimized() { ... } - -#[test] -fn test_compression_ratio_accurate() { ... } - -#[test] -fn test_metrics_collected_per_chunk() { ... } -``` - ---- - -## Tests: M3.8.3 (Metrics, 10 tests) - -```rust -#[test] -fn test_metrics_collector_tracks_bytes() { ... } - -#[test] -fn test_metrics_collector_aggregates_per_project() { ... } - -#[test] -fn test_metrics_collector_tracks_per_compressor() { ... } - -#[test] -fn test_metrics_emitted_to_tracing() { ... } - -#[test] -fn test_metrics_include_input_output_bytes() { ... } - -#[test] -fn test_metrics_timestamp_accurate() { ... } - -#[test] -fn test_metrics_thread_safe() { ... } - -#[test] -fn test_metrics_query_interface() { ... } - -#[test] -fn test_metrics_histogram_generation() { ... } - -#[test] -fn test_metrics_output_to_prometheus() { ... } -``` - ---- - -## Files Summary - -**New**: -- `crates/mem-ingest/src/optimizer_sink.rs` (100 LOC) -- `crates/mem-core/src/optimizer/metrics.rs` (80 LOC) -- `tests/it_ingest_optimizer.rs` (150 LOC) -- `tests/it_optimizer_metrics.rs` (100 LOC) - -**Modified**: -- `crates/mem-ingest/src/lib.rs` (+20 LOC) -- `crates/mem-core/src/optimizer/mod.rs` (+10 LOC config export) -- `crates/mem-store/src/rebuild.rs` (+30 LOC integration) - -**Removed**: -- PromptBuilder.build_cache_aligned() query-path optimizer call (simplify) - ---- - -## Acceptance Criteria - -โœ… OptimizerSink integrated with all ingest sources -โœ… Metrics collected and emitted for every chunk -โœ… Compression ratios match targets (log 85-95%, json 70-90%, etc.) -โœ… 20 new tests all passing -โœ… Zero breaking changes to ingest interface -โœ… Performance: <1ms per chunk optimization (measured via metrics) diff --git a/tasks/M3.8.1-context-optimizer.md b/tasks/M3.8.1-context-optimizer.md index 7312095..aa525f8 100644 --- a/tasks/M3.8.1-context-optimizer.md +++ b/tasks/M3.8.1-context-optimizer.md @@ -4,7 +4,7 @@ |---|---| | Phase | M3.8 โ€” Context optimization | | Size | L โ€” 3โ€“5 days | -| Status | ๐ŸŸก PARTIAL (modules done, needs ingest wiring) | +| Status | โœ… COMPLETE | | Spec | `docs/CONTEXT_OPTIMIZER.md` | | Blocks | M3.8.2 (ingest integration) | | Depends | M3.7.7 (lesson.rs patterns), M3.7.8 (stop words) | diff --git a/tasks/M3.8.2-cache-aligner-headers.md b/tasks/M3.8.2-cache-aligner-headers.md index ed11882..a6c180f 100644 --- a/tasks/M3.8.2-cache-aligner-headers.md +++ b/tasks/M3.8.2-cache-aligner-headers.md @@ -4,7 +4,7 @@ |---|---| | Phase | M3.8 โ€” Context optimization | | Size | M โ€” 1โ€“2 days | -| Status | โฌœ Not started | +| Status | โœ… COMPLETE | | Depends | M3.8.1 (core modules) | | Blocks | M3.8.3 | diff --git a/tasks/M3.8.3-compression-benchmarks.md b/tasks/M3.8.3-compression-benchmarks.md deleted file mode 100644 index 77efd24..0000000 --- a/tasks/M3.8.3-compression-benchmarks.md +++ /dev/null @@ -1,73 +0,0 @@ -# M3.8.3 โ€” Metrics & Monitoring (UPDATED) - -| Field | Value | -|---|---| -| Phase | M3.8 โ€” Context optimization | -| Size | M โ€” 1 day | -| Status | โœ… COMPLETE | -| Depends | M3.8.1, M3.8.2 | -| Blocks | M3.8.4 | - -## Deliverables Completed - -โœ… **MetricsCollector** (7 tests) -- Per-project aggregation of OptimizationMetrics -- Merge metrics from multiple optimization runs -- Structured logging via tracing -- Prometheus-compatible export format - -โœ… **Test Coverage** (7 tests) -- `test_collector_merge_single_project` โ€” Store and retrieve per-project metrics -- `test_collector_merge_multiple_projects` โ€” Aggregate across projects -- `test_collector_merge_aggregates` โ€” Multiple runs per project -- `test_collector_nonexistent_project` โ€” Handle missing project gracefully -- `test_collector_per_compressor_stats` โ€” Track per-compressor breakdowns -- `test_prometheus_export_format` โ€” Valid Prometheus text format -- `test_prometheus_compression_ratio` โ€” Correct ratio calculations - -โœ… **Prometheus Export** -- Counter: `m3_8_optimization_records_total` -- Gauge: `m3_8_optimization_input_bytes_total` -- Gauge: `m3_8_optimization_output_bytes_total` -- Gauge: `m3_8_optimization_compression_ratio` -- Per-compressor stats with labels - -## Integration Pattern - -```rust -use mem_ingest::{MetricsCollector, optimize_record_with_metrics, OptimizationMetrics}; -use std::sync::{Arc, Mutex}; - -let optimizer = ContextOptimizer::from_env()?; -let collector = MetricsCollector::new(); - -for project_id in projects { - let metrics = Arc::new(Mutex::new(OptimizationMetrics::default())); - - for record in source.records() { - let optimized = optimize_record_with_metrics( - record, - &optimizer, - &metrics, - )?; - embed_and_index(&optimized)?; - } - - let final_metrics = metrics.lock().unwrap().clone(); - collector.merge_project(project_id, final_metrics); -} - -// Log summary -collector.log_all_projects(); - -// Export for Prometheus -let prometheus_text = collector.prometheus_export(); -http_server.register_metrics_endpoint("/metrics", prometheus_text); -``` - -## Status - -โœ… **Complete and Production-Ready** -- All 7 tests passing -- Prometheus format validated -- Ready to integrate into rebuild pipeline diff --git a/tasks/M3.8.4-m3.8-gate.md b/tasks/M3.8.4-m3.8-gate.md deleted file mode 100644 index 016768a..0000000 --- a/tasks/M3.8.4-m3.8-gate.md +++ /dev/null @@ -1,66 +0,0 @@ -# M3.8.4 โ€” Query Path Cleanup (IMPLICIT - COMPLETE) - -| Field | Value | -|---|---| -| Phase | M3.8 โ€” Context optimization | -| Size | S โ€” 0 days (no changes needed) | -| Status | โœ… COMPLETE | -| Depends | M3.8.3 | -| Blocks | M3.8.5 | - -## Summary - -โœ… **IMPLICIT COMPLETION** - -When M3.8 architecture was corrected (ingest vs query), the query path was already clean: - -**Current State:** -- โŒ NO query-path compression in `build_cache_aligned()` -- โœ… Only `cache_metrics()` calls optimizer (for metrics collection only, not compression) -- โœ… Actual chunks sent to LLM are unmodified -- โœ… Optimization happens at ingest time via M3.8.2 helpers - -**What Didn't Need Changing:** -- `PromptBuilder::build_cache_aligned()` โ€” Already doesn't compress chunks -- `PromptBuilder::cache_metrics()` โ€” Only uses optimizer for token estimation (okay for metrics) -- `PromptBuilder::build()` โ€” Legacy path, unmodified - -**Why This is Better:** -- Clean separation: ingest optimizes, query doesn't -- LLM gets pre-optimized chunks from search results -- No per-query optimization overhead -- Cache metrics track drift for observability (non-destructive) - -## Verification - -All prompt tests still passing (11 tests): -``` -test prompt::tests::test_cache_aligned_chunk_budget_exceeded ... ok -test prompt::tests::test_cache_aligned_contains_query ... ok -test prompt::tests::test_cache_aligned_memory_budget_exceeded ... ok -test prompt::tests::test_cache_aligned_produces_two_user_messages ... ok -test prompt::tests::test_cache_prefix_is_stable_across_chunks ... ok -test prompt::tests::test_cache_prefix_is_stable_across_memory_changes ... ok -test prompt::tests::test_cache_prefix_tokens_positive ... ok -test prompt::tests::test_headroom_positive_under_budget ... ok -test prompt::tests::test_legacy_build_still_works ... ok -test prompt::tests::test_render_chunk_multiple_roles ... ok -test prompt::tests::test_render_chunk_single_record ... ok -``` - -## Files Verified - -- `crates/mem-core/src/prompt.rs` โ€” No query-path compression -- `crates/mem-core/src/optimizer/mod.rs` โ€” Optimizer available for metrics only -- No changes required - -## Status - -โœ… **Complete** - -This task represents "no work needed" because the architecture was corrected upfront: -- Optimizer at ingest (M3.8.2) โœ… -- Metrics collection in query (M3.8.3) โœ… -- No compression in query โœ… - -The query path is already clean. diff --git a/tasks/M8.3-query-optimizer.md b/tasks/M8.3-query-optimizer.md deleted file mode 100644 index 3b317d2..0000000 --- a/tasks/M8.3-query-optimizer.md +++ /dev/null @@ -1,80 +0,0 @@ -# M8.3 โ€” Query optimizer: context construction + strategy routing - -| Field | Value | -|---|---| -| Phase | M8 โ€” Hybrid Search | -| Size | M โ€” 1โ€“2 days | -| Status | โœ… COMPLETE | -| Flags | โ€” | -| Spec | inlined below | -| Blocks | M8.5 | -| Depends | โ€” (pure logic, no infra dependency) | - -## Goal - -Build a `QueryOptimizer` that analyses a raw user query and produces a `QueryContext` โ€” normalised text, extracted entities, classified question type, and a routed `SearchStrategy`. This runs **before** any database call and determines which engines to use. - -## Design - -**6-stage pipeline:** - -1. **Normalize** โ€” lowercase, trim, collapse whitespace. -2. **Tokenize** โ€” split on whitespace. -3. **Extract entities** โ€” detect years (YYYY), quoted phrases (`"exact"`), tags (`#`, `@`). -4. **Analyse characteristics** โ€” boolean flags: `has_special_syntax`, `has_date_filters`, `has_negation`. -5. **Classify question type** โ€” one of: `Factual`, `Procedural`, `Comparative`, `Troubleshooting`, `Navigational`, `Open`. -6. **Route** โ€” pick `SearchStrategy` with a confidence score (0.0โ€“1.0). - -**Routing rules:** -- Token count < 3 โ†’ `LexicalOnly` (BM25 handles keywords better than embeddings). -- Special syntax (`#tag`, `@mention`, `"phrase"`) โ†’ `LexicalOnly` (preserve exact tokens). -- Date filters present โ†’ `LexicalFirst` (narrow by date in OpenSearch, rerank in pgvector). -- Procedural / Troubleshooting โ†’ `Hybrid` (need both exact errors + semantic understanding). -- Navigational โ†’ `LexicalFirst` (finding specific docs). -- Default โ†’ `Hybrid`. - -**Output struct:** -```rust -pub struct QueryContext { - pub raw_query: String, - pub normalized_query: String, - pub tokens: Vec, - pub entities: HashMap, - pub embedding: Option>, // filled later by worker - pub token_count: usize, - pub has_special_syntax: bool, - pub has_date_filters: bool, - pub has_negation: bool, - pub question_type: QuestionType, - pub search_strategy: SearchStrategy, - pub confidence: f32, -} -``` - -## Steps - -1. Implement `QueryOptimizer` with `optimize_query(&str) -> Result`. -2. Implement each stage as a private method. -3. Write unit tests for every routing rule (โ‰ฅ15 tests). -4. No async, no IO, no dependencies beyond std. Pure logic. - -## Acceptance - -1. `optimize_query("fix port")` โ†’ `LexicalOnly`, confidence โ‰ฅ 0.7. -2. `optimize_query("How do I fix kubernetes port 8080?")` โ†’ `Hybrid`, `Procedural`, confidence โ‰ฅ 0.9. -3. `optimize_query("#networking @devops policy")` โ†’ `LexicalOnly`, `has_special_syntax=true`. -4. `optimize_query("deployment failures in 2024")` โ†’ `LexicalFirst`, `has_date_filters=true`, entity `year=2024`. -5. `optimize_query("Compare Docker and Kubernetes")` โ†’ `Hybrid`, `Comparative`. -6. All 15+ tests pass. - -## Verify - -```bash -cargo test -p mem-cli query_optimizer:: -- --nocapture -``` - -**False pass:** Routing always returns `Hybrid` regardless of input. Check the short-query and special-syntax tests specifically โ€” they must return non-Hybrid strategies. - -## Artifacts - -- `crates/mem-cli/src/query_optimizer.rs` (exists, needs cleanup + test fixes) diff --git a/tasks/M8.4-rrf-fusion.md b/tasks/M8.4-rrf-fusion.md deleted file mode 100644 index 7f978b0..0000000 --- a/tasks/M8.4-rrf-fusion.md +++ /dev/null @@ -1,82 +0,0 @@ -# M8.4 โ€” Reciprocal Rank Fusion engine - -| Field | Value | -|---|---| -| Phase | M8 โ€” Hybrid Search | -| Size | S โ€” 0.5โ€“1 day | -| Status | โœ… COMPLETE | -| Flags | โ€” | -| Spec | inlined below | -| Blocks | M8.5 | -| Depends | โ€” (pure logic, no infra dependency) | - -## Goal - -Implement Reciprocal Rank Fusion (RRF) that merges two ranked lists from different scoring distributions into a single ranked list. No parameter tuning required. - -## Why RRF, not weighted linear - -pgvector returns cosine similarity in `[0.0, 1.0]`. OpenSearch BM25 returns unbounded scores in `[0, 50+]`. These distributions are incomparable. - -**Weighted linear (0.6 * sem + 0.4 * lex)** requires min-max normalisation, which is fragile: one outlier score compresses all other scores to near-zero. It also requires choosing weights, which requires labelled data we don't have yet. - -**RRF** ignores score magnitudes entirely. It uses only **rank positions**: the document that appears first in a list gets rank 1, second gets rank 2, etc. The formula is: - -``` -RRF_score(d) = ฮฃ 1 / (k + rank_i(d)) - lists -``` - -Where `k = 60` is a constant (academic standard, Cormack et al. 2009). A document in rank 1 of both lists gets `1/61 + 1/61 = 0.0328`. A document in rank 1 of only one list gets `1/61 = 0.0164`. The first always outranks the second, regardless of original score magnitudes. - -## Design - -```rust -pub struct RRFConfig { - pub k: f32, // 60.0 (constant, don't tune) - pub retrieve_k: usize, // 50 (top-K from each engine) - pub final_k: usize, // 10 (return top-K) -} - -pub struct RRFFusion { config: RRFConfig } - -impl RRFFusion { - pub fn fuse( - &self, - semantic: Vec<(String, f32)>, // (chunk_id, score) โ€” sorted by score desc - lexical: Vec<(String, f32)>, - ) -> Vec<(String, f32)>; // (chunk_id, rrf_score) โ€” sorted desc, truncated -} -``` - -**Invariants:** -- Input lists must be pre-sorted by score descending (rank = position). -- Output is sorted by RRF score descending. -- Output length โ‰ค `final_k`. -- A document appearing in both lists always outranks one appearing in only one (given same rank positions). - -## Steps - -1. Implement `RRFFusion::fuse()`. -2. Implement `RRFFusion::normalize_scores()` as utility (for optional weighted-linear fallback). -3. Write tests: basic fusion, single-engine input, identical lists, disjoint lists, empty inputs. - -## Acceptance - -1. `fuse([(a,0.9),(b,0.8)], [(a,8.0),(c,7.0)])` โ†’ `a` is rank 1 (appears in both lists). -2. `fuse([(a,0.9)], [])` โ†’ `a` is rank 1 with score `1/61`. -3. `fuse([], [])` โ†’ empty result. -4. `fuse([(a,0.9),(b,0.8)], [(b,8.0),(a,7.0)])` โ†’ `a` and `b` have equal RRF scores (both appear in both at same combined rank sum). Either order is acceptable. -5. Output length never exceeds `final_k`. - -## Verify - -```bash -cargo test -p mem-cli rrf -- --nocapture -``` - -**False pass:** Fusion returns results but sorted by original score, not RRF score. Verify by checking that a document ranked #3 in semantic but #1 in lexical outranks a document ranked #1 in semantic but absent from lexical. - -## Artifacts - -- `crates/mem-cli/src/query_optimizer.rs` (RRFFusion struct, lives alongside QueryOptimizer) diff --git a/tasks/M8.5-hybrid-query-worker.md b/tasks/M8.5-hybrid-query-worker.md deleted file mode 100644 index 9b92086..0000000 --- a/tasks/M8.5-hybrid-query-worker.md +++ /dev/null @@ -1,134 +0,0 @@ -# M8.5 โ€” Hybrid query worker: parallel retrieval + fusion - -| Field | Value | -|---|---| -| Phase | M8 โ€” Hybrid Search | -| Size | L โ€” 2โ€“3 days | -| Status | โœ… COMPLETE | -| Flags | โ€” | -| Spec | inlined below | -| Blocks | M8.6, M8.7 | -| Depends | M8.1 (OpenSearch running), M8.2 (dual-write), M8.3 (query optimizer), M8.4 (RRF) | - -## Goal - -`HybridQueryWorker` orchestrates parallel retrieval from pgvector and OpenSearch, applies RRF fusion, and returns ranked results with score breakdown and latency metrics. It replaces `QueryWorker` as the primary query engine behind `GET /memory/query`. - -## Design - -**Execution flow:** -1. Call `QueryOptimizer::optimize_query()` โ†’ get `QueryContext` with `SearchStrategy`. -2. Generate embedding via `EmbeddingsClient::embed()`. -3. Execute strategy: - - **Hybrid:** `tokio::try_join!` pgvector top-50 + OpenSearch top-50. Fuse with RRF. - - **LexicalFirst:** OpenSearch top-200 โ†’ extract IDs โ†’ pgvector `WHERE id IN (...)` top-10. - - **SemanticOnly:** pgvector top-50 (fallback if OpenSearch unavailable). - - **LexicalOnly:** OpenSearch top-50 (fallback if embedding model unavailable). -4. Build response with score breakdown. - -**Critical: use actual VectorStore API.** - -The existing `VectorStore` exposes `search_l1(project, &embedding, limit)` and `search_l2(project, &embedding)`. The worker must call these โ€” not invented methods. - -For the cascading strategy (`LexicalFirst`), a new `search_l1_by_ids(project, &embedding, limit, &[chunk_id])` method is needed on `VectorStore`. This is a filtered pgvector query: -```sql -SELECT id, 1 - (embedding <=> $1) as score, text, source -FROM chunks -WHERE project = $2 AND id = ANY($3) -ORDER BY embedding <=> $1 -LIMIT $4 -``` - -**OpenSearch query:** -```json -{ - "size": 50, - "query": { - "bool": { - "must": [{ - "multi_match": { - "query": "...", - "fields": ["content^2", "section_title^1.5", "breadcrumb", "source"], - "type": "best_fields", - "fuzziness": "AUTO" - } - }], - "filter": [ - {"term": {"project_id": "..."}}, - {"terms": {"level": ["L0", "L1", "L2"]}} - ] - } - } -} -``` - -**Response struct:** -```rust -pub struct HybridQueryResponse { - pub query: String, - pub project: String, - pub search_strategy: String, - pub results: Vec, - pub metrics: QueryMetrics, -} - -pub struct HybridQueryResult { - pub id: String, - pub text: String, - pub source: String, - pub level: String, - pub breadcrumb: Vec, - pub final_score: f32, - pub semantic_rank: Option, - pub lexical_rank: Option, - pub fusion_method: String, -} - -pub struct QueryMetrics { - pub total_time_ms: u128, - pub semantic_time_ms: Option, - pub lexical_time_ms: Option, - pub fusion_time_ms: u128, - pub semantic_candidates: Option, - pub lexical_candidates: Option, - pub final_count: usize, -} -``` - -## Steps - -1. Add `search_l1_by_ids()` to `VectorStore` (new SQL query with `id = ANY($3)`). -2. Make `OpenSearchClient::lexical_search()` public. -3. Implement `HybridQueryWorker::new()` taking `VectorStore`, `EmbeddingsClient`, `Option`. -4. Implement `query()` method with strategy dispatch. -5. Implement `retrieve_hybrid()` using `tokio::try_join!`. -6. Implement `retrieve_cascading()` (2-stage). -7. Implement fallback methods (`retrieve_semantic()`, `retrieve_lexical()`). -8. Wire RRF fusion into the result pipeline. -9. Build response with per-result rank tracking. -10. Write integration tests with mock VectorStore + mock OpenSearch. - -## Acceptance - -1. Hybrid query returns results from **both** engines โ€” check `semantic_rank` and `lexical_rank` are both `Some` for documents appearing in both lists. -2. Cascading query's pgvector call receives only IDs from the OpenSearch narrowing step โ€” verify with query log or mock. -3. If `OpenSearchClient` is `None`, strategy automatically falls back to `SemanticOnly`. -4. If embedding generation fails, strategy falls back to `LexicalOnly` (if OpenSearch available) or returns error. -5. `metrics.total_time_ms` is populated and < 500ms for test fixtures. -6. `metrics.semantic_time_ms` and `lexical_time_ms` are roughly equal (parallel execution, not serial). -7. Results are sorted by `final_score` descending. -8. `final_count` โ‰ค `RRFConfig.final_k`. - -## Verify - -```bash -cargo test -p mem-cli hybrid_query -- --nocapture -``` - -**False pass:** Worker always returns semantic-only results even when strategy is `Hybrid`. Check by asserting `lexical_rank.is_some()` on at least one result when OpenSearch is configured. Another false pass: serial execution disguised as parallel โ€” assert `max(semantic_time_ms, lexical_time_ms) โ‰ˆ total - fusion_time_ms`, not `sum`. - -## Artifacts - -- `crates/mem-cli/src/hybrid_query_worker.rs` (rewrite from current stub) -- Modified `crates/mem-store/src/lib.rs` (add `search_l1_by_ids`) -- Modified `crates/mem-cli/src/opensearch_client.rs` (make `lexical_search` pub) diff --git a/tasks/M8.6-query-endpoint-upgrade.md b/tasks/M8.6-query-endpoint-upgrade.md deleted file mode 100644 index 0c2ecad..0000000 --- a/tasks/M8.6-query-endpoint-upgrade.md +++ /dev/null @@ -1,127 +0,0 @@ -# M8.6 โ€” Upgrade GET /memory/query to hybrid with fallback - -| Field | Value | -|---|---| -| Phase | M8 โ€” Hybrid Search | -| Size | M โ€” 1โ€“2 days | -| Status | โœ… COMPLETE | -| Flags | โ€” | -| Spec | inlined below | -| Blocks | M8.8, M8.9 | -| Depends | M8.3, M8.4 (query optimizer, RRF fusion complete) | - -## Goal - -Replace the `QueryWorker` call in `GET /memory/query` with `HybridQueryWorker`. Add `?method=` parameter for explicit strategy override. Implement fallback chain: hybrid โ†’ semantic โ†’ error. - -## Design - -**Updated request:** -``` -GET /memory/query?query=...&project=...&limit=10&method=hybrid -Authorization: Bearer -``` - -New parameter: -- `method` (optional) โ€” `hybrid` (default), `semantic`, `lexical`. If omitted, `QueryOptimizer` decides. - -**Handler logic:** -```rust -async fn query_handler(...) -> HttpResponse { - let (claims, token) = validate_auth(...)?; - check_capability(&claims, "memory:read")?; - check_rate_limit(&claims, &state, "/memory/query")?; - - let method_override = query.get("method").map(|m| match m.as_str() { - "semantic" => SearchStrategy::SemanticOnly, - "lexical" => SearchStrategy::LexicalOnly, - _ => SearchStrategy::Hybrid, // includes "hybrid" and unknown values - }); - - // Try hybrid worker first - if let Some(ref hybrid) = state.hybrid_query_worker { - match hybrid.query(&project, &question, limit, &token, method_override).await { - Ok(response) => return HttpResponse::Ok().json(response), - Err(e) => { - tracing::warn!("hybrid query failed, falling back: {}", e); - } - } - } - - // Fallback: existing semantic-only worker - match state.query_worker.query(&project, &question, Some(limit)).await { - Ok(results) => HttpResponse::Ok().json(json!({ - "query": question, - "project": project, - "search_strategy": "semantic_fallback", - "results": results, - })), - Err(e) => HttpResponse::InternalServerError().json(json!({"error": "query_failed"})), - } -} -``` - -**AppState change:** -```rust -pub struct AppState { - // ... existing fields ... - pub hybrid_query_worker: Option>, // None if OpenSearch not configured -} -``` - -Constructed in `start_server()`: -```rust -let hybrid = if std::env::var("OPENSEARCH_HOSTS").is_ok() { - let os_client = OpenSearchClient::new(hosts); - Some(Arc::new(HybridQueryWorker::new(vector_store, embeddings, Some(Arc::new(os_client))))) -} else { - // No OpenSearch configured โ€” hybrid worker without lexical - Some(Arc::new(HybridQueryWorker::new(vector_store, embeddings, None))) -}; -``` - -**Backward compatibility:** If `OPENSEARCH_HOSTS` is not set, the worker still works but always uses `SemanticOnly`. Existing clients see the same results with an added `search_strategy` field. - -## Steps - -1. Add `hybrid_query_worker` field to `AppState`. -2. Construct `HybridQueryWorker` in `start_server()`, gated on `OPENSEARCH_HOSTS` env. -3. Update `query_handler()` with method override + fallback chain. -4. Add `?method=` query parameter parsing. -5. Update response format to include `search_strategy` and `metrics`. -6. Write integration test: query with `method=hybrid`, verify response has `metrics`. -7. Write integration test: query without OpenSearch, verify fallback to semantic. - -## Acceptance - -1. `GET /memory/query?query=test&project=p` returns `"search_strategy": "Hybrid"` when OpenSearch configured. -2. `GET /memory/query?query=test&project=p&method=semantic` forces semantic-only, response says `"search_strategy": "SemanticOnly"`. -3. If OpenSearch is down, query still succeeds with `"search_strategy": "semantic_fallback"`. -4. Response includes `metrics` block with timing data. -5. Existing clients that don't send `method` param get the same results as before (backward compat). -6. JWT token is forwarded to OpenSearch (not a new token, not admin credentials). - -## Verify - -```bash -# With OpenSearch running -curl -H "Authorization: Bearer $TOKEN" \ - "http://localhost:8080/memory/query?query=kubernetes+port&project=poimen" | jq .search_strategy -# Should output: "Hybrid" - -# Force semantic -curl -H "Authorization: Bearer $TOKEN" \ - "http://localhost:8080/memory/query?query=kubernetes+port&project=poimen&method=semantic" | jq .search_strategy -# Should output: "SemanticOnly" - -# Kill OpenSearch, retry -curl -H "Authorization: Bearer $TOKEN" \ - "http://localhost:8080/memory/query?query=kubernetes+port&project=poimen" | jq .search_strategy -# Should output: "semantic_fallback" -``` - -**False pass:** Handler catches the hybrid error silently and always falls back to semantic โ€” user never sees hybrid results even when OpenSearch is healthy. Assert that with OpenSearch up, `metrics.lexical_candidates` is `Some(n)` where `n > 0`. - -## Artifacts - -- Modified `crates/mem-cli/src/http_server.rs` diff --git a/tasks/M8.7-index-optimization.md b/tasks/M8.7-index-optimization.md deleted file mode 100644 index 9087e73..0000000 --- a/tasks/M8.7-index-optimization.md +++ /dev/null @@ -1,103 +0,0 @@ -# M8.7 โ€” Index tuning: HNSW parameters + OpenSearch analyzers - -| Field | Value | -|---|---| -| Phase | M8 โ€” Hybrid Search | -| Size | M โ€” 1โ€“2 days | -| Status | โœ… COMPLETE | -| Flags | โ€” | -| Spec | inlined below | -| Blocks | M8.9 | -| Depends | M8.2 (data in both stores), M8.5 (can query both stores) | - -## Goal - -Tune pgvector index parameters and OpenSearch analyzers for retrieval accuracy. Measure baseline NDCG before and after tuning. This is engineering, not research โ€” change one parameter, measure, keep or revert. - -## Design - -### pgvector tuning - -**Current:** `ivfflat` index with default `lists`. - -**Target:** Switch to `hnsw` index (pgvector 0.5.0+). HNSW provides better recall than IVFFlat at the cost of slower index builds and more memory. - -```sql --- Drop old index -DROP INDEX IF EXISTS chunks_embedding_idx; - --- Create HNSW index -CREATE INDEX chunks_embedding_hnsw_idx -ON chunks USING hnsw (embedding vector_cosine_ops) -WITH (m = 16, ef_construction = 64); -``` - -Parameters: -- `m = 16` โ€” max connections per node (default 16, higher = better recall, more memory). -- `ef_construction = 64` โ€” build-time search width (default 64, higher = better recall, slower build). -- `ef_search = 40` โ€” query-time search width (set via `SET hnsw.ef_search = 40`). - -**Tuning approach:** -1. Baseline: measure recall@50 with IVFFlat. -2. Switch to HNSW with defaults. -3. Measure recall@50 again. -4. If recall@50 โ‰ฅ 0.95, keep defaults. Otherwise increase `ef_construction` to 128. - -### OpenSearch tuning - -**Analyzer changes:** -- Add `edge_ngram` tokenizer for typo tolerance on `content` field. -- Add `synonym` filter for common abbreviations: `k8s โ†’ kubernetes`, `db โ†’ database`, `cfg โ†’ config`. -- Keep `standard` analyzer as primary, add `search_analyzer` for queries. - -**Field boost tuning:** -- `content^2.0` (default โ€” most important). -- `section_title^1.8` (headings are very relevant). -- `source^1.0` (file paths are useful but shouldn't dominate). -- `breadcrumb^0.8` (context, not content). - -**BM25 parameters:** -- `k1 = 1.2` (term frequency saturation โ€” default is fine). -- `b = 0.75` (length normalization โ€” default is fine). -- Don't tune these unless baseline NDCG < 0.7. - -## Steps - -1. Create test query set: 20 queries with known-relevant documents. -2. Measure baseline NDCG@10 for pgvector (semantic-only) and OpenSearch (lexical-only). -3. Switch pgvector from IVFFlat to HNSW. Measure NDCG@10 again. -4. Update OpenSearch index template with synonym filter + edge_ngram. Reindex. Measure. -5. Record all measurements in `docs/INDEX_TUNING_RESULTS.md`. -6. Keep changes that improve NDCG. Revert changes that don't. - -## Acceptance - -1. pgvector uses HNSW index (verify with `\d+ chunks` in psql). -2. OpenSearch index template includes synonym filter. -3. NDCG@10 measurements recorded for before/after each change. -4. No regression: post-tuning NDCG โ‰ฅ pre-tuning NDCG for both engines. -5. pgvector query latency < 150ms (p95) after HNSW switch. -6. OpenSearch query latency < 100ms (p95) after analyzer changes. - -## Verify - -```bash -# Check pgvector index type -psql -c "\d+ chunks" | grep hnsw - -# Check OpenSearch analyzer -curl -k -H "Authorization: Bearer $TOKEN" \ - https://opensearch-internal:9200/vault-test/_settings | jq '.*.settings.index.analysis' - -# Run NDCG measurement -cargo run -- bench-search --queries fixtures/search_queries.yaml --output docs/INDEX_TUNING_RESULTS.md -``` - -**False pass:** HNSW index created but `ef_search` set to 1, making recall worse than IVFFlat. Check recall@50 explicitly โ€” it should be โ‰ฅ 0.95. - -## Artifacts - -- SQL migration (drop IVFFlat, create HNSW) -- Updated OpenSearch index template -- `docs/INDEX_TUNING_RESULTS.md` (measurements) -- `fixtures/search_queries.yaml` (test query set) diff --git a/tasks/M8.8-accuracy-benchmarks.md b/tasks/M8.8-accuracy-benchmarks.md deleted file mode 100644 index e2bbc39..0000000 --- a/tasks/M8.8-accuracy-benchmarks.md +++ /dev/null @@ -1,105 +0,0 @@ -# M8.8 โ€” Accuracy benchmarks: NDCG, MRR, Precision/Recall - -| Field | Value | -|---|---| -| Phase | M8 โ€” Hybrid Search | -| Size | M โ€” 1โ€“2 days | -| Status | โœ… COMPLETE | -| Flags | โ€” | -| Spec | inlined below | -| Blocks | M8.9 | -| Depends | M8.6 (hybrid endpoint working), M8.7 (indices tuned) | - -## Goal - -Build a repeatable benchmark harness that measures retrieval accuracy. Run it against semantic-only, lexical-only, and hybrid strategies. Produce a comparison table that proves hybrid is better (or shows where it isn't). - -## Design - -**Test fixture format:** -```yaml -# fixtures/search_queries.yaml -queries: - - id: q1 - text: "How do I fix kubernetes port 8080 conflict?" - relevant_docs: ["runbooks/networking.md"] - category: troubleshooting - - - id: q2 - text: "What is a StatefulSet?" - relevant_docs: ["docs/kubernetes-concepts.md", "docs/statefulsets.md"] - category: factual - - - id: q3 - text: "#networking firewall rules" - relevant_docs: ["docs/network-policies.md"] - category: navigational -``` - -**Metrics implemented:** -- **NDCG@10** โ€” Are relevant docs ranked near the top? (0.0 worst, 1.0 perfect). -- **MRR** โ€” How early is the first relevant doc? (1/rank of first hit). -- **Precision@5** โ€” What fraction of top-5 results are relevant? -- **Recall@10** โ€” What fraction of all relevant docs appear in top-10? - -**Benchmark runner:** -```rust -pub struct BenchmarkResult { - pub strategy: String, // "hybrid", "semantic", "lexical" - pub avg_ndcg: f32, - pub avg_mrr: f32, - pub avg_precision_at_5: f32, - pub avg_recall_at_10: f32, - pub avg_latency_ms: f32, - pub per_query: Vec, -} -``` - -**Output:** Markdown table written to `docs/BENCHMARK_RESULTS.md`. - -```markdown -| Strategy | NDCG@10 | MRR | P@5 | R@10 | Latency (ms) | -|----------|---------|-----|-----|------|-------------| -| semantic | 0.72 | 0.65 | 0.60 | 0.75 | 95 | -| lexical | 0.68 | 0.70 | 0.55 | 0.70 | 62 | -| hybrid | 0.87 | 0.82 | 0.78 | 0.90 | 210 | -``` - -## Steps - -1. Create `fixtures/search_queries.yaml` with โ‰ฅ 20 queries across categories. -2. Ingest corresponding test documents into both stores. -3. Implement `BenchmarkRunner` that: - a. Loads fixture file. - b. Runs each query against each strategy. - c. Computes NDCG, MRR, Precision, Recall per query. - d. Averages across queries. - e. Writes results to markdown. -4. Implement as CLI command: `cargo run -- bench-search --queries --output `. -5. Run benchmarks. Record results. - -## Acceptance - -1. Benchmark runs to completion on 20+ queries ร— 3 strategies = 60+ query executions. -2. Output markdown table has all 5 columns populated. -3. Hybrid NDCG@10 โ‰ฅ max(semantic NDCG, lexical NDCG) โ€” hybrid must not be worse than best single engine. -4. Per-query results show which categories benefit most from hybrid (expected: troubleshooting, procedural). -5. If hybrid is worse on any category, document why and whether it matters. - -## Verify - -```bash -cargo run -- bench-search \ - --queries fixtures/search_queries.yaml \ - --output docs/BENCHMARK_RESULTS.md - -cat docs/BENCHMARK_RESULTS.md -``` - -**False pass:** Benchmark uses the same documents for queries and ground truth (trivial exact match). Ensure queries use **natural language** and ground truth docs use **technical content** โ€” the match should be semantic, not string equality. - -## Artifacts - -- `fixtures/search_queries.yaml` -- `crates/mem-cli/src/bench_search.rs` (new) -- `docs/BENCHMARK_RESULTS.md` (output) diff --git a/tasks/M8.9-m8-gate.md b/tasks/M8.9-m8-gate.md deleted file mode 100644 index 94498d3..0000000 --- a/tasks/M8.9-m8-gate.md +++ /dev/null @@ -1,120 +0,0 @@ -# M8.9 โ€” M8 composition gate: hybrid search proves its value - -| Field | Value | -|---|---| -| Phase | M8 โ€” Hybrid Search | -| Size | M โ€” 1 day | -| Status | โœ… COMPLETE | -| Flags | gate | -| Spec | inlined below | -| Blocks | โ€” | -| Depends | M8.1โ€“M8.8 all โœ… | - -## Goal - -Prove the hybrid search system works end-to-end and is measurably better than semantic-only. This gate verifies the **composition** โ€” individual tasks pass their own tests, but only the gate proves they compose correctly. - -## Properties to verify - -### P1: Dual-write consistency - -Every chunk in pgvector has a corresponding document in OpenSearch with the same ID, and vice versa. Zero orphans. - -```sql --- pgvector IDs not in OpenSearch -SELECT id FROM chunks WHERE opensearch_pending = true; --- Must return 0 rows (after background retry has run) -``` - -```bash -# OpenSearch document count must equal pgvector chunk count for same project -PG_COUNT=$(psql -t -c "SELECT count(*) FROM chunks WHERE project='test'") -OS_COUNT=$(curl -sk "https://opensearch:9200/vault-test/_count" | jq .count) -# PG_COUNT == OS_COUNT -``` - -### P2: Hybrid outperforms single-engine - -From `docs/BENCHMARK_RESULTS.md`: -- Hybrid NDCG@10 > semantic-only NDCG@10. -- Hybrid NDCG@10 > lexical-only NDCG@10. -- If this fails for a specific query category, it must be documented with reasoning. - -### P3: Fallback works under failure - -1. Stop OpenSearch. Query endpoint still responds with semantic results. -2. Start OpenSearch. Query endpoint returns hybrid results. -3. Response `search_strategy` field accurately reports which mode was used. - -### P4: JWT auth enforced end-to-end - -1. Query without token โ†’ 401. -2. Query with valid token โ†’ 200. -3. OpenSearch rejects requests from non-Memory-Service pods (NetworkPolicy). -4. JWT token forwarded from Memory Service to OpenSearch (not admin credentials). - -### P5: No regression on existing tests - -All pre-existing tests still pass. `cargo test` green. No `#[ignore]` added in M8. - -### P6: Latency budget met - -- Hybrid query: p95 < 500ms. -- Semantic-only query: p95 < 200ms (must not regress from adding hybrid path). -- Fallback to semantic: p95 < 250ms (minimal overhead from failed OpenSearch attempt). - -## Gate test - -```bash -#!/bin/bash -set -euo pipefail - -echo "=== M8 Gate: Hybrid Search ===" - -# P5: All tests pass -cargo test 2>&1 | tail -1 -# Expected: test result: ok. X passed; 0 failed - -# P1: Dual-write consistency -PG=$(psql -t -c "SELECT count(*) FROM chunks WHERE project='test' AND opensearch_pending=false") -OS=$(curl -sk "https://opensearch:9200/vault-test/_count" | jq .count) -[ "$PG" -eq "$OS" ] && echo "P1 PASS: $PG chunks in both stores" || echo "P1 FAIL: pg=$PG os=$OS" - -# P2: Hybrid > single-engine -grep -A1 "hybrid" docs/BENCHMARK_RESULTS.md | grep -oP '[\d.]+' | head -1 -# Must be highest NDCG in the table - -# P3: Fallback -kubectl scale statefulset/opensearch -n poimen --replicas=0 -sleep 5 -STRATEGY=$(curl -s -H "Authorization: Bearer $TOKEN" \ - "http://localhost:8080/memory/query?query=test&project=test" | jq -r .search_strategy) -[ "$STRATEGY" = "semantic_fallback" ] && echo "P3 PASS: fallback works" || echo "P3 FAIL: $STRATEGY" -kubectl scale statefulset/opensearch -n poimen --replicas=2 -sleep 30 -STRATEGY=$(curl -s -H "Authorization: Bearer $TOKEN" \ - "http://localhost:8080/memory/query?query=test&project=test" | jq -r .search_strategy) -[ "$STRATEGY" = "Hybrid" ] && echo "P3 PASS: hybrid restored" || echo "P3 FAIL: $STRATEGY" - -# P4: Auth -STATUS=$(curl -s -o /dev/null -w '%{http_code}' "http://localhost:8080/memory/query?query=test&project=test") -[ "$STATUS" = "401" ] && echo "P4 PASS: no-auth rejected" || echo "P4 FAIL: $STATUS" - -echo "=== M8 Gate Complete ===" -``` - -## Acceptance - -All six properties pass. If P2 fails (hybrid not better), the gate does NOT pass โ€” go back and fix M8.7 (index tuning) or M8.4 (fusion algorithm). - -## False passes to check - -1. **P1 looks green but IDs don't match.** Run a JOIN, not just count comparison. -2. **P3 looks green but fallback latency is 30s** (timeout, not fast fail). Check p95 < 250ms. -3. **P5 looks green but test count dropped.** Compare `cargo test 2>&1 | grep 'test result'` against last known count (currently 239+). - -## Artifacts - -- Gate script (inline above) -- `docs/BENCHMARK_RESULTS.md` (from M8.8) -- All M8.1โ€“M8.8 artifacts