From a96cef7eee1cd053438ca1c451ac5154b4c6d798 Mon Sep 17 00:00:00 2001 From: poimen Date: Fri, 28 Aug 2026 13:42:36 -0700 Subject: [PATCH] feat: Archive M3.8.1, M3.8.2 - remove task files after completion --- tasks/INDEX.md | 2 +- tasks/M3.8.1-context-optimizer.md | 139 -------------------------- tasks/M3.8.2-cache-aligner-headers.md | 126 ----------------------- 3 files changed, 1 insertion(+), 266 deletions(-) delete mode 100644 tasks/M3.8.1-context-optimizer.md delete mode 100644 tasks/M3.8.2-cache-aligner-headers.md diff --git a/tasks/INDEX.md b/tasks/INDEX.md index 9cb4db7..c759c0a 100644 --- a/tasks/INDEX.md +++ b/tasks/INDEX.md @@ -71,7 +71,7 @@ Legend: โฌœ not started ยท ๐ŸŸก in progress ยท โœ… done ยท โ›” blocked | 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | โฌœ M6.6 | | 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | โฌœ M7.10 | | 9 | Hybrid search | M8.x | 9 | 9 | 0 | 0 | โœ… M8.9 | -| | **Total** | | **62** | **61** | **0** | **1** | 11/13 green | +| | **Total** | | **60** | **59** | **0** | **1** | 11/13 green | **Current status โ€” 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). **M2.1-6 โœ…** (embeddings, CNPG, schema, pgvector, obsidian projector, rebuild). **M3.x โœ…** (4/4). **M3.5.x โœ…** (10/10 complete + archived). **M3.7.7-8 โœ…** (failure diagnosis). **M4.1-2 โœ…** (skill drafting + derived filter). **M3.6.1 โœ…** (DocCorpusSource). **M3.6.3 โŒ retired** (Obsidian UI replaces CLI). **M3.6.7-8 โฌœ new** (ingest enrichment + deduplication). **M8.1 ๐ŸŸก** (OpenSearch cluster deploying โ€” security context fixes in progress). diff --git a/tasks/M3.8.1-context-optimizer.md b/tasks/M3.8.1-context-optimizer.md deleted file mode 100644 index aa525f8..0000000 --- a/tasks/M3.8.1-context-optimizer.md +++ /dev/null @@ -1,139 +0,0 @@ -# M3.8.1 โ€” Context Optimizer Core Modules - -| Field | Value | -|---|---| -| Phase | M3.8 โ€” Context optimization | -| Size | L โ€” 3โ€“5 days | -| Status | โœ… COMPLETE | -| Spec | `docs/CONTEXT_OPTIMIZER.md` | -| Blocks | M3.8.2 (ingest integration) | -| Depends | M3.7.7 (lesson.rs patterns), M3.7.8 (stop words) | - -## Status: PARTIAL โš ๏ธ - -โœ… **Core Compressor Modules Complete**: 1,100 LOC, 62 tests -- ContentRouter (Magika ML detection) -- LogCompressor, JsonCrusher, DiffCompressor, TextCompressor -- CacheAligner (drift detection) -- CcrStore (reversible compression) -- ContextOptimizer orchestrator - -โŒ **Integration in Wrong Place**: -- Currently: PromptBuilder.build_cache_aligned() (query path) -- Should be: rebuild.rs ingest pipeline (ingest path) -- Result: Improves only LLM input, not search quality - -## What Was Done Right - -โœ… **Content Detection** (Magika ML + regex) -- <1ms classification -- Detects JSON, code, logs, diffs, config, text -- Thread-safe, ONNX local - -โœ… **5 Compressor Implementations** -- LogCompressor: 85-95% ratio (keep errors + stack traces) -- JsonCrusher: 70-90% ratio (field variance) -- DiffCompressor: 60-80% ratio (change lines only) -- TextCompressor: 30-50% ratio (token importance) -- ConfigCompressor: passthrough (already compact) - -โœ… **Cache Alignment** -- Detects dynamic patterns (timestamps, UUIDs, session IDs) -- Drift metric (0.0-1.0) -- Separates stable prefix from dynamic tail - -โœ… **Reversible Compression** (CCR Store) -- LRU cache with SHA256 -- TTL-based expiry -- Model can retrieve originals via hint injection - -## What Needs Fixing - -### Root Issue: Architecture Misunderstanding - -**Documented** (โŒ Wrong): -``` -Ingest โ†’ pgvector + OpenSearch (full noise) - โ†“ -Query โ†’ M3.8 compression โ†’ LLM -``` - -**Should Be** (โœ… Correct): -``` -Ingest โ†’ M3.8 optimization โ†’ pgvector + OpenSearch (clean) - โ†“ -Query โ†’ retrieve clean results โ†’ LLM -``` - -**Why the correct way is better:** -1. Cleaner text โ†’ better embeddings (pgvector) -2. Signal-rich text โ†’ better BM25 ranking (OpenSearch) -3. One-time processing at ingest, not per-query -4. All users benefit from cleaner search results -5. LLM already gets optimized chunks - -### Next Steps - -**M3.8.2**: Ingest Pipeline Integration (1 day) -- Create OptimizerSink wrapper around ingest sources -- Wire into rebuild.rs -- Test with all source types -- Collect metrics - -**M3.8.3**: Metrics & Monitoring (1 day) -- Track compression ratio per chunk -- Aggregate per project/source/type -- Emit to tracing/Prometheus -- Dashboard visualization - -**M3.8.4**: Query Path Cleanup (0.5 days) -- Remove PromptBuilder.build_cache_aligned() optimizer call -- Keep cache_metrics() for observability (drift tracking) -- Simplify PromptBuilder - -## Test Summary - -โœ… **62 Unit Tests** (all passing) -- Phase 1: 17 (router, log) -- Phase 2: 15 (json, diff) -- Phase 3: 18 (cache align, CCR) -- Phase 4: 12 (text, config) - -โณ **20 New Tests Pending** (M3.8.2-3) -- Ingest source optimization -- Metrics collection -- End-to-end pipeline - -## Files - -**Implemented** (1,100 LOC): -- `crates/mem-core/src/optimizer/mod.rs` -- `crates/mem-core/src/optimizer/router.rs` -- `crates/mem-core/src/optimizer/log.rs` -- `crates/mem-core/src/optimizer/json.rs` -- `crates/mem-core/src/optimizer/diff.rs` -- `crates/mem-core/src/optimizer/text.rs` -- `crates/mem-core/src/optimizer/cache_align.rs` -- `crates/mem-core/src/optimizer/ccr.rs` - -**Pending** (180 LOC): -- `crates/mem-ingest/src/optimizer_sink.rs` (M3.8.2) -- `crates/mem-core/src/optimizer/metrics.rs` (M3.8.3) - -## Commits - -1. `bf13e3a` โ€” Phase 1: ContentRouter + LogCompressor -2. `a903a3f` โ€” Phase 2: JsonCrusher + DiffCompressor -3. `edcc231` โ€” Phase 3: CacheAligner + CcrStore -4. `8d8addc` โ€” Phase 4: TextCompressor + env config - -## Lessons Learned - -1. **Ingest-time optimization > query-time**: Better for entire pipeline -2. **Compression ratios vary widely**: Log 85-95% vs text 30-50% -3. **Reversibility matters**: Model needs originals for detailed analysis -4. **Metrics > assumption**: Need to measure actual improvement in search quality - -## Remediation - -See **`tasks/M3.8-CORRECTED-architecture.md`** for complete re-architecture plan. diff --git a/tasks/M3.8.2-cache-aligner-headers.md b/tasks/M3.8.2-cache-aligner-headers.md deleted file mode 100644 index a6c180f..0000000 --- a/tasks/M3.8.2-cache-aligner-headers.md +++ /dev/null @@ -1,126 +0,0 @@ -# M3.8.2 โ€” Ingest Pipeline Integration (OptimizerSink) - -| Field | Value | -|---|---| -| Phase | M3.8 โ€” Context optimization | -| Size | M โ€” 1โ€“2 days | -| Status | โœ… COMPLETE | -| Depends | M3.8.1 (core modules) | -| Blocks | M3.8.3 | - -## Goal - -Wire M3.8 compressors into the ingest pipeline so that chunks are optimized -BEFORE embedding + indexing, resulting in: -- Better embeddings (clean text) -- Better search ranking (signal-rich documents) -- Cleaner results for all queries - -## Deliverables - -### 1. OptimizerSink Wrapper (100 LOC) - -New module: `crates/mem-ingest/src/optimizer_sink.rs` - -```rust -pub struct OptimizerSink { - inner: Box, - optimizer: ContextOptimizer, - config: OptimizerConfig, - metrics: MetricsCollector, -} - -impl RecordSource for OptimizerSink { - fn next_record(&mut self) -> Option { - let record = self.inner.next_record()?; - let optimized = self.optimizer.optimize(&record.content)?; - - // Track metrics - self.metrics.record(OptimizationMetrics { - input_bytes: record.content.len(), - output_bytes: optimized.compressed.len(), - compressor: optimized.compressor_used, - .. - }); - - // Emit optimized chunk - Some(Record { - content: optimized.compressed, - ..record - }) - } -} - -pub fn optimize_source( - source: Box, - project: &str, -) -> Result -``` - -Tests (3): -- `test_optimizer_sink_preserves_structure` -- `test_optimizer_sink_reduces_bytes` -- `test_optimizer_sink_handles_errors` - -### 2. Rebuild Integration (30 LOC) - -Modify: `crates/mem-store/src/rebuild.rs` - -```rust -let source = DocCorpusSource::new(vault_path)?; -let optimized = optimize_source(Box::new(source), &project)?; // โ† NEW - -for record in optimized { - let embedding = embed(&record.content)?; // clean text - insert_pgvector(embedding, &record)?; - insert_opensearch(&record)?; -} -``` - -Tests (4): -- `test_rebuild_with_optimizer_enabled` -- `test_rebuild_with_optimizer_disabled` -- `test_rebuild_compression_ratio` -- `test_rebuild_pgvector_quality_improves` - -### 3. Source Integration Tests (150 LOC) - -New module: `tests/it_ingest_optimizer.rs` - -Test each ingest source with optimizer: -- `test_pi_session_source_optimized` (Claude transcripts) -- `test_doc_corpus_source_optimized` (markdown files) -- `test_claude_transcript_source_optimized` (agent logs) -- `test_optimizer_preserves_breadcrumb` (M3.6.1 paths) -- `test_optimizer_respects_level` (L0/L1/L2) -- `test_optimizer_disabled_via_env` (MEM_CONTEXT_OPTIMIZER=off) - -Tests (6): -- Per-source integration tests - -### 4. Metrics Collection (NEW) - -Modified: `crates/mem-ingest/src/lib.rs` - -Export MetricsCollector from OptimizerSink: -```rust -pub struct OptimizerMetrics { - pub input_bytes: usize, - pub output_bytes: usize, - pub compression_ratio: f32, - pub compressor_used: String, - pub timestamp: i64, - pub project: String, -} -``` - -No new tests (M3.8.3 handles metrics comprehensively) - -## Acceptance - -โœ… All 13 new tests passing -โœ… OptimizerSink integrated with rebuild.rs -โœ… All ingest sources work with optimizer -โœ… Metrics collected (no performance regression <1ms per chunk) -โœ… Backward compatible (optimizer disableable via env) -โœ… Compression ratios match targets (log 85-95%, json 70-90%, etc.)