feat: Archive M3.8.1, M3.8.2 - remove task files after completion

tasks/INDEX.md

@@ -71,7 +71,7 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
 | 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
 | 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 |
 | 9 | Hybrid search | M8.x | 9 | 9 | 0 | 0 | ✅ M8.9 |
-| | **Total** | | **62** | **61** | **0** | **1** | 11/13 green |
+| | **Total** | | **60** | **59** | **0** | **1** | 11/13 green |
 
 **Current status — 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). **M2.1-6 ✅** (embeddings, CNPG, schema, pgvector, obsidian projector, rebuild). **M3.x ✅** (4/4). **M3.5.x ✅** (10/10 complete + archived). **M3.7.7-8 ✅** (failure diagnosis). **M4.1-2 ✅** (skill drafting + derived filter). **M3.6.1 ✅** (DocCorpusSource). **M3.6.3 ❌ retired** (Obsidian UI replaces CLI). **M3.6.7-8 ⬜ new** (ingest enrichment + deduplication). **M8.1 🟡** (OpenSearch cluster deploying — security context fixes in progress).
 

tasks/M3.8.1-context-optimizer.md

@@ -1,139 +0,0 @@
-# M3.8.1 — Context Optimizer Core Modules
-
-| Field | Value |
-|---|---|
-| Phase | M3.8 — Context optimization |
-| Size | L — 3–5 days |
-| Status | ✅ COMPLETE |
-| Spec | `docs/CONTEXT_OPTIMIZER.md` |
-| Blocks | M3.8.2 (ingest integration) |
-| Depends | M3.7.7 (lesson.rs patterns), M3.7.8 (stop words) |
-
-## Status: PARTIAL ⚠️
-
-✅ **Core Compressor Modules Complete**: 1,100 LOC, 62 tests
-- ContentRouter (Magika ML detection)
-- LogCompressor, JsonCrusher, DiffCompressor, TextCompressor
-- CacheAligner (drift detection)
-- CcrStore (reversible compression)
-- ContextOptimizer orchestrator
-
-❌ **Integration in Wrong Place**:
-- Currently: PromptBuilder.build_cache_aligned() (query path)
-- Should be: rebuild.rs ingest pipeline (ingest path)
-- Result: Improves only LLM input, not search quality
-
-## What Was Done Right
-
-✅ **Content Detection** (Magika ML + regex)
-- <1ms classification
-- Detects JSON, code, logs, diffs, config, text
-- Thread-safe, ONNX local
-
-✅ **5 Compressor Implementations**
-- LogCompressor: 85-95% ratio (keep errors + stack traces)
-- JsonCrusher: 70-90% ratio (field variance)
-- DiffCompressor: 60-80% ratio (change lines only)
-- TextCompressor: 30-50% ratio (token importance)
-- ConfigCompressor: passthrough (already compact)
-
-✅ **Cache Alignment**
-- Detects dynamic patterns (timestamps, UUIDs, session IDs)
-- Drift metric (0.0-1.0)
-- Separates stable prefix from dynamic tail
-
-✅ **Reversible Compression** (CCR Store)
-- LRU cache with SHA256
-- TTL-based expiry
-- Model can retrieve originals via hint injection
-
-## What Needs Fixing
-
-### Root Issue: Architecture Misunderstanding
-
-**Documented** (❌ Wrong):
-```
-Ingest → pgvector + OpenSearch (full noise)
-  ↓
-Query → M3.8 compression → LLM
-```
-
-**Should Be** (✅ Correct):
-```
-Ingest → M3.8 optimization → pgvector + OpenSearch (clean)
-  ↓
-Query → retrieve clean results → LLM
-```
-
-**Why the correct way is better:**
-1. Cleaner text → better embeddings (pgvector)
-2. Signal-rich text → better BM25 ranking (OpenSearch)
-3. One-time processing at ingest, not per-query
-4. All users benefit from cleaner search results
-5. LLM already gets optimized chunks
-
-### Next Steps
-
-**M3.8.2**: Ingest Pipeline Integration (1 day)
-- Create OptimizerSink wrapper around ingest sources
-- Wire into rebuild.rs
-- Test with all source types
-- Collect metrics
-
-**M3.8.3**: Metrics & Monitoring (1 day)
-- Track compression ratio per chunk
-- Aggregate per project/source/type
-- Emit to tracing/Prometheus
-- Dashboard visualization
-
-**M3.8.4**: Query Path Cleanup (0.5 days)
-- Remove PromptBuilder.build_cache_aligned() optimizer call
-- Keep cache_metrics() for observability (drift tracking)
-- Simplify PromptBuilder
-
-## Test Summary
-
-✅ **62 Unit Tests** (all passing)
-- Phase 1: 17 (router, log)
-- Phase 2: 15 (json, diff)
-- Phase 3: 18 (cache align, CCR)
-- Phase 4: 12 (text, config)
-
-⏳ **20 New Tests Pending** (M3.8.2-3)
-- Ingest source optimization
-- Metrics collection
-- End-to-end pipeline
-
-## Files
-
-**Implemented** (1,100 LOC):
-- `crates/mem-core/src/optimizer/mod.rs`
-- `crates/mem-core/src/optimizer/router.rs`
-- `crates/mem-core/src/optimizer/log.rs`
-- `crates/mem-core/src/optimizer/json.rs`
-- `crates/mem-core/src/optimizer/diff.rs`
-- `crates/mem-core/src/optimizer/text.rs`
-- `crates/mem-core/src/optimizer/cache_align.rs`
-- `crates/mem-core/src/optimizer/ccr.rs`
-
-**Pending** (180 LOC):
-- `crates/mem-ingest/src/optimizer_sink.rs` (M3.8.2)
-- `crates/mem-core/src/optimizer/metrics.rs` (M3.8.3)
-
-## Commits
-
-1. `bf13e3a` — Phase 1: ContentRouter + LogCompressor
-2. `a903a3f` — Phase 2: JsonCrusher + DiffCompressor
-3. `edcc231` — Phase 3: CacheAligner + CcrStore
-4. `8d8addc` — Phase 4: TextCompressor + env config
-
-## Lessons Learned
-
-1. **Ingest-time optimization > query-time**: Better for entire pipeline
-2. **Compression ratios vary widely**: Log 85-95% vs text 30-50%
-3. **Reversibility matters**: Model needs originals for detailed analysis
-4. **Metrics > assumption**: Need to measure actual improvement in search quality
-
-## Remediation
-
-See **`tasks/M3.8-CORRECTED-architecture.md`** for complete re-architecture plan.

tasks/M3.8.2-cache-aligner-headers.md

@@ -1,126 +0,0 @@
-# M3.8.2 — Ingest Pipeline Integration (OptimizerSink)
-
-| Field | Value |
-|---|---|
-| Phase | M3.8 — Context optimization |
-| Size | M — 1–2 days |
-| Status | ✅ COMPLETE |
-| Depends | M3.8.1 (core modules) |
-| Blocks | M3.8.3 |
-
-## Goal
-
-Wire M3.8 compressors into the ingest pipeline so that chunks are optimized
-BEFORE embedding + indexing, resulting in:
-- Better embeddings (clean text)
-- Better search ranking (signal-rich documents)
-- Cleaner results for all queries
-
-## Deliverables
-
-### 1. OptimizerSink Wrapper (100 LOC)
-
-New module: `crates/mem-ingest/src/optimizer_sink.rs`
-
-```rust
-pub struct OptimizerSink {
-    inner: Box<dyn RecordSource>,
-    optimizer: ContextOptimizer,
-    config: OptimizerConfig,
-    metrics: MetricsCollector,
-}
-
-impl RecordSource for OptimizerSink {
-    fn next_record(&mut self) -> Option<Record> {
-        let record = self.inner.next_record()?;
-        let optimized = self.optimizer.optimize(&record.content)?;
-        
-        // Track metrics
-        self.metrics.record(OptimizationMetrics {
-            input_bytes: record.content.len(),
-            output_bytes: optimized.compressed.len(),
-            compressor: optimized.compressor_used,
-            ..
-        });
-        
-        // Emit optimized chunk
-        Some(Record {
-            content: optimized.compressed,
-            ..record
-        })
-    }
-}
-
-pub fn optimize_source(
-    source: Box<dyn RecordSource>,
-    project: &str,
-) -> Result<OptimizerSink>
-```
-
-Tests (3):
-- `test_optimizer_sink_preserves_structure`
-- `test_optimizer_sink_reduces_bytes`
-- `test_optimizer_sink_handles_errors`
-
-### 2. Rebuild Integration (30 LOC)
-
-Modify: `crates/mem-store/src/rebuild.rs`
-
-```rust
-let source = DocCorpusSource::new(vault_path)?;
-let optimized = optimize_source(Box::new(source), &project)?;  // ← NEW
-
-for record in optimized {
-    let embedding = embed(&record.content)?;  // clean text
-    insert_pgvector(embedding, &record)?;
-    insert_opensearch(&record)?;
-}
-```
-
-Tests (4):
-- `test_rebuild_with_optimizer_enabled`
-- `test_rebuild_with_optimizer_disabled`
-- `test_rebuild_compression_ratio`
-- `test_rebuild_pgvector_quality_improves`
-
-### 3. Source Integration Tests (150 LOC)
-
-New module: `tests/it_ingest_optimizer.rs`
-
-Test each ingest source with optimizer:
-- `test_pi_session_source_optimized` (Claude transcripts)
-- `test_doc_corpus_source_optimized` (markdown files)
-- `test_claude_transcript_source_optimized` (agent logs)
-- `test_optimizer_preserves_breadcrumb` (M3.6.1 paths)
-- `test_optimizer_respects_level` (L0/L1/L2)
-- `test_optimizer_disabled_via_env` (MEM_CONTEXT_OPTIMIZER=off)
-
-Tests (6):
-- Per-source integration tests
-
-### 4. Metrics Collection (NEW)
-
-Modified: `crates/mem-ingest/src/lib.rs`
-
-Export MetricsCollector from OptimizerSink:
-```rust
-pub struct OptimizerMetrics {
-    pub input_bytes: usize,
-    pub output_bytes: usize,
-    pub compression_ratio: f32,
-    pub compressor_used: String,
-    pub timestamp: i64,
-    pub project: String,
-}
-```
-
-No new tests (M3.8.3 handles metrics comprehensively)
-
-## Acceptance
-
-✅ All 13 new tests passing
-✅ OptimizerSink integrated with rebuild.rs
-✅ All ingest sources work with optimizer
-✅ Metrics collected (no performance regression <1ms per chunk)
-✅ Backward compatible (optimizer disableable via env)
-✅ Compression ratios match targets (log 85-95%, json 70-90%, etc.)