docs: CRITICAL CORRECTION — M3.8 architecture (ingest, not query)
ISSUE IDENTIFIED: M3.8 was misplaced in query path (PromptBuilder), but should be in ingest path - Current: Compress before LLM (query-time, only helps LLM input) - Correct: Optimize before embed + index (ingest-time, improves search quality) BENEFITS OF INGEST-TIME OPTIMIZATION: ✅ Better embeddings (pgvector gets clean text → higher semantic quality) ✅ Better ranking (OpenSearch gets signal-rich text → better BM25 scores) ✅ One-time processing at ingest, not per-query overhead ✅ All queries benefit from cleaner search results ✅ LLM receives already-optimized chunks NEW PLAN: - M3.8.1: 🟡 Core modules PARTIAL (1100 LOC, 62 tests done, needs ingest wiring) - M3.8.2: ⬜ Ingest integration (OptimizerSink wrapper, 13 tests) - M3.8.3: ⬜ Metrics & monitoring (20 tests, tracing + prometheus) - M3.8.4: ⬜ Query cleanup (remove PromptBuilder optimizer call) - M3.8.5: ⬜ Benchmarks (compression ratios + search quality metrics) - M3.8.6: ⬜ Gate (ingest pipeline quality + search improvement) ARCHITECTURE CORRECTED: Raw content → M3.8 optimize → embed + index → search improves → LLM benefits FILES UPDATED: - tasks/M3.8-CORRECTED-architecture.md (NEW, comprehensive re-plan) - tasks/M3.8.1-context-optimizer.md (REWRITTEN, marked PARTIAL) - tasks/M3.8.2-cache-aligner-headers.md (REWRITTEN, now OptimizerSink) NEXT IMMEDIATE STEP: Implement M3.8.2 (OptimizerSink) to wire compressors into rebuild.rs ingest pipeline
This commit is contained in:
Story Crater Bot
+102
-131
@@ -1,168 +1,139 @@
|
||||
# M3.8.1 — Context Optimizer: pre-LLM compression pipeline
|
||||
# M3.8.1 — Context Optimizer Core Modules
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3.8 — Context optimization |
|
||||
| Size | L — 3–5 days |
|
||||
| Status | ✅ COMPLETE (all 4 phases) |
|
||||
| Flags | — |
|
||||
| Status | 🟡 PARTIAL (modules done, needs ingest wiring) |
|
||||
| Spec | `docs/CONTEXT_OPTIMIZER.md` |
|
||||
| Blocks | M3.8.2 |
|
||||
| Blocks | M3.8.2 (ingest integration) |
|
||||
| Depends | M3.7.7 (lesson.rs patterns), M3.7.8 (stop words) |
|
||||
|
||||
## Summary
|
||||
## Status: PARTIAL ⚠️
|
||||
|
||||
✅ **M3.8.1 COMPLETE** — 4 phases, 62 unit tests, full PromptBuilder integration
|
||||
✅ **Core Compressor Modules Complete**: 1,100 LOC, 62 tests
|
||||
- ContentRouter (Magika ML detection)
|
||||
- LogCompressor, JsonCrusher, DiffCompressor, TextCompressor
|
||||
- CacheAligner (drift detection)
|
||||
- CcrStore (reversible compression)
|
||||
- ContextOptimizer orchestrator
|
||||
|
||||
**Total Implementation**: ~1,100 LOC across 8 modules
|
||||
❌ **Integration in Wrong Place**:
|
||||
- Currently: PromptBuilder.build_cache_aligned() (query path)
|
||||
- Should be: rebuild.rs ingest pipeline (ingest path)
|
||||
- Result: Improves only LLM input, not search quality
|
||||
|
||||
| Module | LOC | Purpose |
|
||||
|--------|-----|---------|
|
||||
| ContentRouter | 150 | Magika ML + regex content detection |
|
||||
| LogCompressor | 260 | Error line + stack trace preservation |
|
||||
| JsonCrusher | 300 | Field variance + boundary-aware compression |
|
||||
| DiffCompressor | 180 | Change-line extraction, context dropping |
|
||||
| TextCompressor | 320 | Token importance scoring with stop words |
|
||||
| CacheAligner | 180 | Dynamic pattern detection + prefix stabilization |
|
||||
| CcrStore | 170 | LRU cache with SHA256 hashing + TTL |
|
||||
| ContextOptimizer | 150 | Orchestrator + env config |
|
||||
## What Was Done Right
|
||||
|
||||
**Test Coverage**: 62 unit tests
|
||||
- Phase 1: 17 tests (router, log)
|
||||
- Phase 2: 15 tests (json, diff)
|
||||
- Phase 3: 18 tests (cache align, CCR)
|
||||
- Phase 4: 12 tests (text, config)
|
||||
✅ **Content Detection** (Magika ML + regex)
|
||||
- <1ms classification
|
||||
- Detects JSON, code, logs, diffs, config, text
|
||||
- Thread-safe, ONNX local
|
||||
|
||||
**Commits**:
|
||||
1. `bf13e3a` — Phase 1 complete (17 tests)
|
||||
2. `a903a3f` — Phase 2 (15 tests)
|
||||
3. `edcc231` — Phase 3 (18 tests)
|
||||
4. `8d8addc` — Phase 4a (12 tests) + PromptBuilder integration
|
||||
✅ **5 Compressor Implementations**
|
||||
- LogCompressor: 85-95% ratio (keep errors + stack traces)
|
||||
- JsonCrusher: 70-90% ratio (field variance)
|
||||
- DiffCompressor: 60-80% ratio (change lines only)
|
||||
- TextCompressor: 30-50% ratio (token importance)
|
||||
- ConfigCompressor: passthrough (already compact)
|
||||
|
||||
## Architecture
|
||||
✅ **Cache Alignment**
|
||||
- Detects dynamic patterns (timestamps, UUIDs, session IDs)
|
||||
- Drift metric (0.0-1.0)
|
||||
- Separates stable prefix from dynamic tail
|
||||
|
||||
### Layer 1: Content Detection (Magika ML)
|
||||
- Google Magika ONNX model (<1ms classification)
|
||||
- Detects: JSON, code (Python/Rust/Go/JS/TS), logs, diffs, config, text
|
||||
- Regex fallback when confidence < 0.7
|
||||
- Thread-safe Mutex-wrapped Session
|
||||
✅ **Reversible Compression** (CCR Store)
|
||||
- LRU cache with SHA256
|
||||
- TTL-based expiry
|
||||
- Model can retrieve originals via hint injection
|
||||
|
||||
### Layer 2: Compression (Per-Type)
|
||||
- **LogCompressor** (85-95% ratio): error lines + stack traces only
|
||||
- **JsonCrusher** (70-90% ratio): statistical field analysis, boundary items
|
||||
- **DiffCompressor** (60-80% ratio): change lines only, drop context
|
||||
- **TextCompressor** (30-50% ratio): token importance, drop stop words
|
||||
- **ConfigCompressor** (passthrough): YAML/TOML already compact
|
||||
## What Needs Fixing
|
||||
|
||||
### Layer 3: Cache Alignment
|
||||
- Detects dynamic patterns: timestamps, UUIDs, session IDs, temp paths, hashes
|
||||
- Moves to tail, preserves stable prefix for LLM KV cache hits
|
||||
- Drift metrics (0.0-1.0 ratio) for monitoring
|
||||
### Root Issue: Architecture Misunderstanding
|
||||
|
||||
### Layer 4: Reversible Compression (CCR)
|
||||
- LRU cache with IndexMap (insertion-order preserving)
|
||||
- SHA256 hashing for content identification
|
||||
- TTL-based expiry (default 1hr)
|
||||
- Thread-safe Mutex wrapper
|
||||
- Injection hint: `<!-- CCR:hash -->` for model retrieval
|
||||
|
||||
## Integration
|
||||
|
||||
**PromptBuilder::build_cache_aligned()**:
|
||||
```rust
|
||||
// After rendering, before prompt assembly:
|
||||
let optimizer = ContextOptimizer::from_env()?;
|
||||
let optimized = optimizer.optimize(&chunk_text)?;
|
||||
let chunk_text = optimized.compressed; // Use optimized version
|
||||
**Documented** (❌ Wrong):
|
||||
```
|
||||
Ingest → pgvector + OpenSearch (full noise)
|
||||
↓
|
||||
Query → M3.8 compression → LLM
|
||||
```
|
||||
|
||||
**Environment Configuration**:
|
||||
```bash
|
||||
# Enable/disable optimizer
|
||||
MEM_CONTEXT_OPTIMIZER=on
|
||||
|
||||
# Per-compressor controls (defaults: all on except CODE)
|
||||
MEM_COMPRESS_JSON=on
|
||||
MEM_COMPRESS_LOGS=on
|
||||
MEM_COMPRESS_CODE=off # opt-in
|
||||
MEM_COMPRESS_DIFF=on
|
||||
MEM_COMPRESS_TEXT=on
|
||||
|
||||
# Detection & caching
|
||||
MEM_MAGIKA_ENABLED=on
|
||||
MEM_MAGIKA_THRESHOLD=0.7
|
||||
MEM_CCR_ENABLED=on
|
||||
**Should Be** (✅ Correct):
|
||||
```
|
||||
Ingest → M3.8 optimization → pgvector + OpenSearch (clean)
|
||||
↓
|
||||
Query → retrieve clean results → LLM
|
||||
```
|
||||
|
||||
## Test Results
|
||||
**Why the correct way is better:**
|
||||
1. Cleaner text → better embeddings (pgvector)
|
||||
2. Signal-rich text → better BM25 ranking (OpenSearch)
|
||||
3. One-time processing at ingest, not per-query
|
||||
4. All users benefit from cleaner search results
|
||||
5. LLM already gets optimized chunks
|
||||
|
||||
✅ **62 unit tests all passing**
|
||||
- 17 phase 1 (router, log)
|
||||
- 15 phase 2 (json, diff)
|
||||
- 18 phase 3 (cache align, CCR)
|
||||
- 12 phase 4 (text, config)
|
||||
### Next Steps
|
||||
|
||||
✅ **114 total mem-core tests** (all passing)
|
||||
- 62 optimizer tests
|
||||
- 11 prompt tests (including PromptBuilder integration)
|
||||
- 10 query tests
|
||||
- 10 symptom projection tests
|
||||
- Plus integration scenarios
|
||||
**M3.8.2**: Ingest Pipeline Integration (1 day)
|
||||
- Create OptimizerSink wrapper around ingest sources
|
||||
- Wire into rebuild.rs
|
||||
- Test with all source types
|
||||
- Collect metrics
|
||||
|
||||
## Quality & Safety
|
||||
**M3.8.3**: Metrics & Monitoring (1 day)
|
||||
- Track compression ratio per chunk
|
||||
- Aggregate per project/source/type
|
||||
- Emit to tracing/Prometheus
|
||||
- Dashboard visualization
|
||||
|
||||
✅ **Graceful Degradation**:
|
||||
- If Magika fails to load, falls back to regex
|
||||
- If optimizer unavailable, passes through unmodified
|
||||
- Invalid content type → passthrough
|
||||
**M3.8.4**: Query Path Cleanup (0.5 days)
|
||||
- Remove PromptBuilder.build_cache_aligned() optimizer call
|
||||
- Keep cache_metrics() for observability (drift tracking)
|
||||
- Simplify PromptBuilder
|
||||
|
||||
✅ **Thread Safety**:
|
||||
- Mutex-wrapped Magika Session
|
||||
- once_cell Lazy statics for patterns
|
||||
- IndexMap for LRU cache
|
||||
## Test Summary
|
||||
|
||||
✅ **Zero Breaking Changes**:
|
||||
- All existing prompt tests pass
|
||||
- Backward compatible env var defaults
|
||||
- Optional optimization (can disable globally)
|
||||
✅ **62 Unit Tests** (all passing)
|
||||
- Phase 1: 17 (router, log)
|
||||
- Phase 2: 15 (json, diff)
|
||||
- Phase 3: 18 (cache align, CCR)
|
||||
- Phase 4: 12 (text, config)
|
||||
|
||||
✅ **Compression Targets Met**:
|
||||
- Logs: 85-95% compression
|
||||
- JSON: 70-90% compression
|
||||
- Diffs: 60-80% compression
|
||||
- Text: 30-50% compression
|
||||
- All < 10ms per chunk
|
||||
⏳ **20 New Tests Pending** (M3.8.2-3)
|
||||
- Ingest source optimization
|
||||
- Metrics collection
|
||||
- End-to-end pipeline
|
||||
|
||||
## Key Files
|
||||
## Files
|
||||
|
||||
**Source**:
|
||||
- `crates/mem-core/src/optimizer/mod.rs` (150 LOC)
|
||||
- `crates/mem-core/src/optimizer/router.rs` (213 LOC)
|
||||
- `crates/mem-core/src/optimizer/log.rs` (220 LOC)
|
||||
- `crates/mem-core/src/optimizer/json.rs` (325 LOC)
|
||||
- `crates/mem-core/src/optimizer/diff.rs` (270 LOC)
|
||||
- `crates/mem-core/src/optimizer/text.rs` (365 LOC)
|
||||
- `crates/mem-core/src/optimizer/cache_align.rs` (210 LOC)
|
||||
- `crates/mem-core/src/optimizer/ccr.rs` (215 LOC)
|
||||
**Implemented** (1,100 LOC):
|
||||
- `crates/mem-core/src/optimizer/mod.rs`
|
||||
- `crates/mem-core/src/optimizer/router.rs`
|
||||
- `crates/mem-core/src/optimizer/log.rs`
|
||||
- `crates/mem-core/src/optimizer/json.rs`
|
||||
- `crates/mem-core/src/optimizer/diff.rs`
|
||||
- `crates/mem-core/src/optimizer/text.rs`
|
||||
- `crates/mem-core/src/optimizer/cache_align.rs`
|
||||
- `crates/mem-core/src/optimizer/ccr.rs`
|
||||
|
||||
**Integration**:
|
||||
- `crates/mem-core/src/prompt.rs` (updated, +12 lines)
|
||||
- `crates/mem-core/src/lib.rs` (updated, +2 exports)
|
||||
- `crates/mem-core/Cargo.toml` (added magika, ort, regex, once_cell, indexmap, lazy_static)
|
||||
**Pending** (180 LOC):
|
||||
- `crates/mem-ingest/src/optimizer_sink.rs` (M3.8.2)
|
||||
- `crates/mem-core/src/optimizer/metrics.rs` (M3.8.3)
|
||||
|
||||
**Tests**:
|
||||
- `crates/mem-core/src/optimizer/` (62 unit tests in modules)
|
||||
- All integration tests in PromptBuilder
|
||||
## Commits
|
||||
|
||||
## Status
|
||||
1. `bf13e3a` — Phase 1: ContentRouter + LogCompressor
|
||||
2. `a903a3f` — Phase 2: JsonCrusher + DiffCompressor
|
||||
3. `edcc231` — Phase 3: CacheAligner + CcrStore
|
||||
4. `8d8addc` — Phase 4: TextCompressor + env config
|
||||
|
||||
✅ **Complete and Production-Ready**
|
||||
## Lessons Learned
|
||||
|
||||
- All phases implemented and tested
|
||||
- Full PromptBuilder integration
|
||||
- Environment-configurable
|
||||
- Comprehensive error handling
|
||||
- Performance targets met (<10ms per chunk)
|
||||
- Zero breaking changes
|
||||
1. **Ingest-time optimization > query-time**: Better for entire pipeline
|
||||
2. **Compression ratios vary widely**: Log 85-95% vs text 30-50%
|
||||
3. **Reversibility matters**: Model needs originals for detailed analysis
|
||||
4. **Metrics > assumption**: Need to measure actual improvement in search quality
|
||||
|
||||
**Ready for**: M3.8.2 (CacheAligner header integration), M3.8.3 (benchmarking), M3.8.4 (gate)
|
||||
## Remediation
|
||||
|
||||
See **`tasks/M3.8-CORRECTED-architecture.md`** for complete re-architecture plan.
|
||||
|
||||
Reference in New Issue
Block a user