Story Crater Bot
f1917e1260
ISSUE IDENTIFIED: M3.8 was misplaced in query path (PromptBuilder), but should be in ingest path - Current: Compress before LLM (query-time, only helps LLM input) - Correct: Optimize before embed + index (ingest-time, improves search quality) BENEFITS OF INGEST-TIME OPTIMIZATION: ✅ Better embeddings (pgvector gets clean text → higher semantic quality) ✅ Better ranking (OpenSearch gets signal-rich text → better BM25 scores) ✅ One-time processing at ingest, not per-query overhead ✅ All queries benefit from cleaner search results ✅ LLM receives already-optimized chunks NEW PLAN: - M3.8.1: 🟡 Core modules PARTIAL (1100 LOC, 62 tests done, needs ingest wiring) - M3.8.2: ⬜ Ingest integration (OptimizerSink wrapper, 13 tests) - M3.8.3: ⬜ Metrics & monitoring (20 tests, tracing + prometheus) - M3.8.4: ⬜ Query cleanup (remove PromptBuilder optimizer call) - M3.8.5: ⬜ Benchmarks (compression ratios + search quality metrics) - M3.8.6: ⬜ Gate (ingest pipeline quality + search improvement) ARCHITECTURE CORRECTED: Raw content → M3.8 optimize → embed + index → search improves → LLM benefits FILES UPDATED: - tasks/M3.8-CORRECTED-architecture.md (NEW, comprehensive re-plan) - tasks/M3.8.1-context-optimizer.md (REWRITTEN, marked PARTIAL) - tasks/M3.8.2-cache-aligner-headers.md (REWRITTEN, now OptimizerSink) NEXT IMMEDIATE STEP: Implement M3.8.2 (OptimizerSink) to wire compressors into rebuild.rs ingest pipeline
140 lines
4.1 KiB
Markdown
140 lines
4.1 KiB
Markdown
# M3.8.1 — Context Optimizer Core Modules
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M3.8 — Context optimization |
|
||
| Size | L — 3–5 days |
|
||
| Status | 🟡 PARTIAL (modules done, needs ingest wiring) |
|
||
| Spec | `docs/CONTEXT_OPTIMIZER.md` |
|
||
| Blocks | M3.8.2 (ingest integration) |
|
||
| Depends | M3.7.7 (lesson.rs patterns), M3.7.8 (stop words) |
|
||
|
||
## Status: PARTIAL ⚠️
|
||
|
||
✅ **Core Compressor Modules Complete**: 1,100 LOC, 62 tests
|
||
- ContentRouter (Magika ML detection)
|
||
- LogCompressor, JsonCrusher, DiffCompressor, TextCompressor
|
||
- CacheAligner (drift detection)
|
||
- CcrStore (reversible compression)
|
||
- ContextOptimizer orchestrator
|
||
|
||
❌ **Integration in Wrong Place**:
|
||
- Currently: PromptBuilder.build_cache_aligned() (query path)
|
||
- Should be: rebuild.rs ingest pipeline (ingest path)
|
||
- Result: Improves only LLM input, not search quality
|
||
|
||
## What Was Done Right
|
||
|
||
✅ **Content Detection** (Magika ML + regex)
|
||
- <1ms classification
|
||
- Detects JSON, code, logs, diffs, config, text
|
||
- Thread-safe, ONNX local
|
||
|
||
✅ **5 Compressor Implementations**
|
||
- LogCompressor: 85-95% ratio (keep errors + stack traces)
|
||
- JsonCrusher: 70-90% ratio (field variance)
|
||
- DiffCompressor: 60-80% ratio (change lines only)
|
||
- TextCompressor: 30-50% ratio (token importance)
|
||
- ConfigCompressor: passthrough (already compact)
|
||
|
||
✅ **Cache Alignment**
|
||
- Detects dynamic patterns (timestamps, UUIDs, session IDs)
|
||
- Drift metric (0.0-1.0)
|
||
- Separates stable prefix from dynamic tail
|
||
|
||
✅ **Reversible Compression** (CCR Store)
|
||
- LRU cache with SHA256
|
||
- TTL-based expiry
|
||
- Model can retrieve originals via hint injection
|
||
|
||
## What Needs Fixing
|
||
|
||
### Root Issue: Architecture Misunderstanding
|
||
|
||
**Documented** (❌ Wrong):
|
||
```
|
||
Ingest → pgvector + OpenSearch (full noise)
|
||
↓
|
||
Query → M3.8 compression → LLM
|
||
```
|
||
|
||
**Should Be** (✅ Correct):
|
||
```
|
||
Ingest → M3.8 optimization → pgvector + OpenSearch (clean)
|
||
↓
|
||
Query → retrieve clean results → LLM
|
||
```
|
||
|
||
**Why the correct way is better:**
|
||
1. Cleaner text → better embeddings (pgvector)
|
||
2. Signal-rich text → better BM25 ranking (OpenSearch)
|
||
3. One-time processing at ingest, not per-query
|
||
4. All users benefit from cleaner search results
|
||
5. LLM already gets optimized chunks
|
||
|
||
### Next Steps
|
||
|
||
**M3.8.2**: Ingest Pipeline Integration (1 day)
|
||
- Create OptimizerSink wrapper around ingest sources
|
||
- Wire into rebuild.rs
|
||
- Test with all source types
|
||
- Collect metrics
|
||
|
||
**M3.8.3**: Metrics & Monitoring (1 day)
|
||
- Track compression ratio per chunk
|
||
- Aggregate per project/source/type
|
||
- Emit to tracing/Prometheus
|
||
- Dashboard visualization
|
||
|
||
**M3.8.4**: Query Path Cleanup (0.5 days)
|
||
- Remove PromptBuilder.build_cache_aligned() optimizer call
|
||
- Keep cache_metrics() for observability (drift tracking)
|
||
- Simplify PromptBuilder
|
||
|
||
## Test Summary
|
||
|
||
✅ **62 Unit Tests** (all passing)
|
||
- Phase 1: 17 (router, log)
|
||
- Phase 2: 15 (json, diff)
|
||
- Phase 3: 18 (cache align, CCR)
|
||
- Phase 4: 12 (text, config)
|
||
|
||
⏳ **20 New Tests Pending** (M3.8.2-3)
|
||
- Ingest source optimization
|
||
- Metrics collection
|
||
- End-to-end pipeline
|
||
|
||
## Files
|
||
|
||
**Implemented** (1,100 LOC):
|
||
- `crates/mem-core/src/optimizer/mod.rs`
|
||
- `crates/mem-core/src/optimizer/router.rs`
|
||
- `crates/mem-core/src/optimizer/log.rs`
|
||
- `crates/mem-core/src/optimizer/json.rs`
|
||
- `crates/mem-core/src/optimizer/diff.rs`
|
||
- `crates/mem-core/src/optimizer/text.rs`
|
||
- `crates/mem-core/src/optimizer/cache_align.rs`
|
||
- `crates/mem-core/src/optimizer/ccr.rs`
|
||
|
||
**Pending** (180 LOC):
|
||
- `crates/mem-ingest/src/optimizer_sink.rs` (M3.8.2)
|
||
- `crates/mem-core/src/optimizer/metrics.rs` (M3.8.3)
|
||
|
||
## Commits
|
||
|
||
1. `bf13e3a` — Phase 1: ContentRouter + LogCompressor
|
||
2. `a903a3f` — Phase 2: JsonCrusher + DiffCompressor
|
||
3. `edcc231` — Phase 3: CacheAligner + CcrStore
|
||
4. `8d8addc` — Phase 4: TextCompressor + env config
|
||
|
||
## Lessons Learned
|
||
|
||
1. **Ingest-time optimization > query-time**: Better for entire pipeline
|
||
2. **Compression ratios vary widely**: Log 85-95% vs text 30-50%
|
||
3. **Reversibility matters**: Model needs originals for detailed analysis
|
||
4. **Metrics > assumption**: Need to measure actual improvement in search quality
|
||
|
||
## Remediation
|
||
|
||
See **`tasks/M3.8-CORRECTED-architecture.md`** for complete re-architecture plan.
|