Story Crater Bot
f1917e1260
ISSUE IDENTIFIED: M3.8 was misplaced in query path (PromptBuilder), but should be in ingest path - Current: Compress before LLM (query-time, only helps LLM input) - Correct: Optimize before embed + index (ingest-time, improves search quality) BENEFITS OF INGEST-TIME OPTIMIZATION: ✅ Better embeddings (pgvector gets clean text → higher semantic quality) ✅ Better ranking (OpenSearch gets signal-rich text → better BM25 scores) ✅ One-time processing at ingest, not per-query overhead ✅ All queries benefit from cleaner search results ✅ LLM receives already-optimized chunks NEW PLAN: - M3.8.1: 🟡 Core modules PARTIAL (1100 LOC, 62 tests done, needs ingest wiring) - M3.8.2: ⬜ Ingest integration (OptimizerSink wrapper, 13 tests) - M3.8.3: ⬜ Metrics & monitoring (20 tests, tracing + prometheus) - M3.8.4: ⬜ Query cleanup (remove PromptBuilder optimizer call) - M3.8.5: ⬜ Benchmarks (compression ratios + search quality metrics) - M3.8.6: ⬜ Gate (ingest pipeline quality + search improvement) ARCHITECTURE CORRECTED: Raw content → M3.8 optimize → embed + index → search improves → LLM benefits FILES UPDATED: - tasks/M3.8-CORRECTED-architecture.md (NEW, comprehensive re-plan) - tasks/M3.8.1-context-optimizer.md (REWRITTEN, marked PARTIAL) - tasks/M3.8.2-cache-aligner-headers.md (REWRITTEN, now OptimizerSink) NEXT IMMEDIATE STEP: Implement M3.8.2 (OptimizerSink) to wire compressors into rebuild.rs ingest pipeline
127 lines
3.3 KiB
Markdown
127 lines
3.3 KiB
Markdown
# M3.8.2 — Ingest Pipeline Integration (OptimizerSink)
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M3.8 — Context optimization |
|
||
| Size | M — 1–2 days |
|
||
| Status | ⬜ Not started |
|
||
| Depends | M3.8.1 (core modules) |
|
||
| Blocks | M3.8.3 |
|
||
|
||
## Goal
|
||
|
||
Wire M3.8 compressors into the ingest pipeline so that chunks are optimized
|
||
BEFORE embedding + indexing, resulting in:
|
||
- Better embeddings (clean text)
|
||
- Better search ranking (signal-rich documents)
|
||
- Cleaner results for all queries
|
||
|
||
## Deliverables
|
||
|
||
### 1. OptimizerSink Wrapper (100 LOC)
|
||
|
||
New module: `crates/mem-ingest/src/optimizer_sink.rs`
|
||
|
||
```rust
|
||
pub struct OptimizerSink {
|
||
inner: Box<dyn RecordSource>,
|
||
optimizer: ContextOptimizer,
|
||
config: OptimizerConfig,
|
||
metrics: MetricsCollector,
|
||
}
|
||
|
||
impl RecordSource for OptimizerSink {
|
||
fn next_record(&mut self) -> Option<Record> {
|
||
let record = self.inner.next_record()?;
|
||
let optimized = self.optimizer.optimize(&record.content)?;
|
||
|
||
// Track metrics
|
||
self.metrics.record(OptimizationMetrics {
|
||
input_bytes: record.content.len(),
|
||
output_bytes: optimized.compressed.len(),
|
||
compressor: optimized.compressor_used,
|
||
..
|
||
});
|
||
|
||
// Emit optimized chunk
|
||
Some(Record {
|
||
content: optimized.compressed,
|
||
..record
|
||
})
|
||
}
|
||
}
|
||
|
||
pub fn optimize_source(
|
||
source: Box<dyn RecordSource>,
|
||
project: &str,
|
||
) -> Result<OptimizerSink>
|
||
```
|
||
|
||
Tests (3):
|
||
- `test_optimizer_sink_preserves_structure`
|
||
- `test_optimizer_sink_reduces_bytes`
|
||
- `test_optimizer_sink_handles_errors`
|
||
|
||
### 2. Rebuild Integration (30 LOC)
|
||
|
||
Modify: `crates/mem-store/src/rebuild.rs`
|
||
|
||
```rust
|
||
let source = DocCorpusSource::new(vault_path)?;
|
||
let optimized = optimize_source(Box::new(source), &project)?; // ← NEW
|
||
|
||
for record in optimized {
|
||
let embedding = embed(&record.content)?; // clean text
|
||
insert_pgvector(embedding, &record)?;
|
||
insert_opensearch(&record)?;
|
||
}
|
||
```
|
||
|
||
Tests (4):
|
||
- `test_rebuild_with_optimizer_enabled`
|
||
- `test_rebuild_with_optimizer_disabled`
|
||
- `test_rebuild_compression_ratio`
|
||
- `test_rebuild_pgvector_quality_improves`
|
||
|
||
### 3. Source Integration Tests (150 LOC)
|
||
|
||
New module: `tests/it_ingest_optimizer.rs`
|
||
|
||
Test each ingest source with optimizer:
|
||
- `test_pi_session_source_optimized` (Claude transcripts)
|
||
- `test_doc_corpus_source_optimized` (markdown files)
|
||
- `test_claude_transcript_source_optimized` (agent logs)
|
||
- `test_optimizer_preserves_breadcrumb` (M3.6.1 paths)
|
||
- `test_optimizer_respects_level` (L0/L1/L2)
|
||
- `test_optimizer_disabled_via_env` (MEM_CONTEXT_OPTIMIZER=off)
|
||
|
||
Tests (6):
|
||
- Per-source integration tests
|
||
|
||
### 4. Metrics Collection (NEW)
|
||
|
||
Modified: `crates/mem-ingest/src/lib.rs`
|
||
|
||
Export MetricsCollector from OptimizerSink:
|
||
```rust
|
||
pub struct OptimizerMetrics {
|
||
pub input_bytes: usize,
|
||
pub output_bytes: usize,
|
||
pub compression_ratio: f32,
|
||
pub compressor_used: String,
|
||
pub timestamp: i64,
|
||
pub project: String,
|
||
}
|
||
```
|
||
|
||
No new tests (M3.8.3 handles metrics comprehensively)
|
||
|
||
## Acceptance
|
||
|
||
✅ All 13 new tests passing
|
||
✅ OptimizerSink integrated with rebuild.rs
|
||
✅ All ingest sources work with optimizer
|
||
✅ Metrics collected (no performance regression <1ms per chunk)
|
||
✅ Backward compatible (optimizer disableable via env)
|
||
✅ Compression ratios match targets (log 85-95%, json 70-90%, etc.)
|