rock/poimen-memory

Fork 0

Files

T

Story Crater Bot f1917e1260

Build and Push / Test (push) Failing after 1m53s

Details

Build and Push / Build and push image (push) Skipped

Details

docs: CRITICAL CORRECTION — M3.8 architecture (ingest, not query)

ISSUE IDENTIFIED:
M3.8 was misplaced in query path (PromptBuilder), but should be in ingest path
- Current: Compress before LLM (query-time, only helps LLM input)
- Correct: Optimize before embed + index (ingest-time, improves search quality)

BENEFITS OF INGEST-TIME OPTIMIZATION:
✅ Better embeddings (pgvector gets clean text → higher semantic quality)
✅ Better ranking (OpenSearch gets signal-rich text → better BM25 scores)
✅ One-time processing at ingest, not per-query overhead
✅ All queries benefit from cleaner search results
✅ LLM receives already-optimized chunks

NEW PLAN:
- M3.8.1: 🟡 Core modules PARTIAL (1100 LOC, 62 tests done, needs ingest wiring)
- M3.8.2: ⬜ Ingest integration (OptimizerSink wrapper, 13 tests)
- M3.8.3: ⬜ Metrics & monitoring (20 tests, tracing + prometheus)
- M3.8.4: ⬜ Query cleanup (remove PromptBuilder optimizer call)
- M3.8.5: ⬜ Benchmarks (compression ratios + search quality metrics)
- M3.8.6: ⬜ Gate (ingest pipeline quality + search improvement)

ARCHITECTURE CORRECTED:
Raw content → M3.8 optimize → embed + index → search improves → LLM benefits

FILES UPDATED:
- tasks/M3.8-CORRECTED-architecture.md (NEW, comprehensive re-plan)
- tasks/M3.8.1-context-optimizer.md (REWRITTEN, marked PARTIAL)
- tasks/M3.8.2-cache-aligner-headers.md (REWRITTEN, now OptimizerSink)

NEXT IMMEDIATE STEP:
Implement M3.8.2 (OptimizerSink) to wire compressors into rebuild.rs ingest pipeline

2026-08-28 10:25:31 -07:00

3.3 KiB

Raw Blame History

M3.8.2 — Ingest Pipeline Integration (OptimizerSink)

Field	Value
Phase	M3.8 — Context optimization
Size	M — 1–2 days
Status	⬜ Not started
Depends	M3.8.1 (core modules)
Blocks	M3.8.3

Goal

Wire M3.8 compressors into the ingest pipeline so that chunks are optimized BEFORE embedding + indexing, resulting in:

Better embeddings (clean text)
Better search ranking (signal-rich documents)
Cleaner results for all queries

Deliverables

1. OptimizerSink Wrapper (100 LOC)

New module: crates/mem-ingest/src/optimizer_sink.rs

pub struct OptimizerSink {
    inner: Box<dyn RecordSource>,
    optimizer: ContextOptimizer,
    config: OptimizerConfig,
    metrics: MetricsCollector,
}

impl RecordSource for OptimizerSink {
    fn next_record(&mut self) -> Option<Record> {
        let record = self.inner.next_record()?;
        let optimized = self.optimizer.optimize(&record.content)?;
        
        // Track metrics
        self.metrics.record(OptimizationMetrics {
            input_bytes: record.content.len(),
            output_bytes: optimized.compressed.len(),
            compressor: optimized.compressor_used,
            ..
        });
        
        // Emit optimized chunk
        Some(Record {
            content: optimized.compressed,
            ..record
        })
    }
}

pub fn optimize_source(
    source: Box<dyn RecordSource>,
    project: &str,
) -> Result<OptimizerSink>

Tests (3):

test_optimizer_sink_preserves_structure
test_optimizer_sink_reduces_bytes
test_optimizer_sink_handles_errors

2. Rebuild Integration (30 LOC)

Modify: crates/mem-store/src/rebuild.rs

let source = DocCorpusSource::new(vault_path)?;
let optimized = optimize_source(Box::new(source), &project)?;  // ← NEW

for record in optimized {
    let embedding = embed(&record.content)?;  // clean text
    insert_pgvector(embedding, &record)?;
    insert_opensearch(&record)?;
}

Tests (4):

test_rebuild_with_optimizer_enabled
test_rebuild_with_optimizer_disabled
test_rebuild_compression_ratio
test_rebuild_pgvector_quality_improves

3. Source Integration Tests (150 LOC)

New module: tests/it_ingest_optimizer.rs

Test each ingest source with optimizer:

test_pi_session_source_optimized (Claude transcripts)
test_doc_corpus_source_optimized (markdown files)
test_claude_transcript_source_optimized (agent logs)
test_optimizer_preserves_breadcrumb (M3.6.1 paths)
test_optimizer_respects_level (L0/L1/L2)
test_optimizer_disabled_via_env (MEM_CONTEXT_OPTIMIZER=off)

Tests (6):

Per-source integration tests

4. Metrics Collection (NEW)

Modified: crates/mem-ingest/src/lib.rs

Export MetricsCollector from OptimizerSink:

pub struct OptimizerMetrics {
    pub input_bytes: usize,
    pub output_bytes: usize,
    pub compression_ratio: f32,
    pub compressor_used: String,
    pub timestamp: i64,
    pub project: String,
}

No new tests (M3.8.3 handles metrics comprehensively)

Acceptance

✅ All 13 new tests passing ✅ OptimizerSink integrated with rebuild.rs ✅ All ingest sources work with optimizer ✅ Metrics collected (no performance regression <1ms per chunk) ✅ Backward compatible (optimizer disableable via env) ✅ Compression ratios match targets (log 85-95%, json 70-90%, etc.)

3.3 KiB Raw Blame History Unescape Escape