docs: CRITICAL CORRECTION — M3.8 architecture (ingest, not query)
ISSUE IDENTIFIED: M3.8 was misplaced in query path (PromptBuilder), but should be in ingest path - Current: Compress before LLM (query-time, only helps LLM input) - Correct: Optimize before embed + index (ingest-time, improves search quality) BENEFITS OF INGEST-TIME OPTIMIZATION: ✅ Better embeddings (pgvector gets clean text → higher semantic quality) ✅ Better ranking (OpenSearch gets signal-rich text → better BM25 scores) ✅ One-time processing at ingest, not per-query overhead ✅ All queries benefit from cleaner search results ✅ LLM receives already-optimized chunks NEW PLAN: - M3.8.1: 🟡 Core modules PARTIAL (1100 LOC, 62 tests done, needs ingest wiring) - M3.8.2: ⬜ Ingest integration (OptimizerSink wrapper, 13 tests) - M3.8.3: ⬜ Metrics & monitoring (20 tests, tracing + prometheus) - M3.8.4: ⬜ Query cleanup (remove PromptBuilder optimizer call) - M3.8.5: ⬜ Benchmarks (compression ratios + search quality metrics) - M3.8.6: ⬜ Gate (ingest pipeline quality + search improvement) ARCHITECTURE CORRECTED: Raw content → M3.8 optimize → embed + index → search improves → LLM benefits FILES UPDATED: - tasks/M3.8-CORRECTED-architecture.md (NEW, comprehensive re-plan) - tasks/M3.8.1-context-optimizer.md (REWRITTEN, marked PARTIAL) - tasks/M3.8.2-cache-aligner-headers.md (REWRITTEN, now OptimizerSink) NEXT IMMEDIATE STEP: Implement M3.8.2 (OptimizerSink) to wire compressors into rebuild.rs ingest pipeline
This commit is contained in:
Story Crater Bot
@@ -1,81 +1,126 @@
|
||||
# M3.8.2 — CacheAligner: HTTP Headers + Metrics
|
||||
# M3.8.2 — Ingest Pipeline Integration (OptimizerSink)
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3.8 — Context optimization |
|
||||
| Size | M — 1–2 days |
|
||||
| Status | ✅ COMPLETE |
|
||||
| Depends | M3.8.1 (CacheAligner complete) |
|
||||
| Status | ⬜ Not started |
|
||||
| Depends | M3.8.1 (core modules) |
|
||||
| Blocks | M3.8.3 |
|
||||
|
||||
## Goal
|
||||
|
||||
Integrate CacheAligner output into HTTP response headers and observability metrics
|
||||
so that:
|
||||
1. LLM provider can use cache hints
|
||||
2. Monitoring can track cache effectiveness
|
||||
3. Debugging can identify cache misses
|
||||
Wire M3.8 compressors into the ingest pipeline so that chunks are optimized
|
||||
BEFORE embedding + indexing, resulting in:
|
||||
- Better embeddings (clean text)
|
||||
- Better search ranking (signal-rich documents)
|
||||
- Cleaner results for all queries
|
||||
|
||||
## Deliverables
|
||||
|
||||
### 1. PromptBuilder::cache_metrics()
|
||||
### 1. OptimizerSink Wrapper (100 LOC)
|
||||
|
||||
New method returning cache metadata:
|
||||
New module: `crates/mem-ingest/src/optimizer_sink.rs`
|
||||
|
||||
```rust
|
||||
pub struct CacheMetrics {
|
||||
pub stable_prefix_bytes: usize,
|
||||
pub dynamic_tail_bytes: usize,
|
||||
pub drift_metric: f32, // 0.0-1.0 ratio
|
||||
pub cache_eligible: bool, // true if drift < 0.3
|
||||
pub struct OptimizerSink {
|
||||
inner: Box<dyn RecordSource>,
|
||||
optimizer: ContextOptimizer,
|
||||
config: OptimizerConfig,
|
||||
metrics: MetricsCollector,
|
||||
}
|
||||
|
||||
impl PromptBuilder {
|
||||
pub fn cache_metrics(query: &Query, chunk: &Chunk) -> Result<CacheMetrics>
|
||||
impl RecordSource for OptimizerSink {
|
||||
fn next_record(&mut self) -> Option<Record> {
|
||||
let record = self.inner.next_record()?;
|
||||
let optimized = self.optimizer.optimize(&record.content)?;
|
||||
|
||||
// Track metrics
|
||||
self.metrics.record(OptimizationMetrics {
|
||||
input_bytes: record.content.len(),
|
||||
output_bytes: optimized.compressed.len(),
|
||||
compressor: optimized.compressor_used,
|
||||
..
|
||||
});
|
||||
|
||||
// Emit optimized chunk
|
||||
Some(Record {
|
||||
content: optimized.compressed,
|
||||
..record
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
pub fn optimize_source(
|
||||
source: Box<dyn RecordSource>,
|
||||
project: &str,
|
||||
) -> Result<OptimizerSink>
|
||||
```
|
||||
|
||||
Tests (3):
|
||||
- `test_cache_metrics_stable_query`
|
||||
- `test_cache_metrics_high_drift`
|
||||
- `test_cache_metrics_zero_drift`
|
||||
- `test_optimizer_sink_preserves_structure`
|
||||
- `test_optimizer_sink_reduces_bytes`
|
||||
- `test_optimizer_sink_handles_errors`
|
||||
|
||||
### 2. HTTP Response Headers
|
||||
### 2. Rebuild Integration (30 LOC)
|
||||
|
||||
Add to PromptBuilder output:
|
||||
- `X-Cache-Stable-Bytes`: size of cacheable prefix
|
||||
- `X-Cache-Drift`: 0.0-1.0 ratio
|
||||
- `X-Cache-Eligible`: "true"/"false"
|
||||
- `X-Compression-Ratio`: original vs. compressed
|
||||
Modify: `crates/mem-store/src/rebuild.rs`
|
||||
|
||||
Tests (4):
|
||||
- `test_headers_present_in_response`
|
||||
- `test_headers_accurate_values`
|
||||
- `test_headers_skipped_when_disabled`
|
||||
- `test_headers_format_valid`
|
||||
|
||||
### 3. Observability Hooks
|
||||
|
||||
Integrate with logging:
|
||||
```rust
|
||||
pub fn log_cache_metrics(metrics: &CacheMetrics) {
|
||||
tracing::info!(
|
||||
stable_bytes = metrics.stable_prefix_bytes,
|
||||
drift = metrics.drift_metric,
|
||||
eligible = metrics.cache_eligible,
|
||||
"cache_alignment"
|
||||
);
|
||||
let source = DocCorpusSource::new(vault_path)?;
|
||||
let optimized = optimize_source(Box::new(source), &project)?; // ← NEW
|
||||
|
||||
for record in optimized {
|
||||
let embedding = embed(&record.content)?; // clean text
|
||||
insert_pgvector(embedding, &record)?;
|
||||
insert_opensearch(&record)?;
|
||||
}
|
||||
```
|
||||
|
||||
Tests (2):
|
||||
- `test_metrics_logged_on_alignment`
|
||||
- `test_drift_high_triggers_warning`
|
||||
Tests (4):
|
||||
- `test_rebuild_with_optimizer_enabled`
|
||||
- `test_rebuild_with_optimizer_disabled`
|
||||
- `test_rebuild_compression_ratio`
|
||||
- `test_rebuild_pgvector_quality_improves`
|
||||
|
||||
### 3. Source Integration Tests (150 LOC)
|
||||
|
||||
New module: `tests/it_ingest_optimizer.rs`
|
||||
|
||||
Test each ingest source with optimizer:
|
||||
- `test_pi_session_source_optimized` (Claude transcripts)
|
||||
- `test_doc_corpus_source_optimized` (markdown files)
|
||||
- `test_claude_transcript_source_optimized` (agent logs)
|
||||
- `test_optimizer_preserves_breadcrumb` (M3.6.1 paths)
|
||||
- `test_optimizer_respects_level` (L0/L1/L2)
|
||||
- `test_optimizer_disabled_via_env` (MEM_CONTEXT_OPTIMIZER=off)
|
||||
|
||||
Tests (6):
|
||||
- Per-source integration tests
|
||||
|
||||
### 4. Metrics Collection (NEW)
|
||||
|
||||
Modified: `crates/mem-ingest/src/lib.rs`
|
||||
|
||||
Export MetricsCollector from OptimizerSink:
|
||||
```rust
|
||||
pub struct OptimizerMetrics {
|
||||
pub input_bytes: usize,
|
||||
pub output_bytes: usize,
|
||||
pub compression_ratio: f32,
|
||||
pub compressor_used: String,
|
||||
pub timestamp: i64,
|
||||
pub project: String,
|
||||
}
|
||||
```
|
||||
|
||||
No new tests (M3.8.3 handles metrics comprehensively)
|
||||
|
||||
## Acceptance
|
||||
|
||||
- All 9 new tests passing
|
||||
- Existing 114 mem-core tests still pass
|
||||
- Cache metrics accurately reflect alignment
|
||||
- HTTP headers present and valid
|
||||
- Zero performance overhead (< 1ms additional)
|
||||
✅ All 13 new tests passing
|
||||
✅ OptimizerSink integrated with rebuild.rs
|
||||
✅ All ingest sources work with optimizer
|
||||
✅ Metrics collected (no performance regression <1ms per chunk)
|
||||
✅ Backward compatible (optimizer disableable via env)
|
||||
✅ Compression ratios match targets (log 85-95%, json 70-90%, etc.)
|
||||
|
||||
Reference in New Issue
Block a user