rock/poimen-memory

Fork 0

Files

T

Story Crater Bot f1917e1260

Build and Push / Test (push) Failing after 1m53s

Details

Build and Push / Build and push image (push) Skipped

Details

docs: CRITICAL CORRECTION — M3.8 architecture (ingest, not query)

ISSUE IDENTIFIED:
M3.8 was misplaced in query path (PromptBuilder), but should be in ingest path
- Current: Compress before LLM (query-time, only helps LLM input)
- Correct: Optimize before embed + index (ingest-time, improves search quality)

BENEFITS OF INGEST-TIME OPTIMIZATION:
✅ Better embeddings (pgvector gets clean text → higher semantic quality)
✅ Better ranking (OpenSearch gets signal-rich text → better BM25 scores)
✅ One-time processing at ingest, not per-query overhead
✅ All queries benefit from cleaner search results
✅ LLM receives already-optimized chunks

NEW PLAN:
- M3.8.1: 🟡 Core modules PARTIAL (1100 LOC, 62 tests done, needs ingest wiring)
- M3.8.2: ⬜ Ingest integration (OptimizerSink wrapper, 13 tests)
- M3.8.3: ⬜ Metrics & monitoring (20 tests, tracing + prometheus)
- M3.8.4: ⬜ Query cleanup (remove PromptBuilder optimizer call)
- M3.8.5: ⬜ Benchmarks (compression ratios + search quality metrics)
- M3.8.6: ⬜ Gate (ingest pipeline quality + search improvement)

ARCHITECTURE CORRECTED:
Raw content → M3.8 optimize → embed + index → search improves → LLM benefits

FILES UPDATED:
- tasks/M3.8-CORRECTED-architecture.md (NEW, comprehensive re-plan)
- tasks/M3.8.1-context-optimizer.md (REWRITTEN, marked PARTIAL)
- tasks/M3.8.2-cache-aligner-headers.md (REWRITTEN, now OptimizerSink)

NEXT IMMEDIATE STEP:
Implement M3.8.2 (OptimizerSink) to wire compressors into rebuild.rs ingest pipeline

2026-08-28 10:25:31 -07:00

4.1 KiB

Raw Blame History

M3.8.1 — Context Optimizer Core Modules

Field	Value
Phase	M3.8 — Context optimization
Size	L — 3–5 days
Status	🟡 PARTIAL (modules done, needs ingest wiring)
Spec	`docs/CONTEXT_OPTIMIZER.md`
Blocks	M3.8.2 (ingest integration)
Depends	M3.7.7 (lesson.rs patterns), M3.7.8 (stop words)

Status: PARTIAL ⚠️

✅ Core Compressor Modules Complete: 1,100 LOC, 62 tests

ContentRouter (Magika ML detection)
LogCompressor, JsonCrusher, DiffCompressor, TextCompressor
CacheAligner (drift detection)
CcrStore (reversible compression)
ContextOptimizer orchestrator

❌ Integration in Wrong Place:

Currently: PromptBuilder.build_cache_aligned() (query path)
Should be: rebuild.rs ingest pipeline (ingest path)
Result: Improves only LLM input, not search quality

What Was Done Right

✅ Content Detection (Magika ML + regex)

<1ms classification
Detects JSON, code, logs, diffs, config, text
Thread-safe, ONNX local

✅ 5 Compressor Implementations

LogCompressor: 85-95% ratio (keep errors + stack traces)
JsonCrusher: 70-90% ratio (field variance)
DiffCompressor: 60-80% ratio (change lines only)
TextCompressor: 30-50% ratio (token importance)
ConfigCompressor: passthrough (already compact)

✅ Cache Alignment

Detects dynamic patterns (timestamps, UUIDs, session IDs)
Drift metric (0.0-1.0)
Separates stable prefix from dynamic tail

✅ Reversible Compression (CCR Store)

LRU cache with SHA256
TTL-based expiry
Model can retrieve originals via hint injection

What Needs Fixing

Root Issue: Architecture Misunderstanding

Documented (❌ Wrong):

Ingest → pgvector + OpenSearch (full noise)
  ↓
Query → M3.8 compression → LLM

Should Be (✅ Correct):

Ingest → M3.8 optimization → pgvector + OpenSearch (clean)
  ↓
Query → retrieve clean results → LLM

Why the correct way is better:

Cleaner text → better embeddings (pgvector)
Signal-rich text → better BM25 ranking (OpenSearch)
One-time processing at ingest, not per-query
All users benefit from cleaner search results
LLM already gets optimized chunks

Next Steps

M3.8.2: Ingest Pipeline Integration (1 day)

Create OptimizerSink wrapper around ingest sources
Wire into rebuild.rs
Test with all source types
Collect metrics

M3.8.3: Metrics & Monitoring (1 day)

Track compression ratio per chunk
Aggregate per project/source/type
Emit to tracing/Prometheus
Dashboard visualization

M3.8.4: Query Path Cleanup (0.5 days)

Remove PromptBuilder.build_cache_aligned() optimizer call
Keep cache_metrics() for observability (drift tracking)
Simplify PromptBuilder

Test Summary

✅ 62 Unit Tests (all passing)

Phase 1: 17 (router, log)
Phase 2: 15 (json, diff)
Phase 3: 18 (cache align, CCR)
Phase 4: 12 (text, config)

⏳ 20 New Tests Pending (M3.8.2-3)

Ingest source optimization
Metrics collection
End-to-end pipeline

Files

Implemented (1,100 LOC):

crates/mem-core/src/optimizer/mod.rs
crates/mem-core/src/optimizer/router.rs
crates/mem-core/src/optimizer/log.rs
crates/mem-core/src/optimizer/json.rs
crates/mem-core/src/optimizer/diff.rs
crates/mem-core/src/optimizer/text.rs
crates/mem-core/src/optimizer/cache_align.rs
crates/mem-core/src/optimizer/ccr.rs

Pending (180 LOC):

crates/mem-ingest/src/optimizer_sink.rs (M3.8.2)
crates/mem-core/src/optimizer/metrics.rs (M3.8.3)

Commits

bf13e3a — Phase 1: ContentRouter + LogCompressor
a903a3f — Phase 2: JsonCrusher + DiffCompressor
edcc231 — Phase 3: CacheAligner + CcrStore
8d8addc — Phase 4: TextCompressor + env config

Lessons Learned

Ingest-time optimization > query-time: Better for entire pipeline
Compression ratios vary widely: Log 85-95% vs text 30-50%
Reversibility matters: Model needs originals for detailed analysis
Metrics > assumption: Need to measure actual improvement in search quality

Remediation

See tasks/M3.8-CORRECTED-architecture.md for complete re-architecture plan.

4.1 KiB Raw Blame History Unescape Escape