4 Commits

Include renames

Author	SHA1	Message	Date
rock	fd9f73230a	feat: Mark M3.8.1, M3.8.2 complete, verify optimizer infrastructure	2026-08-28 13:40:17 -07:00
Story Crater Bot	f1917e1260	docs: CRITICAL CORRECTION — M3.8 architecture (ingest, not query) ... ISSUE IDENTIFIED: M3.8 was misplaced in query path (PromptBuilder), but should be in ingest path - Current: Compress before LLM (query-time, only helps LLM input) - Correct: Optimize before embed + index (ingest-time, improves search quality) BENEFITS OF INGEST-TIME OPTIMIZATION: ✅ Better embeddings (pgvector gets clean text → higher semantic quality) ✅ Better ranking (OpenSearch gets signal-rich text → better BM25 scores) ✅ One-time processing at ingest, not per-query overhead ✅ All queries benefit from cleaner search results ✅ LLM receives already-optimized chunks NEW PLAN: - M3.8.1: 🟡 Core modules PARTIAL (1100 LOC, 62 tests done, needs ingest wiring) - M3.8.2: ⬜ Ingest integration (OptimizerSink wrapper, 13 tests) - M3.8.3: ⬜ Metrics & monitoring (20 tests, tracing + prometheus) - M3.8.4: ⬜ Query cleanup (remove PromptBuilder optimizer call) - M3.8.5: ⬜ Benchmarks (compression ratios + search quality metrics) - M3.8.6: ⬜ Gate (ingest pipeline quality + search improvement) ARCHITECTURE CORRECTED: Raw content → M3.8 optimize → embed + index → search improves → LLM benefits FILES UPDATED: - tasks/M3.8-CORRECTED-architecture.md (NEW, comprehensive re-plan) - tasks/M3.8.1-context-optimizer.md (REWRITTEN, marked PARTIAL) - tasks/M3.8.2-cache-aligner-headers.md (REWRITTEN, now OptimizerSink) NEXT IMMEDIATE STEP: Implement M3.8.2 (OptimizerSink) to wire compressors into rebuild.rs ingest pipeline	2026-08-28 10:25:31 -07:00
Story Crater Bot	0acbbd09b4	chore: mark M3.8.2 complete (3 cache metrics tests, 117 total)	2026-08-28 10:05:55 -07:00
Story Crater Bot	a854ea69e4	feat: M3.8.2 cache aligner integration — metrics + headers (3 tests) ... CacheMetrics struct (40 LOC): - stable_prefix_bytes, dynamic_tail_bytes - drift_metric (0.0-1.0 ratio) - cache_eligible flag (drift < 0.3) - compression_ratio() and header_* methods PromptBuilder::cache_metrics() (40 LOC): - Calculates cache alignment metrics for query+chunk pairs - Integrates CacheAligner output - Gets compression ratio from ContextOptimizer - Used for HTTP headers and observability HTTP headers ready for client integration: - X-Cache-Stable-Bytes - X-Cache-Drift - X-Cache-Eligible - X-Compression-Ratio 3 new tests: - test_cache_metrics_stable_query - test_cache_metrics_compression_ratio - test_cache_metrics_header_drift Total: 117 mem-core tests (114 before + 3 new)	2026-08-28 10:05:45 -07:00