From f0beb7fff165692e47777324a7ee2fbc8b4ecd18 Mon Sep 17 00:00:00 2001 From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com> Date: Fri, 28 Aug 2026 09:04:40 -0700 Subject: [PATCH] plan: M3.8 context optimizer (4 tasks, Headroom-inspired) --- tasks/INDEX.md | 17 +++- tasks/M3.8.1-context-optimizer.md | 158 ++++++++++++++++++++++++++++++ 2 files changed, 174 insertions(+), 1 deletion(-) create mode 100644 tasks/M3.8.1-context-optimizer.md diff --git a/tasks/INDEX.md b/tasks/INDEX.md index 3ba7857..9cbe285 100644 --- a/tasks/INDEX.md +++ b/tasks/INDEX.md @@ -66,15 +66,17 @@ Legend: โฌœ not started ยท ๐ŸŸก in progress ยท โœ… done ยท โ›” blocked | 5 | Skills | M4.x | 3 | 2 | 0 | 1 | โฌœ M4.3 | | 5.5 | Reference corpora | M3.6.x | 5 | 1 | 0 | 4 | โฌœ M3.6.6 | | 5.6 | Tool context | M3.7.x | 4 | 2 | 0 | 2 | โฌœ M3.7.6 | +| 5.7 | Context optimization | M3.8.x | 4 | 0 | 0 | 4 | โฌœ M3.8.4 | | 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | โฌœ M5.6 | | 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | โฌœ M6.6 | | 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | โฌœ M7.10 | | 9 | Hybrid search | M8.x | 9 | 1 | 1 | 7 | โฌœ M8.9 | -| | **Total** | | **70** | **60** | **0** | **10** | 6/11 green | +| | **Total** | | **74** | **60** | **0** | **14** | 6/12 green | **Current status โ€” 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). **M2.1-6 โœ…** (embeddings, CNPG, schema, pgvector, obsidian projector, rebuild). **M3.x โœ…** (4/4). **M3.5.x โœ…** (10/10 complete + archived). **M3.7.7-8 โœ…** (failure diagnosis). **M4.1-2 โœ…** (skill drafting + derived filter). **M3.6.1 โœ…** (DocCorpusSource). **M3.6.3 โŒ retired** (Obsidian UI replaces CLI). **M8.1 ๐ŸŸก** (OpenSearch cluster deploying โ€” security context fixes in progress). **Current work:** +- M3.8.1: Context optimizer (Headroom-inspired pre-LLM compression) - M8.1: OpenSearch StatefulSet (pod security baseline, fsGroup perms) โ€” deploying - M3.7.4: Context endpoint glue (uses tier logic + hybrid search) - Obsidian service: โœ… Deployed (ppatlabs/obsidian:latest, REST API on 27124) @@ -194,6 +196,19 @@ Ids are `M3.7.x` and frozen. `M3.7.1`, `M3.7.2`, `M3.7.3`, `M3.7.5` retired. | M3.7.7 | Failure signature extraction + normalisation | M | โ€” | โœ… ARCHIVED | | M3.7.8 | Symptom projection at ingest | M | โ€” | โœ… ARCHIVED | +## 5.7 โ€” Context optimization ยท M3.8.x + +Headroom-inspired pre-LLM compression. Sits between hybrid search retrieval +and the LLM gateway. Search indexes (pgvector + OpenSearch) stay at full +fidelity; only evidence chunks entering the prompt get optimized. + +| Task | Title | Size | Flags | Status | +|---|---|---|---|---| +| [M3.8.1](M3.8.1-context-optimizer.md) | Context optimizer pipeline (router + compressors + CCR) | L | โ€” | โฌœ | +| M3.8.2 | CacheAligner integration with prompt builder | M | โ€” | โฌœ | +| M3.8.3 | Compression benchmarks + tuning | M | โ€” | โฌœ | +| M3.8.4 | **M3.8 composition gate** | M | gate | โฌœ | + ## 6 โ€” Post-training ยท M5.x Python, separate from the Rust workspace. The boundary is the JSONL log. diff --git a/tasks/M3.8.1-context-optimizer.md b/tasks/M3.8.1-context-optimizer.md new file mode 100644 index 0000000..9f87570 --- /dev/null +++ b/tasks/M3.8.1-context-optimizer.md @@ -0,0 +1,158 @@ +# M3.8.1 โ€” Context Optimizer: pre-LLM compression pipeline + +| Field | Value | +|---|---| +| Phase | M3.8 โ€” Context optimization | +| Size | L โ€” 3โ€“5 days | +| Status | โฌœ Not started | +| Flags | โ€” | +| Spec | `docs/CONTEXT_OPTIMIZER.md` | +| Blocks | M3.8.4 | +| Depends | M3.7.7 (lesson.rs patterns), M3.7.8 (stop words) | + +## Goal + +Build a 4-stage pre-LLM pipeline that compresses retrieved evidence chunks +before they enter the GRU-Mem prompt. Search indexes stay untouched. + +## Deliverables + +### Phase 1: ContentRouter + LogCompressor (day 1) + +**Files:** +- `crates/mem-core/src/optimizer/mod.rs` โ€” orchestrator +- `crates/mem-core/src/optimizer/router.rs` โ€” content type detection +- `crates/mem-core/src/optimizer/log.rs` โ€” log compression + +**ContentRouter** detects content type via heuristics: + +| Type | Signal | +|---|---| +| Json | starts with `{` or `[`, valid JSON parse | +| Log | timestamp patterns, log levels, `error:`, `npm ERR!` | +| Diff | `---`/`+++`/`@@` markers | +| Code | `import`/`use`/`fn`/`def`/`class` + indentation | +| Text | fallback | + +**LogCompressor** reuses M3.7.7 `lesson.rs`: +- `markers()` for error line detection +- `is_cascade()` for noise suppression +- `strip_ansi()` for cleanup +- Keep: error lines, stack traces, exit codes +- Drop: INFO/DEBUG noise, passing tests, repeated patterns + +**Tests (10):** +- `detect_json`, `detect_log`, `detect_diff`, `detect_code`, `detect_text` +- `log_keeps_errors`, `log_drops_info_noise`, `log_keeps_stack_traces` +- `log_compression_ratio_above_80pct`, `log_strips_ansi` + +### Phase 2: JsonCrusher + DiffCompressor (day 2) + +**Files:** +- `crates/mem-core/src/optimizer/json.rs` +- `crates/mem-core/src/optimizer/diff.rs` + +**JsonCrusher:** +- Parse JSON array โ†’ analyse field variance per key +- Allocation: 30% start (schema), 15% end (recency), 55% importance +- Keep: all keys, structure, error/null/boolean fields, boundary items +- Drop: mid-array homogeneous elements, long string values + +**DiffCompressor:** +- Keep: `+`/`-` lines (actual changes), hunk headers (`@@`) +- Drop: unchanged context lines, file mode changes +- Preserve enough context for the model to understand the change + +**Tests (10):** +- `json_keeps_keys`, `json_drops_mid_array`, `json_keeps_boundaries` +- `json_preserves_errors`, `json_ratio_70_90pct` +- `diff_keeps_changes`, `diff_drops_context`, `diff_keeps_headers` +- `diff_preserves_additions`, `diff_ratio_60_80pct` + +### Phase 3: CacheAligner + CCR Store (day 3) + +**Files:** +- `crates/mem-core/src/optimizer/cache_align.rs` +- `crates/mem-core/src/optimizer/ccr.rs` + +**CacheAligner:** +- Detect dynamic patterns in prompt prefix (timestamps, UUIDs, temp paths, SHAs) +- Reuse `lesson.rs` normalise() regex patterns +- Move dynamic content to tail, keep static prefix stable +- Report drift metrics + +**CCR Store:** +- LRU cache (bounded, default 1000 entries) +- `store(content) โ†’ hash` / `retrieve(hash) โ†’ content` +- TTL-based expiry (default 1hr, matches gate run duration) +- Inject retrieval hint: `` + +**Tests (10):** +- `align_moves_timestamps_to_tail`, `align_moves_uuids_to_tail` +- `align_preserves_static_prefix`, `align_stable_across_calls` +- `align_drift_metrics_reported` +- `ccr_store_and_retrieve`, `ccr_lru_eviction`, `ccr_ttl_expiry` +- `ccr_hash_deterministic`, `ccr_inject_hint` + +### Phase 4: TextCompressor + Integration (day 4) + +**Files:** +- `crates/mem-core/src/optimizer/text.rs` +- Update `crates/mem-core/src/prompt.rs` โ€” wire optimizer into `build_cache_aligned` + +**TextCompressor:** +- Reuse M3.7.8 stop words for low-value token detection +- Keep: high-entropy tokens (IDs, hashes, error codes, numbers) +- Drop: filler words, repeated phrases, low-information prose + +**Integration:** +- `ContextOptimizer::optimize(chunk, config) โ†’ OptimizedChunk` +- Called inside `PromptBuilder::build_cache_aligned()` before assembly +- Config: enable/disable per-compressor, token budget, CCR toggle +- Env var: `MEM_CONTEXT_OPTIMIZER=on|off` (default: on) + +**Tests (10):** +- `text_keeps_high_entropy`, `text_drops_filler`, `text_ratio_30_50pct` +- `optimizer_end_to_end_json`, `optimizer_end_to_end_log` +- `optimizer_skips_under_budget`, `optimizer_respects_budget` +- `optimizer_off_passthrough`, `optimizer_with_ccr` +- `prompt_builder_uses_optimizer` + +## Verify + +**Integration test** โ€” `tests/it_context_optimizer.rs`: + +1. `a1_log_compression` โ€” 50-line npm error log compressed to <10 lines, + all error lines preserved, ratio >80%. +2. `a2_json_compression` โ€” 100-element JSON array compressed to <20 items, + keys preserved, boundary items present, ratio >70%. +3. `a3_diff_compression` โ€” 200-line unified diff compressed to changes only, + context lines dropped, ratio >60%. +4. `a4_cache_prefix_stable` โ€” Same query across 20 chunks produces identical + cache prefix (system + query) after alignment. +5. `a5_ccr_roundtrip` โ€” Compressed chunk has CCR hash, retrieve returns + original byte-identical content. +6. `a6_search_untouched` โ€” Verify optimizer is NOT called during indexing, + only during prompt assembly. +7. `a7_under_budget_passthrough` โ€” Chunk already under token budget passes + through unmodified (zero overhead). +8. `a8_no_false_negatives` โ€” Error lines, stack traces, and exit codes + survive compression across all content types. + +**Command:** `cargo test -p mem-core optimizer` + +**False pass:** +- Compression ratio measured on already-small input. Fixtures must be + realistic size (50+ lines for logs, 100+ elements for JSON arrays). +- Testing CCR retrieve without first compressing. The store must be + populated by the compression step, not manually seeded. + +## Acceptance + +- Log compression: >80% ratio, zero error lines lost +- JSON compression: >70% ratio, all keys preserved +- Diff compression: >60% ratio, all change lines preserved +- Cache prefix stable across chunks within a run +- CCR retrieval returns byte-identical originals +- Search indexes never see compressed content +- `MEM_CONTEXT_OPTIMIZER=off` disables entirely (passthrough)