feat: Archive M3.8.1, M3.8.2 - remove task files after completion
This commit is contained in:
+1
-1
@@ -71,7 +71,7 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
|
||||
| 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
|
||||
| 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 |
|
||||
| 9 | Hybrid search | M8.x | 9 | 9 | 0 | 0 | ✅ M8.9 |
|
||||
| | **Total** | | **62** | **61** | **0** | **1** | 11/13 green |
|
||||
| | **Total** | | **60** | **59** | **0** | **1** | 11/13 green |
|
||||
|
||||
**Current status — 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). **M2.1-6 ✅** (embeddings, CNPG, schema, pgvector, obsidian projector, rebuild). **M3.x ✅** (4/4). **M3.5.x ✅** (10/10 complete + archived). **M3.7.7-8 ✅** (failure diagnosis). **M4.1-2 ✅** (skill drafting + derived filter). **M3.6.1 ✅** (DocCorpusSource). **M3.6.3 ❌ retired** (Obsidian UI replaces CLI). **M3.6.7-8 ⬜ new** (ingest enrichment + deduplication). **M8.1 🟡** (OpenSearch cluster deploying — security context fixes in progress).
|
||||
|
||||
|
||||
@@ -1,139 +0,0 @@
|
||||
# M3.8.1 — Context Optimizer Core Modules
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3.8 — Context optimization |
|
||||
| Size | L — 3–5 days |
|
||||
| Status | ✅ COMPLETE |
|
||||
| Spec | `docs/CONTEXT_OPTIMIZER.md` |
|
||||
| Blocks | M3.8.2 (ingest integration) |
|
||||
| Depends | M3.7.7 (lesson.rs patterns), M3.7.8 (stop words) |
|
||||
|
||||
## Status: PARTIAL ⚠️
|
||||
|
||||
✅ **Core Compressor Modules Complete**: 1,100 LOC, 62 tests
|
||||
- ContentRouter (Magika ML detection)
|
||||
- LogCompressor, JsonCrusher, DiffCompressor, TextCompressor
|
||||
- CacheAligner (drift detection)
|
||||
- CcrStore (reversible compression)
|
||||
- ContextOptimizer orchestrator
|
||||
|
||||
❌ **Integration in Wrong Place**:
|
||||
- Currently: PromptBuilder.build_cache_aligned() (query path)
|
||||
- Should be: rebuild.rs ingest pipeline (ingest path)
|
||||
- Result: Improves only LLM input, not search quality
|
||||
|
||||
## What Was Done Right
|
||||
|
||||
✅ **Content Detection** (Magika ML + regex)
|
||||
- <1ms classification
|
||||
- Detects JSON, code, logs, diffs, config, text
|
||||
- Thread-safe, ONNX local
|
||||
|
||||
✅ **5 Compressor Implementations**
|
||||
- LogCompressor: 85-95% ratio (keep errors + stack traces)
|
||||
- JsonCrusher: 70-90% ratio (field variance)
|
||||
- DiffCompressor: 60-80% ratio (change lines only)
|
||||
- TextCompressor: 30-50% ratio (token importance)
|
||||
- ConfigCompressor: passthrough (already compact)
|
||||
|
||||
✅ **Cache Alignment**
|
||||
- Detects dynamic patterns (timestamps, UUIDs, session IDs)
|
||||
- Drift metric (0.0-1.0)
|
||||
- Separates stable prefix from dynamic tail
|
||||
|
||||
✅ **Reversible Compression** (CCR Store)
|
||||
- LRU cache with SHA256
|
||||
- TTL-based expiry
|
||||
- Model can retrieve originals via hint injection
|
||||
|
||||
## What Needs Fixing
|
||||
|
||||
### Root Issue: Architecture Misunderstanding
|
||||
|
||||
**Documented** (❌ Wrong):
|
||||
```
|
||||
Ingest → pgvector + OpenSearch (full noise)
|
||||
↓
|
||||
Query → M3.8 compression → LLM
|
||||
```
|
||||
|
||||
**Should Be** (✅ Correct):
|
||||
```
|
||||
Ingest → M3.8 optimization → pgvector + OpenSearch (clean)
|
||||
↓
|
||||
Query → retrieve clean results → LLM
|
||||
```
|
||||
|
||||
**Why the correct way is better:**
|
||||
1. Cleaner text → better embeddings (pgvector)
|
||||
2. Signal-rich text → better BM25 ranking (OpenSearch)
|
||||
3. One-time processing at ingest, not per-query
|
||||
4. All users benefit from cleaner search results
|
||||
5. LLM already gets optimized chunks
|
||||
|
||||
### Next Steps
|
||||
|
||||
**M3.8.2**: Ingest Pipeline Integration (1 day)
|
||||
- Create OptimizerSink wrapper around ingest sources
|
||||
- Wire into rebuild.rs
|
||||
- Test with all source types
|
||||
- Collect metrics
|
||||
|
||||
**M3.8.3**: Metrics & Monitoring (1 day)
|
||||
- Track compression ratio per chunk
|
||||
- Aggregate per project/source/type
|
||||
- Emit to tracing/Prometheus
|
||||
- Dashboard visualization
|
||||
|
||||
**M3.8.4**: Query Path Cleanup (0.5 days)
|
||||
- Remove PromptBuilder.build_cache_aligned() optimizer call
|
||||
- Keep cache_metrics() for observability (drift tracking)
|
||||
- Simplify PromptBuilder
|
||||
|
||||
## Test Summary
|
||||
|
||||
✅ **62 Unit Tests** (all passing)
|
||||
- Phase 1: 17 (router, log)
|
||||
- Phase 2: 15 (json, diff)
|
||||
- Phase 3: 18 (cache align, CCR)
|
||||
- Phase 4: 12 (text, config)
|
||||
|
||||
⏳ **20 New Tests Pending** (M3.8.2-3)
|
||||
- Ingest source optimization
|
||||
- Metrics collection
|
||||
- End-to-end pipeline
|
||||
|
||||
## Files
|
||||
|
||||
**Implemented** (1,100 LOC):
|
||||
- `crates/mem-core/src/optimizer/mod.rs`
|
||||
- `crates/mem-core/src/optimizer/router.rs`
|
||||
- `crates/mem-core/src/optimizer/log.rs`
|
||||
- `crates/mem-core/src/optimizer/json.rs`
|
||||
- `crates/mem-core/src/optimizer/diff.rs`
|
||||
- `crates/mem-core/src/optimizer/text.rs`
|
||||
- `crates/mem-core/src/optimizer/cache_align.rs`
|
||||
- `crates/mem-core/src/optimizer/ccr.rs`
|
||||
|
||||
**Pending** (180 LOC):
|
||||
- `crates/mem-ingest/src/optimizer_sink.rs` (M3.8.2)
|
||||
- `crates/mem-core/src/optimizer/metrics.rs` (M3.8.3)
|
||||
|
||||
## Commits
|
||||
|
||||
1. `bf13e3a` — Phase 1: ContentRouter + LogCompressor
|
||||
2. `a903a3f` — Phase 2: JsonCrusher + DiffCompressor
|
||||
3. `edcc231` — Phase 3: CacheAligner + CcrStore
|
||||
4. `8d8addc` — Phase 4: TextCompressor + env config
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **Ingest-time optimization > query-time**: Better for entire pipeline
|
||||
2. **Compression ratios vary widely**: Log 85-95% vs text 30-50%
|
||||
3. **Reversibility matters**: Model needs originals for detailed analysis
|
||||
4. **Metrics > assumption**: Need to measure actual improvement in search quality
|
||||
|
||||
## Remediation
|
||||
|
||||
See **`tasks/M3.8-CORRECTED-architecture.md`** for complete re-architecture plan.
|
||||
@@ -1,126 +0,0 @@
|
||||
# M3.8.2 — Ingest Pipeline Integration (OptimizerSink)
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3.8 — Context optimization |
|
||||
| Size | M — 1–2 days |
|
||||
| Status | ✅ COMPLETE |
|
||||
| Depends | M3.8.1 (core modules) |
|
||||
| Blocks | M3.8.3 |
|
||||
|
||||
## Goal
|
||||
|
||||
Wire M3.8 compressors into the ingest pipeline so that chunks are optimized
|
||||
BEFORE embedding + indexing, resulting in:
|
||||
- Better embeddings (clean text)
|
||||
- Better search ranking (signal-rich documents)
|
||||
- Cleaner results for all queries
|
||||
|
||||
## Deliverables
|
||||
|
||||
### 1. OptimizerSink Wrapper (100 LOC)
|
||||
|
||||
New module: `crates/mem-ingest/src/optimizer_sink.rs`
|
||||
|
||||
```rust
|
||||
pub struct OptimizerSink {
|
||||
inner: Box<dyn RecordSource>,
|
||||
optimizer: ContextOptimizer,
|
||||
config: OptimizerConfig,
|
||||
metrics: MetricsCollector,
|
||||
}
|
||||
|
||||
impl RecordSource for OptimizerSink {
|
||||
fn next_record(&mut self) -> Option<Record> {
|
||||
let record = self.inner.next_record()?;
|
||||
let optimized = self.optimizer.optimize(&record.content)?;
|
||||
|
||||
// Track metrics
|
||||
self.metrics.record(OptimizationMetrics {
|
||||
input_bytes: record.content.len(),
|
||||
output_bytes: optimized.compressed.len(),
|
||||
compressor: optimized.compressor_used,
|
||||
..
|
||||
});
|
||||
|
||||
// Emit optimized chunk
|
||||
Some(Record {
|
||||
content: optimized.compressed,
|
||||
..record
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
pub fn optimize_source(
|
||||
source: Box<dyn RecordSource>,
|
||||
project: &str,
|
||||
) -> Result<OptimizerSink>
|
||||
```
|
||||
|
||||
Tests (3):
|
||||
- `test_optimizer_sink_preserves_structure`
|
||||
- `test_optimizer_sink_reduces_bytes`
|
||||
- `test_optimizer_sink_handles_errors`
|
||||
|
||||
### 2. Rebuild Integration (30 LOC)
|
||||
|
||||
Modify: `crates/mem-store/src/rebuild.rs`
|
||||
|
||||
```rust
|
||||
let source = DocCorpusSource::new(vault_path)?;
|
||||
let optimized = optimize_source(Box::new(source), &project)?; // ← NEW
|
||||
|
||||
for record in optimized {
|
||||
let embedding = embed(&record.content)?; // clean text
|
||||
insert_pgvector(embedding, &record)?;
|
||||
insert_opensearch(&record)?;
|
||||
}
|
||||
```
|
||||
|
||||
Tests (4):
|
||||
- `test_rebuild_with_optimizer_enabled`
|
||||
- `test_rebuild_with_optimizer_disabled`
|
||||
- `test_rebuild_compression_ratio`
|
||||
- `test_rebuild_pgvector_quality_improves`
|
||||
|
||||
### 3. Source Integration Tests (150 LOC)
|
||||
|
||||
New module: `tests/it_ingest_optimizer.rs`
|
||||
|
||||
Test each ingest source with optimizer:
|
||||
- `test_pi_session_source_optimized` (Claude transcripts)
|
||||
- `test_doc_corpus_source_optimized` (markdown files)
|
||||
- `test_claude_transcript_source_optimized` (agent logs)
|
||||
- `test_optimizer_preserves_breadcrumb` (M3.6.1 paths)
|
||||
- `test_optimizer_respects_level` (L0/L1/L2)
|
||||
- `test_optimizer_disabled_via_env` (MEM_CONTEXT_OPTIMIZER=off)
|
||||
|
||||
Tests (6):
|
||||
- Per-source integration tests
|
||||
|
||||
### 4. Metrics Collection (NEW)
|
||||
|
||||
Modified: `crates/mem-ingest/src/lib.rs`
|
||||
|
||||
Export MetricsCollector from OptimizerSink:
|
||||
```rust
|
||||
pub struct OptimizerMetrics {
|
||||
pub input_bytes: usize,
|
||||
pub output_bytes: usize,
|
||||
pub compression_ratio: f32,
|
||||
pub compressor_used: String,
|
||||
pub timestamp: i64,
|
||||
pub project: String,
|
||||
}
|
||||
```
|
||||
|
||||
No new tests (M3.8.3 handles metrics comprehensively)
|
||||
|
||||
## Acceptance
|
||||
|
||||
✅ All 13 new tests passing
|
||||
✅ OptimizerSink integrated with rebuild.rs
|
||||
✅ All ingest sources work with optimizer
|
||||
✅ Metrics collected (no performance regression <1ms per chunk)
|
||||
✅ Backward compatible (optimizer disableable via env)
|
||||
✅ Compression ratios match targets (log 85-95%, json 70-90%, etc.)
|
||||
Reference in New Issue
Block a user