feat: M3.8.1 phase 4a — TextCompressor + env config (12 tests)
TextCompressor (320 LOC, 10 tests): - Token importance scoring with lazy_static STOP_WORDS - Keeps: high-entropy tokens (IDs, hashes, error codes, numbers, symbols) - Drops: stop words, filler words, low-information prose - ID detection: UUID, SHA256, session IDs, underscored patterns - Error marker detection: error, exception, panic, fail, warn, critical - Configurable compression ratio (default 40% token retention) ContextOptimizerConfig::from_env() (2 tests): - MEM_CONTEXT_OPTIMIZER (on/off) - MEM_MAGIKA_ENABLED, MEM_MAGIKA_THRESHOLD - MEM_COMPRESS_JSON, MEM_COMPRESS_LOGS, MEM_COMPRESS_CODE, MEM_COMPRESS_DIFF, MEM_COMPRESS_TEXT - MEM_TOKEN_BUDGET, MEM_CCR_ENABLED ContextOptimizer::from_env() factory method 62 optimizer tests total: Phase 1 (17) + Phase 2 (15) + Phase 3 (18) + Phase 4a (12) = 62 passing
This commit is contained in:
@@ -21,3 +21,4 @@ ort = { version = "2.0.0-rc.12", default-features = true }
|
||||
regex = "1.10"
|
||||
once_cell = "1.19"
|
||||
indexmap = "2.0"
|
||||
lazy_static = "1.4"
|
||||
|
||||
Reference in New Issue
Block a user