feat: M3.8.1 phase 4a — TextCompressor + env config (12 tests)
TextCompressor (320 LOC, 10 tests): - Token importance scoring with lazy_static STOP_WORDS - Keeps: high-entropy tokens (IDs, hashes, error codes, numbers, symbols) - Drops: stop words, filler words, low-information prose - ID detection: UUID, SHA256, session IDs, underscored patterns - Error marker detection: error, exception, panic, fail, warn, critical - Configurable compression ratio (default 40% token retention) ContextOptimizerConfig::from_env() (2 tests): - MEM_CONTEXT_OPTIMIZER (on/off) - MEM_MAGIKA_ENABLED, MEM_MAGIKA_THRESHOLD - MEM_COMPRESS_JSON, MEM_COMPRESS_LOGS, MEM_COMPRESS_CODE, MEM_COMPRESS_DIFF, MEM_COMPRESS_TEXT - MEM_TOKEN_BUDGET, MEM_CCR_ENABLED ContextOptimizer::from_env() factory method 62 optimizer tests total: Phase 1 (17) + Phase 2 (15) + Phase 3 (18) + Phase 4a (12) = 62 passing
This commit is contained in:
Generated
+1
@@ -2064,6 +2064,7 @@ dependencies = [
|
||||
"futures",
|
||||
"hex",
|
||||
"indexmap",
|
||||
"lazy_static",
|
||||
"magika",
|
||||
"once_cell",
|
||||
"ort",
|
||||
|
||||
Reference in New Issue
Block a user