Complete documentation for the pluggable optimizer architecture: Architecture Overview: - SOLID principles (S: OptimizerPlugin, F: FormatHandler | O: Registry trait) - DRY code (generic Registry<T>, reusable pattern) - Dependency injection (PluginLocator strategy, OptimizerService) Core Concepts: 1. OptimizerPlugin - custom optimization strategies 2. FormatHandler - output formats (JSON, JSONL, Raw, CSV, YAML) 3. Registry<T> - generic plugin/format storage 4. PluginLocator - extensible lookup strategies 5. OptimizerService - orchestrator with dependency injection Usage Patterns: 1. Built-in optimizer (no custom code) 2. Custom optimizer + format 3. Ingest-time optimization (rebuild.rs) 4. Query-time optimization (query_executor.rs) Full Integration Guide: - Environment variables - Ingest pipeline wiring - Query path wiring - Monitoring (Prometheus + logging) Examples: - Semantic pruning optimizer - Code formatter optimizer Performance Targets: - Ingest: <1ms/record, 1000+/sec - Query: <50ms P95, graceful fallback - Compression: 85-95% logs, 70-90% JSON, 30-50% text Metrics: Prometheus counters + structured logging + health checks
