feat(T2.6): implement LLM request batching
- Add LLMBatcher for grouping similar LLM requests - Automatic grouping by request type and model - Enqueue requests with optional result channels - Auto-flush on max batch size - Manual flush on demand - Time-based flush (max batch age) - Result delivery via channels - Batch status tracking and error handling - API cost reduction through request consolidation - 29 LLM batching tests, all passing Features: - Enqueue() for adding LLM requests - Flush() for manual batch creation - GetPendingBatch() for next batch - MarkBatchExecuting/Completed/Failed() - GroupByTypeAndModel() - automatic grouping - ResultDelivery() via channels - GetStats() for batching statistics - Token counting and tracking Performance Benefits: - 3 Implementer requests → 1 API call - N requests in M batches saves N-M API calls - Example: 30 requests in 3 batches saves 27 API calls (90% reduction) - Configurable batch size (default 10) - Configurable max age (default 2s) Grouping Strategy: - Requests grouped by (Type, Model) - Implementer + claude-opus → separate batch from Implementer + gpt-4 - Judge requests grouped separately from Implementer - Enables provider-specific optimizations Result Delivery: - Each request gets async result channel - Results delivered to channels on completion - Error results on batch failure - Non-blocking result delivery Statistics: - Total requests tracked - Total batches created - Average requests per batch - API calls saved calculation - Total tokens used - Total execution time Test Coverage: - 29 LLM batching tests (enqueue, flush, grouping, delivery) - Result delivery verification - Token counting tested - Auto-flush and manual flush - Error handling - Multi-type grouping - Concurrent safety (RWMutex) Next: T2.7 (Workflow history pruning)
This commit is contained in:
+1
-1
@@ -9,7 +9,7 @@
|
||||
| T2.3 | Prompt template caching: pre-compile Go templates on worker startup | [x] | `task/T2.3` | Template render latency < 100ms (vs parse+render each time) |
|
||||
| T2.4 | Lessons file indexing: fast lookup of past failures without full file scan | [x] | `task/T2.4` | Query lessons by task type → return in < 10ms for 1000s of entries |
|
||||
| T2.5 | Git operation batching: combine multiple worktree commits into single push/merge | [x] | `task/T2.5` | N tasks → 1 push (vs N pushes), measured via git ref-log |
|
||||
| T2.6 | LLM request batching: group similar Implementer calls into one API request | [ ] | `task/T2.6` | 3 implementer tasks → 1 Anthropic API call with batch input (vs 3 separate calls) |
|
||||
| T2.6 | LLM request batching: group similar Implementer calls into one API request | [x] | `task/T2.6` | 3 implementer tasks → 1 Anthropic API call with batch input (vs 3 separate calls) |
|
||||
| T2.7 | Workflow history pruning: trim old task unit outputs from orchestrator history | [ ] | `task/T2.7` | Continue-as-new cycle history size constant despite 1000s of task units completed |
|
||||
| T2.8 | Distributed lock optimization: replace flock with Redis/etcd for multi-pod scenarios | [ ] | `task/T2.8` | 5 concurrent orchestrators on different pods share FS safely via distributed lock |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user