Test
927835cb0e
- Add internal/tuning package with intelligent timeout analysis - Implement TimeoutAnalyzer for tracking activity execution metrics - Calculate percentile-based timeout recommendations (P95, P99) - Generate confidence scores based on sample size and failure rate - Implement TimeoutLessonsStore for persistent lesson tracking - Store lessons in per-task JSONL files with effectiveness tracking - Generate TimeoutTuningSignal objects for planner integration - Generate human-readable lesson format for planner context - Support three-tier priority signaling (high/medium/low) - Analyze multiple activities concurrently Analysis Features: - Track duration, success/failure, timestamps for each execution - Identify undertuned activities (P99 exceeds timeout) - Detect overtuned activities (timeout > 2x P99) - Calculate confidence scores (40% sample data + 60% reliability) - Generate recommendations with reasoning Lesson Management: - Persist lessons per task in JSONL format - Support lesson effectiveness tracking - Format lessons for planner input - Enable feedback loop for timeout optimization Test Coverage: - 14 analyzer tests (metrics, analysis, persistence) - 22 lessons tests (storage, signals, formatting) - 36 total tuning tests, all passing - Edge cases: empty metrics, all failures, multiple activities Key Design: - P99 + 20% buffer for safe timeout values - Weighted confidence scoring for reliable recommendations - Separation: Analyzer (metrics), Lessons (storage), Signals (integration) - Thread-safe analyzer with RWMutex - No external dependencies added Closes T1.3
20 lines
1.8 KiB
Markdown
20 lines
1.8 KiB
Markdown
# Task Board — Milestone T1: Production Hardening
|
||
|
||
**Submilestone:** T1 (Error recovery, observability, metrics, reliability)
|
||
|
||
| ID | Scope | Status | Branch | Verification |
|
||
|----|-------|--------|--------|--------------|
|
||
| T1.1 | Workflow error recovery: retry policies, deadletter handling, graceful shutdown | [x] | `task/T1.1` | Simulate orchestrator crash mid-cycle, resume without data loss |
|
||
| T1.2 | Structured logging + metrics export (Prometheus/OpenTelemetry integration) | [x] | `task/T1.2` | Metrics visible in homelab Grafana, logs queryable in Loki |
|
||
| T1.3 | Activity timeout tuning automation: learn from historical failures, recommend overrides | [x] | `task/T1.3` | Planner reads lessons file, suggests `update-tuning` signal based on patterns |
|
||
| T1.4 | Board state validation: detect corruption, auto-heal from board divergence | [ ] | `task/T1.4` | Corrupt board file recovered without manual intervention |
|
||
| T1.5 | Workflow pause/resume with state snapshot: serialize mid-cycle state to persistent store | [ ] | `task/T1.5` | Pause signal, restart pod, resume signal → workflow continues from exact point |
|
||
| T1.6 | Comprehensive integration tests: multi-pod concurrency, network flakiness simulation | [ ] | `task/T1.6` | Concurrent orchestrator instances on shared repo pass e2e without conflicts |
|
||
| T1.7 | Audit logging: all planner decisions, judge verdicts, implementer changes logged immutably | [ ] | `task/T1.7` | Audit log persists across workflow restarts, queryable by task/timestamp |
|
||
| T1.8 | Health checks: Temporal connectivity, git repo accessibility, LLM API availability | [x] | `task/T1.8` | Periodic health probes, liveness/readiness endpoints for K8s |
|
||
|
||
---
|
||
|
||
## Submission Criteria
|
||
All T1.1–T1.8 marked `[x]` → submilestone complete → squash-merge `task/T1.*` to main.
|