Add future milestones: T1 (hardening), T2 (scale), T3 (features)
T1: Error recovery, observability, metrics, audit logging (8 tasks) T2: Caching, parallelism, distributed locking (8 tasks) T3: Plugins, templates, dependencies, custom judges, nested workflows (8 tasks) Total project timeline: ~3 months T0→T3.
This commit is contained in:
@@ -0,0 +1,102 @@
|
||||
# Project Roadmap
|
||||
|
||||
Multi-agent dev orchestrator for Temporal, building toward a fully autonomous software development system.
|
||||
|
||||
## Milestones
|
||||
|
||||
### T0: Core System (IN PROGRESS)
|
||||
9 tasks, foundational: Planner/Judge/Implementer orchestration, git workflow, LLM integration, e2e validation.
|
||||
|
||||
**Status:** 9 subtasks defined, testable criteria in place.
|
||||
**Timeline:** ~2-4 weeks to implementation.
|
||||
**Done:** All T0.1–T0.9 pass verification → squash-merge to main.
|
||||
|
||||
**Board:** `tasks/board.md`
|
||||
|
||||
---
|
||||
|
||||
### T1: Production Hardening (PLANNED)
|
||||
8 tasks: Error recovery, observability, metrics, audit logging, health checks.
|
||||
|
||||
**Focus:** Reliability for long-running orchestrators in homelab.
|
||||
**Key wins:**
|
||||
- Resume from crash without data loss
|
||||
- Structured logging + Grafana metrics
|
||||
- Auto-tuning based on historical failures
|
||||
- Audit trail for compliance
|
||||
|
||||
**Board:** `tasks/board-T1.md`
|
||||
|
||||
---
|
||||
|
||||
### T2: Scale & Performance (PLANNED)
|
||||
8 tasks: Caching, parallelism, batching, distributed locking.
|
||||
|
||||
**Focus:** Handle 100s of concurrent tasks, reduce API call overhead.
|
||||
**Key wins:**
|
||||
- Result caching deduplicates LLM calls
|
||||
- Parallel task dispatch (9x wall-clock speedup)
|
||||
- Git operation batching (fewer network round-trips)
|
||||
- Distributed lock for multi-pod safety
|
||||
|
||||
**Board:** `tasks/board-T2.md`
|
||||
|
||||
---
|
||||
|
||||
### T3: Feature Expansion (PLANNED)
|
||||
8 tasks: Plugins, templates, dependencies, human gates, custom judges, nested workflows.
|
||||
|
||||
**Focus:** Extensibility + domain specialization.
|
||||
**Key wins:**
|
||||
- Load custom skill plugins
|
||||
- Save/load orchestrator config as templates
|
||||
- Task dependency ordering
|
||||
- Swap Judge for domain-specific validator (security auditor, code reviewer, etc.)
|
||||
- Nest orchestrators (multi-level hierarchy)
|
||||
- Import tasks from GitHub/Linear/JIRA
|
||||
|
||||
**Board:** `tasks/board-T3.md`
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. **T0 (2-4 weeks):** Core system working end-to-end.
|
||||
2. **T1 (2 weeks):** Production-harden the core.
|
||||
3. **T2 (3 weeks):** Scale & optimize.
|
||||
4. **T3 (4 weeks):** Advanced features.
|
||||
|
||||
**Total:** ~3 months to full feature parity.
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
- **T0:** System runs unsupervised on 9-task milestone, all subtasks auto-complete, squash-merge to main succeeds.
|
||||
- **T1:** 1000+ tasks completed, zero data loss across pod restarts, metrics queryable.
|
||||
- **T2:** 1000 concurrent tasks complete 10x faster than T0, < 5 API calls/task (vs current ~20).
|
||||
- **T3:** Custom Judge plugin loads and validates tasks, workflow templates save/restore state.
|
||||
|
||||
---
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| Temporal cluster unavailability | Implement activity heartbeat recovery + resumption logic (T1.1) |
|
||||
| Git conflicts on shared FS | Test multi-pod concurrent access (T1.6) → upgrade to distributed lock if needed (T2.8) |
|
||||
| LLM API latency | Implement result caching + batching (T2.1, T2.6) |
|
||||
| Board state corruption | Detect + auto-heal (T1.4) |
|
||||
| Audit trail tampering | Sign audit log with workflow key (T3.6) |
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- Each milestone is independent: T1 can start once T0 core is complete, doesn't need full T0 cleanup.
|
||||
- T0 tasks are foundational; changes will ripple into T1–T3 test suites, but core API should remain stable.
|
||||
- Future milestones (T4+) could focus on: web UI, real-time dashboard, automated deployment of orchestrator itself (meta!), multi-cluster orchestration.
|
||||
|
||||
---
|
||||
|
||||
**Next:** Start T0.1 (repo scaffold).
|
||||
@@ -0,0 +1,19 @@
|
||||
# Task Board — Milestone T1: Production Hardening
|
||||
|
||||
**Submilestone:** T1 (Error recovery, observability, metrics, reliability)
|
||||
|
||||
| ID | Scope | Status | Branch | Verification |
|
||||
|----|-------|--------|--------|--------------|
|
||||
| T1.1 | Workflow error recovery: retry policies, deadletter handling, graceful shutdown | [ ] | `task/T1.1` | Simulate orchestrator crash mid-cycle, resume without data loss |
|
||||
| T1.2 | Structured logging + metrics export (Prometheus/OpenTelemetry integration) | [ ] | `task/T1.2` | Metrics visible in homelab Grafana, logs queryable in Loki |
|
||||
| T1.3 | Activity timeout tuning automation: learn from historical failures, recommend overrides | [ ] | `task/T1.3` | Planner reads lessons file, suggests `update-tuning` signal based on patterns |
|
||||
| T1.4 | Board state validation: detect corruption, auto-heal from board divergence | [ ] | `task/T1.4` | Corrupt board file recovered without manual intervention |
|
||||
| T1.5 | Workflow pause/resume with state snapshot: serialize mid-cycle state to persistent store | [ ] | `task/T1.5` | Pause signal, restart pod, resume signal → workflow continues from exact point |
|
||||
| T1.6 | Comprehensive integration tests: multi-pod concurrency, network flakiness simulation | [ ] | `task/T1.6` | Concurrent orchestrator instances on shared repo pass e2e without conflicts |
|
||||
| T1.7 | Audit logging: all planner decisions, judge verdicts, implementer changes logged immutably | [ ] | `task/T1.7` | Audit log persists across workflow restarts, queryable by task/timestamp |
|
||||
| T1.8 | Health checks: Temporal connectivity, git repo accessibility, LLM API availability | [ ] | `task/T1.8` | Periodic health probes, liveness/readiness endpoints for K8s |
|
||||
|
||||
---
|
||||
|
||||
## Submission Criteria
|
||||
All T1.1–T1.8 marked `[x]` → submilestone complete → squash-merge `task/T1.*` to main.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Task Board — Milestone T2: Scale & Performance
|
||||
|
||||
**Submilestone:** T2 (Distributed execution, caching, performance optimization)
|
||||
|
||||
| ID | Scope | Status | Branch | Verification |
|
||||
|----|-------|--------|--------|--------------|
|
||||
| T2.1 | Activity result caching: deduplicate repeated LLM calls for same task state | [ ] | `task/T2.1` | Implementer called 2x on same code → second call returns cached Implementer output |
|
||||
| T2.2 | Parallel task dispatch: multiple T0.x tasks execute truly concurrently (not sequential) | [ ] | `task/T2.2` | 9 tasks complete in ~1/9 total time (wall-clock speedup measured) |
|
||||
| T2.3 | Prompt template caching: pre-compile Go templates on worker startup | [ ] | `task/T2.3` | Template render latency < 100ms (vs parse+render each time) |
|
||||
| T2.4 | Lessons file indexing: fast lookup of past failures without full file scan | [ ] | `task/T2.4` | Query lessons by task type → return in < 10ms for 1000s of entries |
|
||||
| T2.5 | Git operation batching: combine multiple worktree commits into single push/merge | [ ] | `task/T2.5` | N tasks → 1 push (vs N pushes), measured via git ref-log |
|
||||
| T2.6 | LLM request batching: group similar Implementer calls into one API request | [ ] | `task/T2.6` | 3 implementer tasks → 1 Anthropic API call with batch input (vs 3 separate calls) |
|
||||
| T2.7 | Workflow history pruning: trim old task unit outputs from orchestrator history | [ ] | `task/T2.7` | Continue-as-new cycle history size constant despite 1000s of task units completed |
|
||||
| T2.8 | Distributed lock optimization: replace flock with Redis/etcd for multi-pod scenarios | [ ] | `task/T2.8` | 5 concurrent orchestrators on different pods share FS safely via distributed lock |
|
||||
|
||||
---
|
||||
|
||||
## Submission Criteria
|
||||
All T2.1–T2.8 marked `[x]` → submilestone complete → squash-merge `task/T2.*` to main.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Task Board — Milestone T3: Feature Expansion
|
||||
|
||||
**Submilestone:** T3 (Custom plugins, workflow templates, audit, advanced features)
|
||||
|
||||
| ID | Scope | Status | Branch | Verification |
|
||||
|----|-------|--------|--------|--------------|
|
||||
| T3.1 | Custom skill plugins: load user-defined skills from plugin registry (not just pi clone) | [ ] | `task/T3.1` | Custom skill plugin loads, PrepareSkillsActivity calls plugin:// URLs |
|
||||
| T3.2 | Workflow templates: save/load orchestrator config as YAML templates (not CLI flags only) | [ ] | `task/T3.2` | Load template `templates/golang-project.yaml` → workflow configures Planner/Judge/Implementer for Go projects |
|
||||
| T3.3 | Task dependency graph: specify task order (T0.2 must complete before T0.3 can start) | [ ] | `task/T3.3` | Board supports `depends_on: [T0.1]` field, orchestrator respects ordering |
|
||||
| T3.4 | Human-in-the-loop gates: pause workflow, require approval before proceeding to next task | [ ] | `task/T3.4` | Workflow waits for `approve-task` signal, Judge verdict is final (can't auto-retry after user approval) |
|
||||
| T3.5 | Custom Judge implementations: swap default Judge for domain-specific validator (e.g., security auditor) | [ ] | `task/T3.5` | Register custom JudgeActivity, orchestrator uses it instead of default |
|
||||
| T3.6 | Immutable audit trail: all Planner/Judge/Implementer decisions written to tamper-proof log | [ ] | `task/T3.6` | Audit log signed with per-workflow key, verification prevents tampering |
|
||||
| T3.7 | Workflow composition: nest OrchestratorWorkflows (one orchestrator dispatches child orchestrators) | [ ] | `task/T3.7` | Multi-level task hierarchy: T0 milestone → T0.a/T0.b sub-milestones, each with own orchestrator |
|
||||
| T3.8 | Integration with external task systems: import tasks from Linear, GitHub Issues, JIRA | [ ] | `task/T3.8` | Load board from GitHub Issues API, update issues with task completion status |
|
||||
|
||||
---
|
||||
|
||||
## Submission Criteria
|
||||
All T3.1–T3.8 marked `[x]` → submilestone complete → squash-merge `task/T3.*` to main.
|
||||
Reference in New Issue
Block a user