- Better describes the intent: awaiting task completion via queue - More descriptive than 'await-kmsvc' - Applied to all workflow design documents - Reflects the actual operation: wait for external task to complete Updated files: - DESIGN_MASTER_REVIEW.md - SERVICE_INTEGRATION.md - IMPLEMENTATION_PLAN.md - CLI_DESIGN.md - FINAL_COMPREHENSIVE_DESIGN.md
This commit is contained in:
Test
@@ -0,0 +1,226 @@
|
||||
# Poimen Declarative Workflow Implementation Plan
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Current State**: Hardcoded workflows (OrchestratorWorkflow, TaskUnitWorkflow)
|
||||
- Fixed flow: Clone → Plan → Implement → Judge → Commit
|
||||
- New flows require recompilation
|
||||
- No customer-specified sequencing
|
||||
|
||||
**Target State**: API-driven dynamic workflows
|
||||
- Customer sends JSON specifying activity sequence
|
||||
- Routing engine executes dynamically
|
||||
- Support 3 patterns: Sequential, Await-Task-Complete, Retry
|
||||
- Backward compatible with existing cmd/starter
|
||||
|
||||
**Timeline**: 4 weeks | **Effort**: ~80 story points | **Team**: 2-3 engineers
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Core Routing Engine (Week 1) — 16 hours
|
||||
|
||||
**Goal**: Build engine that parses WorkflowSpec and executes activity sequences dynamically
|
||||
|
||||
**Files to Create**:
|
||||
- `internal/routing/types.go` — Data structures for WorkflowSpec, ActivityStep, RetryConfig
|
||||
- `internal/routing/jsonpath_evaluator.go` — Parameter interpolation using JSONPath
|
||||
- `internal/routing/execution_context.go` — Execution state management
|
||||
- `internal/routing/activity_registry.go` — Activity name → implementation mapper
|
||||
- `statemachine/routing_workflow.go` — **Core RoutingWorkflow that ties everything together**
|
||||
- Tests: `*_test.go` files for each module
|
||||
|
||||
**Key Implementation Details**:
|
||||
- RoutingWorkflow is the main Temporal workflow
|
||||
- It parses activity steps and executes them sequentially
|
||||
- Uses JSONPath to interpolate parameters: `${step_1.output.path}`
|
||||
- Supports retry logic, error catching, and conditional jumps
|
||||
- All activities are pre-registered in worker pool
|
||||
|
||||
**Backward Compatibility**: ✅ Keep OrchestratorWorkflow, TaskUnitWorkflow unchanged
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: API Server (Week 2) — 10 hours
|
||||
|
||||
**Goal**: HTTP API that accepts WorkflowSpec JSON and manages workflow execution
|
||||
|
||||
**Files to Create**:
|
||||
- `cmd/api-server/main.go` — Server bootstrap, Temporal client connection
|
||||
- `cmd/api-server/handlers.go` — POST /api/v1/workflows, GET /api/v1/workflows/{id}/status
|
||||
- `cmd/api-server/validation.go` — WorkflowSpec validation (references, timeouts, etc.)
|
||||
- `cmd/api-server/server.go` — Server config and lifecycle
|
||||
- Tests: handlers, validation, integration tests
|
||||
|
||||
**Endpoints**:
|
||||
```
|
||||
POST /api/v1/workflows
|
||||
→ Submit WorkflowSpec
|
||||
→ Returns: {workflow_id, status, polling_url}
|
||||
|
||||
GET /api/v1/workflows/{workflow_id}/status
|
||||
→ Poll execution status
|
||||
→ Returns: {status, steps_completed, results, final_output}
|
||||
```
|
||||
|
||||
**Key Details**:
|
||||
- Validates WorkflowSpec before submission
|
||||
- Submits to RoutingWorkflow (not OrchestratorWorkflow)
|
||||
- Caches workflow metadata locally
|
||||
- Polls Temporal for status
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Homelab Frontend Integration (Week 3) — 5 hours
|
||||
|
||||
**Goal**: Expose API through ingress, connect to gateway
|
||||
|
||||
**Files to Create**:
|
||||
- `k8s/api-server-deployment.yaml` — K8s Deployment, 2 replicas
|
||||
- `k8s/api-server-service.yaml` — K8s Service for api-server
|
||||
- `k8s/api-server-networkpolicy.yaml` — Allow ingress traffic, deny by default
|
||||
- `homelab-frontend/k8s/gateway-routes.yaml` — Route /api/v1/workflows to api-server
|
||||
|
||||
**Key Details**:
|
||||
- api-server connects to Temporal: `temporal-frontend.temporal:7233`
|
||||
- Ingress routes external traffic to api-server:8080
|
||||
- NetworkPolicy allows only ingress-nginx → api-server traffic
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Migration & Docs (Week 4) — 7 hours
|
||||
|
||||
**Files to Create**:
|
||||
- `docs/API.md` — API reference (endpoints, request/response examples)
|
||||
- `docs/MIGRATION.md` — How to migrate from cmd/starter to API
|
||||
- `docs/examples/sequential-workflow.json` — Example: Clone → Analyze → Implement → Judge
|
||||
- `docs/examples/retry-workflow.json` — Example: with retry + error handling
|
||||
- `docs/examples/conditional-workflow.json` — Example: with catch blocks
|
||||
|
||||
**Key Details**:
|
||||
- Comprehensive examples for all 3 patterns
|
||||
- cmd/starter continues to work (backward compatible)
|
||||
- Documentation emphasizes JSONPath parameter passing
|
||||
|
||||
---
|
||||
|
||||
## Three Composite Patterns
|
||||
|
||||
### Pattern A: Sequential (Request-Response)
|
||||
```json
|
||||
{
|
||||
"activities": [
|
||||
{"name": "Step1", "resource": "ActivityA", "parameters": {...}, "next": "Step2"},
|
||||
{"name": "Step2", "resource": "ActivityB", "parameters": {"input": "${Step1.output}"}, "next": "Step3"},
|
||||
{"name": "Step3", "resource": "ActivityC", "parameters": {...}, "end": true}
|
||||
]
|
||||
}
|
||||
```
|
||||
**Flow**: Step1 → Step2 (receives Step1's output) → Step3
|
||||
|
||||
---
|
||||
|
||||
### Pattern B: Await-Task-Complete (Callback)
|
||||
```json
|
||||
{
|
||||
"activities": [
|
||||
{"name": "LaunchJob", "type": "await-callback", "resource": "LaunchActivity", "heartbeat_timeout": "5m", "next": "ProcessResult"},
|
||||
{"name": "ProcessResult", "resource": "ProcessActivity", "parameters": {"result": "${LaunchJob.callback}"}, "end": true}
|
||||
]
|
||||
}
|
||||
```
|
||||
**Flow**: LaunchJob (waits for callback) → ProcessResult (receives callback data)
|
||||
|
||||
---
|
||||
|
||||
### Pattern C: Retry with Error Handling
|
||||
```json
|
||||
{
|
||||
"activities": [
|
||||
{"name": "Generate", "resource": "ImplementerActivity", "retry": {"maxAttempts": 3, "backoffRate": 2.0}, "catch": [{"errorEquals": ["Timeout"], "next": "HandleTimeout"}], "next": "Verify"},
|
||||
{"name": "Verify", "resource": "JudgeActivity", "end": true},
|
||||
{"name": "HandleTimeout", "type": "pass", "result": {"status": "timeout"}, "end": true}
|
||||
]
|
||||
}
|
||||
```
|
||||
**Flow**: Generate (retry 3x) → if success: Verify, else: HandleTimeout
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- ✅ RoutingWorkflow executes dynamic sequences correctly
|
||||
- ✅ JSONPath parameter passing works (nested references, arrays)
|
||||
- ✅ Error handling and catch blocks work
|
||||
- ✅ Retry logic respects backoff intervals
|
||||
- ✅ API validates WorkflowSpec (no invalid refs)
|
||||
- ✅ Status polling returns step-by-step results
|
||||
- ✅ All tests pass (>90% coverage)
|
||||
- ✅ Backward compatible (cmd/starter still works)
|
||||
- ✅ API exposed via ingress at `/api/v1/workflows`
|
||||
- ✅ Performance: <200ms submit, <100ms poll
|
||||
|
||||
---
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```
|
||||
┌─ Customer/Frontend ─────────────────────────┐
|
||||
│ POST /api/v1/workflows (JSON WorkflowSpec) │
|
||||
└────────────────────┬────────────────────────┘
|
||||
↓
|
||||
┌──────────────────────┐
|
||||
│ API Server │
|
||||
│ • Validate spec │
|
||||
│ • Submit to Temporal │
|
||||
│ • Cache metadata │
|
||||
└────────────┬─────────┘
|
||||
↓
|
||||
┌────────────────────────────────┐
|
||||
│ Temporal Server │
|
||||
│ RoutingWorkflow │
|
||||
│ ├─ Parse activities │
|
||||
│ ├─ Evaluate JSONPath │
|
||||
│ ├─ Execute sequence │
|
||||
│ └─ Handle errors/retries │
|
||||
└────────────┬───────────────────┘
|
||||
↓
|
||||
┌────────────────────────────────┐
|
||||
│ Temporal Worker Pool │
|
||||
│ • Poll queue │
|
||||
│ • Execute activities │
|
||||
│ • Return results │
|
||||
└────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| JSONPath bugs | Unit tests + property-based testing |
|
||||
| Activity registry mismatch | Validate registered activities match spec |
|
||||
| Temporal network issues | Retry logic + timeouts in API server |
|
||||
| Cache staleness | Event-driven cache invalidation |
|
||||
|
||||
---
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
| Phase | Rollback Strategy |
|
||||
|-------|------------------|
|
||||
| Phase 1 | Keep old workflows, don't use RoutingWorkflow |
|
||||
| Phase 2 | Disable api-server, fall back to cmd/starter |
|
||||
| Phase 3 | Remove ingress, access api-server directly |
|
||||
| Phase 4 | Continue supporting both APIs (cmd/starter + API) |
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Week 1**: Start Phase 1 implementation (RoutingWorkflow)
|
||||
2. **Week 2**: Start Phase 2 (API Server)
|
||||
3. **Week 3**: Deploy to K8s (Phase 3)
|
||||
4. **Week 4**: Document, test, release (Phase 4)
|
||||
5. **Post-Release**: Monitor, gather feedback, plan Phase 2 enhancements (parallel, conditional branching)
|
||||
|
||||
Reference in New Issue
Block a user