# Poimen Service Integration & Dynamic Workflows — MASTER REVIEW DOCUMENT **Status**: Ready for Approval **Version**: 1.0 **Date**: 2025-01-31 **Duration**: 52-58 hours (4 weeks) --- ## EXECUTIVE SUMMARY (5 MINUTES) ### Problem Current workflows are **hardcoded in Go**. Changing activity sequences requires: - Code modification - Recompilation - K8s deployment - Pod restart - **Total: 5-10 minutes** ### Solution New **RoutingWorkflow** reads JSON WorkflowSpec and executes dynamically: - **Three entry points**: CLI + HTTP API + Legacy CLI (backward compatible) - **Three patterns**: Sequential, Await-Task-Complete (KMSvc queue), Retry with error handling - **JSONPath parameters**: `${step1.output.path}` instead of hardcoded values - **Result**: Workflow changes in **seconds** (API call only) ### Impact ✅ Instant changes | ✅ Three interfaces | ✅ Full compatibility | ❌ +KMSvc complexity --- ## PART 1: THE THREE PATTERNS ### Pattern 1: Sequential (A → B → C) Activities execute in sequence. Each step passes output to next via JSONPath. ```json { "activities": [ { "name": "Clone", "resource": "CloneRepoActivity", "parameters": {"repo": "${input.repo}"}, "timeout": "5m", "next": "Analyze" }, { "name": "Analyze", "resource": "AnalyzeCodeActivity", "parameters": {"path": "${Clone.output.path}"}, "timeout": "10m", "next": "Judge" }, { "name": "Judge", "resource": "JudgeActivity", "parameters": {"code": "${Analyze.output.code}"}, "timeout": "5m", "end": true } ] } ``` **Execution Flow**: ``` Clone ──(output: {path: /tmp/repo})──> Analyze ──(output: {code: ...})──> Judge ──(final result) ``` **Use Case**: Code review pipeline (clone → analyze → judge) --- ### Pattern 2: Await-Task-Complete (Launch → Poll Queue → Process) Launch long-running job, poll KMSvc queue for result with correlation ID matching, then proceed. ```json { "name": "LaunchJob", "type": "awaitTaskComplete", "resource": "LaunchJobActivity", "parameters": {"job_id": "job-123"}, "queue": "job-completions", "correlation_key": "${LaunchJob.output.correlation_id}", "timeout": "5m", "next": "ProcessResult" } ``` **Execution Flow**: ``` LaunchJobActivity ──(returns {correlation_id: "xyz"})──> Poll Queue "job-completions" ↓ [External system completes job] [Publishes {correlation_id: "xyz", result: ...}] ↓ [Workflow receives message] [Proceeds to ProcessResult] ``` **Use Case**: Long-running batch jobs (model training, data processing) --- ### Pattern 3: Retry with Error Handling Activity retries N times with exponential backoff. On failure, jumps to catch block. ```json { "name": "Generate", "resource": "ImplementerActivity", "parameters": {"task": "${input.task}"}, "timeout": "30m", "retry": { "maxAttempts": 3, "backoffRate": 2.0, "initialInterval": "1s" }, "catch": [ {"errorEquals": ["Timeout"], "next": "HandleTimeout"} ], "next": "Verify" } ``` **Execution Flow**: ``` Generate (attempt 1) ──TIMEOUT──> [Wait 1s] Generate (attempt 2) ──TIMEOUT──> [Wait 2s] Generate (attempt 3) ──TIMEOUT──> [Wait 4s] Max retries exceeded ──> Jump to HandleTimeout ``` **Use Case**: Code generation (retry on timeout, handle gracefully) --- ## PART 2: THE THREE ENTRY POINTS ### Entry Point 1: CLI (New Command-Line) ```bash # Submit workflow from JSON file $ poimen-cli submit workflow.json Workflow submitted: wf-abc123 Poll status: poimen-cli status wf-abc123 # Submit and wait for results $ poimen-cli submit workflow.json --wait Waiting for workflow to complete... ✅ Workflow completed Output: {...} # Execute template immediately (sync, no polling) $ poimen-cli execute template code-review-v1 --input-file input.json Executing template... Step 1/3: Clone ... OK (45s) Step 2/3: Analyze ... OK (120s) Step 3/3: Judge ... OK (30s) ✅ COMPLETED Output: {...} # Check status later $ poimen-cli status wf-abc123 Workflow ID: wf-abc123 Status: COMPLETED Started: 2025-01-31 15:04:05 Completed: 2025-01-31 15:05:00 # List templates $ poimen-cli template list Available Templates: - code-review-v1 - clone-analyze-v1 ``` **Implementation**: `cmd/cli/` (new directory) --- ### Entry Point 2: HTTP API (New) #### Route 1: Submit Workflow (Async) ```bash curl -X POST http://api.example.com/api/v1/workflows \ -H "Content-Type: application/json" \ -d @workflow.json ``` **Response**: ```json { "workflow_id": "wf-abc123", "status": "RUNNING", "polling_url": "/api/v1/workflows/wf-abc123/status" } ``` #### Check Status ```bash curl http://api.example.com/api/v1/workflows/wf-abc123/status ``` **Response**: ```json { "workflow_id": "wf-abc123", "status": "COMPLETED", "steps_completed": 3, "results": { "Clone": { "status": "COMPLETED", "output": {...} }, "Analyze": { "status": "COMPLETED", "output": {...} }, "Judge": { "status": "COMPLETED", "output": {...} } }, "final_output": {...} } ``` #### Route 2: Execute Template (Sync) ```bash curl -X POST http://api.example.com/api/v1/execute \ -H "Content-Type: application/json" \ -d '{ "template": "code-review-v1", "input": {"repo": "https://github.com/...", "branch": "feature/x"}, "timeout": "2m" }' ``` **Response** (immediate): ```json { "status": "COMPLETED", "results": {...}, "final_output": {...} } ``` **Implementation**: `cmd/api-server/` (new directory) --- ### Entry Point 3: Legacy CLI (Unchanged) ```bash # Old way still works, completely backward compatible $ go run ./cmd/starter \ --repo https://github.com/rockliang/poimen \ --remote file:///tmp/poimen-output \ --milestone T0 \ --planner-model ornith \ --judge-model ornith \ --implementer-model claude-sonnet-5 Workflow submitted: orch-poimen Status: RUNNING ``` **Implementation**: `cmd/starter/` (existing, unchanged) --- ## PART 3: ARCHITECTURE OVERVIEW ``` ┌─────────────────────────────────────────────────────────────────┐ │ THREE ENTRY POINTS │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ CLI (JSON file) HTTP API (JSON POST) Legacy CLI │ │ ┌──────────────┐ ┌──────────────┐ ┌─────────┐ │ │ │ poimen-cli │ │ /api/v1/ │ │ cmd/ │ │ │ │ submit │ │ workflows │ │ starter │ │ │ └──────┬───────┘ └──────┬───────┘ └────┬────┘ │ │ │ │ │ │ └─────────┼─────────────────────────┼─────────────────────┼────────┘ │ │ │ └─────────────────────────┼─────────────────────┘ ↓ ┌───────────────────────────┐ │ RoutingWorkflow │ │ (NEW - generic) │ │ │ │ 1. Parse WorkflowSpec │ │ 2. For each activity: │ │ - Resolve JSONPath │ │ - Execute activity │ │ - Handle errors │ │ - Store result │ │ 3. Return results │ └───────────┬───────────────┘ ↓ ┌───────────────────────────┐ │ Temporal Server │ │ │ │ Worker Pool: │ │ - Executes activities │ │ - Returns results │ │ - Retries with backoff │ └───────────────────────────┘ ``` --- ## PART 4: BEFORE vs AFTER ### Before (Hardcoded Workflows) ```bash # To change activity sequence: 1. Modify Go code (OrchestratorWorkflow in statemachine/) 2. Compile: go build ./cmd/worker 3. Push docker image 4. Deploy to K8s: kubectl set image deployment/poimen-worker 5. Wait for rollout Time: 5-10 minutes ``` **Workflow definition**: ```go // Hard to change, tight coupling to specific sequence if err := workflow.ExecuteActivity(ctx, "CloneRepoActivity", params).Get(ctx, nil); err != nil { return err } if err := workflow.ExecuteActivity(ctx, "PlanningActivity", params2).Get(ctx, nil); err != nil { return err } // ... more activities hardcoded ``` --- ### After (Dynamic Workflows) ```bash # To change activity sequence: 1. Edit workflow.json (define sequence) 2. Call API or CLI 3. Done Time: Seconds ``` **Workflow definition**: ```json { "activities": [ {"name": "Clone", "resource": "CloneRepoActivity", "parameters": {...}, "next": "Analyze"}, {"name": "Analyze", "resource": "AnalyzeCodeActivity", "parameters": {...}, "next": "Judge"}, {"name": "Judge", "resource": "JudgeActivity", "parameters": {...}, "end": true} ] } ``` --- ## PART 5: IMPLEMENTATION PHASES ### Phase 1: Routing Engine (Week 1) — 20-22 Hours **Goal**: Core RoutingWorkflow that executes JSON sequences dynamically **Files to Create**: ``` internal/routing/ ├─ types.go (2h) - WorkflowSpec, ActivityStep ├─ jsonpath_evaluator.go (3h) - ${step1.output.path} resolution ├─ execution_context.go (2h) - State tracking (vars, step results) ├─ activity_registry.go (2h) - Activity name → function mapping ├─ kmsvc_client.go (2-3h)- KMSvc queue polling (await-continue) └─ *_test.go (tests) statemachine/ ├─ routing_workflow.go (4-5h)- ⭐ Main workflow logic └─ routing_workflow_test.go (2h) - Tests cmd/worker/main.go: Register RoutingWorkflow (1h) ``` **Deliverable**: RoutingWorkflow executes activity sequences with: - ✅ Sequential execution (A → B → C) - ✅ Await-continue pattern (KMSvc queue polling) - ✅ Retry with exponential backoff - ✅ Error catch blocks - ✅ JSONPath parameter resolution **Risk**: Low-Medium (KMSvc queue polling adds complexity) --- ### Phase 2: API Server (Week 2) — 12-14 Hours **Goal**: HTTP endpoints for workflow submission and status polling **Files to Create**: ``` cmd/api-server/ ├─ main.go (2h) - Server bootstrap ├─ handlers.go (4h) - Routes: /workflows, /execute, status ├─ validation.go (2h) - Validate WorkflowSpec ├─ template_loader.go (2-3h)- Load predefined templates └─ *_test.go (2h) - Tests ``` **Deliverable**: - `POST /api/v1/workflows` — Submit workflow (async) - `GET /api/v1/workflows/{id}/status` — Check status - `POST /api/v1/execute` — Execute template (sync) **Risk**: Low (isolated, no breaking changes) --- ### Phase 3: CLI (Week 2) — 8-10 Hours **Goal**: Command-line interface for workflow submission **Files to Create**: ``` cmd/cli/ ├─ main.go (2h) - Entry point, command routing └─ commands/ ├─ submit.go (2h) - poimen-cli submit workflow.json ├─ status.go (2h) - poimen-cli status wf-id ├─ template.go (1h) - poimen-cli template list/show ├─ execute.go (1h) - poimen-cli execute template └─ history.go (1h) - poimen-cli history wf-id ``` **Deliverable**: - `poimen-cli submit workflow.json [--wait] [--watch]` - `poimen-cli status wf-id [--wait]` - `poimen-cli execute template name --input-file input.json` - `poimen-cli template list/show` **Risk**: Low --- ### Phase 4: Kubernetes Integration (Week 3) — 5 Hours **Goal**: Deploy api-server to K8s, expose via ingress **Files to Create**: ``` k8s/ ├─ api-server-deployment.yaml (2h) - 2 replicas, Temporal connection ├─ api-server-service.yaml (1h) - Port 8080 └─ api-server-networkpolicy.yaml(1h) - Allow ingress traffic homelab-frontend/k8s/ └─ gateway-routes.yaml (1h) - Route /api/v1/workflows to api-server ``` **Deliverable**: API exposed at `https://api.riotpiao.com/api/v1/workflows` **Risk**: Medium (network policy changes) --- ### Phase 5: Documentation (Week 4) — 7 Hours **Files to Create**: ``` docs/ ├─ API.md (2h) - Complete API reference ├─ CLI.md (2h) - CLI commands + examples ├─ TEMPLATES.md (1h) - How to create/manage templates ├─ MIGRATION.md (1h) - Migration from cmd/starter └─ examples/ ├─ sequential-workflow.json ├─ retry-workflow.json └─ template-execute.json ``` **Deliverable**: Complete documentation with working examples **Risk**: Low --- ### Timeline Summary ``` Phase 1: Routing Engine 20-22 hours (Week 1) Phase 2: API Server 12-14 hours (Week 2) Phase 3: CLI 8-10 hours (Week 2) Phase 4: K8s Integration 5 hours (Week 3) Phase 5: Documentation 7 hours (Week 4) ───────────────────────────────────────────────── TOTAL 52-58 hours (~1.5 weeks) Team: 2-3 engineers Duration: 4 weeks (with overlapping phases) ``` --- ## PART 6: CRITICAL KMSVC QUESTIONS These answers **block implementation** of Phase 1: ### Q1: What is KMSvc? - Is it Kafka? Redis? AWS SQS? Custom system? - What Go client library should we use? - Example: `github.com/segmentio/kafka-go`? ### Q2: Message Format - Is it JSON? - What fields are required? (e.g., `correlation_id`, `result`, `status`?) - Example: ```json { "correlation_id": "xyz123", "status": "completed", "result": {...} } ``` ### Q3: Correlation ID Management - Who generates the correlation_id? (LaunchJobActivity? Or RoutingWorkflow?) - Is it returned in LaunchJobActivity output? - Example: LaunchJobActivity returns: ```json { "correlation_id": "abc123-generated-by-activity", "job_id": "job-123" } ``` ### Q4: Queue Polling Strategy - Active polling loop (pull messages every N seconds)? OR - Temporal Signal (push-based, external system sends signal to workflow)? - If polling: What interval? (1s, 5s, 10s?) ### Q5: Message Timeout Behavior - If no message arrives within timeout (e.g., 5m), what happens? - Option A: Jump to catch block (error handler) - Option B: Fail entire workflow - Option C: Infinite wait (only human intervention can stop) ### Q6: Consumer Group Strategy - Single shared consumer (all workflows share one connection)? - Per-workflow consumer (each workflow gets own connection)? - Dead-letter queue for unmatched messages? --- ## PART 7: KEY DESIGN DECISIONS ### Decision 1: Three Patterns (Sequential, Await-Task-Complete, Retry) **Rationale**: Covers 90% of use cases. Retry critical for production. Await-Task-Complete enables long-running jobs. **Alternative Rejected**: Just Sequential (too limited) ### Decision 2: KMSvc Queue for Await-Task-Complete **Rationale**: Decouples external systems from Temporal. External systems only need queue client, not Temporal SDK. **Alternative Rejected**: Temporal callback/signal (tight coupling) ### Decision 3: JSONPath for Parameters **Rationale**: Standard notation (AWS Step Functions, Kubernetes). Supports nested refs and arrays. **Alternative Rejected**: Go templating (overkill, security risks) ### Decision 4: RoutingWorkflow (not Activity) **Rationale**: Workflows can call activities with retries. Durable execution + replay guarantee. **Alternative Rejected**: RoutingActivity (can't call other activities, no retries) ### Decision 5: Static Activity Registry **Rationale**: Type-safe, simpler. Worker pod must compile all activities anyway. **Alternative Rejected**: Dynamic registration (complex, unsafe) ### Decision 6: Keep cmd/starter Forever **Rationale**: No need to break existing deployments. Let customers choose. **Alternative Rejected**: Deprecate (breaking change) ### Decision 7: Shared api-server (2 replicas) **Rationale**: Simpler, resource-efficient, easier to scale. **Alternative Rejected**: Per-namespace servers (overkill) --- ## PART 8: RISKS & MITIGATIONS | Risk | Severity | Mitigation | |------|----------|-----------| | JSONPath evaluation bugs | Medium | Unit tests (>90% coverage), property-based testing | | **KMSvc queue reliability** | **High** | Dead-letter queue, timeouts, persistence checks | | **Correlation ID mismatch** | **Medium** | Strict matching, schema validation, detailed logging | | **Consumer group conflicts** | **Medium** | Clear strategy, rebalancing, message redelivery | | Activity registry mismatch | Low | Validate at spec submission time | | Temporal network issues | Low | Retry + timeout in api-server | | Cache staleness (api-server) | Low | 24h TTL, event-driven invalidation | | Performance regression | Low | Measure baseline: target <200ms submit, <100ms poll | **Total Risk Score**: Medium-High (up from Low-Medium due to KMSvc) --- ## PART 9: SUCCESS CRITERIA ### Functional ✅ - RoutingWorkflow executes activity sequences dynamically - JSONPath parameters resolve correctly (nested, arrays) - Error catch blocks jump to handlers - Retry logic with exponential backoff (1s → 2s → 4s) - API validates WorkflowSpec (no dangling refs) - CLI reads JSON (no hardcoded flags) - Status polling returns step-by-step results - Backward compatible (cmd/starter unchanged) ### Non-Functional ✅ - Performance: <200ms workflow submit, <100ms poll - Uptime: 99.9% (2 replicas, rolling updates) - Test coverage: >90% - Documentation: Complete + 3 working examples --- ## PART 10: BACKWARD COMPATIBILITY ### What Stays Unchanged ✅ - `cmd/starter/` (old CLI works exactly as before) - All existing activities - All existing workflows (OrchestratorWorkflow, TaskUnitWorkflow) - Temporal worker registration ### What's New ✅ - `cmd/cli/` (new CLI) - `cmd/api-server/` (new HTTP API) - `internal/routing/` (new routing engine) - `statemachine/routing_workflow.go` (new workflow) ### Migration Path ✅ - Customers can use OLD way or NEW way - No forced upgrades - No breaking changes --- ## PART 11: FILES SUMMARY ### New Files (Phase 1-5) **Routing Engine** (Phase 1): ``` internal/routing/types.go internal/routing/jsonpath_evaluator.go internal/routing/execution_context.go internal/routing/activity_registry.go internal/routing/kmsvc_client.go statemachine/routing_workflow.go ``` **API Server** (Phase 2): ``` cmd/api-server/main.go cmd/api-server/handlers.go cmd/api-server/validation.go cmd/api-server/template_loader.go ``` **CLI** (Phase 3): ``` cmd/cli/main.go cmd/cli/commands/submit.go cmd/cli/commands/status.go cmd/cli/commands/template.go cmd/cli/commands/execute.go cmd/cli/commands/history.go ``` **K8s** (Phase 4): ``` k8s/api-server-deployment.yaml k8s/api-server-service.yaml k8s/api-server-networkpolicy.yaml homelab-frontend/k8s/gateway-routes.yaml ``` **Docs** (Phase 5): ``` docs/API.md docs/CLI.md docs/TEMPLATES.md docs/MIGRATION.md docs/examples/*.json ``` ### Modified Files - `cmd/worker/main.go` (register RoutingWorkflow) - `homelab-frontend/k8s/network-policy.yaml` (already done ✅) --- ## PART 12: COMPARISON TABLE | Aspect | Before (Hardcoded) | After (Dynamic) | |--------|---|---| | **Workflow Definition** | Go code (OrchestratorWorkflow) | JSON (customer-specified) | | **Sequencing** | Fixed (Clone→Plan→Impl→Judge) | Any (customer defines in JSON) | | **Parameters** | Direct: `in.RemoteURL` | JSONPath: `${input.repo}` | | **Change Time** | 5-10 min (recompile+deploy) | Seconds (API call) | | **Entry Points** | CLI only (hardcoded flags) | CLI + API + Legacy CLI | | **Error Handling** | Try-catch (per-activity) | Catch blocks (conditional jumps) | | **Retry Policy** | Temporal default | Configurable per-activity | | **Backward Compat** | N/A | 100% (cmd/starter unchanged) | | **Code Coupling** | Tight (to specific sequence) | Loose (generic RoutingWorkflow) | --- ## PART 13: APPROVAL CHECKLIST ### Must Answer (Blocks Phase 1) - [ ] Q1: What is KMSvc system? - [ ] Q2: Message format for KMSvc? - [ ] Q3: Who generates correlation_id? - [ ] Q4: Polling strategy (active or signal)? - [ ] Q5: Timeout behavior? - [ ] Q6: Consumer group strategy? ### Should Approve - [ ] Timeline OK? (52-58 hours) - [ ] Architecture OK? (RoutingWorkflow + 3 entry points) - [ ] Risk level acceptable? (Medium-High) - [ ] Backward compatibility OK? (Keep cmd/starter) ### Sign-Off - [ ] Tech Lead approval - [ ] Stakeholder sign-off - [ ] Ready to start Phase 1 --- ## PART 14: NEXT STEPS 1. **Review This Document** (1-2 hours) - Read Parts 1-7 (understanding) - Read Parts 8-9 (risks & success) - Read Part 13 (approval checklist) 2. **Answer Critical Questions** (Q1-Q6) - Email or Slack responses required - Needed before Phase 1 can start 3. **Stakeholder Approval** - Architecture review meeting - Risk acknowledgment - Sign-off on timeline 4. **Start Implementation** - Week 1: Phase 1 (RoutingWorkflow) - Week 2: Phase 2 (API) + Phase 3 (CLI) - Week 3: Phase 4 (K8s deploy) - Week 4: Phase 5 (Docs) --- ## APPENDIX: EXAMPLE WORKFLOW JSON ```json { "name": "code-review-pipeline", "input": { "repo": "https://github.com/rockliang/poimen", "branch": "feature/new-cli" }, "activities": [ { "name": "Clone", "resource": "CloneRepoActivity", "parameters": { "repo": "${input.repo}", "branch": "${input.branch}" }, "timeout": "5m", "next": "Analyze" }, { "name": "Analyze", "resource": "AnalyzeCodeActivity", "parameters": { "path": "${Clone.output.path}" }, "timeout": "10m", "retry": { "maxAttempts": 3, "backoffRate": 2.0, "initialInterval": "1s" }, "catch": [ { "errorEquals": ["Timeout"], "next": "HandleTimeout" } ], "next": "Judge" }, { "name": "Judge", "resource": "JudgeActivity", "parameters": { "code": "${Analyze.output.code}", "analysis": "${Analyze.output.analysis}" }, "timeout": "5m", "end": true }, { "name": "HandleTimeout", "type": "pass", "result": { "status": "failed", "reason": "Analysis timed out after 10m" }, "end": true } ] } ``` --- ## APPENDIX: CLI USAGE EXAMPLES ```bash # Submit workflow from JSON file $ poimen-cli submit workflow.json Workflow submitted: wf-abc123 Poll status: poimen-cli status wf-abc123 # Submit and wait for results $ poimen-cli submit workflow.json --wait Waiting for workflow to complete... ✅ Workflow completed Output: { "verdict": "approved", "score": 0.95, "issues": [] } # Check status $ poimen-cli status wf-abc123 Workflow ID: wf-abc123 Status: COMPLETED Started: 2025-01-31 15:04:05 Completed: 2025-01-31 15:05:00 # Execute template immediately (sync) $ poimen-cli execute template code-review-v1 --input-file input.json Executing template: code-review-v1 [15:04:05] Step 1/3: Clone ... OK (45s) [15:04:50] Step 2/3: Analyze ... OK (120s) [15:05:50] Step 3/3: Judge ... OK (30s) ✅ COMPLETED Output: {...} # List templates $ poimen-cli template list Available Templates: - code-review-v1 - clone-analyze-v1 - simple-test-v1 # Show template $ poimen-cli template show code-review-v1 Template: code-review-v1 Activities: 3 1. Clone (CloneRepoActivity) Timeout: 5m 2. Analyze (AnalyzeCodeActivity) Timeout: 10m Retry: 3 attempts, backoff 2.0 3. Judge (JudgeActivity) Timeout: 5m ``` --- **END OF MASTER REVIEW DOCUMENT** For detailed technical documentation, see: - SERVICE_INTEGRATION.md (patterns + API spec) - IMPLEMENTATION_PLAN.md (phase breakdown) - CLI_DESIGN.md (CLI implementation) - FINAL_COMPREHENSIVE_DESIGN.md (1-2 hour read)