diff --git a/CLI_DESIGN.md b/CLI_DESIGN.md new file mode 100644 index 0000000..702641d --- /dev/null +++ b/CLI_DESIGN.md @@ -0,0 +1,81 @@ +# Poimen CLI — Dynamic JSON-Based Design + +## Overview + +New CLI (`cmd/cli`) that accepts JSON WorkflowSpecs and dynamically understands activity sequences—same mechanism as HTTP API, but command-line interface. + +## Key Commands + +```bash +# Submit workflow from JSON +poimen-cli submit workflow.json +poimen-cli submit workflow.json --wait +poimen-cli submit workflow.json --watch + +# Check status +poimen-cli status wf-abc123 +poimen-cli status wf-abc123 --wait + +# Template management +poimen-cli template list +poimen-cli template show code-review-v1 + +# Execute immediately (sync) +poimen-cli execute template code-review-v1 --input-file input.json +``` + +## Workflow JSON Example + +```json +{ + "name": "code-review-pipeline", + "input": {"repo": "https://github.com/..."}, + "activities": [ + { + "name": "Clone", + "resource": "CloneRepoActivity", + "parameters": {"repo": "${input.repo}"}, + "timeout": "5m", + "next": "Analyze" + }, + { + "name": "Analyze", + "resource": "AnalyzeCodeActivity", + "parameters": {"path": "${Clone.output.path}"}, + "timeout": "10m", + "retry": {"maxAttempts": 3}, + "catch": [{"errorEquals": ["Timeout"], "next": "HandleTimeout"}], + "next": "Judge" + } + ] +} +``` + +## File Structure + +``` +cmd/ + ├─ cli/ (NEW) + │ ├─ main.go + │ └─ commands/ + │ ├─ submit.go + │ ├─ status.go + │ ├─ template.go + │ ├─ execute.go + │ └─ history.go + ├─ starter/ (OLD: keep for backward compat) + └─ worker/ +``` + +## Three Entry Points (Same Core) + +| Interface | Endpoint | Input | Output | +|-----------|----------|-------|--------| +| **CLI** | `submit workflow.json` | JSON file | workflow_id | +| **CLI** | `execute template T` | Template name | Results (sync) | +| **API** | `POST /api/v1/workflows` | JSON body | workflow_id | +| **API** | `POST /api/v1/execute` | JSON body | Results (sync) | + +All use same `RoutingWorkflow` internally—just different interfaces. + +See CLI_DESIGN.md in repo for full implementation details. diff --git a/DESIGN_MASTER_REVIEW.md b/DESIGN_MASTER_REVIEW.md new file mode 100644 index 0000000..9fef2d2 --- /dev/null +++ b/DESIGN_MASTER_REVIEW.md @@ -0,0 +1,889 @@ +# Poimen Service Integration & Dynamic Workflows — MASTER REVIEW DOCUMENT + +**Status**: Ready for Approval +**Version**: 1.0 +**Date**: 2025-01-31 +**Duration**: 52-58 hours (4 weeks) + +--- + +## EXECUTIVE SUMMARY (5 MINUTES) + +### Problem +Current workflows are **hardcoded in Go**. Changing activity sequences requires: +- Code modification +- Recompilation +- K8s deployment +- Pod restart +- **Total: 5-10 minutes** + +### Solution +New **RoutingWorkflow** reads JSON WorkflowSpec and executes dynamically: +- **Three entry points**: CLI + HTTP API + Legacy CLI (backward compatible) +- **Three patterns**: Sequential, Await-Task-Complete (KMSvc queue), Retry with error handling +- **JSONPath parameters**: `${step1.output.path}` instead of hardcoded values +- **Result**: Workflow changes in **seconds** (API call only) + +### Impact +✅ Instant changes | ✅ Three interfaces | ✅ Full compatibility | ❌ +KMSvc complexity + +--- + +## PART 1: THE THREE PATTERNS + +### Pattern 1: Sequential (A → B → C) + +Activities execute in sequence. Each step passes output to next via JSONPath. + +```json +{ + "activities": [ + { + "name": "Clone", + "resource": "CloneRepoActivity", + "parameters": {"repo": "${input.repo}"}, + "timeout": "5m", + "next": "Analyze" + }, + { + "name": "Analyze", + "resource": "AnalyzeCodeActivity", + "parameters": {"path": "${Clone.output.path}"}, + "timeout": "10m", + "next": "Judge" + }, + { + "name": "Judge", + "resource": "JudgeActivity", + "parameters": {"code": "${Analyze.output.code}"}, + "timeout": "5m", + "end": true + } + ] +} +``` + +**Execution Flow**: +``` +Clone ──(output: {path: /tmp/repo})──> Analyze ──(output: {code: ...})──> Judge ──(final result) +``` + +**Use Case**: Code review pipeline (clone → analyze → judge) + +--- + +### Pattern 2: Await-Task-Complete (Launch → Poll Queue → Process) + +Launch long-running job, poll KMSvc queue for result with correlation ID matching, then proceed. + +```json +{ + "name": "LaunchJob", + "type": "awaitTaskComplete", + "resource": "LaunchJobActivity", + "parameters": {"job_id": "job-123"}, + "queue": "job-completions", + "correlation_key": "${LaunchJob.output.correlation_id}", + "timeout": "5m", + "next": "ProcessResult" +} +``` + +**Execution Flow**: +``` +LaunchJobActivity ──(returns {correlation_id: "xyz"})──> Poll Queue "job-completions" + ↓ + [External system completes job] + [Publishes {correlation_id: "xyz", result: ...}] + ↓ + [Workflow receives message] + [Proceeds to ProcessResult] +``` + +**Use Case**: Long-running batch jobs (model training, data processing) + +--- + +### Pattern 3: Retry with Error Handling + +Activity retries N times with exponential backoff. On failure, jumps to catch block. + +```json +{ + "name": "Generate", + "resource": "ImplementerActivity", + "parameters": {"task": "${input.task}"}, + "timeout": "30m", + "retry": { + "maxAttempts": 3, + "backoffRate": 2.0, + "initialInterval": "1s" + }, + "catch": [ + {"errorEquals": ["Timeout"], "next": "HandleTimeout"} + ], + "next": "Verify" +} +``` + +**Execution Flow**: +``` +Generate (attempt 1) ──TIMEOUT──> [Wait 1s] +Generate (attempt 2) ──TIMEOUT──> [Wait 2s] +Generate (attempt 3) ──TIMEOUT──> [Wait 4s] +Max retries exceeded ──> Jump to HandleTimeout +``` + +**Use Case**: Code generation (retry on timeout, handle gracefully) + +--- + +## PART 2: THE THREE ENTRY POINTS + +### Entry Point 1: CLI (New Command-Line) + +```bash +# Submit workflow from JSON file +$ poimen-cli submit workflow.json +Workflow submitted: wf-abc123 +Poll status: poimen-cli status wf-abc123 + +# Submit and wait for results +$ poimen-cli submit workflow.json --wait +Waiting for workflow to complete... +✅ Workflow completed +Output: {...} + +# Execute template immediately (sync, no polling) +$ poimen-cli execute template code-review-v1 --input-file input.json +Executing template... +Step 1/3: Clone ... OK (45s) +Step 2/3: Analyze ... OK (120s) +Step 3/3: Judge ... OK (30s) +✅ COMPLETED +Output: {...} + +# Check status later +$ poimen-cli status wf-abc123 +Workflow ID: wf-abc123 +Status: COMPLETED +Started: 2025-01-31 15:04:05 +Completed: 2025-01-31 15:05:00 + +# List templates +$ poimen-cli template list +Available Templates: + - code-review-v1 + - clone-analyze-v1 +``` + +**Implementation**: `cmd/cli/` (new directory) + +--- + +### Entry Point 2: HTTP API (New) + +#### Route 1: Submit Workflow (Async) +```bash +curl -X POST http://api.example.com/api/v1/workflows \ + -H "Content-Type: application/json" \ + -d @workflow.json +``` + +**Response**: +```json +{ + "workflow_id": "wf-abc123", + "status": "RUNNING", + "polling_url": "/api/v1/workflows/wf-abc123/status" +} +``` + +#### Check Status +```bash +curl http://api.example.com/api/v1/workflows/wf-abc123/status +``` + +**Response**: +```json +{ + "workflow_id": "wf-abc123", + "status": "COMPLETED", + "steps_completed": 3, + "results": { + "Clone": { "status": "COMPLETED", "output": {...} }, + "Analyze": { "status": "COMPLETED", "output": {...} }, + "Judge": { "status": "COMPLETED", "output": {...} } + }, + "final_output": {...} +} +``` + +#### Route 2: Execute Template (Sync) +```bash +curl -X POST http://api.example.com/api/v1/execute \ + -H "Content-Type: application/json" \ + -d '{ + "template": "code-review-v1", + "input": {"repo": "https://github.com/...", "branch": "feature/x"}, + "timeout": "2m" + }' +``` + +**Response** (immediate): +```json +{ + "status": "COMPLETED", + "results": {...}, + "final_output": {...} +} +``` + +**Implementation**: `cmd/api-server/` (new directory) + +--- + +### Entry Point 3: Legacy CLI (Unchanged) + +```bash +# Old way still works, completely backward compatible +$ go run ./cmd/starter \ + --repo https://github.com/rockliang/poimen \ + --remote file:///tmp/poimen-output \ + --milestone T0 \ + --planner-model ornith \ + --judge-model ornith \ + --implementer-model claude-sonnet-5 + +Workflow submitted: orch-poimen +Status: RUNNING +``` + +**Implementation**: `cmd/starter/` (existing, unchanged) + +--- + +## PART 3: ARCHITECTURE OVERVIEW + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ THREE ENTRY POINTS │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ CLI (JSON file) HTTP API (JSON POST) Legacy CLI │ +│ ┌──────────────┐ ┌──────────────┐ ┌─────────┐ │ +│ │ poimen-cli │ │ /api/v1/ │ │ cmd/ │ │ +│ │ submit │ │ workflows │ │ starter │ │ +│ └──────┬───────┘ └──────┬───────┘ └────┬────┘ │ +│ │ │ │ │ +└─────────┼─────────────────────────┼─────────────────────┼────────┘ + │ │ │ + └─────────────────────────┼─────────────────────┘ + ↓ + ┌───────────────────────────┐ + │ RoutingWorkflow │ + │ (NEW - generic) │ + │ │ + │ 1. Parse WorkflowSpec │ + │ 2. For each activity: │ + │ - Resolve JSONPath │ + │ - Execute activity │ + │ - Handle errors │ + │ - Store result │ + │ 3. Return results │ + └───────────┬───────────────┘ + ↓ + ┌───────────────────────────┐ + │ Temporal Server │ + │ │ + │ Worker Pool: │ + │ - Executes activities │ + │ - Returns results │ + │ - Retries with backoff │ + └───────────────────────────┘ +``` + +--- + +## PART 4: BEFORE vs AFTER + +### Before (Hardcoded Workflows) + +```bash +# To change activity sequence: +1. Modify Go code (OrchestratorWorkflow in statemachine/) +2. Compile: go build ./cmd/worker +3. Push docker image +4. Deploy to K8s: kubectl set image deployment/poimen-worker +5. Wait for rollout +Time: 5-10 minutes +``` + +**Workflow definition**: +```go +// Hard to change, tight coupling to specific sequence +if err := workflow.ExecuteActivity(ctx, "CloneRepoActivity", params).Get(ctx, nil); err != nil { + return err +} +if err := workflow.ExecuteActivity(ctx, "PlanningActivity", params2).Get(ctx, nil); err != nil { + return err +} +// ... more activities hardcoded +``` + +--- + +### After (Dynamic Workflows) + +```bash +# To change activity sequence: +1. Edit workflow.json (define sequence) +2. Call API or CLI +3. Done +Time: Seconds +``` + +**Workflow definition**: +```json +{ + "activities": [ + {"name": "Clone", "resource": "CloneRepoActivity", "parameters": {...}, "next": "Analyze"}, + {"name": "Analyze", "resource": "AnalyzeCodeActivity", "parameters": {...}, "next": "Judge"}, + {"name": "Judge", "resource": "JudgeActivity", "parameters": {...}, "end": true} + ] +} +``` + +--- + +## PART 5: IMPLEMENTATION PHASES + +### Phase 1: Routing Engine (Week 1) — 20-22 Hours + +**Goal**: Core RoutingWorkflow that executes JSON sequences dynamically + +**Files to Create**: +``` +internal/routing/ + ├─ types.go (2h) - WorkflowSpec, ActivityStep + ├─ jsonpath_evaluator.go (3h) - ${step1.output.path} resolution + ├─ execution_context.go (2h) - State tracking (vars, step results) + ├─ activity_registry.go (2h) - Activity name → function mapping + ├─ kmsvc_client.go (2-3h)- KMSvc queue polling (await-continue) + └─ *_test.go (tests) + +statemachine/ + ├─ routing_workflow.go (4-5h)- ⭐ Main workflow logic + └─ routing_workflow_test.go (2h) - Tests + +cmd/worker/main.go: Register RoutingWorkflow (1h) +``` + +**Deliverable**: RoutingWorkflow executes activity sequences with: +- ✅ Sequential execution (A → B → C) +- ✅ Await-continue pattern (KMSvc queue polling) +- ✅ Retry with exponential backoff +- ✅ Error catch blocks +- ✅ JSONPath parameter resolution + +**Risk**: Low-Medium (KMSvc queue polling adds complexity) + +--- + +### Phase 2: API Server (Week 2) — 12-14 Hours + +**Goal**: HTTP endpoints for workflow submission and status polling + +**Files to Create**: +``` +cmd/api-server/ + ├─ main.go (2h) - Server bootstrap + ├─ handlers.go (4h) - Routes: /workflows, /execute, status + ├─ validation.go (2h) - Validate WorkflowSpec + ├─ template_loader.go (2-3h)- Load predefined templates + └─ *_test.go (2h) - Tests +``` + +**Deliverable**: +- `POST /api/v1/workflows` — Submit workflow (async) +- `GET /api/v1/workflows/{id}/status` — Check status +- `POST /api/v1/execute` — Execute template (sync) + +**Risk**: Low (isolated, no breaking changes) + +--- + +### Phase 3: CLI (Week 2) — 8-10 Hours + +**Goal**: Command-line interface for workflow submission + +**Files to Create**: +``` +cmd/cli/ + ├─ main.go (2h) - Entry point, command routing + └─ commands/ + ├─ submit.go (2h) - poimen-cli submit workflow.json + ├─ status.go (2h) - poimen-cli status wf-id + ├─ template.go (1h) - poimen-cli template list/show + ├─ execute.go (1h) - poimen-cli execute template + └─ history.go (1h) - poimen-cli history wf-id +``` + +**Deliverable**: +- `poimen-cli submit workflow.json [--wait] [--watch]` +- `poimen-cli status wf-id [--wait]` +- `poimen-cli execute template name --input-file input.json` +- `poimen-cli template list/show` + +**Risk**: Low + +--- + +### Phase 4: Kubernetes Integration (Week 3) — 5 Hours + +**Goal**: Deploy api-server to K8s, expose via ingress + +**Files to Create**: +``` +k8s/ + ├─ api-server-deployment.yaml (2h) - 2 replicas, Temporal connection + ├─ api-server-service.yaml (1h) - Port 8080 + └─ api-server-networkpolicy.yaml(1h) - Allow ingress traffic + +homelab-frontend/k8s/ + └─ gateway-routes.yaml (1h) - Route /api/v1/workflows to api-server +``` + +**Deliverable**: API exposed at `https://api.riotpiao.com/api/v1/workflows` + +**Risk**: Medium (network policy changes) + +--- + +### Phase 5: Documentation (Week 4) — 7 Hours + +**Files to Create**: +``` +docs/ + ├─ API.md (2h) - Complete API reference + ├─ CLI.md (2h) - CLI commands + examples + ├─ TEMPLATES.md (1h) - How to create/manage templates + ├─ MIGRATION.md (1h) - Migration from cmd/starter + └─ examples/ + ├─ sequential-workflow.json + ├─ retry-workflow.json + └─ template-execute.json +``` + +**Deliverable**: Complete documentation with working examples + +**Risk**: Low + +--- + +### Timeline Summary + +``` +Phase 1: Routing Engine 20-22 hours (Week 1) +Phase 2: API Server 12-14 hours (Week 2) +Phase 3: CLI 8-10 hours (Week 2) +Phase 4: K8s Integration 5 hours (Week 3) +Phase 5: Documentation 7 hours (Week 4) +───────────────────────────────────────────────── +TOTAL 52-58 hours (~1.5 weeks) + +Team: 2-3 engineers +Duration: 4 weeks (with overlapping phases) +``` + +--- + +## PART 6: CRITICAL KMSVC QUESTIONS + +These answers **block implementation** of Phase 1: + +### Q1: What is KMSvc? +- Is it Kafka? Redis? AWS SQS? Custom system? +- What Go client library should we use? +- Example: `github.com/segmentio/kafka-go`? + +### Q2: Message Format +- Is it JSON? +- What fields are required? (e.g., `correlation_id`, `result`, `status`?) +- Example: +```json +{ + "correlation_id": "xyz123", + "status": "completed", + "result": {...} +} +``` + +### Q3: Correlation ID Management +- Who generates the correlation_id? (LaunchJobActivity? Or RoutingWorkflow?) +- Is it returned in LaunchJobActivity output? +- Example: LaunchJobActivity returns: +```json +{ + "correlation_id": "abc123-generated-by-activity", + "job_id": "job-123" +} +``` + +### Q4: Queue Polling Strategy +- Active polling loop (pull messages every N seconds)? OR +- Temporal Signal (push-based, external system sends signal to workflow)? +- If polling: What interval? (1s, 5s, 10s?) + +### Q5: Message Timeout Behavior +- If no message arrives within timeout (e.g., 5m), what happens? + - Option A: Jump to catch block (error handler) + - Option B: Fail entire workflow + - Option C: Infinite wait (only human intervention can stop) + +### Q6: Consumer Group Strategy +- Single shared consumer (all workflows share one connection)? +- Per-workflow consumer (each workflow gets own connection)? +- Dead-letter queue for unmatched messages? + +--- + +## PART 7: KEY DESIGN DECISIONS + +### Decision 1: Three Patterns (Sequential, Await-Task-Complete, Retry) +**Rationale**: Covers 90% of use cases. Retry critical for production. Await-Task-Complete enables long-running jobs. +**Alternative Rejected**: Just Sequential (too limited) + +### Decision 2: KMSvc Queue for Await-Task-Complete +**Rationale**: Decouples external systems from Temporal. External systems only need queue client, not Temporal SDK. +**Alternative Rejected**: Temporal callback/signal (tight coupling) + +### Decision 3: JSONPath for Parameters +**Rationale**: Standard notation (AWS Step Functions, Kubernetes). Supports nested refs and arrays. +**Alternative Rejected**: Go templating (overkill, security risks) + +### Decision 4: RoutingWorkflow (not Activity) +**Rationale**: Workflows can call activities with retries. Durable execution + replay guarantee. +**Alternative Rejected**: RoutingActivity (can't call other activities, no retries) + +### Decision 5: Static Activity Registry +**Rationale**: Type-safe, simpler. Worker pod must compile all activities anyway. +**Alternative Rejected**: Dynamic registration (complex, unsafe) + +### Decision 6: Keep cmd/starter Forever +**Rationale**: No need to break existing deployments. Let customers choose. +**Alternative Rejected**: Deprecate (breaking change) + +### Decision 7: Shared api-server (2 replicas) +**Rationale**: Simpler, resource-efficient, easier to scale. +**Alternative Rejected**: Per-namespace servers (overkill) + +--- + +## PART 8: RISKS & MITIGATIONS + +| Risk | Severity | Mitigation | +|------|----------|-----------| +| JSONPath evaluation bugs | Medium | Unit tests (>90% coverage), property-based testing | +| **KMSvc queue reliability** | **High** | Dead-letter queue, timeouts, persistence checks | +| **Correlation ID mismatch** | **Medium** | Strict matching, schema validation, detailed logging | +| **Consumer group conflicts** | **Medium** | Clear strategy, rebalancing, message redelivery | +| Activity registry mismatch | Low | Validate at spec submission time | +| Temporal network issues | Low | Retry + timeout in api-server | +| Cache staleness (api-server) | Low | 24h TTL, event-driven invalidation | +| Performance regression | Low | Measure baseline: target <200ms submit, <100ms poll | + +**Total Risk Score**: Medium-High (up from Low-Medium due to KMSvc) + +--- + +## PART 9: SUCCESS CRITERIA + +### Functional ✅ +- RoutingWorkflow executes activity sequences dynamically +- JSONPath parameters resolve correctly (nested, arrays) +- Error catch blocks jump to handlers +- Retry logic with exponential backoff (1s → 2s → 4s) +- API validates WorkflowSpec (no dangling refs) +- CLI reads JSON (no hardcoded flags) +- Status polling returns step-by-step results +- Backward compatible (cmd/starter unchanged) + +### Non-Functional ✅ +- Performance: <200ms workflow submit, <100ms poll +- Uptime: 99.9% (2 replicas, rolling updates) +- Test coverage: >90% +- Documentation: Complete + 3 working examples + +--- + +## PART 10: BACKWARD COMPATIBILITY + +### What Stays Unchanged ✅ +- `cmd/starter/` (old CLI works exactly as before) +- All existing activities +- All existing workflows (OrchestratorWorkflow, TaskUnitWorkflow) +- Temporal worker registration + +### What's New ✅ +- `cmd/cli/` (new CLI) +- `cmd/api-server/` (new HTTP API) +- `internal/routing/` (new routing engine) +- `statemachine/routing_workflow.go` (new workflow) + +### Migration Path ✅ +- Customers can use OLD way or NEW way +- No forced upgrades +- No breaking changes + +--- + +## PART 11: FILES SUMMARY + +### New Files (Phase 1-5) + +**Routing Engine** (Phase 1): +``` +internal/routing/types.go +internal/routing/jsonpath_evaluator.go +internal/routing/execution_context.go +internal/routing/activity_registry.go +internal/routing/kmsvc_client.go +statemachine/routing_workflow.go +``` + +**API Server** (Phase 2): +``` +cmd/api-server/main.go +cmd/api-server/handlers.go +cmd/api-server/validation.go +cmd/api-server/template_loader.go +``` + +**CLI** (Phase 3): +``` +cmd/cli/main.go +cmd/cli/commands/submit.go +cmd/cli/commands/status.go +cmd/cli/commands/template.go +cmd/cli/commands/execute.go +cmd/cli/commands/history.go +``` + +**K8s** (Phase 4): +``` +k8s/api-server-deployment.yaml +k8s/api-server-service.yaml +k8s/api-server-networkpolicy.yaml +homelab-frontend/k8s/gateway-routes.yaml +``` + +**Docs** (Phase 5): +``` +docs/API.md +docs/CLI.md +docs/TEMPLATES.md +docs/MIGRATION.md +docs/examples/*.json +``` + +### Modified Files +- `cmd/worker/main.go` (register RoutingWorkflow) +- `homelab-frontend/k8s/network-policy.yaml` (already done ✅) + +--- + +## PART 12: COMPARISON TABLE + +| Aspect | Before (Hardcoded) | After (Dynamic) | +|--------|---|---| +| **Workflow Definition** | Go code (OrchestratorWorkflow) | JSON (customer-specified) | +| **Sequencing** | Fixed (Clone→Plan→Impl→Judge) | Any (customer defines in JSON) | +| **Parameters** | Direct: `in.RemoteURL` | JSONPath: `${input.repo}` | +| **Change Time** | 5-10 min (recompile+deploy) | Seconds (API call) | +| **Entry Points** | CLI only (hardcoded flags) | CLI + API + Legacy CLI | +| **Error Handling** | Try-catch (per-activity) | Catch blocks (conditional jumps) | +| **Retry Policy** | Temporal default | Configurable per-activity | +| **Backward Compat** | N/A | 100% (cmd/starter unchanged) | +| **Code Coupling** | Tight (to specific sequence) | Loose (generic RoutingWorkflow) | + +--- + +## PART 13: APPROVAL CHECKLIST + +### Must Answer (Blocks Phase 1) +- [ ] Q1: What is KMSvc system? +- [ ] Q2: Message format for KMSvc? +- [ ] Q3: Who generates correlation_id? +- [ ] Q4: Polling strategy (active or signal)? +- [ ] Q5: Timeout behavior? +- [ ] Q6: Consumer group strategy? + +### Should Approve +- [ ] Timeline OK? (52-58 hours) +- [ ] Architecture OK? (RoutingWorkflow + 3 entry points) +- [ ] Risk level acceptable? (Medium-High) +- [ ] Backward compatibility OK? (Keep cmd/starter) + +### Sign-Off +- [ ] Tech Lead approval +- [ ] Stakeholder sign-off +- [ ] Ready to start Phase 1 + +--- + +## PART 14: NEXT STEPS + +1. **Review This Document** (1-2 hours) + - Read Parts 1-7 (understanding) + - Read Parts 8-9 (risks & success) + - Read Part 13 (approval checklist) + +2. **Answer Critical Questions** (Q1-Q6) + - Email or Slack responses required + - Needed before Phase 1 can start + +3. **Stakeholder Approval** + - Architecture review meeting + - Risk acknowledgment + - Sign-off on timeline + +4. **Start Implementation** + - Week 1: Phase 1 (RoutingWorkflow) + - Week 2: Phase 2 (API) + Phase 3 (CLI) + - Week 3: Phase 4 (K8s deploy) + - Week 4: Phase 5 (Docs) + +--- + +## APPENDIX: EXAMPLE WORKFLOW JSON + +```json +{ + "name": "code-review-pipeline", + "input": { + "repo": "https://github.com/rockliang/poimen", + "branch": "feature/new-cli" + }, + "activities": [ + { + "name": "Clone", + "resource": "CloneRepoActivity", + "parameters": { + "repo": "${input.repo}", + "branch": "${input.branch}" + }, + "timeout": "5m", + "next": "Analyze" + }, + { + "name": "Analyze", + "resource": "AnalyzeCodeActivity", + "parameters": { + "path": "${Clone.output.path}" + }, + "timeout": "10m", + "retry": { + "maxAttempts": 3, + "backoffRate": 2.0, + "initialInterval": "1s" + }, + "catch": [ + { + "errorEquals": ["Timeout"], + "next": "HandleTimeout" + } + ], + "next": "Judge" + }, + { + "name": "Judge", + "resource": "JudgeActivity", + "parameters": { + "code": "${Analyze.output.code}", + "analysis": "${Analyze.output.analysis}" + }, + "timeout": "5m", + "end": true + }, + { + "name": "HandleTimeout", + "type": "pass", + "result": { + "status": "failed", + "reason": "Analysis timed out after 10m" + }, + "end": true + } + ] +} +``` + +--- + +## APPENDIX: CLI USAGE EXAMPLES + +```bash +# Submit workflow from JSON file +$ poimen-cli submit workflow.json +Workflow submitted: wf-abc123 +Poll status: poimen-cli status wf-abc123 + +# Submit and wait for results +$ poimen-cli submit workflow.json --wait +Waiting for workflow to complete... +✅ Workflow completed +Output: +{ + "verdict": "approved", + "score": 0.95, + "issues": [] +} + +# Check status +$ poimen-cli status wf-abc123 +Workflow ID: wf-abc123 +Status: COMPLETED +Started: 2025-01-31 15:04:05 +Completed: 2025-01-31 15:05:00 + +# Execute template immediately (sync) +$ poimen-cli execute template code-review-v1 --input-file input.json +Executing template: code-review-v1 +[15:04:05] Step 1/3: Clone ... OK (45s) +[15:04:50] Step 2/3: Analyze ... OK (120s) +[15:05:50] Step 3/3: Judge ... OK (30s) +✅ COMPLETED +Output: {...} + +# List templates +$ poimen-cli template list +Available Templates: + - code-review-v1 + - clone-analyze-v1 + - simple-test-v1 + +# Show template +$ poimen-cli template show code-review-v1 +Template: code-review-v1 +Activities: 3 + +1. Clone (CloneRepoActivity) + Timeout: 5m +2. Analyze (AnalyzeCodeActivity) + Timeout: 10m + Retry: 3 attempts, backoff 2.0 +3. Judge (JudgeActivity) + Timeout: 5m +``` + +--- + +**END OF MASTER REVIEW DOCUMENT** + +For detailed technical documentation, see: +- SERVICE_INTEGRATION.md (patterns + API spec) +- IMPLEMENTATION_PLAN.md (phase breakdown) +- CLI_DESIGN.md (CLI implementation) +- FINAL_COMPREHENSIVE_DESIGN.md (1-2 hour read) + diff --git a/FINAL_COMPREHENSIVE_DESIGN.md b/FINAL_COMPREHENSIVE_DESIGN.md new file mode 100644 index 0000000..dbf3025 --- /dev/null +++ b/FINAL_COMPREHENSIVE_DESIGN.md @@ -0,0 +1,119 @@ +# Poimen Service Integration & Dynamic Workflows — FINAL DESIGN + +**Status**: Ready for Implementation Review +**Updated**: 2025-01-31 +**Scope**: Replace hardcoded workflows with API-driven, JSON-based, customer-specified sequences + +--- + +## 1-Minute Summary + +### Problem +Hardcoded workflows require code changes → recompile → deploy (5-10 min) to change activity sequences. + +### Solution +- New **RoutingWorkflow** reads JSON WorkflowSpec and executes dynamically +- Three interfaces: **CLI** + **HTTP API** + **legacy CLI** (backward compat) +- Three patterns: Sequential, Await-Task-Complete (KMSvc), Retry with error handling +- JSONPath parameters: `${step1.output.path}` + +### Result +Workflow changes in **seconds** (API call) instead of 5-10 min (recompile) + +--- + +## The Three Patterns + +### 1. Sequential (A → B → C) +Activities execute in sequence, each step passes output to next via JSONPath. + +### 2. Await-Task-Complete (Launch → Poll Queue → Process) +Launch long-running job, poll KMSvc queue for result with correlation ID matching, then proceed. + +### 3. Retry with Error Handling +Activity retry N times with exponential backoff, jump to catch block on failure. + +--- + +## The Three Entry Points + +### CLI (New) +```bash +poimen-cli submit workflow.json --wait +poimen-cli execute template code-review-v1 --input-file input.json +poimen-cli status wf-abc123 +``` + +### HTTP API (New) +```bash +curl -X POST /api/v1/workflows -d @workflow.json +curl -X POST /api/v1/execute -d '{"template": "...", "input": {...}}' +curl /api/v1/workflows/wf-abc123/status +``` + +### Legacy CLI (Unchanged) +```bash +go run ./cmd/starter --repo ... --milestone T0 (still works) +``` + +--- + +## Implementation: 5 Phases + +| Phase | Files | Hours | Week | +|-------|-------|-------|------| +| 1. Routing Engine | `internal/routing/`, `statemachine/routing_workflow.go` | 20-22 | 1 | +| 2. API Server | `cmd/api-server/` | 12-14 | 2 | +| 3. CLI | `cmd/cli/` | 8-10 | 2 | +| 4. K8s Deploy | `k8s/`, `homelab-frontend/k8s/` | 5 | 3 | +| 5. Docs | `docs/`, examples | 7 | 4 | +| **TOTAL** | **~15 files** | **52-58** | **4 weeks** | + +--- + +## Critical KMSvc Questions + +1. **What is KMSvc?** (Kafka/Redis/SQS/custom?) +2. **Message format?** (JSON with correlation_id field?) +3. **Who generates correlation_id?** (Activity or Workflow?) +4. **Polling strategy?** (Active loop or Temporal Signal?) +5. **Timeout behavior?** (Catch block, fail, or infinite wait?) +6. **Consumer group?** (Shared or per-workflow?) + +--- + +## Success Criteria + +✅ RoutingWorkflow executes sequences dynamically +✅ JSONPath parameters resolve correctly +✅ Error catch blocks work +✅ Retry with backoff works +✅ CLI reads JSON (no flags) +✅ API validates specs +✅ Backward compatible +✅ Performance: <200ms submit +✅ Test coverage: >90% + +--- + +## Key Files + +See full design doc (FINAL_COMPREHENSIVE_DESIGN.md in repo) for: +- Detailed phase breakdown +- Code examples +- Risk assessment +- Design decisions +- Complete file list +- Timeline details + +--- + +## Next Steps + +1. Answer KMSvc Q1-Q6 +2. Review 52-58 hour estimate +3. Stakeholder approval +4. Start Phase 1 + +See DESIGN_REVIEW_UPDATED.md for detailed questions. + diff --git a/IMPLEMENTATION_PLAN.md b/IMPLEMENTATION_PLAN.md new file mode 100644 index 0000000..e3b29e6 --- /dev/null +++ b/IMPLEMENTATION_PLAN.md @@ -0,0 +1,226 @@ +# Poimen Declarative Workflow Implementation Plan + +## Executive Summary + +**Current State**: Hardcoded workflows (OrchestratorWorkflow, TaskUnitWorkflow) +- Fixed flow: Clone → Plan → Implement → Judge → Commit +- New flows require recompilation +- No customer-specified sequencing + +**Target State**: API-driven dynamic workflows +- Customer sends JSON specifying activity sequence +- Routing engine executes dynamically +- Support 3 patterns: Sequential, Await-Task-Complete, Retry +- Backward compatible with existing cmd/starter + +**Timeline**: 4 weeks | **Effort**: ~80 story points | **Team**: 2-3 engineers + +--- + +## Phase 1: Core Routing Engine (Week 1) — 16 hours + +**Goal**: Build engine that parses WorkflowSpec and executes activity sequences dynamically + +**Files to Create**: +- `internal/routing/types.go` — Data structures for WorkflowSpec, ActivityStep, RetryConfig +- `internal/routing/jsonpath_evaluator.go` — Parameter interpolation using JSONPath +- `internal/routing/execution_context.go` — Execution state management +- `internal/routing/activity_registry.go` — Activity name → implementation mapper +- `statemachine/routing_workflow.go` — **Core RoutingWorkflow that ties everything together** +- Tests: `*_test.go` files for each module + +**Key Implementation Details**: +- RoutingWorkflow is the main Temporal workflow +- It parses activity steps and executes them sequentially +- Uses JSONPath to interpolate parameters: `${step_1.output.path}` +- Supports retry logic, error catching, and conditional jumps +- All activities are pre-registered in worker pool + +**Backward Compatibility**: ✅ Keep OrchestratorWorkflow, TaskUnitWorkflow unchanged + +--- + +## Phase 2: API Server (Week 2) — 10 hours + +**Goal**: HTTP API that accepts WorkflowSpec JSON and manages workflow execution + +**Files to Create**: +- `cmd/api-server/main.go` — Server bootstrap, Temporal client connection +- `cmd/api-server/handlers.go` — POST /api/v1/workflows, GET /api/v1/workflows/{id}/status +- `cmd/api-server/validation.go` — WorkflowSpec validation (references, timeouts, etc.) +- `cmd/api-server/server.go` — Server config and lifecycle +- Tests: handlers, validation, integration tests + +**Endpoints**: +``` +POST /api/v1/workflows + → Submit WorkflowSpec + → Returns: {workflow_id, status, polling_url} + +GET /api/v1/workflows/{workflow_id}/status + → Poll execution status + → Returns: {status, steps_completed, results, final_output} +``` + +**Key Details**: +- Validates WorkflowSpec before submission +- Submits to RoutingWorkflow (not OrchestratorWorkflow) +- Caches workflow metadata locally +- Polls Temporal for status + +--- + +## Phase 3: Homelab Frontend Integration (Week 3) — 5 hours + +**Goal**: Expose API through ingress, connect to gateway + +**Files to Create**: +- `k8s/api-server-deployment.yaml` — K8s Deployment, 2 replicas +- `k8s/api-server-service.yaml` — K8s Service for api-server +- `k8s/api-server-networkpolicy.yaml` — Allow ingress traffic, deny by default +- `homelab-frontend/k8s/gateway-routes.yaml` — Route /api/v1/workflows to api-server + +**Key Details**: +- api-server connects to Temporal: `temporal-frontend.temporal:7233` +- Ingress routes external traffic to api-server:8080 +- NetworkPolicy allows only ingress-nginx → api-server traffic + +--- + +## Phase 4: Migration & Docs (Week 4) — 7 hours + +**Files to Create**: +- `docs/API.md` — API reference (endpoints, request/response examples) +- `docs/MIGRATION.md` — How to migrate from cmd/starter to API +- `docs/examples/sequential-workflow.json` — Example: Clone → Analyze → Implement → Judge +- `docs/examples/retry-workflow.json` — Example: with retry + error handling +- `docs/examples/conditional-workflow.json` — Example: with catch blocks + +**Key Details**: +- Comprehensive examples for all 3 patterns +- cmd/starter continues to work (backward compatible) +- Documentation emphasizes JSONPath parameter passing + +--- + +## Three Composite Patterns + +### Pattern A: Sequential (Request-Response) +```json +{ + "activities": [ + {"name": "Step1", "resource": "ActivityA", "parameters": {...}, "next": "Step2"}, + {"name": "Step2", "resource": "ActivityB", "parameters": {"input": "${Step1.output}"}, "next": "Step3"}, + {"name": "Step3", "resource": "ActivityC", "parameters": {...}, "end": true} + ] +} +``` +**Flow**: Step1 → Step2 (receives Step1's output) → Step3 + +--- + +### Pattern B: Await-Task-Complete (Callback) +```json +{ + "activities": [ + {"name": "LaunchJob", "type": "await-callback", "resource": "LaunchActivity", "heartbeat_timeout": "5m", "next": "ProcessResult"}, + {"name": "ProcessResult", "resource": "ProcessActivity", "parameters": {"result": "${LaunchJob.callback}"}, "end": true} + ] +} +``` +**Flow**: LaunchJob (waits for callback) → ProcessResult (receives callback data) + +--- + +### Pattern C: Retry with Error Handling +```json +{ + "activities": [ + {"name": "Generate", "resource": "ImplementerActivity", "retry": {"maxAttempts": 3, "backoffRate": 2.0}, "catch": [{"errorEquals": ["Timeout"], "next": "HandleTimeout"}], "next": "Verify"}, + {"name": "Verify", "resource": "JudgeActivity", "end": true}, + {"name": "HandleTimeout", "type": "pass", "result": {"status": "timeout"}, "end": true} + ] +} +``` +**Flow**: Generate (retry 3x) → if success: Verify, else: HandleTimeout + +--- + +## Success Criteria + +- ✅ RoutingWorkflow executes dynamic sequences correctly +- ✅ JSONPath parameter passing works (nested references, arrays) +- ✅ Error handling and catch blocks work +- ✅ Retry logic respects backoff intervals +- ✅ API validates WorkflowSpec (no invalid refs) +- ✅ Status polling returns step-by-step results +- ✅ All tests pass (>90% coverage) +- ✅ Backward compatible (cmd/starter still works) +- ✅ API exposed via ingress at `/api/v1/workflows` +- ✅ Performance: <200ms submit, <100ms poll + +--- + +## Architecture Diagram + +``` +┌─ Customer/Frontend ─────────────────────────┐ +│ POST /api/v1/workflows (JSON WorkflowSpec) │ +└────────────────────┬────────────────────────┘ + ↓ + ┌──────────────────────┐ + │ API Server │ + │ • Validate spec │ + │ • Submit to Temporal │ + │ • Cache metadata │ + └────────────┬─────────┘ + ↓ + ┌────────────────────────────────┐ + │ Temporal Server │ + │ RoutingWorkflow │ + │ ├─ Parse activities │ + │ ├─ Evaluate JSONPath │ + │ ├─ Execute sequence │ + │ └─ Handle errors/retries │ + └────────────┬───────────────────┘ + ↓ + ┌────────────────────────────────┐ + │ Temporal Worker Pool │ + │ • Poll queue │ + │ • Execute activities │ + │ • Return results │ + └────────────────────────────────┘ +``` + +--- + +## Risk Mitigation + +| Risk | Mitigation | +|------|-----------| +| JSONPath bugs | Unit tests + property-based testing | +| Activity registry mismatch | Validate registered activities match spec | +| Temporal network issues | Retry logic + timeouts in API server | +| Cache staleness | Event-driven cache invalidation | + +--- + +## Rollback Plan + +| Phase | Rollback Strategy | +|-------|------------------| +| Phase 1 | Keep old workflows, don't use RoutingWorkflow | +| Phase 2 | Disable api-server, fall back to cmd/starter | +| Phase 3 | Remove ingress, access api-server directly | +| Phase 4 | Continue supporting both APIs (cmd/starter + API) | + +--- + +## Next Steps + +1. **Week 1**: Start Phase 1 implementation (RoutingWorkflow) +2. **Week 2**: Start Phase 2 (API Server) +3. **Week 3**: Deploy to K8s (Phase 3) +4. **Week 4**: Document, test, release (Phase 4) +5. **Post-Release**: Monitor, gather feedback, plan Phase 2 enhancements (parallel, conditional branching) + diff --git a/SERVICE_INTEGRATION.md b/SERVICE_INTEGRATION.md new file mode 100644 index 0000000..8994948 --- /dev/null +++ b/SERVICE_INTEGRATION.md @@ -0,0 +1,452 @@ +# Poimen Service Integration & Declarative Workflow Design + +## Overview + +**Current (Bad)**: Hardcoded workflows +``` +CLI flags → OrchestratorWorkflow (fixed flow) +``` + +**Proposed (Better)**: API-driven, customer-specified workflows +``` +API Request (JSON) → Routing Agent → Dynamic Activity Sequence → Results +``` + +--- + +## Part 1: Service Integration Patterns (AWS Step Functions Inspired) + +### Pattern 1: Sequential (Request-Response) + +```yaml +# Customer specifies: "Call A, then call B with A's output" +activities: + - name: CloneRepo + type: request-response + resource: CloneRepoActivity + parameters: + repo: "https://github.com/..." + path: "/tmp/clone" + timeout: 5m + retry: + maxAttempts: 3 + backoff: exponential + + - name: Analyze + type: request-response + resource: AnalyzeCodeActivity + parameters: + path: $.CloneRepo.path # ← Reference previous output + inputPath: "$.CloneRepo" # ← What to pass + outputPath: "$" # ← What to return + timeout: 10m +``` + +**Execution**: CloneRepo waits → outputs result → Analyze receives it → outputs + +--- + +### Pattern 2: Await-Task-Complete (Callback) + +```yaml +# Customer specifies: "Call A, wait for callback, then call B" +activities: + - name: LaunchJob + type: await-callback + resource: LaunchLongRunningJobActivity + parameters: + job_id: "job-123" + heartbeat_timeout: 5m + next: OnJobComplete + + - name: OnJobComplete + type: request-response + resource: ProcessResultActivity + parameters: + result: $.LaunchJob.result +``` + +**Execution**: LaunchJob sent → waits for callback (heartbeat) → ProcessResult receives callback data → outputs + +--- + +### Pattern 3: Retry with Backoff + +```yaml +# Customer specifies: "Call A, if fails retry 3x with exponential backoff, else B" +activities: + - name: GenerateCode + type: request-response + resource: ImplementerActivity + parameters: + task: $.input.taskID + model: $.input.model + retry: + maxAttempts: 3 + backoffRate: 2.0 # 1s → 2s → 4s + initialInterval: 1s + maxInterval: 10m + catch: + - errorEquals: ["Timeout", "ThrottlingException"] + next: HandleError + - errorEquals: ["States.ALL"] + next: HandleError + next: JudgeCode + + - name: JudgeCode + type: request-response + resource: JudgeActivity + parameters: + implementation: $.GenerateCode.code + + - name: HandleError + type: pass + result: { status: "failed", reason: $.error } + resultPath: "$.error_output" + end: true +``` + +**Execution**: GenerateCode fails → retry 1 (wait 1s) → fails → retry 2 (wait 2s) → fails → HandleError + +--- + +## Part 2: Routing Agent Architecture + +### 2.1 Routing Agent Activity (New) + +```go +// internal/routing/routing_agent.go + +type WorkflowSpec struct { + WorkflowID string `json:"workflow_id"` + Name string `json:"name"` + Input map[string]interface{} `json:"input"` + Activities []ActivityStep `json:"activities"` + RetryPolicy RetryConfig `json:"retry_policy,omitempty"` +} + +type ActivityStep struct { + Name string `json:"name"` + Type string `json:"type"` // request-response, await-callback, parallel, choice + Resource string `json:"resource"` // Activity name + Parameters map[string]interface{} `json:"parameters"` + InputPath string `json:"input_path"` // JSONPath: "$" or "$.previous.output" + OutputPath string `json:"output_path"` // What to extract + Timeout time.Duration `json:"timeout"` + Retry *RetryConfig `json:"retry,omitempty"` + Catch []CatchBlock `json:"catch,omitempty"` + Next string `json:"next,omitempty"` + End bool `json:"end,omitempty"` +} + +type RetryConfig struct { + MaxAttempts int `json:"max_attempts"` + BackoffRate float64 `json:"backoff_rate"` + InitialInterval time.Duration `json:"initial_interval"` + MaxInterval time.Duration `json:"max_interval"` +} + +type CatchBlock struct { + ErrorEquals []string `json:"error_equals"` + Next string `json:"next"` + ResultPath string `json:"result_path,omitempty"` +} + +// RoutingAgentActivity: Receives WorkflowSpec, executes activities sequentially +func RoutingAgentActivity(ctx context.Context, spec WorkflowSpec) (map[string]interface{}, error) { + // 1. Parse WorkflowSpec + // 2. Build execution plan (DAG of activities) + // 3. For each step: + // a. Evaluate JSONPath inputs + // b. Execute activity with retries + // c. Handle errors (catch blocks) + // d. Store output in context + // e. Move to next step + // 4. Return final result +} +``` + +### 2.2 Activity Mapper Registry (New) + +```go +// internal/routing/activity_registry.go + +type ActivityMapper struct { + activities map[string]interface{} +} + +func NewActivityMapper() *ActivityMapper { + return &ActivityMapper{ + activities: map[string]interface{}{ + "CloneRepoActivity": action.CloneRepoActivity, + "AnalyzeCodeActivity": action.AnalyzeCodeActivity, + "ImplementerActivity": action.ImplementerActivity, + "JudgeActivity": action.JudgeActivity, + "GitCommitActivity": action.GitCommitActivity, + // ... all activities registered here + }, + } +} + +func (am *ActivityMapper) Get(name string) (interface{}, error) { + if activity, ok := am.activities[name]; ok { + return activity, nil + } + return nil, fmt.Errorf("activity %s not found", name) +} + +// cmd/worker/main.go +w.RegisterActivity(routing.RoutingAgentActivity) // ← Only register this +w.RegisterActivity(action.CloneRepoActivity) // ← Still register all individual activities +// ... rest of activities +``` + +--- + +## Part 3: API Message Format (General Purpose) + +### 3.1 Workflow Request Format + +```json +POST /api/workflows +{ + "workflow": { + "id": "workflow-123", + "name": "code-review-pipeline", + "version": "v1", + "input": { + "repo": "https://github.com/rockliang/poimen", + "branch": "feature/new-feature", + "milestone": "T0" + }, + "activities": [ + { + "id": "step_1", + "name": "CloneRepo", + "type": "request-response", + "resource": "CloneRepoActivity", + "parameters": { + "repo": "${input.repo}", + "path": "/tmp/work" + }, + "timeout": "5m", + "next": "step_2" + }, + { + "id": "step_2", + "name": "AnalyzeCode", + "type": "request-response", + "resource": "AnalyzeCodeActivity", + "parameters": { + "path": "${step_1.output.path}" + }, + "inputPath": "$.step_1", + "outputPath": "$", + "timeout": "10m", + "retry": { + "maxAttempts": 3, + "backoffRate": 2.0, + "initialInterval": "1s" + }, + "catch": [ + { + "errorEquals": ["Timeout"], + "next": "handle_timeout" + } + ], + "next": "step_3" + }, + { + "id": "step_3", + "name": "GenerateImplementation", + "type": "request-response", + "resource": "ImplementerActivity", + "parameters": { + "analysis": "${step_2.output.analysis}", + "model": "claude-sonnet-5" + }, + "timeout": "30m", + "retry": { + "maxAttempts": 3, + "backoffRate": 2.0 + }, + "next": "step_4" + }, + { + "id": "step_4", + "name": "Review", + "type": "request-response", + "resource": "JudgeActivity", + "parameters": { + "code": "${step_3.output.code}", + "analysis": "${step_2.output.analysis}" + }, + "timeout": "10m", + "end": true + }, + { + "id": "handle_timeout", + "name": "TimeoutHandler", + "type": "pass", + "result": { + "status": "failed", + "reason": "Analysis timed out" + }, + "end": true + } + ] + } +} +``` + +### 3.2 Workflow Response Format + +```json +{ + "workflow_id": "workflow-123", + "status": "RUNNING", + "temporal_workflow_id": "orch-workflow-123", + "task_queue": "poimen-taskqueue", + "execution_started_at": "2025-01-29T10:00:00Z", + "monitoring": { + "web_ui": "http://temporal.local:8080/namespaces/poimen/workflows/orch-workflow-123", + "poll_url": "/api/workflows/workflow-123/status" + } +} +``` + +### 3.3 Status Poll Format + +```json +GET /api/workflows/workflow-123/status +{ + "workflow_id": "workflow-123", + "status": "COMPLETED", + "steps_completed": 4, + "total_steps": 4, + "results": { + "step_1": { + "name": "CloneRepo", + "status": "COMPLETED", + "duration": "45s", + "output": { + "path": "/tmp/work/poimen", + "commit": "abc123" + } + }, + "step_2": { + "name": "AnalyzeCode", + "status": "COMPLETED", + "duration": "120s", + "output": { + "analysis": "..." + } + }, + "step_3": { + "name": "GenerateImplementation", + "status": "COMPLETED", + "duration": "300s", + "output": { + "code": "..." + } + }, + "step_4": { + "name": "Review", + "status": "COMPLETED", + "duration": "60s", + "output": { + "verdict": "approved", + "score": 0.95 + } + } + }, + "final_output": { + "verdict": "approved", + "score": 0.95 + }, + "completed_at": "2025-01-29T10:10:00Z" +} +``` + +--- + +## Part 4: Implementation Plan + +### Phase 1: Core Routing Engine (Week 1) +- [ ] Create `internal/routing/types.go` (WorkflowSpec, ActivityStep structs) +- [ ] Create `internal/routing/jsonpath_evaluator.go` (parameter interpolation) +- [ ] Create `internal/routing/execution_context.go` (execution state management) +- [ ] Create `internal/routing/activity_registry.go` (activity mapper) +- [ ] Create `statemachine/routing_workflow.go` (RoutingWorkflow implementation) +- [ ] Update `cmd/worker/main.go` (register RoutingWorkflow) +- [ ] Write tests for routing engine + +### Phase 2: API Server (Week 2) +- [ ] Create `cmd/api-server/main.go` (HTTP server bootstrap) +- [ ] Create `cmd/api-server/handlers.go` (POST /workflows, GET /status) +- [ ] Create `cmd/api-server/validation.go` (WorkflowSpec validation) +- [ ] Create `internal/routing/client.go` (Temporal client wrapper) +- [ ] Write integration tests + +### Phase 3: Homelab Frontend Integration (Week 3) +- [ ] Create `k8s/gateway-routes.yaml` (route /api/v1/workflows to api-server) +- [ ] Deploy api-server as K8s Service +- [ ] Update NetworkPolicy to allow traffic to api-server +- [ ] Test end-to-end API flow + +### Phase 4: Migration & Cleanup (Week 4) +- [ ] Keep cmd/starter for backward compatibility +- [ ] Document migration path +- [ ] Update examples to use new API +- [ ] Performance tuning + +--- + +## Files to Create + +``` +NEW: + internal/routing/ + ├─ types.go (WorkflowSpec, ActivityStep, RetryConfig) + ├─ jsonpath_evaluator.go (Parameter interpolation) + ├─ execution_context.go (State management) + ├─ activity_registry.go (Activity mapper) + └─ routing_workflow.go (RoutingWorkflow implementation) + + statemachine/ + └─ routing_workflow.go (See Part 5 below) + + cmd/api-server/ + ├─ main.go (Server bootstrap) + ├─ handlers.go (HTTP handlers) + └─ validation.go (Spec validation) + +MODIFY: + cmd/worker/main.go (Register RoutingWorkflow) + +NEW (Homelab Frontend): + k8s/ + └─ gateway-routes.yaml (Route definitions) +``` + +--- + +## Backward Compatibility + +✅ **Existing cmd/starter** still works (calls OrchestratorWorkflow) +✅ **New API** calls RoutingWorkflow +✅ **Worker** registers both workflows +✅ **No breaking changes** to current deployments + +--- + +## Benefits + +✅ Customer-driven workflows (JSON API) +✅ Service Integration patterns (Sequential, Await-Task-Complete, Retry) +✅ No recompilation needed +✅ Full Temporal observability +✅ JSONPath-based parameter passing +✅ Composable activities +✅ Error handling & retries built-in + diff --git a/k8s/git-commit.yaml b/k8s/git-commit.yaml index a4a4b51..8e04b44 100644 --- a/k8s/git-commit.yaml +++ b/k8s/git-commit.yaml @@ -9,6 +9,6 @@ metadata: app.kubernetes.io/name: poimen app.kubernetes.io/component: orchestrator data: - GIT_COMMIT: "74a8f2e8" # Updated automatically by CI/CD + GIT_COMMIT: "04792563" # Updated automatically by CI/CD GIT_BRANCH: "main" DEPLOYMENT_DATE: "2026-08-31" diff --git a/k8s/worker-deployment.yaml b/k8s/worker-deployment.yaml index 73df1f5..c90ea42 100644 --- a/k8s/worker-deployment.yaml +++ b/k8s/worker-deployment.yaml @@ -13,7 +13,7 @@ spec: labels: app: poimen-worker annotations: - git-commit: "74a8f2e8" # ✅ Updated on each push, triggers rolling restart + git-commit: "04792563" # ✅ Updated on each push, triggers rolling restart deployment-date: "2026-08-31" spec: containers: