- Better describes the intent: awaiting task completion via queue - More descriptive than 'await-kmsvc' - Applied to all workflow design documents - Reflects the actual operation: wait for external task to complete Updated files: - DESIGN_MASTER_REVIEW.md - SERVICE_INTEGRATION.md - IMPLEMENTATION_PLAN.md - CLI_DESIGN.md - FINAL_COMPREHENSIVE_DESIGN.md
This commit is contained in:
Test
@@ -0,0 +1,81 @@
|
||||
# Poimen CLI — Dynamic JSON-Based Design
|
||||
|
||||
## Overview
|
||||
|
||||
New CLI (`cmd/cli`) that accepts JSON WorkflowSpecs and dynamically understands activity sequences—same mechanism as HTTP API, but command-line interface.
|
||||
|
||||
## Key Commands
|
||||
|
||||
```bash
|
||||
# Submit workflow from JSON
|
||||
poimen-cli submit workflow.json
|
||||
poimen-cli submit workflow.json --wait
|
||||
poimen-cli submit workflow.json --watch
|
||||
|
||||
# Check status
|
||||
poimen-cli status wf-abc123
|
||||
poimen-cli status wf-abc123 --wait
|
||||
|
||||
# Template management
|
||||
poimen-cli template list
|
||||
poimen-cli template show code-review-v1
|
||||
|
||||
# Execute immediately (sync)
|
||||
poimen-cli execute template code-review-v1 --input-file input.json
|
||||
```
|
||||
|
||||
## Workflow JSON Example
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "code-review-pipeline",
|
||||
"input": {"repo": "https://github.com/..."},
|
||||
"activities": [
|
||||
{
|
||||
"name": "Clone",
|
||||
"resource": "CloneRepoActivity",
|
||||
"parameters": {"repo": "${input.repo}"},
|
||||
"timeout": "5m",
|
||||
"next": "Analyze"
|
||||
},
|
||||
{
|
||||
"name": "Analyze",
|
||||
"resource": "AnalyzeCodeActivity",
|
||||
"parameters": {"path": "${Clone.output.path}"},
|
||||
"timeout": "10m",
|
||||
"retry": {"maxAttempts": 3},
|
||||
"catch": [{"errorEquals": ["Timeout"], "next": "HandleTimeout"}],
|
||||
"next": "Judge"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
cmd/
|
||||
├─ cli/ (NEW)
|
||||
│ ├─ main.go
|
||||
│ └─ commands/
|
||||
│ ├─ submit.go
|
||||
│ ├─ status.go
|
||||
│ ├─ template.go
|
||||
│ ├─ execute.go
|
||||
│ └─ history.go
|
||||
├─ starter/ (OLD: keep for backward compat)
|
||||
└─ worker/
|
||||
```
|
||||
|
||||
## Three Entry Points (Same Core)
|
||||
|
||||
| Interface | Endpoint | Input | Output |
|
||||
|-----------|----------|-------|--------|
|
||||
| **CLI** | `submit workflow.json` | JSON file | workflow_id |
|
||||
| **CLI** | `execute template T` | Template name | Results (sync) |
|
||||
| **API** | `POST /api/v1/workflows` | JSON body | workflow_id |
|
||||
| **API** | `POST /api/v1/execute` | JSON body | Results (sync) |
|
||||
|
||||
All use same `RoutingWorkflow` internally—just different interfaces.
|
||||
|
||||
See CLI_DESIGN.md in repo for full implementation details.
|
||||
@@ -0,0 +1,889 @@
|
||||
# Poimen Service Integration & Dynamic Workflows — MASTER REVIEW DOCUMENT
|
||||
|
||||
**Status**: Ready for Approval
|
||||
**Version**: 1.0
|
||||
**Date**: 2025-01-31
|
||||
**Duration**: 52-58 hours (4 weeks)
|
||||
|
||||
---
|
||||
|
||||
## EXECUTIVE SUMMARY (5 MINUTES)
|
||||
|
||||
### Problem
|
||||
Current workflows are **hardcoded in Go**. Changing activity sequences requires:
|
||||
- Code modification
|
||||
- Recompilation
|
||||
- K8s deployment
|
||||
- Pod restart
|
||||
- **Total: 5-10 minutes**
|
||||
|
||||
### Solution
|
||||
New **RoutingWorkflow** reads JSON WorkflowSpec and executes dynamically:
|
||||
- **Three entry points**: CLI + HTTP API + Legacy CLI (backward compatible)
|
||||
- **Three patterns**: Sequential, Await-Task-Complete (KMSvc queue), Retry with error handling
|
||||
- **JSONPath parameters**: `${step1.output.path}` instead of hardcoded values
|
||||
- **Result**: Workflow changes in **seconds** (API call only)
|
||||
|
||||
### Impact
|
||||
✅ Instant changes | ✅ Three interfaces | ✅ Full compatibility | ❌ +KMSvc complexity
|
||||
|
||||
---
|
||||
|
||||
## PART 1: THE THREE PATTERNS
|
||||
|
||||
### Pattern 1: Sequential (A → B → C)
|
||||
|
||||
Activities execute in sequence. Each step passes output to next via JSONPath.
|
||||
|
||||
```json
|
||||
{
|
||||
"activities": [
|
||||
{
|
||||
"name": "Clone",
|
||||
"resource": "CloneRepoActivity",
|
||||
"parameters": {"repo": "${input.repo}"},
|
||||
"timeout": "5m",
|
||||
"next": "Analyze"
|
||||
},
|
||||
{
|
||||
"name": "Analyze",
|
||||
"resource": "AnalyzeCodeActivity",
|
||||
"parameters": {"path": "${Clone.output.path}"},
|
||||
"timeout": "10m",
|
||||
"next": "Judge"
|
||||
},
|
||||
{
|
||||
"name": "Judge",
|
||||
"resource": "JudgeActivity",
|
||||
"parameters": {"code": "${Analyze.output.code}"},
|
||||
"timeout": "5m",
|
||||
"end": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Execution Flow**:
|
||||
```
|
||||
Clone ──(output: {path: /tmp/repo})──> Analyze ──(output: {code: ...})──> Judge ──(final result)
|
||||
```
|
||||
|
||||
**Use Case**: Code review pipeline (clone → analyze → judge)
|
||||
|
||||
---
|
||||
|
||||
### Pattern 2: Await-Task-Complete (Launch → Poll Queue → Process)
|
||||
|
||||
Launch long-running job, poll KMSvc queue for result with correlation ID matching, then proceed.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "LaunchJob",
|
||||
"type": "awaitTaskComplete",
|
||||
"resource": "LaunchJobActivity",
|
||||
"parameters": {"job_id": "job-123"},
|
||||
"queue": "job-completions",
|
||||
"correlation_key": "${LaunchJob.output.correlation_id}",
|
||||
"timeout": "5m",
|
||||
"next": "ProcessResult"
|
||||
}
|
||||
```
|
||||
|
||||
**Execution Flow**:
|
||||
```
|
||||
LaunchJobActivity ──(returns {correlation_id: "xyz"})──> Poll Queue "job-completions"
|
||||
↓
|
||||
[External system completes job]
|
||||
[Publishes {correlation_id: "xyz", result: ...}]
|
||||
↓
|
||||
[Workflow receives message]
|
||||
[Proceeds to ProcessResult]
|
||||
```
|
||||
|
||||
**Use Case**: Long-running batch jobs (model training, data processing)
|
||||
|
||||
---
|
||||
|
||||
### Pattern 3: Retry with Error Handling
|
||||
|
||||
Activity retries N times with exponential backoff. On failure, jumps to catch block.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Generate",
|
||||
"resource": "ImplementerActivity",
|
||||
"parameters": {"task": "${input.task}"},
|
||||
"timeout": "30m",
|
||||
"retry": {
|
||||
"maxAttempts": 3,
|
||||
"backoffRate": 2.0,
|
||||
"initialInterval": "1s"
|
||||
},
|
||||
"catch": [
|
||||
{"errorEquals": ["Timeout"], "next": "HandleTimeout"}
|
||||
],
|
||||
"next": "Verify"
|
||||
}
|
||||
```
|
||||
|
||||
**Execution Flow**:
|
||||
```
|
||||
Generate (attempt 1) ──TIMEOUT──> [Wait 1s]
|
||||
Generate (attempt 2) ──TIMEOUT──> [Wait 2s]
|
||||
Generate (attempt 3) ──TIMEOUT──> [Wait 4s]
|
||||
Max retries exceeded ──> Jump to HandleTimeout
|
||||
```
|
||||
|
||||
**Use Case**: Code generation (retry on timeout, handle gracefully)
|
||||
|
||||
---
|
||||
|
||||
## PART 2: THE THREE ENTRY POINTS
|
||||
|
||||
### Entry Point 1: CLI (New Command-Line)
|
||||
|
||||
```bash
|
||||
# Submit workflow from JSON file
|
||||
$ poimen-cli submit workflow.json
|
||||
Workflow submitted: wf-abc123
|
||||
Poll status: poimen-cli status wf-abc123
|
||||
|
||||
# Submit and wait for results
|
||||
$ poimen-cli submit workflow.json --wait
|
||||
Waiting for workflow to complete...
|
||||
✅ Workflow completed
|
||||
Output: {...}
|
||||
|
||||
# Execute template immediately (sync, no polling)
|
||||
$ poimen-cli execute template code-review-v1 --input-file input.json
|
||||
Executing template...
|
||||
Step 1/3: Clone ... OK (45s)
|
||||
Step 2/3: Analyze ... OK (120s)
|
||||
Step 3/3: Judge ... OK (30s)
|
||||
✅ COMPLETED
|
||||
Output: {...}
|
||||
|
||||
# Check status later
|
||||
$ poimen-cli status wf-abc123
|
||||
Workflow ID: wf-abc123
|
||||
Status: COMPLETED
|
||||
Started: 2025-01-31 15:04:05
|
||||
Completed: 2025-01-31 15:05:00
|
||||
|
||||
# List templates
|
||||
$ poimen-cli template list
|
||||
Available Templates:
|
||||
- code-review-v1
|
||||
- clone-analyze-v1
|
||||
```
|
||||
|
||||
**Implementation**: `cmd/cli/` (new directory)
|
||||
|
||||
---
|
||||
|
||||
### Entry Point 2: HTTP API (New)
|
||||
|
||||
#### Route 1: Submit Workflow (Async)
|
||||
```bash
|
||||
curl -X POST http://api.example.com/api/v1/workflows \
|
||||
-H "Content-Type: application/json" \
|
||||
-d @workflow.json
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"workflow_id": "wf-abc123",
|
||||
"status": "RUNNING",
|
||||
"polling_url": "/api/v1/workflows/wf-abc123/status"
|
||||
}
|
||||
```
|
||||
|
||||
#### Check Status
|
||||
```bash
|
||||
curl http://api.example.com/api/v1/workflows/wf-abc123/status
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"workflow_id": "wf-abc123",
|
||||
"status": "COMPLETED",
|
||||
"steps_completed": 3,
|
||||
"results": {
|
||||
"Clone": { "status": "COMPLETED", "output": {...} },
|
||||
"Analyze": { "status": "COMPLETED", "output": {...} },
|
||||
"Judge": { "status": "COMPLETED", "output": {...} }
|
||||
},
|
||||
"final_output": {...}
|
||||
}
|
||||
```
|
||||
|
||||
#### Route 2: Execute Template (Sync)
|
||||
```bash
|
||||
curl -X POST http://api.example.com/api/v1/execute \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"template": "code-review-v1",
|
||||
"input": {"repo": "https://github.com/...", "branch": "feature/x"},
|
||||
"timeout": "2m"
|
||||
}'
|
||||
```
|
||||
|
||||
**Response** (immediate):
|
||||
```json
|
||||
{
|
||||
"status": "COMPLETED",
|
||||
"results": {...},
|
||||
"final_output": {...}
|
||||
}
|
||||
```
|
||||
|
||||
**Implementation**: `cmd/api-server/` (new directory)
|
||||
|
||||
---
|
||||
|
||||
### Entry Point 3: Legacy CLI (Unchanged)
|
||||
|
||||
```bash
|
||||
# Old way still works, completely backward compatible
|
||||
$ go run ./cmd/starter \
|
||||
--repo https://github.com/rockliang/poimen \
|
||||
--remote file:///tmp/poimen-output \
|
||||
--milestone T0 \
|
||||
--planner-model ornith \
|
||||
--judge-model ornith \
|
||||
--implementer-model claude-sonnet-5
|
||||
|
||||
Workflow submitted: orch-poimen
|
||||
Status: RUNNING
|
||||
```
|
||||
|
||||
**Implementation**: `cmd/starter/` (existing, unchanged)
|
||||
|
||||
---
|
||||
|
||||
## PART 3: ARCHITECTURE OVERVIEW
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ THREE ENTRY POINTS │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ CLI (JSON file) HTTP API (JSON POST) Legacy CLI │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌─────────┐ │
|
||||
│ │ poimen-cli │ │ /api/v1/ │ │ cmd/ │ │
|
||||
│ │ submit │ │ workflows │ │ starter │ │
|
||||
│ └──────┬───────┘ └──────┬───────┘ └────┬────┘ │
|
||||
│ │ │ │ │
|
||||
└─────────┼─────────────────────────┼─────────────────────┼────────┘
|
||||
│ │ │
|
||||
└─────────────────────────┼─────────────────────┘
|
||||
↓
|
||||
┌───────────────────────────┐
|
||||
│ RoutingWorkflow │
|
||||
│ (NEW - generic) │
|
||||
│ │
|
||||
│ 1. Parse WorkflowSpec │
|
||||
│ 2. For each activity: │
|
||||
│ - Resolve JSONPath │
|
||||
│ - Execute activity │
|
||||
│ - Handle errors │
|
||||
│ - Store result │
|
||||
│ 3. Return results │
|
||||
└───────────┬───────────────┘
|
||||
↓
|
||||
┌───────────────────────────┐
|
||||
│ Temporal Server │
|
||||
│ │
|
||||
│ Worker Pool: │
|
||||
│ - Executes activities │
|
||||
│ - Returns results │
|
||||
│ - Retries with backoff │
|
||||
└───────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PART 4: BEFORE vs AFTER
|
||||
|
||||
### Before (Hardcoded Workflows)
|
||||
|
||||
```bash
|
||||
# To change activity sequence:
|
||||
1. Modify Go code (OrchestratorWorkflow in statemachine/)
|
||||
2. Compile: go build ./cmd/worker
|
||||
3. Push docker image
|
||||
4. Deploy to K8s: kubectl set image deployment/poimen-worker
|
||||
5. Wait for rollout
|
||||
Time: 5-10 minutes
|
||||
```
|
||||
|
||||
**Workflow definition**:
|
||||
```go
|
||||
// Hard to change, tight coupling to specific sequence
|
||||
if err := workflow.ExecuteActivity(ctx, "CloneRepoActivity", params).Get(ctx, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := workflow.ExecuteActivity(ctx, "PlanningActivity", params2).Get(ctx, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
// ... more activities hardcoded
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### After (Dynamic Workflows)
|
||||
|
||||
```bash
|
||||
# To change activity sequence:
|
||||
1. Edit workflow.json (define sequence)
|
||||
2. Call API or CLI
|
||||
3. Done
|
||||
Time: Seconds
|
||||
```
|
||||
|
||||
**Workflow definition**:
|
||||
```json
|
||||
{
|
||||
"activities": [
|
||||
{"name": "Clone", "resource": "CloneRepoActivity", "parameters": {...}, "next": "Analyze"},
|
||||
{"name": "Analyze", "resource": "AnalyzeCodeActivity", "parameters": {...}, "next": "Judge"},
|
||||
{"name": "Judge", "resource": "JudgeActivity", "parameters": {...}, "end": true}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PART 5: IMPLEMENTATION PHASES
|
||||
|
||||
### Phase 1: Routing Engine (Week 1) — 20-22 Hours
|
||||
|
||||
**Goal**: Core RoutingWorkflow that executes JSON sequences dynamically
|
||||
|
||||
**Files to Create**:
|
||||
```
|
||||
internal/routing/
|
||||
├─ types.go (2h) - WorkflowSpec, ActivityStep
|
||||
├─ jsonpath_evaluator.go (3h) - ${step1.output.path} resolution
|
||||
├─ execution_context.go (2h) - State tracking (vars, step results)
|
||||
├─ activity_registry.go (2h) - Activity name → function mapping
|
||||
├─ kmsvc_client.go (2-3h)- KMSvc queue polling (await-continue)
|
||||
└─ *_test.go (tests)
|
||||
|
||||
statemachine/
|
||||
├─ routing_workflow.go (4-5h)- ⭐ Main workflow logic
|
||||
└─ routing_workflow_test.go (2h) - Tests
|
||||
|
||||
cmd/worker/main.go: Register RoutingWorkflow (1h)
|
||||
```
|
||||
|
||||
**Deliverable**: RoutingWorkflow executes activity sequences with:
|
||||
- ✅ Sequential execution (A → B → C)
|
||||
- ✅ Await-continue pattern (KMSvc queue polling)
|
||||
- ✅ Retry with exponential backoff
|
||||
- ✅ Error catch blocks
|
||||
- ✅ JSONPath parameter resolution
|
||||
|
||||
**Risk**: Low-Medium (KMSvc queue polling adds complexity)
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: API Server (Week 2) — 12-14 Hours
|
||||
|
||||
**Goal**: HTTP endpoints for workflow submission and status polling
|
||||
|
||||
**Files to Create**:
|
||||
```
|
||||
cmd/api-server/
|
||||
├─ main.go (2h) - Server bootstrap
|
||||
├─ handlers.go (4h) - Routes: /workflows, /execute, status
|
||||
├─ validation.go (2h) - Validate WorkflowSpec
|
||||
├─ template_loader.go (2-3h)- Load predefined templates
|
||||
└─ *_test.go (2h) - Tests
|
||||
```
|
||||
|
||||
**Deliverable**:
|
||||
- `POST /api/v1/workflows` — Submit workflow (async)
|
||||
- `GET /api/v1/workflows/{id}/status` — Check status
|
||||
- `POST /api/v1/execute` — Execute template (sync)
|
||||
|
||||
**Risk**: Low (isolated, no breaking changes)
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: CLI (Week 2) — 8-10 Hours
|
||||
|
||||
**Goal**: Command-line interface for workflow submission
|
||||
|
||||
**Files to Create**:
|
||||
```
|
||||
cmd/cli/
|
||||
├─ main.go (2h) - Entry point, command routing
|
||||
└─ commands/
|
||||
├─ submit.go (2h) - poimen-cli submit workflow.json
|
||||
├─ status.go (2h) - poimen-cli status wf-id
|
||||
├─ template.go (1h) - poimen-cli template list/show
|
||||
├─ execute.go (1h) - poimen-cli execute template
|
||||
└─ history.go (1h) - poimen-cli history wf-id
|
||||
```
|
||||
|
||||
**Deliverable**:
|
||||
- `poimen-cli submit workflow.json [--wait] [--watch]`
|
||||
- `poimen-cli status wf-id [--wait]`
|
||||
- `poimen-cli execute template name --input-file input.json`
|
||||
- `poimen-cli template list/show`
|
||||
|
||||
**Risk**: Low
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Kubernetes Integration (Week 3) — 5 Hours
|
||||
|
||||
**Goal**: Deploy api-server to K8s, expose via ingress
|
||||
|
||||
**Files to Create**:
|
||||
```
|
||||
k8s/
|
||||
├─ api-server-deployment.yaml (2h) - 2 replicas, Temporal connection
|
||||
├─ api-server-service.yaml (1h) - Port 8080
|
||||
└─ api-server-networkpolicy.yaml(1h) - Allow ingress traffic
|
||||
|
||||
homelab-frontend/k8s/
|
||||
└─ gateway-routes.yaml (1h) - Route /api/v1/workflows to api-server
|
||||
```
|
||||
|
||||
**Deliverable**: API exposed at `https://api.riotpiao.com/api/v1/workflows`
|
||||
|
||||
**Risk**: Medium (network policy changes)
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Documentation (Week 4) — 7 Hours
|
||||
|
||||
**Files to Create**:
|
||||
```
|
||||
docs/
|
||||
├─ API.md (2h) - Complete API reference
|
||||
├─ CLI.md (2h) - CLI commands + examples
|
||||
├─ TEMPLATES.md (1h) - How to create/manage templates
|
||||
├─ MIGRATION.md (1h) - Migration from cmd/starter
|
||||
└─ examples/
|
||||
├─ sequential-workflow.json
|
||||
├─ retry-workflow.json
|
||||
└─ template-execute.json
|
||||
```
|
||||
|
||||
**Deliverable**: Complete documentation with working examples
|
||||
|
||||
**Risk**: Low
|
||||
|
||||
---
|
||||
|
||||
### Timeline Summary
|
||||
|
||||
```
|
||||
Phase 1: Routing Engine 20-22 hours (Week 1)
|
||||
Phase 2: API Server 12-14 hours (Week 2)
|
||||
Phase 3: CLI 8-10 hours (Week 2)
|
||||
Phase 4: K8s Integration 5 hours (Week 3)
|
||||
Phase 5: Documentation 7 hours (Week 4)
|
||||
─────────────────────────────────────────────────
|
||||
TOTAL 52-58 hours (~1.5 weeks)
|
||||
|
||||
Team: 2-3 engineers
|
||||
Duration: 4 weeks (with overlapping phases)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PART 6: CRITICAL KMSVC QUESTIONS
|
||||
|
||||
These answers **block implementation** of Phase 1:
|
||||
|
||||
### Q1: What is KMSvc?
|
||||
- Is it Kafka? Redis? AWS SQS? Custom system?
|
||||
- What Go client library should we use?
|
||||
- Example: `github.com/segmentio/kafka-go`?
|
||||
|
||||
### Q2: Message Format
|
||||
- Is it JSON?
|
||||
- What fields are required? (e.g., `correlation_id`, `result`, `status`?)
|
||||
- Example:
|
||||
```json
|
||||
{
|
||||
"correlation_id": "xyz123",
|
||||
"status": "completed",
|
||||
"result": {...}
|
||||
}
|
||||
```
|
||||
|
||||
### Q3: Correlation ID Management
|
||||
- Who generates the correlation_id? (LaunchJobActivity? Or RoutingWorkflow?)
|
||||
- Is it returned in LaunchJobActivity output?
|
||||
- Example: LaunchJobActivity returns:
|
||||
```json
|
||||
{
|
||||
"correlation_id": "abc123-generated-by-activity",
|
||||
"job_id": "job-123"
|
||||
}
|
||||
```
|
||||
|
||||
### Q4: Queue Polling Strategy
|
||||
- Active polling loop (pull messages every N seconds)? OR
|
||||
- Temporal Signal (push-based, external system sends signal to workflow)?
|
||||
- If polling: What interval? (1s, 5s, 10s?)
|
||||
|
||||
### Q5: Message Timeout Behavior
|
||||
- If no message arrives within timeout (e.g., 5m), what happens?
|
||||
- Option A: Jump to catch block (error handler)
|
||||
- Option B: Fail entire workflow
|
||||
- Option C: Infinite wait (only human intervention can stop)
|
||||
|
||||
### Q6: Consumer Group Strategy
|
||||
- Single shared consumer (all workflows share one connection)?
|
||||
- Per-workflow consumer (each workflow gets own connection)?
|
||||
- Dead-letter queue for unmatched messages?
|
||||
|
||||
---
|
||||
|
||||
## PART 7: KEY DESIGN DECISIONS
|
||||
|
||||
### Decision 1: Three Patterns (Sequential, Await-Task-Complete, Retry)
|
||||
**Rationale**: Covers 90% of use cases. Retry critical for production. Await-Task-Complete enables long-running jobs.
|
||||
**Alternative Rejected**: Just Sequential (too limited)
|
||||
|
||||
### Decision 2: KMSvc Queue for Await-Task-Complete
|
||||
**Rationale**: Decouples external systems from Temporal. External systems only need queue client, not Temporal SDK.
|
||||
**Alternative Rejected**: Temporal callback/signal (tight coupling)
|
||||
|
||||
### Decision 3: JSONPath for Parameters
|
||||
**Rationale**: Standard notation (AWS Step Functions, Kubernetes). Supports nested refs and arrays.
|
||||
**Alternative Rejected**: Go templating (overkill, security risks)
|
||||
|
||||
### Decision 4: RoutingWorkflow (not Activity)
|
||||
**Rationale**: Workflows can call activities with retries. Durable execution + replay guarantee.
|
||||
**Alternative Rejected**: RoutingActivity (can't call other activities, no retries)
|
||||
|
||||
### Decision 5: Static Activity Registry
|
||||
**Rationale**: Type-safe, simpler. Worker pod must compile all activities anyway.
|
||||
**Alternative Rejected**: Dynamic registration (complex, unsafe)
|
||||
|
||||
### Decision 6: Keep cmd/starter Forever
|
||||
**Rationale**: No need to break existing deployments. Let customers choose.
|
||||
**Alternative Rejected**: Deprecate (breaking change)
|
||||
|
||||
### Decision 7: Shared api-server (2 replicas)
|
||||
**Rationale**: Simpler, resource-efficient, easier to scale.
|
||||
**Alternative Rejected**: Per-namespace servers (overkill)
|
||||
|
||||
---
|
||||
|
||||
## PART 8: RISKS & MITIGATIONS
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|------|----------|-----------|
|
||||
| JSONPath evaluation bugs | Medium | Unit tests (>90% coverage), property-based testing |
|
||||
| **KMSvc queue reliability** | **High** | Dead-letter queue, timeouts, persistence checks |
|
||||
| **Correlation ID mismatch** | **Medium** | Strict matching, schema validation, detailed logging |
|
||||
| **Consumer group conflicts** | **Medium** | Clear strategy, rebalancing, message redelivery |
|
||||
| Activity registry mismatch | Low | Validate at spec submission time |
|
||||
| Temporal network issues | Low | Retry + timeout in api-server |
|
||||
| Cache staleness (api-server) | Low | 24h TTL, event-driven invalidation |
|
||||
| Performance regression | Low | Measure baseline: target <200ms submit, <100ms poll |
|
||||
|
||||
**Total Risk Score**: Medium-High (up from Low-Medium due to KMSvc)
|
||||
|
||||
---
|
||||
|
||||
## PART 9: SUCCESS CRITERIA
|
||||
|
||||
### Functional ✅
|
||||
- RoutingWorkflow executes activity sequences dynamically
|
||||
- JSONPath parameters resolve correctly (nested, arrays)
|
||||
- Error catch blocks jump to handlers
|
||||
- Retry logic with exponential backoff (1s → 2s → 4s)
|
||||
- API validates WorkflowSpec (no dangling refs)
|
||||
- CLI reads JSON (no hardcoded flags)
|
||||
- Status polling returns step-by-step results
|
||||
- Backward compatible (cmd/starter unchanged)
|
||||
|
||||
### Non-Functional ✅
|
||||
- Performance: <200ms workflow submit, <100ms poll
|
||||
- Uptime: 99.9% (2 replicas, rolling updates)
|
||||
- Test coverage: >90%
|
||||
- Documentation: Complete + 3 working examples
|
||||
|
||||
---
|
||||
|
||||
## PART 10: BACKWARD COMPATIBILITY
|
||||
|
||||
### What Stays Unchanged ✅
|
||||
- `cmd/starter/` (old CLI works exactly as before)
|
||||
- All existing activities
|
||||
- All existing workflows (OrchestratorWorkflow, TaskUnitWorkflow)
|
||||
- Temporal worker registration
|
||||
|
||||
### What's New ✅
|
||||
- `cmd/cli/` (new CLI)
|
||||
- `cmd/api-server/` (new HTTP API)
|
||||
- `internal/routing/` (new routing engine)
|
||||
- `statemachine/routing_workflow.go` (new workflow)
|
||||
|
||||
### Migration Path ✅
|
||||
- Customers can use OLD way or NEW way
|
||||
- No forced upgrades
|
||||
- No breaking changes
|
||||
|
||||
---
|
||||
|
||||
## PART 11: FILES SUMMARY
|
||||
|
||||
### New Files (Phase 1-5)
|
||||
|
||||
**Routing Engine** (Phase 1):
|
||||
```
|
||||
internal/routing/types.go
|
||||
internal/routing/jsonpath_evaluator.go
|
||||
internal/routing/execution_context.go
|
||||
internal/routing/activity_registry.go
|
||||
internal/routing/kmsvc_client.go
|
||||
statemachine/routing_workflow.go
|
||||
```
|
||||
|
||||
**API Server** (Phase 2):
|
||||
```
|
||||
cmd/api-server/main.go
|
||||
cmd/api-server/handlers.go
|
||||
cmd/api-server/validation.go
|
||||
cmd/api-server/template_loader.go
|
||||
```
|
||||
|
||||
**CLI** (Phase 3):
|
||||
```
|
||||
cmd/cli/main.go
|
||||
cmd/cli/commands/submit.go
|
||||
cmd/cli/commands/status.go
|
||||
cmd/cli/commands/template.go
|
||||
cmd/cli/commands/execute.go
|
||||
cmd/cli/commands/history.go
|
||||
```
|
||||
|
||||
**K8s** (Phase 4):
|
||||
```
|
||||
k8s/api-server-deployment.yaml
|
||||
k8s/api-server-service.yaml
|
||||
k8s/api-server-networkpolicy.yaml
|
||||
homelab-frontend/k8s/gateway-routes.yaml
|
||||
```
|
||||
|
||||
**Docs** (Phase 5):
|
||||
```
|
||||
docs/API.md
|
||||
docs/CLI.md
|
||||
docs/TEMPLATES.md
|
||||
docs/MIGRATION.md
|
||||
docs/examples/*.json
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
- `cmd/worker/main.go` (register RoutingWorkflow)
|
||||
- `homelab-frontend/k8s/network-policy.yaml` (already done ✅)
|
||||
|
||||
---
|
||||
|
||||
## PART 12: COMPARISON TABLE
|
||||
|
||||
| Aspect | Before (Hardcoded) | After (Dynamic) |
|
||||
|--------|---|---|
|
||||
| **Workflow Definition** | Go code (OrchestratorWorkflow) | JSON (customer-specified) |
|
||||
| **Sequencing** | Fixed (Clone→Plan→Impl→Judge) | Any (customer defines in JSON) |
|
||||
| **Parameters** | Direct: `in.RemoteURL` | JSONPath: `${input.repo}` |
|
||||
| **Change Time** | 5-10 min (recompile+deploy) | Seconds (API call) |
|
||||
| **Entry Points** | CLI only (hardcoded flags) | CLI + API + Legacy CLI |
|
||||
| **Error Handling** | Try-catch (per-activity) | Catch blocks (conditional jumps) |
|
||||
| **Retry Policy** | Temporal default | Configurable per-activity |
|
||||
| **Backward Compat** | N/A | 100% (cmd/starter unchanged) |
|
||||
| **Code Coupling** | Tight (to specific sequence) | Loose (generic RoutingWorkflow) |
|
||||
|
||||
---
|
||||
|
||||
## PART 13: APPROVAL CHECKLIST
|
||||
|
||||
### Must Answer (Blocks Phase 1)
|
||||
- [ ] Q1: What is KMSvc system?
|
||||
- [ ] Q2: Message format for KMSvc?
|
||||
- [ ] Q3: Who generates correlation_id?
|
||||
- [ ] Q4: Polling strategy (active or signal)?
|
||||
- [ ] Q5: Timeout behavior?
|
||||
- [ ] Q6: Consumer group strategy?
|
||||
|
||||
### Should Approve
|
||||
- [ ] Timeline OK? (52-58 hours)
|
||||
- [ ] Architecture OK? (RoutingWorkflow + 3 entry points)
|
||||
- [ ] Risk level acceptable? (Medium-High)
|
||||
- [ ] Backward compatibility OK? (Keep cmd/starter)
|
||||
|
||||
### Sign-Off
|
||||
- [ ] Tech Lead approval
|
||||
- [ ] Stakeholder sign-off
|
||||
- [ ] Ready to start Phase 1
|
||||
|
||||
---
|
||||
|
||||
## PART 14: NEXT STEPS
|
||||
|
||||
1. **Review This Document** (1-2 hours)
|
||||
- Read Parts 1-7 (understanding)
|
||||
- Read Parts 8-9 (risks & success)
|
||||
- Read Part 13 (approval checklist)
|
||||
|
||||
2. **Answer Critical Questions** (Q1-Q6)
|
||||
- Email or Slack responses required
|
||||
- Needed before Phase 1 can start
|
||||
|
||||
3. **Stakeholder Approval**
|
||||
- Architecture review meeting
|
||||
- Risk acknowledgment
|
||||
- Sign-off on timeline
|
||||
|
||||
4. **Start Implementation**
|
||||
- Week 1: Phase 1 (RoutingWorkflow)
|
||||
- Week 2: Phase 2 (API) + Phase 3 (CLI)
|
||||
- Week 3: Phase 4 (K8s deploy)
|
||||
- Week 4: Phase 5 (Docs)
|
||||
|
||||
---
|
||||
|
||||
## APPENDIX: EXAMPLE WORKFLOW JSON
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "code-review-pipeline",
|
||||
"input": {
|
||||
"repo": "https://github.com/rockliang/poimen",
|
||||
"branch": "feature/new-cli"
|
||||
},
|
||||
"activities": [
|
||||
{
|
||||
"name": "Clone",
|
||||
"resource": "CloneRepoActivity",
|
||||
"parameters": {
|
||||
"repo": "${input.repo}",
|
||||
"branch": "${input.branch}"
|
||||
},
|
||||
"timeout": "5m",
|
||||
"next": "Analyze"
|
||||
},
|
||||
{
|
||||
"name": "Analyze",
|
||||
"resource": "AnalyzeCodeActivity",
|
||||
"parameters": {
|
||||
"path": "${Clone.output.path}"
|
||||
},
|
||||
"timeout": "10m",
|
||||
"retry": {
|
||||
"maxAttempts": 3,
|
||||
"backoffRate": 2.0,
|
||||
"initialInterval": "1s"
|
||||
},
|
||||
"catch": [
|
||||
{
|
||||
"errorEquals": ["Timeout"],
|
||||
"next": "HandleTimeout"
|
||||
}
|
||||
],
|
||||
"next": "Judge"
|
||||
},
|
||||
{
|
||||
"name": "Judge",
|
||||
"resource": "JudgeActivity",
|
||||
"parameters": {
|
||||
"code": "${Analyze.output.code}",
|
||||
"analysis": "${Analyze.output.analysis}"
|
||||
},
|
||||
"timeout": "5m",
|
||||
"end": true
|
||||
},
|
||||
{
|
||||
"name": "HandleTimeout",
|
||||
"type": "pass",
|
||||
"result": {
|
||||
"status": "failed",
|
||||
"reason": "Analysis timed out after 10m"
|
||||
},
|
||||
"end": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## APPENDIX: CLI USAGE EXAMPLES
|
||||
|
||||
```bash
|
||||
# Submit workflow from JSON file
|
||||
$ poimen-cli submit workflow.json
|
||||
Workflow submitted: wf-abc123
|
||||
Poll status: poimen-cli status wf-abc123
|
||||
|
||||
# Submit and wait for results
|
||||
$ poimen-cli submit workflow.json --wait
|
||||
Waiting for workflow to complete...
|
||||
✅ Workflow completed
|
||||
Output:
|
||||
{
|
||||
"verdict": "approved",
|
||||
"score": 0.95,
|
||||
"issues": []
|
||||
}
|
||||
|
||||
# Check status
|
||||
$ poimen-cli status wf-abc123
|
||||
Workflow ID: wf-abc123
|
||||
Status: COMPLETED
|
||||
Started: 2025-01-31 15:04:05
|
||||
Completed: 2025-01-31 15:05:00
|
||||
|
||||
# Execute template immediately (sync)
|
||||
$ poimen-cli execute template code-review-v1 --input-file input.json
|
||||
Executing template: code-review-v1
|
||||
[15:04:05] Step 1/3: Clone ... OK (45s)
|
||||
[15:04:50] Step 2/3: Analyze ... OK (120s)
|
||||
[15:05:50] Step 3/3: Judge ... OK (30s)
|
||||
✅ COMPLETED
|
||||
Output: {...}
|
||||
|
||||
# List templates
|
||||
$ poimen-cli template list
|
||||
Available Templates:
|
||||
- code-review-v1
|
||||
- clone-analyze-v1
|
||||
- simple-test-v1
|
||||
|
||||
# Show template
|
||||
$ poimen-cli template show code-review-v1
|
||||
Template: code-review-v1
|
||||
Activities: 3
|
||||
|
||||
1. Clone (CloneRepoActivity)
|
||||
Timeout: 5m
|
||||
2. Analyze (AnalyzeCodeActivity)
|
||||
Timeout: 10m
|
||||
Retry: 3 attempts, backoff 2.0
|
||||
3. Judge (JudgeActivity)
|
||||
Timeout: 5m
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**END OF MASTER REVIEW DOCUMENT**
|
||||
|
||||
For detailed technical documentation, see:
|
||||
- SERVICE_INTEGRATION.md (patterns + API spec)
|
||||
- IMPLEMENTATION_PLAN.md (phase breakdown)
|
||||
- CLI_DESIGN.md (CLI implementation)
|
||||
- FINAL_COMPREHENSIVE_DESIGN.md (1-2 hour read)
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
# Poimen Service Integration & Dynamic Workflows — FINAL DESIGN
|
||||
|
||||
**Status**: Ready for Implementation Review
|
||||
**Updated**: 2025-01-31
|
||||
**Scope**: Replace hardcoded workflows with API-driven, JSON-based, customer-specified sequences
|
||||
|
||||
---
|
||||
|
||||
## 1-Minute Summary
|
||||
|
||||
### Problem
|
||||
Hardcoded workflows require code changes → recompile → deploy (5-10 min) to change activity sequences.
|
||||
|
||||
### Solution
|
||||
- New **RoutingWorkflow** reads JSON WorkflowSpec and executes dynamically
|
||||
- Three interfaces: **CLI** + **HTTP API** + **legacy CLI** (backward compat)
|
||||
- Three patterns: Sequential, Await-Task-Complete (KMSvc), Retry with error handling
|
||||
- JSONPath parameters: `${step1.output.path}`
|
||||
|
||||
### Result
|
||||
Workflow changes in **seconds** (API call) instead of 5-10 min (recompile)
|
||||
|
||||
---
|
||||
|
||||
## The Three Patterns
|
||||
|
||||
### 1. Sequential (A → B → C)
|
||||
Activities execute in sequence, each step passes output to next via JSONPath.
|
||||
|
||||
### 2. Await-Task-Complete (Launch → Poll Queue → Process)
|
||||
Launch long-running job, poll KMSvc queue for result with correlation ID matching, then proceed.
|
||||
|
||||
### 3. Retry with Error Handling
|
||||
Activity retry N times with exponential backoff, jump to catch block on failure.
|
||||
|
||||
---
|
||||
|
||||
## The Three Entry Points
|
||||
|
||||
### CLI (New)
|
||||
```bash
|
||||
poimen-cli submit workflow.json --wait
|
||||
poimen-cli execute template code-review-v1 --input-file input.json
|
||||
poimen-cli status wf-abc123
|
||||
```
|
||||
|
||||
### HTTP API (New)
|
||||
```bash
|
||||
curl -X POST /api/v1/workflows -d @workflow.json
|
||||
curl -X POST /api/v1/execute -d '{"template": "...", "input": {...}}'
|
||||
curl /api/v1/workflows/wf-abc123/status
|
||||
```
|
||||
|
||||
### Legacy CLI (Unchanged)
|
||||
```bash
|
||||
go run ./cmd/starter --repo ... --milestone T0 (still works)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation: 5 Phases
|
||||
|
||||
| Phase | Files | Hours | Week |
|
||||
|-------|-------|-------|------|
|
||||
| 1. Routing Engine | `internal/routing/`, `statemachine/routing_workflow.go` | 20-22 | 1 |
|
||||
| 2. API Server | `cmd/api-server/` | 12-14 | 2 |
|
||||
| 3. CLI | `cmd/cli/` | 8-10 | 2 |
|
||||
| 4. K8s Deploy | `k8s/`, `homelab-frontend/k8s/` | 5 | 3 |
|
||||
| 5. Docs | `docs/`, examples | 7 | 4 |
|
||||
| **TOTAL** | **~15 files** | **52-58** | **4 weeks** |
|
||||
|
||||
---
|
||||
|
||||
## Critical KMSvc Questions
|
||||
|
||||
1. **What is KMSvc?** (Kafka/Redis/SQS/custom?)
|
||||
2. **Message format?** (JSON with correlation_id field?)
|
||||
3. **Who generates correlation_id?** (Activity or Workflow?)
|
||||
4. **Polling strategy?** (Active loop or Temporal Signal?)
|
||||
5. **Timeout behavior?** (Catch block, fail, or infinite wait?)
|
||||
6. **Consumer group?** (Shared or per-workflow?)
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
✅ RoutingWorkflow executes sequences dynamically
|
||||
✅ JSONPath parameters resolve correctly
|
||||
✅ Error catch blocks work
|
||||
✅ Retry with backoff works
|
||||
✅ CLI reads JSON (no flags)
|
||||
✅ API validates specs
|
||||
✅ Backward compatible
|
||||
✅ Performance: <200ms submit
|
||||
✅ Test coverage: >90%
|
||||
|
||||
---
|
||||
|
||||
## Key Files
|
||||
|
||||
See full design doc (FINAL_COMPREHENSIVE_DESIGN.md in repo) for:
|
||||
- Detailed phase breakdown
|
||||
- Code examples
|
||||
- Risk assessment
|
||||
- Design decisions
|
||||
- Complete file list
|
||||
- Timeline details
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Answer KMSvc Q1-Q6
|
||||
2. Review 52-58 hour estimate
|
||||
3. Stakeholder approval
|
||||
4. Start Phase 1
|
||||
|
||||
See DESIGN_REVIEW_UPDATED.md for detailed questions.
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
# Poimen Declarative Workflow Implementation Plan
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Current State**: Hardcoded workflows (OrchestratorWorkflow, TaskUnitWorkflow)
|
||||
- Fixed flow: Clone → Plan → Implement → Judge → Commit
|
||||
- New flows require recompilation
|
||||
- No customer-specified sequencing
|
||||
|
||||
**Target State**: API-driven dynamic workflows
|
||||
- Customer sends JSON specifying activity sequence
|
||||
- Routing engine executes dynamically
|
||||
- Support 3 patterns: Sequential, Await-Task-Complete, Retry
|
||||
- Backward compatible with existing cmd/starter
|
||||
|
||||
**Timeline**: 4 weeks | **Effort**: ~80 story points | **Team**: 2-3 engineers
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Core Routing Engine (Week 1) — 16 hours
|
||||
|
||||
**Goal**: Build engine that parses WorkflowSpec and executes activity sequences dynamically
|
||||
|
||||
**Files to Create**:
|
||||
- `internal/routing/types.go` — Data structures for WorkflowSpec, ActivityStep, RetryConfig
|
||||
- `internal/routing/jsonpath_evaluator.go` — Parameter interpolation using JSONPath
|
||||
- `internal/routing/execution_context.go` — Execution state management
|
||||
- `internal/routing/activity_registry.go` — Activity name → implementation mapper
|
||||
- `statemachine/routing_workflow.go` — **Core RoutingWorkflow that ties everything together**
|
||||
- Tests: `*_test.go` files for each module
|
||||
|
||||
**Key Implementation Details**:
|
||||
- RoutingWorkflow is the main Temporal workflow
|
||||
- It parses activity steps and executes them sequentially
|
||||
- Uses JSONPath to interpolate parameters: `${step_1.output.path}`
|
||||
- Supports retry logic, error catching, and conditional jumps
|
||||
- All activities are pre-registered in worker pool
|
||||
|
||||
**Backward Compatibility**: ✅ Keep OrchestratorWorkflow, TaskUnitWorkflow unchanged
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: API Server (Week 2) — 10 hours
|
||||
|
||||
**Goal**: HTTP API that accepts WorkflowSpec JSON and manages workflow execution
|
||||
|
||||
**Files to Create**:
|
||||
- `cmd/api-server/main.go` — Server bootstrap, Temporal client connection
|
||||
- `cmd/api-server/handlers.go` — POST /api/v1/workflows, GET /api/v1/workflows/{id}/status
|
||||
- `cmd/api-server/validation.go` — WorkflowSpec validation (references, timeouts, etc.)
|
||||
- `cmd/api-server/server.go` — Server config and lifecycle
|
||||
- Tests: handlers, validation, integration tests
|
||||
|
||||
**Endpoints**:
|
||||
```
|
||||
POST /api/v1/workflows
|
||||
→ Submit WorkflowSpec
|
||||
→ Returns: {workflow_id, status, polling_url}
|
||||
|
||||
GET /api/v1/workflows/{workflow_id}/status
|
||||
→ Poll execution status
|
||||
→ Returns: {status, steps_completed, results, final_output}
|
||||
```
|
||||
|
||||
**Key Details**:
|
||||
- Validates WorkflowSpec before submission
|
||||
- Submits to RoutingWorkflow (not OrchestratorWorkflow)
|
||||
- Caches workflow metadata locally
|
||||
- Polls Temporal for status
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Homelab Frontend Integration (Week 3) — 5 hours
|
||||
|
||||
**Goal**: Expose API through ingress, connect to gateway
|
||||
|
||||
**Files to Create**:
|
||||
- `k8s/api-server-deployment.yaml` — K8s Deployment, 2 replicas
|
||||
- `k8s/api-server-service.yaml` — K8s Service for api-server
|
||||
- `k8s/api-server-networkpolicy.yaml` — Allow ingress traffic, deny by default
|
||||
- `homelab-frontend/k8s/gateway-routes.yaml` — Route /api/v1/workflows to api-server
|
||||
|
||||
**Key Details**:
|
||||
- api-server connects to Temporal: `temporal-frontend.temporal:7233`
|
||||
- Ingress routes external traffic to api-server:8080
|
||||
- NetworkPolicy allows only ingress-nginx → api-server traffic
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Migration & Docs (Week 4) — 7 hours
|
||||
|
||||
**Files to Create**:
|
||||
- `docs/API.md` — API reference (endpoints, request/response examples)
|
||||
- `docs/MIGRATION.md` — How to migrate from cmd/starter to API
|
||||
- `docs/examples/sequential-workflow.json` — Example: Clone → Analyze → Implement → Judge
|
||||
- `docs/examples/retry-workflow.json` — Example: with retry + error handling
|
||||
- `docs/examples/conditional-workflow.json` — Example: with catch blocks
|
||||
|
||||
**Key Details**:
|
||||
- Comprehensive examples for all 3 patterns
|
||||
- cmd/starter continues to work (backward compatible)
|
||||
- Documentation emphasizes JSONPath parameter passing
|
||||
|
||||
---
|
||||
|
||||
## Three Composite Patterns
|
||||
|
||||
### Pattern A: Sequential (Request-Response)
|
||||
```json
|
||||
{
|
||||
"activities": [
|
||||
{"name": "Step1", "resource": "ActivityA", "parameters": {...}, "next": "Step2"},
|
||||
{"name": "Step2", "resource": "ActivityB", "parameters": {"input": "${Step1.output}"}, "next": "Step3"},
|
||||
{"name": "Step3", "resource": "ActivityC", "parameters": {...}, "end": true}
|
||||
]
|
||||
}
|
||||
```
|
||||
**Flow**: Step1 → Step2 (receives Step1's output) → Step3
|
||||
|
||||
---
|
||||
|
||||
### Pattern B: Await-Task-Complete (Callback)
|
||||
```json
|
||||
{
|
||||
"activities": [
|
||||
{"name": "LaunchJob", "type": "await-callback", "resource": "LaunchActivity", "heartbeat_timeout": "5m", "next": "ProcessResult"},
|
||||
{"name": "ProcessResult", "resource": "ProcessActivity", "parameters": {"result": "${LaunchJob.callback}"}, "end": true}
|
||||
]
|
||||
}
|
||||
```
|
||||
**Flow**: LaunchJob (waits for callback) → ProcessResult (receives callback data)
|
||||
|
||||
---
|
||||
|
||||
### Pattern C: Retry with Error Handling
|
||||
```json
|
||||
{
|
||||
"activities": [
|
||||
{"name": "Generate", "resource": "ImplementerActivity", "retry": {"maxAttempts": 3, "backoffRate": 2.0}, "catch": [{"errorEquals": ["Timeout"], "next": "HandleTimeout"}], "next": "Verify"},
|
||||
{"name": "Verify", "resource": "JudgeActivity", "end": true},
|
||||
{"name": "HandleTimeout", "type": "pass", "result": {"status": "timeout"}, "end": true}
|
||||
]
|
||||
}
|
||||
```
|
||||
**Flow**: Generate (retry 3x) → if success: Verify, else: HandleTimeout
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- ✅ RoutingWorkflow executes dynamic sequences correctly
|
||||
- ✅ JSONPath parameter passing works (nested references, arrays)
|
||||
- ✅ Error handling and catch blocks work
|
||||
- ✅ Retry logic respects backoff intervals
|
||||
- ✅ API validates WorkflowSpec (no invalid refs)
|
||||
- ✅ Status polling returns step-by-step results
|
||||
- ✅ All tests pass (>90% coverage)
|
||||
- ✅ Backward compatible (cmd/starter still works)
|
||||
- ✅ API exposed via ingress at `/api/v1/workflows`
|
||||
- ✅ Performance: <200ms submit, <100ms poll
|
||||
|
||||
---
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```
|
||||
┌─ Customer/Frontend ─────────────────────────┐
|
||||
│ POST /api/v1/workflows (JSON WorkflowSpec) │
|
||||
└────────────────────┬────────────────────────┘
|
||||
↓
|
||||
┌──────────────────────┐
|
||||
│ API Server │
|
||||
│ • Validate spec │
|
||||
│ • Submit to Temporal │
|
||||
│ • Cache metadata │
|
||||
└────────────┬─────────┘
|
||||
↓
|
||||
┌────────────────────────────────┐
|
||||
│ Temporal Server │
|
||||
│ RoutingWorkflow │
|
||||
│ ├─ Parse activities │
|
||||
│ ├─ Evaluate JSONPath │
|
||||
│ ├─ Execute sequence │
|
||||
│ └─ Handle errors/retries │
|
||||
└────────────┬───────────────────┘
|
||||
↓
|
||||
┌────────────────────────────────┐
|
||||
│ Temporal Worker Pool │
|
||||
│ • Poll queue │
|
||||
│ • Execute activities │
|
||||
│ • Return results │
|
||||
└────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| JSONPath bugs | Unit tests + property-based testing |
|
||||
| Activity registry mismatch | Validate registered activities match spec |
|
||||
| Temporal network issues | Retry logic + timeouts in API server |
|
||||
| Cache staleness | Event-driven cache invalidation |
|
||||
|
||||
---
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
| Phase | Rollback Strategy |
|
||||
|-------|------------------|
|
||||
| Phase 1 | Keep old workflows, don't use RoutingWorkflow |
|
||||
| Phase 2 | Disable api-server, fall back to cmd/starter |
|
||||
| Phase 3 | Remove ingress, access api-server directly |
|
||||
| Phase 4 | Continue supporting both APIs (cmd/starter + API) |
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Week 1**: Start Phase 1 implementation (RoutingWorkflow)
|
||||
2. **Week 2**: Start Phase 2 (API Server)
|
||||
3. **Week 3**: Deploy to K8s (Phase 3)
|
||||
4. **Week 4**: Document, test, release (Phase 4)
|
||||
5. **Post-Release**: Monitor, gather feedback, plan Phase 2 enhancements (parallel, conditional branching)
|
||||
|
||||
@@ -0,0 +1,452 @@
|
||||
# Poimen Service Integration & Declarative Workflow Design
|
||||
|
||||
## Overview
|
||||
|
||||
**Current (Bad)**: Hardcoded workflows
|
||||
```
|
||||
CLI flags → OrchestratorWorkflow (fixed flow)
|
||||
```
|
||||
|
||||
**Proposed (Better)**: API-driven, customer-specified workflows
|
||||
```
|
||||
API Request (JSON) → Routing Agent → Dynamic Activity Sequence → Results
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 1: Service Integration Patterns (AWS Step Functions Inspired)
|
||||
|
||||
### Pattern 1: Sequential (Request-Response)
|
||||
|
||||
```yaml
|
||||
# Customer specifies: "Call A, then call B with A's output"
|
||||
activities:
|
||||
- name: CloneRepo
|
||||
type: request-response
|
||||
resource: CloneRepoActivity
|
||||
parameters:
|
||||
repo: "https://github.com/..."
|
||||
path: "/tmp/clone"
|
||||
timeout: 5m
|
||||
retry:
|
||||
maxAttempts: 3
|
||||
backoff: exponential
|
||||
|
||||
- name: Analyze
|
||||
type: request-response
|
||||
resource: AnalyzeCodeActivity
|
||||
parameters:
|
||||
path: $.CloneRepo.path # ← Reference previous output
|
||||
inputPath: "$.CloneRepo" # ← What to pass
|
||||
outputPath: "$" # ← What to return
|
||||
timeout: 10m
|
||||
```
|
||||
|
||||
**Execution**: CloneRepo waits → outputs result → Analyze receives it → outputs
|
||||
|
||||
---
|
||||
|
||||
### Pattern 2: Await-Task-Complete (Callback)
|
||||
|
||||
```yaml
|
||||
# Customer specifies: "Call A, wait for callback, then call B"
|
||||
activities:
|
||||
- name: LaunchJob
|
||||
type: await-callback
|
||||
resource: LaunchLongRunningJobActivity
|
||||
parameters:
|
||||
job_id: "job-123"
|
||||
heartbeat_timeout: 5m
|
||||
next: OnJobComplete
|
||||
|
||||
- name: OnJobComplete
|
||||
type: request-response
|
||||
resource: ProcessResultActivity
|
||||
parameters:
|
||||
result: $.LaunchJob.result
|
||||
```
|
||||
|
||||
**Execution**: LaunchJob sent → waits for callback (heartbeat) → ProcessResult receives callback data → outputs
|
||||
|
||||
---
|
||||
|
||||
### Pattern 3: Retry with Backoff
|
||||
|
||||
```yaml
|
||||
# Customer specifies: "Call A, if fails retry 3x with exponential backoff, else B"
|
||||
activities:
|
||||
- name: GenerateCode
|
||||
type: request-response
|
||||
resource: ImplementerActivity
|
||||
parameters:
|
||||
task: $.input.taskID
|
||||
model: $.input.model
|
||||
retry:
|
||||
maxAttempts: 3
|
||||
backoffRate: 2.0 # 1s → 2s → 4s
|
||||
initialInterval: 1s
|
||||
maxInterval: 10m
|
||||
catch:
|
||||
- errorEquals: ["Timeout", "ThrottlingException"]
|
||||
next: HandleError
|
||||
- errorEquals: ["States.ALL"]
|
||||
next: HandleError
|
||||
next: JudgeCode
|
||||
|
||||
- name: JudgeCode
|
||||
type: request-response
|
||||
resource: JudgeActivity
|
||||
parameters:
|
||||
implementation: $.GenerateCode.code
|
||||
|
||||
- name: HandleError
|
||||
type: pass
|
||||
result: { status: "failed", reason: $.error }
|
||||
resultPath: "$.error_output"
|
||||
end: true
|
||||
```
|
||||
|
||||
**Execution**: GenerateCode fails → retry 1 (wait 1s) → fails → retry 2 (wait 2s) → fails → HandleError
|
||||
|
||||
---
|
||||
|
||||
## Part 2: Routing Agent Architecture
|
||||
|
||||
### 2.1 Routing Agent Activity (New)
|
||||
|
||||
```go
|
||||
// internal/routing/routing_agent.go
|
||||
|
||||
type WorkflowSpec struct {
|
||||
WorkflowID string `json:"workflow_id"`
|
||||
Name string `json:"name"`
|
||||
Input map[string]interface{} `json:"input"`
|
||||
Activities []ActivityStep `json:"activities"`
|
||||
RetryPolicy RetryConfig `json:"retry_policy,omitempty"`
|
||||
}
|
||||
|
||||
type ActivityStep struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"` // request-response, await-callback, parallel, choice
|
||||
Resource string `json:"resource"` // Activity name
|
||||
Parameters map[string]interface{} `json:"parameters"`
|
||||
InputPath string `json:"input_path"` // JSONPath: "$" or "$.previous.output"
|
||||
OutputPath string `json:"output_path"` // What to extract
|
||||
Timeout time.Duration `json:"timeout"`
|
||||
Retry *RetryConfig `json:"retry,omitempty"`
|
||||
Catch []CatchBlock `json:"catch,omitempty"`
|
||||
Next string `json:"next,omitempty"`
|
||||
End bool `json:"end,omitempty"`
|
||||
}
|
||||
|
||||
type RetryConfig struct {
|
||||
MaxAttempts int `json:"max_attempts"`
|
||||
BackoffRate float64 `json:"backoff_rate"`
|
||||
InitialInterval time.Duration `json:"initial_interval"`
|
||||
MaxInterval time.Duration `json:"max_interval"`
|
||||
}
|
||||
|
||||
type CatchBlock struct {
|
||||
ErrorEquals []string `json:"error_equals"`
|
||||
Next string `json:"next"`
|
||||
ResultPath string `json:"result_path,omitempty"`
|
||||
}
|
||||
|
||||
// RoutingAgentActivity: Receives WorkflowSpec, executes activities sequentially
|
||||
func RoutingAgentActivity(ctx context.Context, spec WorkflowSpec) (map[string]interface{}, error) {
|
||||
// 1. Parse WorkflowSpec
|
||||
// 2. Build execution plan (DAG of activities)
|
||||
// 3. For each step:
|
||||
// a. Evaluate JSONPath inputs
|
||||
// b. Execute activity with retries
|
||||
// c. Handle errors (catch blocks)
|
||||
// d. Store output in context
|
||||
// e. Move to next step
|
||||
// 4. Return final result
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 Activity Mapper Registry (New)
|
||||
|
||||
```go
|
||||
// internal/routing/activity_registry.go
|
||||
|
||||
type ActivityMapper struct {
|
||||
activities map[string]interface{}
|
||||
}
|
||||
|
||||
func NewActivityMapper() *ActivityMapper {
|
||||
return &ActivityMapper{
|
||||
activities: map[string]interface{}{
|
||||
"CloneRepoActivity": action.CloneRepoActivity,
|
||||
"AnalyzeCodeActivity": action.AnalyzeCodeActivity,
|
||||
"ImplementerActivity": action.ImplementerActivity,
|
||||
"JudgeActivity": action.JudgeActivity,
|
||||
"GitCommitActivity": action.GitCommitActivity,
|
||||
// ... all activities registered here
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (am *ActivityMapper) Get(name string) (interface{}, error) {
|
||||
if activity, ok := am.activities[name]; ok {
|
||||
return activity, nil
|
||||
}
|
||||
return nil, fmt.Errorf("activity %s not found", name)
|
||||
}
|
||||
|
||||
// cmd/worker/main.go
|
||||
w.RegisterActivity(routing.RoutingAgentActivity) // ← Only register this
|
||||
w.RegisterActivity(action.CloneRepoActivity) // ← Still register all individual activities
|
||||
// ... rest of activities
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 3: API Message Format (General Purpose)
|
||||
|
||||
### 3.1 Workflow Request Format
|
||||
|
||||
```json
|
||||
POST /api/workflows
|
||||
{
|
||||
"workflow": {
|
||||
"id": "workflow-123",
|
||||
"name": "code-review-pipeline",
|
||||
"version": "v1",
|
||||
"input": {
|
||||
"repo": "https://github.com/rockliang/poimen",
|
||||
"branch": "feature/new-feature",
|
||||
"milestone": "T0"
|
||||
},
|
||||
"activities": [
|
||||
{
|
||||
"id": "step_1",
|
||||
"name": "CloneRepo",
|
||||
"type": "request-response",
|
||||
"resource": "CloneRepoActivity",
|
||||
"parameters": {
|
||||
"repo": "${input.repo}",
|
||||
"path": "/tmp/work"
|
||||
},
|
||||
"timeout": "5m",
|
||||
"next": "step_2"
|
||||
},
|
||||
{
|
||||
"id": "step_2",
|
||||
"name": "AnalyzeCode",
|
||||
"type": "request-response",
|
||||
"resource": "AnalyzeCodeActivity",
|
||||
"parameters": {
|
||||
"path": "${step_1.output.path}"
|
||||
},
|
||||
"inputPath": "$.step_1",
|
||||
"outputPath": "$",
|
||||
"timeout": "10m",
|
||||
"retry": {
|
||||
"maxAttempts": 3,
|
||||
"backoffRate": 2.0,
|
||||
"initialInterval": "1s"
|
||||
},
|
||||
"catch": [
|
||||
{
|
||||
"errorEquals": ["Timeout"],
|
||||
"next": "handle_timeout"
|
||||
}
|
||||
],
|
||||
"next": "step_3"
|
||||
},
|
||||
{
|
||||
"id": "step_3",
|
||||
"name": "GenerateImplementation",
|
||||
"type": "request-response",
|
||||
"resource": "ImplementerActivity",
|
||||
"parameters": {
|
||||
"analysis": "${step_2.output.analysis}",
|
||||
"model": "claude-sonnet-5"
|
||||
},
|
||||
"timeout": "30m",
|
||||
"retry": {
|
||||
"maxAttempts": 3,
|
||||
"backoffRate": 2.0
|
||||
},
|
||||
"next": "step_4"
|
||||
},
|
||||
{
|
||||
"id": "step_4",
|
||||
"name": "Review",
|
||||
"type": "request-response",
|
||||
"resource": "JudgeActivity",
|
||||
"parameters": {
|
||||
"code": "${step_3.output.code}",
|
||||
"analysis": "${step_2.output.analysis}"
|
||||
},
|
||||
"timeout": "10m",
|
||||
"end": true
|
||||
},
|
||||
{
|
||||
"id": "handle_timeout",
|
||||
"name": "TimeoutHandler",
|
||||
"type": "pass",
|
||||
"result": {
|
||||
"status": "failed",
|
||||
"reason": "Analysis timed out"
|
||||
},
|
||||
"end": true
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 Workflow Response Format
|
||||
|
||||
```json
|
||||
{
|
||||
"workflow_id": "workflow-123",
|
||||
"status": "RUNNING",
|
||||
"temporal_workflow_id": "orch-workflow-123",
|
||||
"task_queue": "poimen-taskqueue",
|
||||
"execution_started_at": "2025-01-29T10:00:00Z",
|
||||
"monitoring": {
|
||||
"web_ui": "http://temporal.local:8080/namespaces/poimen/workflows/orch-workflow-123",
|
||||
"poll_url": "/api/workflows/workflow-123/status"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 Status Poll Format
|
||||
|
||||
```json
|
||||
GET /api/workflows/workflow-123/status
|
||||
{
|
||||
"workflow_id": "workflow-123",
|
||||
"status": "COMPLETED",
|
||||
"steps_completed": 4,
|
||||
"total_steps": 4,
|
||||
"results": {
|
||||
"step_1": {
|
||||
"name": "CloneRepo",
|
||||
"status": "COMPLETED",
|
||||
"duration": "45s",
|
||||
"output": {
|
||||
"path": "/tmp/work/poimen",
|
||||
"commit": "abc123"
|
||||
}
|
||||
},
|
||||
"step_2": {
|
||||
"name": "AnalyzeCode",
|
||||
"status": "COMPLETED",
|
||||
"duration": "120s",
|
||||
"output": {
|
||||
"analysis": "..."
|
||||
}
|
||||
},
|
||||
"step_3": {
|
||||
"name": "GenerateImplementation",
|
||||
"status": "COMPLETED",
|
||||
"duration": "300s",
|
||||
"output": {
|
||||
"code": "..."
|
||||
}
|
||||
},
|
||||
"step_4": {
|
||||
"name": "Review",
|
||||
"status": "COMPLETED",
|
||||
"duration": "60s",
|
||||
"output": {
|
||||
"verdict": "approved",
|
||||
"score": 0.95
|
||||
}
|
||||
}
|
||||
},
|
||||
"final_output": {
|
||||
"verdict": "approved",
|
||||
"score": 0.95
|
||||
},
|
||||
"completed_at": "2025-01-29T10:10:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Part 4: Implementation Plan
|
||||
|
||||
### Phase 1: Core Routing Engine (Week 1)
|
||||
- [ ] Create `internal/routing/types.go` (WorkflowSpec, ActivityStep structs)
|
||||
- [ ] Create `internal/routing/jsonpath_evaluator.go` (parameter interpolation)
|
||||
- [ ] Create `internal/routing/execution_context.go` (execution state management)
|
||||
- [ ] Create `internal/routing/activity_registry.go` (activity mapper)
|
||||
- [ ] Create `statemachine/routing_workflow.go` (RoutingWorkflow implementation)
|
||||
- [ ] Update `cmd/worker/main.go` (register RoutingWorkflow)
|
||||
- [ ] Write tests for routing engine
|
||||
|
||||
### Phase 2: API Server (Week 2)
|
||||
- [ ] Create `cmd/api-server/main.go` (HTTP server bootstrap)
|
||||
- [ ] Create `cmd/api-server/handlers.go` (POST /workflows, GET /status)
|
||||
- [ ] Create `cmd/api-server/validation.go` (WorkflowSpec validation)
|
||||
- [ ] Create `internal/routing/client.go` (Temporal client wrapper)
|
||||
- [ ] Write integration tests
|
||||
|
||||
### Phase 3: Homelab Frontend Integration (Week 3)
|
||||
- [ ] Create `k8s/gateway-routes.yaml` (route /api/v1/workflows to api-server)
|
||||
- [ ] Deploy api-server as K8s Service
|
||||
- [ ] Update NetworkPolicy to allow traffic to api-server
|
||||
- [ ] Test end-to-end API flow
|
||||
|
||||
### Phase 4: Migration & Cleanup (Week 4)
|
||||
- [ ] Keep cmd/starter for backward compatibility
|
||||
- [ ] Document migration path
|
||||
- [ ] Update examples to use new API
|
||||
- [ ] Performance tuning
|
||||
|
||||
---
|
||||
|
||||
## Files to Create
|
||||
|
||||
```
|
||||
NEW:
|
||||
internal/routing/
|
||||
├─ types.go (WorkflowSpec, ActivityStep, RetryConfig)
|
||||
├─ jsonpath_evaluator.go (Parameter interpolation)
|
||||
├─ execution_context.go (State management)
|
||||
├─ activity_registry.go (Activity mapper)
|
||||
└─ routing_workflow.go (RoutingWorkflow implementation)
|
||||
|
||||
statemachine/
|
||||
└─ routing_workflow.go (See Part 5 below)
|
||||
|
||||
cmd/api-server/
|
||||
├─ main.go (Server bootstrap)
|
||||
├─ handlers.go (HTTP handlers)
|
||||
└─ validation.go (Spec validation)
|
||||
|
||||
MODIFY:
|
||||
cmd/worker/main.go (Register RoutingWorkflow)
|
||||
|
||||
NEW (Homelab Frontend):
|
||||
k8s/
|
||||
└─ gateway-routes.yaml (Route definitions)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
✅ **Existing cmd/starter** still works (calls OrchestratorWorkflow)
|
||||
✅ **New API** calls RoutingWorkflow
|
||||
✅ **Worker** registers both workflows
|
||||
✅ **No breaking changes** to current deployments
|
||||
|
||||
---
|
||||
|
||||
## Benefits
|
||||
|
||||
✅ Customer-driven workflows (JSON API)
|
||||
✅ Service Integration patterns (Sequential, Await-Task-Complete, Retry)
|
||||
✅ No recompilation needed
|
||||
✅ Full Temporal observability
|
||||
✅ JSONPath-based parameter passing
|
||||
✅ Composable activities
|
||||
✅ Error handling & retries built-in
|
||||
|
||||
+1
-1
@@ -9,6 +9,6 @@ metadata:
|
||||
app.kubernetes.io/name: poimen
|
||||
app.kubernetes.io/component: orchestrator
|
||||
data:
|
||||
GIT_COMMIT: "74a8f2e8" # Updated automatically by CI/CD
|
||||
GIT_COMMIT: "04792563" # Updated automatically by CI/CD
|
||||
GIT_BRANCH: "main"
|
||||
DEPLOYMENT_DATE: "2026-08-31"
|
||||
|
||||
@@ -13,7 +13,7 @@ spec:
|
||||
labels:
|
||||
app: poimen-worker
|
||||
annotations:
|
||||
git-commit: "74a8f2e8" # ✅ Updated on each push, triggers rolling restart
|
||||
git-commit: "04792563" # ✅ Updated on each push, triggers rolling restart
|
||||
deployment-date: "2026-08-31"
|
||||
spec:
|
||||
containers:
|
||||
|
||||
Reference in New Issue
Block a user