poimen-workflows/DESIGN_MASTER_REVIEW.md

Poimen Service Integration & Dynamic Workflows — MASTER REVIEW DOCUMENT

Status: Ready for Approval
Version: 1.0
Date: 2025-01-31
Duration: 52-58 hours (4 weeks)

---

EXECUTIVE SUMMARY (5 MINUTES)

Problem

Current workflows are hardcoded in Go. Changing activity sequences requires:

• Code modification
• Recompilation
• K8s deployment
• Pod restart
• Total: 5-10 minutes

Solution

New RoutingWorkflow reads JSON WorkflowSpec and executes dynamically:

• Three entry points: CLI + HTTP API + Legacy CLI (backward compatible)
• Three patterns: Sequential, Await-Task-Complete (KMSvc queue), Retry with error handling
• JSONPath parameters: ${step1.output.path} instead of hardcoded values
• Result: Workflow changes in seconds (API call only)

Impact

✅ Instant changes | ✅ Three interfaces | ✅ Full compatibility | ❌ +KMSvc complexity

---

PART 1: THE THREE PATTERNS

Pattern 1: Sequential (A → B → C)

Activities execute in sequence. Each step passes output to next via JSONPath.

{
  "activities": [
    {
      "name": "Clone",
      "resource": "CloneRepoActivity",
      "parameters": {"repo": "${input.repo}"},
      "timeout": "5m",
      "next": "Analyze"
    },
    {
      "name": "Analyze",
      "resource": "AnalyzeCodeActivity",
      "parameters": {"path": "${Clone.output.path}"},
      "timeout": "10m",
      "next": "Judge"
    },
    {
      "name": "Judge",
      "resource": "JudgeActivity",
      "parameters": {"code": "${Analyze.output.code}"},
      "timeout": "5m",
      "end": true
    }
  ]
}

Execution Flow:

Clone ──(output: {path: /tmp/repo})──> Analyze ──(output: {code: ...})──> Judge ──(final result)

Use Case: Code review pipeline (clone → analyze → judge)

---

Pattern 2: Await-Task-Complete (Launch → Poll Queue → Process)

Launch long-running job, poll KMSvc queue for result with correlation ID matching, then proceed.

{
  "name": "LaunchJob",
  "type": "awaitTaskComplete",
  "resource": "LaunchJobActivity",
  "parameters": {"job_id": "job-123"},
  "queue": "job-completions",
  "correlation_key": "${LaunchJob.output.correlation_id}",
  "timeout": "5m",
  "next": "ProcessResult"
}

Execution Flow:

LaunchJobActivity ──(returns {correlation_id: "xyz"})──> Poll Queue "job-completions"
                                                            ↓
                                                    [External system completes job]
                                                    [Publishes {correlation_id: "xyz", result: ...}]
                                                            ↓
                                                    [Workflow receives message]
                                                    [Proceeds to ProcessResult]

Note: Pattern 2 type name is awaitTaskComplete (awaits task completion via queue)

Use Case: Long-running batch jobs (model training, data processing)

---

Pattern 3: Retry with Error Handling (Retry N times with backoff)

Activity retries N times with exponential backoff. On failure, jumps to catch block.

{
  "name": "Generate",
  "resource": "ImplementerActivity",
  "parameters": {"task": "${input.task}"},
  "timeout": "30m",
  "retry": {
    "maxAttempts": 3,
    "backoffRate": 2.0,
    "initialInterval": "1s"
  },
  "catch": [
    {"errorEquals": ["Timeout"], "next": "HandleTimeout"}
  ],
  "next": "Verify"
}

Execution Flow:

Generate (attempt 1) ──TIMEOUT──> [Wait 1s]
Generate (attempt 2) ──TIMEOUT──> [Wait 2s]
Generate (attempt 3) ──TIMEOUT──> [Wait 4s]
Max retries exceeded ──> Jump to HandleTimeout

Use Case: Code generation (retry on timeout, handle gracefully)

---

PART 2: THE THREE ENTRY POINTS

Entry Point 1: CLI (New Command-Line)

# Submit workflow from JSON file
$ poimen-cli submit workflow.json
Workflow submitted: wf-abc123
Poll status: poimen-cli status wf-abc123

# Submit and wait for results
$ poimen-cli submit workflow.json --wait
Waiting for workflow to complete...
✅ Workflow completed
Output: {...}

# Execute template immediately (sync, no polling)
$ poimen-cli execute template code-review-v1 --input-file input.json
Executing template...
Step 1/3: Clone ... OK (45s)
Step 2/3: Analyze ... OK (120s)
Step 3/3: Judge ... OK (30s)
✅ COMPLETED
Output: {...}

# Check status later
$ poimen-cli status wf-abc123
Workflow ID: wf-abc123
Status: COMPLETED
Started: 2025-01-31 15:04:05
Completed: 2025-01-31 15:05:00

# List templates
$ poimen-cli template list
Available Templates:
  - code-review-v1
  - clone-analyze-v1

Implementation: cmd/cli/ (new directory)

---

Entry Point 2: HTTP API (New)

Route 1: Submit Workflow (Async)

curl -X POST http://api.example.com/api/v1/workflows \
  -H "Content-Type: application/json" \
  -d @workflow.json

Response:

{
  "workflow_id": "wf-abc123",
  "status": "RUNNING",
  "polling_url": "/api/v1/workflows/wf-abc123/status"
}

Check Status

curl http://api.example.com/api/v1/workflows/wf-abc123/status

Response:

{
  "workflow_id": "wf-abc123",
  "status": "COMPLETED",
  "steps_completed": 3,
  "results": {
    "Clone": { "status": "COMPLETED", "output": {...} },
    "Analyze": { "status": "COMPLETED", "output": {...} },
    "Judge": { "status": "COMPLETED", "output": {...} }
  },
  "final_output": {...}
}

Route 2: Execute Template (Sync)

curl -X POST http://api.example.com/api/v1/execute \
  -H "Content-Type: application/json" \
  -d '{
    "template": "code-review-v1",
    "input": {"repo": "https://github.com/...", "branch": "feature/x"},
    "timeout": "2m"
  }'

Response (immediate):

{
  "status": "COMPLETED",
  "results": {...},
  "final_output": {...}
}

Implementation: cmd/api-server/ (new directory)

---

Entry Point 3: Legacy CLI (Unchanged)

# Old way still works, completely backward compatible
$ go run ./cmd/starter \
  --repo https://github.com/rockliang/poimen \
  --remote file:///tmp/poimen-output \
  --milestone T0 \
  --planner-model ornith \
  --judge-model ornith \
  --implementer-model claude-sonnet-5

Workflow submitted: orch-poimen
Status: RUNNING

Implementation: cmd/starter/ (existing, unchanged)

---

PART 3: ARCHITECTURE OVERVIEW

┌─────────────────────────────────────────────────────────────────┐
│                   THREE ENTRY POINTS                             │
├─────────────────────────────────────────────────────────────────┤
│                                                                   │
│  CLI (JSON file)           HTTP API (JSON POST)    Legacy CLI   │
│  ┌──────────────┐          ┌──────────────┐        ┌─────────┐  │
│  │ poimen-cli   │          │ /api/v1/     │        │ cmd/    │  │
│  │ submit       │          │ workflows    │        │ starter │  │
│  └──────┬───────┘          └──────┬───────┘        └────┬────┘  │
│         │                         │                     │        │
└─────────┼─────────────────────────┼─────────────────────┼────────┘
          │                         │                     │
          └─────────────────────────┼─────────────────────┘
                                    ↓
                    ┌───────────────────────────┐
                    │   RoutingWorkflow         │
                    │   (NEW - generic)         │
                    │                           │
                    │ 1. Parse WorkflowSpec     │
                    │ 2. For each activity:     │
                    │    - Resolve JSONPath     │
                    │    - Execute activity     │
                    │    - Handle errors        │
                    │    - Store result         │
                    │ 3. Return results         │
                    └───────────┬───────────────┘
                                ↓
                    ┌───────────────────────────┐
                    │  Temporal Server          │
                    │                           │
                    │  Worker Pool:             │
                    │  - Executes activities    │
                    │  - Returns results        │
                    │  - Retries with backoff   │
                    └───────────────────────────┘

---

PART 4: BEFORE vs AFTER

Before (Hardcoded Workflows)

# To change activity sequence:
1. Modify Go code (OrchestratorWorkflow in statemachine/)
2. Compile: go build ./cmd/worker
3. Push docker image
4. Deploy to K8s: kubectl set image deployment/poimen-worker
5. Wait for rollout
Time: 5-10 minutes

Workflow definition:

// Hard to change, tight coupling to specific sequence
if err := workflow.ExecuteActivity(ctx, "CloneRepoActivity", params).Get(ctx, nil); err != nil {
    return err
}
if err := workflow.ExecuteActivity(ctx, "PlanningActivity", params2).Get(ctx, nil); err != nil {
    return err
}
// ... more activities hardcoded

---

After (Dynamic Workflows)

# To change activity sequence:
1. Edit workflow.json (define sequence)
2. Call API or CLI
3. Done
Time: Seconds

Workflow definition:

{
  "activities": [
    {"name": "Clone", "resource": "CloneRepoActivity", "parameters": {...}, "next": "Analyze"},
    {"name": "Analyze", "resource": "AnalyzeCodeActivity", "parameters": {...}, "next": "Judge"},
    {"name": "Judge", "resource": "JudgeActivity", "parameters": {...}, "end": true}
  ]
}

---

PART 5: IMPLEMENTATION PHASES

Phase 1: Routing Engine (Week 1) — 20-22 Hours

Goal: Core RoutingWorkflow that executes JSON sequences dynamically

Files to Create:

internal/routing/
  ├─ types.go                    (2h)  - WorkflowSpec, ActivityStep
  ├─ jsonpath_evaluator.go       (3h)  - ${step1.output.path} resolution
  ├─ execution_context.go        (2h)  - State tracking (vars, step results)
  ├─ activity_registry.go        (2h)  - Activity name → function mapping
  ├─ kmsvc_client.go             (2-3h)- KMSvc queue polling (awaitTaskComplete)
  └─ *_test.go                   (tests)

statemachine/
  ├─ routing_workflow.go         (4-5h)- ⭐ Main workflow logic
  └─ routing_workflow_test.go    (2h)  - Tests

cmd/worker/main.go: Register RoutingWorkflow (1h)

Deliverable: RoutingWorkflow executes activity sequences with:

• ✅ Sequential execution (A → B → C)
• ✅ Await-Task-Complete pattern (KMSvc queue polling)
• ✅ Retry with exponential backoff
• ✅ Error catch blocks
• ✅ JSONPath parameter resolution

Risk: Low-Medium (KMSvc queue polling adds complexity)

---

Phase 2: API Server (Week 2) — 12-14 Hours

Goal: HTTP endpoints for workflow submission and status polling

Files to Create:

cmd/api-server/
  ├─ main.go                     (2h)  - Server bootstrap
  ├─ handlers.go                 (4h)  - Routes: /workflows, /execute, status
  ├─ validation.go               (2h)  - Validate WorkflowSpec
  ├─ template_loader.go          (2-3h)- Load predefined templates
  └─ *_test.go                   (2h)  - Tests

Deliverable:

• POST /api/v1/workflows — Submit workflow (async)
• GET /api/v1/workflows/{id}/status — Check status
• POST /api/v1/execute — Execute template (sync)

Risk: Low (isolated, no breaking changes)

---

Phase 3: CLI (Week 2) — 8-10 Hours

Goal: Command-line interface for workflow submission

Files to Create:

cmd/cli/
  ├─ main.go                     (2h)  - Entry point, command routing
  └─ commands/
     ├─ submit.go                (2h)  - poimen-cli submit workflow.json
     ├─ status.go                (2h)  - poimen-cli status wf-id
     ├─ template.go              (1h)  - poimen-cli template list/show
     ├─ execute.go               (1h)  - poimen-cli execute template
     └─ history.go               (1h)  - poimen-cli history wf-id

Deliverable:

• poimen-cli submit workflow.json [--wait] [--watch]
• poimen-cli status wf-id [--wait]
• poimen-cli execute template name --input-file input.json
• poimen-cli template list/show

Risk: Low

---

Phase 4: Kubernetes Integration (Week 3) — 5 Hours

Goal: Deploy api-server to K8s, expose via ingress

Files to Create:

k8s/
  ├─ api-server-deployment.yaml  (2h)  - 2 replicas, Temporal connection
  ├─ api-server-service.yaml     (1h)  - Port 8080
  └─ api-server-networkpolicy.yaml(1h) - Allow ingress traffic

homelab-frontend/k8s/
  └─ gateway-routes.yaml         (1h)  - Route /api/v1/workflows to api-server

Deliverable: API exposed at https://api.riotpiao.com/api/v1/workflows

Risk: Medium (network policy changes)

---

Phase 5: Documentation (Week 4) — 7 Hours

Files to Create:

docs/
  ├─ API.md                      (2h)  - Complete API reference
  ├─ CLI.md                      (2h)  - CLI commands + examples
  ├─ TEMPLATES.md                (1h)  - How to create/manage templates
  ├─ MIGRATION.md                (1h)  - Migration from cmd/starter
  └─ examples/
     ├─ sequential-workflow.json
     ├─ retry-workflow.json
     └─ template-execute.json

Deliverable: Complete documentation with working examples

Risk: Low

---

Timeline Summary

Phase 1: Routing Engine      20-22 hours  (Week 1)
Phase 2: API Server          12-14 hours  (Week 2)
Phase 3: CLI                  8-10 hours  (Week 2)
Phase 4: K8s Integration       5 hours    (Week 3)
Phase 5: Documentation         7 hours    (Week 4)
─────────────────────────────────────────────────────
TOTAL                         52-58 hours (~1.5 weeks)

Team: 2-3 engineers
Duration: 4 weeks (with overlapping phases)

---

PART 6: CRITICAL KMSVC QUESTIONS

These answers block implementation of Phase 1:

Q1: What is KMSvc?

• Is it Kafka? Redis? AWS SQS? Custom system?
• What Go client library should we use?
• Example: github.com/segmentio/kafka-go?

Q2: Message Format

• Is it JSON?
• What fields are required? (e.g., correlation_id, result, status?)
• Example:

{
  "correlation_id": "xyz123",
  "status": "completed",
  "result": {...}
}

Q3: Correlation ID Management

• Who generates the correlation_id? (LaunchJobActivity? Or RoutingWorkflow?)
• Is it returned in LaunchJobActivity output?
• Example: LaunchJobActivity returns:

{
  "correlation_id": "abc123-generated-by-activity",
  "job_id": "job-123"
}

Q4: Queue Polling Strategy

• Active polling loop (pull messages every N seconds)? OR
• Temporal Signal (push-based, external system sends signal to workflow)?
• If polling: What interval? (1s, 5s, 10s?)

Q5: Message Timeout Behavior

• If no message arrives within timeout (e.g., 5m), what happens?
  • Option A: Jump to catch block (error handler)
  • Option B: Fail entire workflow
  • Option C: Infinite wait (only human intervention can stop)

Q6: Consumer Group Strategy

• Single shared consumer (all workflows share one connection)?
• Per-workflow consumer (each workflow gets own connection)?
• Dead-letter queue for unmatched messages?

---

PART 7: KEY DESIGN DECISIONS

Decision 1: Three Patterns (Sequential, Await-Task-Complete, Retry)

Rationale: Covers 90% of use cases. Retry critical for production. Await-Task-Complete enables long-running jobs. Alternative Rejected: Just Sequential (too limited)

Decision 2: KMSvc Queue for Await-Task-Complete Pattern

Rationale: Decouples external systems from Temporal. External systems only need queue client, not Temporal SDK. Alternative Rejected: Temporal callback/signal (tight coupling)

Decision 3: JSONPath for Parameters

Rationale: Standard notation (AWS Step Functions, Kubernetes). Supports nested refs and arrays. Alternative Rejected: Go templating (overkill, security risks)

Decision 4: RoutingWorkflow (not Activity)

Rationale: Workflows can call activities with retries. Durable execution + replay guarantee. Alternative Rejected: RoutingActivity (can't call other activities, no retries)

Decision 5: Static Activity Registry

Rationale: Type-safe, simpler. Worker pod must compile all activities anyway. Alternative Rejected: Dynamic registration (complex, unsafe)

Decision 6: Keep cmd/starter Forever

Rationale: No need to break existing deployments. Let customers choose. Alternative Rejected: Deprecate (breaking change)

Decision 7: Shared api-server (2 replicas)

Rationale: Simpler, resource-efficient, easier to scale. Alternative Rejected: Per-namespace servers (overkill)

---

PART 8: RISKS & MITIGATIONS

Risk	Severity	Mitigation
JSONPath evaluation bugs	Medium	Unit tests (>90% coverage), property-based testing
KMSvc queue reliability	High	Dead-letter queue, timeouts, persistence checks
Correlation ID mismatch	Medium	Strict matching, schema validation, detailed logging
Consumer group conflicts	Medium	Clear strategy, rebalancing, message redelivery
Activity registry mismatch	Low	Validate at spec submission time
Temporal network issues	Low	Retry + timeout in api-server
Cache staleness (api-server)	Low	24h TTL, event-driven invalidation
Performance regression	Low	Measure baseline: target <200ms submit, <100ms poll

Total Risk Score: Medium-High (up from Low-Medium due to KMSvc)

---

PART 9: SUCCESS CRITERIA

Functional ✅

• RoutingWorkflow executes activity sequences dynamically
• JSONPath parameters resolve correctly (nested, arrays)
• Error catch blocks jump to handlers
• Retry logic with exponential backoff (1s → 2s → 4s)
• API validates WorkflowSpec (no dangling refs)
• CLI reads JSON (no hardcoded flags)
• Status polling returns step-by-step results
• Backward compatible (cmd/starter unchanged)

Non-Functional ✅

• Performance: <200ms workflow submit, <100ms poll
• Uptime: 99.9% (2 replicas, rolling updates)
• Test coverage: >90%
• Documentation: Complete + 3 working examples

---

PART 10: BACKWARD COMPATIBILITY

What Stays Unchanged ✅

• cmd/starter/ (old CLI works exactly as before)
• All existing activities
• All existing workflows (OrchestratorWorkflow, TaskUnitWorkflow)
• Temporal worker registration

What's New ✅

• cmd/cli/ (new CLI)
• cmd/api-server/ (new HTTP API)
• internal/routing/ (new routing engine)
• statemachine/routing_workflow.go (new workflow)

Migration Path ✅

• Customers can use OLD way or NEW way
• No forced upgrades
• No breaking changes

---

PART 11: FILES SUMMARY

New Files (Phase 1-5)

Routing Engine (Phase 1):

internal/routing/types.go
internal/routing/jsonpath_evaluator.go
internal/routing/execution_context.go
internal/routing/activity_registry.go
internal/routing/kmsvc_client.go
statemachine/routing_workflow.go

API Server (Phase 2):

cmd/api-server/main.go
cmd/api-server/handlers.go
cmd/api-server/validation.go
cmd/api-server/template_loader.go

CLI (Phase 3):

cmd/cli/main.go
cmd/cli/commands/submit.go
cmd/cli/commands/status.go
cmd/cli/commands/template.go
cmd/cli/commands/execute.go
cmd/cli/commands/history.go

K8s (Phase 4):

k8s/api-server-deployment.yaml
k8s/api-server-service.yaml
k8s/api-server-networkpolicy.yaml
homelab-frontend/k8s/gateway-routes.yaml

Docs (Phase 5):

docs/API.md
docs/CLI.md
docs/TEMPLATES.md
docs/MIGRATION.md
docs/examples/*.json

Modified Files

• cmd/worker/main.go (register RoutingWorkflow)
• homelab-frontend/k8s/network-policy.yaml (already done ✅)

---

PART 12: COMPARISON TABLE

Aspect	Before (Hardcoded)	After (Dynamic)
Workflow Definition	Go code (OrchestratorWorkflow)	JSON (customer-specified)
Sequencing	Fixed (Clone→Plan→Impl→Judge)	Any (customer defines in JSON)
Parameters	Direct: in.RemoteURL	JSONPath: ${input.repo}
Change Time	5-10 min (recompile+deploy)	Seconds (API call)
Entry Points	CLI only (hardcoded flags)	CLI + API + Legacy CLI
Error Handling	Try-catch (per-activity)	Catch blocks (conditional jumps)
Retry Policy	Temporal default	Configurable per-activity
Long-Running Jobs	Not supported	awaitTaskComplete pattern
Backward Compat	N/A	100% (cmd/starter unchanged)
Code Coupling	Tight (to specific sequence)	Loose (generic RoutingWorkflow)

---

PART 13: APPROVAL CHECKLIST

Must Answer (Blocks Phase 1)

• Q1: What is KMSvc system?
• Q2: Message format for KMSvc?
• Q3: Who generates correlation_id?
• Q4: Polling strategy (active or signal)?
• Q5: Timeout behavior?
• Q6: Consumer group strategy?

Should Approve

• Timeline OK? (52-58 hours)
• Architecture OK? (RoutingWorkflow + 3 entry points)
• Risk level acceptable? (Medium-High)
• Backward compatibility OK? (Keep cmd/starter)

Sign-Off

• Tech Lead approval
• Stakeholder sign-off
• Ready to start Phase 1

---

PART 14: NEXT STEPS

1. Review This Document (1-2 hours)

   • Read Parts 1-7 (understanding)
   • Read Parts 8-9 (risks & success)
   • Read Part 13 (approval checklist)

2. Answer Critical Questions (Q1-Q6)

   • Email or Slack responses required
   • Needed before Phase 1 can start

3. Stakeholder Approval

   • Architecture review meeting
   • Risk acknowledgment
   • Sign-off on timeline

4. Start Implementation

   • Week 1: Phase 1 (RoutingWorkflow)
   • Week 2: Phase 2 (API) + Phase 3 (CLI)
   • Week 3: Phase 4 (K8s deploy)
   • Week 4: Phase 5 (Docs)

---

APPENDIX: EXAMPLE WORKFLOW JSON

{
  "name": "code-review-pipeline",
  "input": {
    "repo": "https://github.com/rockliang/poimen",
    "branch": "feature/new-cli"
  },
  "activities": [
    {
      "name": "Clone",
      "resource": "CloneRepoActivity",
      "parameters": {
        "repo": "${input.repo}",
        "branch": "${input.branch}"
      },
      "timeout": "5m",
      "next": "Analyze"
    },
    {
      "name": "Analyze",
      "resource": "AnalyzeCodeActivity",
      "parameters": {
        "path": "${Clone.output.path}"
      },
      "timeout": "10m",
      "retry": {
        "maxAttempts": 3,
        "backoffRate": 2.0,
        "initialInterval": "1s"
      },
      "catch": [
        {
          "errorEquals": ["Timeout"],
          "next": "HandleTimeout"
        }
      ],
      "next": "Judge"
    },
    {
      "name": "Judge",
      "resource": "JudgeActivity",
      "parameters": {
        "code": "${Analyze.output.code}",
        "analysis": "${Analyze.output.analysis}"
      },
      "timeout": "5m",
      "end": true
    },
    {
      "name": "HandleTimeout",
      "type": "pass",
      "result": {
        "status": "failed",
        "reason": "Analysis timed out after 10m"
      },
      "end": true
    }
  ]
}

---

APPENDIX: CLI USAGE EXAMPLES

# Submit workflow from JSON file
$ poimen-cli submit workflow.json
Workflow submitted: wf-abc123
Poll status: poimen-cli status wf-abc123

# Submit and wait for results
$ poimen-cli submit workflow.json --wait
Waiting for workflow to complete...
✅ Workflow completed
Output:
{
  "verdict": "approved",
  "score": 0.95,
  "issues": []
}

# Check status
$ poimen-cli status wf-abc123
Workflow ID: wf-abc123
Status: COMPLETED
Started: 2025-01-31 15:04:05
Completed: 2025-01-31 15:05:00

# Execute template immediately (sync)
$ poimen-cli execute template code-review-v1 --input-file input.json
Executing template: code-review-v1
[15:04:05] Step 1/3: Clone ... OK (45s)
[15:04:50] Step 2/3: Analyze ... OK (120s)
[15:05:50] Step 3/3: Judge ... OK (30s)
✅ COMPLETED
Output: {...}

# List templates
$ poimen-cli template list
Available Templates:
  - code-review-v1
  - clone-analyze-v1
  - simple-test-v1

# Show template
$ poimen-cli template show code-review-v1
Template: code-review-v1
Activities: 3

1. Clone (CloneRepoActivity)
   Timeout: 5m
2. Analyze (AnalyzeCodeActivity)
   Timeout: 10m
   Retry: 3 attempts, backoff 2.0
3. Judge (JudgeActivity)
   Timeout: 5m

---

END OF MASTER REVIEW DOCUMENT

For detailed technical documentation, see existing design docs in the repo.