poimen-workflows/FINAL_COMPREHENSIVE_DESIGN.md

Poimen Service Integration & Dynamic Workflows — FINAL DESIGN

Status: Ready for Implementation Review
Updated: 2025-01-31
Scope: Replace hardcoded workflows with API-driven, JSON-based, customer-specified sequences

---

1-Minute Summary

Problem

Hardcoded workflows require code changes → recompile → deploy (5-10 min) to change activity sequences.

Solution

• New RoutingWorkflow reads JSON WorkflowSpec and executes dynamically
• Three interfaces: CLI + HTTP API + legacy CLI (backward compat)
• Three patterns: Sequential, Await-Task-Complete (KMSvc), Retry with error handling
• JSONPath parameters: ${step1.output.path}

Result

Workflow changes in seconds (API call) instead of 5-10 min (recompile)

---

The Three Patterns

1. Sequential (A → B → C)

Activities execute in sequence, each step passes output to next via JSONPath.

2. Await-Task-Complete (Launch → Poll Queue → Process)

Launch long-running job, poll KMSvc queue for result with correlation ID matching, then proceed.

3. Retry with Error Handling

Activity retry N times with exponential backoff, jump to catch block on failure.

---

The Three Entry Points

CLI (New)

poimen-cli submit workflow.json --wait
poimen-cli execute template code-review-v1 --input-file input.json
poimen-cli status wf-abc123

HTTP API (New)

curl -X POST /api/v1/workflows -d @workflow.json
curl -X POST /api/v1/execute -d '{"template": "...", "input": {...}}'
curl /api/v1/workflows/wf-abc123/status

Legacy CLI (Unchanged)

go run ./cmd/starter --repo ... --milestone T0 (still works)

---

Implementation: 5 Phases

Phase	Files	Hours	Week
1. Routing Engine	internal/routing/, statemachine/routing_workflow.go	20-22	1
2. API Server	cmd/api-server/	12-14	2
3. CLI	cmd/cli/	8-10	2
4. K8s Deploy	k8s/, homelab-frontend/k8s/	5	3
5. Docs	docs/, examples	7	4
TOTAL	~15 files	52-58	4 weeks

---

Critical KMSvc Questions

1. What is KMSvc? (Kafka/Redis/SQS/custom?)
2. Message format? (JSON with correlation_id field?)
3. Who generates correlation_id? (Activity or Workflow?)
4. Polling strategy? (Active loop or Temporal Signal?)
5. Timeout behavior? (Catch block, fail, or infinite wait?)
6. Consumer group? (Shared or per-workflow?)

---

Success Criteria

✅ RoutingWorkflow executes sequences dynamically
✅ JSONPath parameters resolve correctly
✅ Error catch blocks work
✅ Retry with backoff works
✅ CLI reads JSON (no flags)
✅ API validates specs
✅ Backward compatible
✅ Performance: <200ms submit
✅ Test coverage: >90%

---

Key Files

See full design doc (FINAL_COMPREHENSIVE_DESIGN.md in repo) for:

• Detailed phase breakdown
• Code examples
• Risk assessment
• Design decisions
• Complete file list
• Timeline details

---

Next Steps

1. Answer KMSvc Q1-Q6
2. Review 52-58 hour estimate
3. Stakeholder approval
4. Start Phase 1

See DESIGN_REVIEW_UPDATED.md for detailed questions.