Scaffold: PLAN.md, tasks/INDEX.md, tasks/board.md

Initial project structure documentation and task breakdown for Multi-Agent Dev Orchestrator (Temporal + Go).
This commit is contained in:
Test
2026-08-20 22:10:14 -07:00
commit 3a8f946105
3 changed files with 500 additions and 0 deletions
+337
View File
@@ -0,0 +1,337 @@
# Multi-Agent Dev Orchestrator (Temporal + Go) — Implementation Plan
See the full design doc at `/Users/rockliang/.claude/plans/considered-u-are-a-curried-reef.md`.
## Quick Summary
Build a Temporal-based orchestrator that drives multi-agent software dev work on arbitrary target repos. Three roles (Planner/reasoning, Judge/reasoning, Implementer/cheaper model) collaborate on hierarchical tasks (`T0` milestone split into `T0.1`-`T0.9` subtasks). Orchestrator owns config (system prompt, skills, activity timeouts/retries), live-updatable via signals. All code runs on shared FS where target repo sits; git concurrency handled via worktrees + advisory lock. System testable with mocked activities + real end-to-end against `temporal.riotpiao.com`.
## Implementation Track
**Milestone T0**: Nine subtasks, each with its own verification gate = completion criterion.
| Task | Scope | Status |
|---|---|---|
| [T0.1](#t01-repo-scaffold) | Repo scaffold: `go.mod`, `statemachine/`, `action/`, `cmd/`, `prompts/`, `internal/`, `tests/` | [ ] |
| [T0.2](#t02-shared-types) | Shared types: `ModelSpec`, `PromptSpec`, `OrchestratorConfig`, `ActivityTuning`, `PiRetryPolicy` | [ ] |
| [T0.3](#t03-git-and-locking) | Git & locking: `CloneRepoActivity`, worktrees, squash-merge, `orchestrator.lock` | [ ] |
| [T0.4](#t04-pi-and-error-classification) | `PrepareSkillsActivity`, `classifyPiErr` (4xx/5xx/504), stream timeout learning | [ ] |
| [T0.5](#t05-llm-agents-and-prompts) | Planner/Judge/Implementer activities, LLM client, prompt templates + live customization | [ ] |
| [T0.6](#t06-taskunit-workflow) | TaskUnit workflow: retry loops, timeout escalation, lessons injection | [ ] |
| [T0.7](#t07-orchestrator-workflow) | Orchestrator workflow: config state, signals, fan-out/fan-in, `continue-as-new`, 504 learning | [ ] |
| [T0.8](#t08-worker-and-starter) | Worker & starter CLIs, env/config loading, Temporal registration | [ ] |
| [T0.9](#t09-end-to-end) | Full e2e against real cluster + scratch repo: all 7 verification items | [ ] |
---
### T0.1: Repo Scaffold
Create directory structure, `go.mod`, empty stubs.
**Verification:** `go build ./...` succeeds; layout matches plan.
<details>
<summary>Details</summary>
```
/go.mod
/cmd/worker/main.go
/cmd/starter/main.go
/statemachine/types.go types.go signals.go orchestrator.go taskunit.go
/action/planner.go implementer.go judge.go git.go skills.go integration_test.go lessons.go llm/client.go
/prompts/registry.go planner/default.tmpl judge/default.tmpl implementer/default.tmpl
/internal/config/config.go lock/flock.go
/tests/taskunit_workflow_test.go orchestrator_workflow_test.go
```
</details>
---
### T0.2: Shared Types
Implement `statemachine/types.go` with all config/input/output structs. Document defaults.
**Verification:** Unit test asserts all defaults (5m / 2s / 30s / 2.0 / 30s stream / 2m stream-max).
<details>
<summary>Details</summary>
- `ModelSpec`: ModelID, Thinking, Effort
- `PromptSpec`: TemplateRef, RawTemplate, Variables, Model, LessonsRef
- `OrchestratorInput`, `OrchestratorOutput`
- `TaskUnitInput`, `TaskUnitOutput`
- `ActivityTuning`: ImplementerBaseTimeout, ImplementerMaxRetries, JudgeTimeout, PiRetry
- `PiRetryPolicy`: ScheduleToCloseTimeout (5m), InitialInterval (2s), MaximumInterval (30s), BackoffCoefficient (2.0), StreamTimeout (30s), StreamTimeoutMax (2m)
- `OrchestratorConfig`: SystemPrompt, Skills, RolePrompts, Tuning
</details>
---
### T0.3: Git & Locking
Implement `action/git.go` + `internal/lock/flock.go`.
**Verification:** Test against local scratch repo: clone-if-empty vs fetch-if-exists, worktree lifecycle (add/commit/remove), squash-merge produces exactly one commit on main.
<details>
<summary>Details</summary>
**Activities:**
- `CloneRepoActivity(ctx, {RemoteURL, TargetRepoPath}) error` — idempotent `git clone` or `git fetch`
- `GitWorktreeAddActivity(ctx, {RepoPath, TaskID}) (string, error)` — returns worktree path, guarded by lock
- `GitCommitActivity(ctx, {WorktreePath, Message}) error` — commits in worktree (no lock needed)
- `GitPushActivity(ctx, {RepoPath}) error` — guarded by lock
- `GitSquashMergeActivity(ctx, {RepoPath, Branches, Message}) error` — guarded by lock
**Lock helper (`internal/lock/`):**
- `Lock(path string) error`, `Unlock(path string) error` using `golang.org/x/sys/unix.Flock` or `fcntl` equivalent
**Squash-merge sequence:**
```
fetch origin main
checkout main && pull --ff-only origin main
for b in branches:
merge --squash $b
commit -m "T0: squash merge subtasks..."
push origin main
for b in branches:
worktree remove worktrees/$id --force
branch -D $b
```
</details>
---
### T0.4: Pi & Error Classification
Implement `action/skills.go` with `PrepareSkillsActivity` and `classifyPiErr`.
**Verification:** Unit test all three error buckets (4xx/5xx/504) against a mocked `pi` HTTP client.
<details>
<summary>Details</summary>
**`PrepareSkillsActivity`:**
- Input: `{Skills []SkillRef, StreamTimeout time.Duration}`
- For each skill, `pi clone-or-fetch <skill-url>` (idempotent)
- Each skill guarded by its own lock
**Error classification:**
```go
func classifyPiErr(err error) error {
// 4xx -> NonRetryableApplicationError "PiClientError"
// 504 -> ApplicationError "PiStreamTimeout"
// others -> retryable
}
```
</details>
---
### T0.5: LLM Agents & Prompts
Implement LLM activities + prompt templates.
**Verification:** Unit test renders a `PromptSpec` (system prompt + template override + raw template) and calls mock Anthropic client.
<details>
<summary>Details</summary>
**Activities:**
- `PlanningActivity(ctx, {OrchestratorConfig, BoardState}) (TaskDispatch, error)` — reads board/INDEX.md, calls Planner
- `ImplementerActivity(ctx, {PromptSpec, WortkreeePath, Lessons}) (ImplementOutput, error)` — tool-call agent loop
- `JudgeActivity(ctx, {PromptSpec, Diff, IntegrationTestResult}) (Verdict, Critique, error)` — reviews correctness
- `RunIntegrationTestActivity(ctx, {WortkreeePath, TestCmd}) (pass/fail, logs, error)` — shells out
**Prompt templates:**
- `planner/default.tmpl`: expects `{{.SystemPrompt}}`, `{{.TaskBoard}}`, etc.
- `judge/default.tmpl`: expects `{{.SystemPrompt}}`, `{{.Diff}}`, `{{.TestResult}}`
- `implementer/default.tmpl`: expects `{{.SystemPrompt}}`, `{{.Task}}`, `{{.Lessons}}`
**`prompts/registry.go`:**
- `go:embed prompts/*.tmpl`
- `Render(templateRef string, variables map[string]any) (string, error)`
**`action/llm/client.go`:**
- Thin Anthropic client wrapper
- Read `ANTHROPIC_API_KEY` from env
- Call `messages.Create` with model/thinking/effort from `ModelSpec`
</details>
---
### T0.6: TaskUnit Workflow
Implement `statemachine/taskunit.go` with retry loops & timeout escalation.
**Verification (Testsuite):**
- Pass-first-try
- Fail-then-pass-after-lesson-injection
- Retries-exhausted
- Timeout-escalation (both judges)
<details>
<summary>Details</summary>
**Flow:**
1. `GitWorktreeAddActivity` → get isolated working tree
2. Retry loop:
- Track `timeoutAttempt`, `judgeAttempt` separately
- `ImplementerActivity` with timeout = `BaseTimeout * timeoutAttempt`
- If timeout, increment `timeoutAttempt` and retry (duration grows)
- If success, call `RunIntegrationTestActivity`
- Call `JudgeActivity`
- If judge pass, commit in worktree and return
- If judge fail, append to lessons, increment `judgeAttempt`, retry (lessons injected next time)
- If retries exhausted, return fail verdict to orchestrator
**Key detail:** HeartbeatTimeout = (BaseTimeout * timeoutAttempt) / 4, scales with escalation.
</details>
---
### T0.7: Orchestrator Workflow
Implement `statemachine/orchestrator.go` with config state, signals, fan-out/fan-in, `continue-as-new`, 504 learning.
**Verification (Testsuite):**
- Fan-out/fan-in correctness
- Squash-merge triggers on submilestone complete
- `continue-as-new` at cycle cap, carries `OrchestratorConfig` forward
- `update-*` signals mutate config without touching in-flight TaskUnit
- `PiStreamTimeout` doubles `StreamTimeout` and persists it
<details>
<summary>Details</summary>
**Per-cycle logic:**
1. If `config.Skills` changed, call `PrepareSkillsActivity` once (wraps it for 504 learning)
2. Call `PlanningActivity` → get dispatch decision
3. Fan out: `workflow.ExecuteChildWorkflow(TaskUnitWorkflow, ...)` for each dispatched T0.x
4. Await all via `workflow.Selector`
5. Call `PlanningActivity` again to update board + commit + push
6. If submilestone complete, call `GitSquashMergeActivity`
7. Increment cycle count
8. If cycle count >= cap, `workflow.NewContinueAsNewError(ctx, ..., nextInput)`
**Signal handlers:**
- `pause`, `resume`: gate the cycle loop
- `abort-task(taskID)`: forward via `SignalExternalWorkflow` to TaskUnit
- `inject-lesson`: append to lessons store
- `update-system-prompt`, `update-skills`, `update-role-prompt`, `update-tuning`: mutate `config.*`
**504 learning wrapper (pseudo-code):**
```go
for {
r := config.Tuning.PiRetry
err := ExecuteActivity(..., PrepareSkillsActivity, Input{...StreamTimeout: r.StreamTimeout})
if isPiStreamTimeout(err) && r.StreamTimeout < r.StreamTimeoutMax {
config.Tuning.PiRetry.StreamTimeout *= 2
continue
}
break
}
```
</details>
---
### T0.8: Worker & Starter CLIs
Implement `cmd/worker/main.go` and `cmd/starter/main.go`.
**Verification:**
- `go run ./cmd/worker` connects to `temporal.riotpiao.com:7233` without error
- `go run ./cmd/starter --dry-run` starts a workflow that appears in Temporal Web UI
<details>
<summary>Details</summary>
**`cmd/worker/main.go`:**
```go
config := loadConfig() // reads env: TEMPORAL_NAMESPACE, TEMPORAL_TLS_CERT, TEMPORAL_TLS_KEY, ANTHROPIC_API_KEY
c, err := client.Dial(client.Options{HostPort: "temporal.riotpiao.com:7233", ...TLS...})
w, err := worker.New(c, "default", worker.Options{})
// register both workflows
w.RegisterWorkflow(statemachine.OrchestratorWorkflow)
w.RegisterWorkflow(statemachine.TaskUnitWorkflow)
// register all activities
w.RegisterActivity(action.CloneRepoActivity)
w.RegisterActivity(action.GitWorktreeAddActivity)
// ... etc
w.Run()
```
**`cmd/starter/main.go`:**
```go
flag.String("repo", "", "target repo path")
flag.String("remote", "", "remote URL")
flag.String("milestone", "T0", "milestone ID")
flag.Bool("dry-run", false, "disable git push/merge")
flag.String("planner-model", "claude-opus-5", "planner model ID")
// ... judge, implementer models
// build OrchestratorInput, call client.ExecuteWorkflow
```
**`internal/config/config.go`:**
- Load Temporal settings from env
- Load ANTHROPIC_API_KEY from env
- Return filled config struct
</details>
---
### T0.9: End-to-End Test
Run against real `temporal.riotpiao.com` + disposable scratch repo.
**Verification (all 7 items in the plan):**
1. Clone bootstrap: fresh clone when repo path empty
2. Full cycle: dispatch subtasks, judge pass/fail, commit, squash-merge
3. Live signal updates: change prompt/skills mid-run, next dispatch sees them
4. 5xx retry-then-succeed + always-503 exhausts at 5m mark
5. 504 stream-timeout learning: doubles and is actually used, capped at max
6. `continue-as-new` history bounded
7. Squash-merge result: main has one squashed commit per submilestone
<details>
<summary>Details</summary>
**Fixture repo structure:**
```
tasks/
INDEX.md (guidelines)
board.json (task list, T0.1-T0.3 with trivial definitions)
```
**Example subtask:** "Create file `output.txt` with content 'hello world'"
**Run sequence:**
1. `go run ./cmd/starter --repo /tmp/fixture --remote [email protected]:scratch/workflow-test.git --dry-run`
2. Monitor Temporal Web UI for workflow progress
3. Midway, send signals: `temporal workflow signal --workflow-id orch-... --name update-role-prompt ...`
4. Confirm next dispatch uses new prompt (assert marker in output file)
5. Confirm board updated, lessons file exists (if any failure happened)
6. Remove `--dry-run`, repeat against real remote
7. Assert final state: real commits on remote, squash-merge on main
</details>
---
## Next Steps
1. Approve this scaffold (PLAN.md + tasks/INDEX.md + board)
2. Start T0.1 → checkout branch `task/T0.1` → scaffold repo structure
3. Each task: implement, test locally, verify against criterion
4. Mark on board: [x] when verification passes
5. T0.9: final e2e run
6. Squash all T0.* branches into main, push
+125
View File
@@ -0,0 +1,125 @@
# Poimen Workflows — Development Guidelines
This project orchestrates multi-agent software development tasks using Temporal + Go. The repo is organized as:
- `statemachine/` — Temporal workflow definitions (deterministic state machines)
- `action/` — Temporal activity definitions (units of work / LLM calls)
- `cmd/` — CLI entry points (worker registration, workflow starter)
- `prompts/` — LLM prompt templates (Go-embedded, live-updatable)
- `internal/` — Shared config/locking utilities
- `tests/` — Unit tests via `go.temporal.io/sdk/testsuite`
## Code Standards
### Go Style
Follow `golang-skills` conventions from `~/.claude/skills/golang-skills/`:
- Error handling mandatory; no naked `_ =` discards.
- Idiomatic naming: `ctx` for context, `err` for error returns.
- Interfaces kept narrow; one struct per concern.
- No over-generalization for hypothetical use.
- Documentation via comments only where the WHY is non-obvious.
### Testing
- Every `statemachine/*` change ships with a `testsuite`-based test in `tests/`.
- Unit tests use `go.temporal.io/sdk/testsuite.WorkflowTestEnvironment` with mocked activities.
- Activities are tested in isolation before integrating into workflows.
- No literal timeout/retry values in `action/*` code — all come from `OrchestratorConfig.Tuning` at runtime.
### Activity Design
Activities stay side-effect-isolated (one concern per file) so they remain independently reusable across projects.
- `action/git.go` — all git operations
- `action/skills.go` — skill prep / pi command
- `action/planner.go` — Planner LLM call
- `action/judge.go` — Judge LLM call
- `action/implementer.go` — Implementer LLM + tool-call loop
- `action/lessons.go` — Lessons store read/write
No composite activities like `PrepareAndImplement`; that's a workflow's job, not an activity's.
### Config as Data
**No hardcoded numbers.** Every timeout, retry count, backoff coefficient, stream timeout — all come from `OrchestratorConfig.Tuning` (or role-specific `PromptSpec.Model`), read at execution time. This lets:
- `update-tuning` signal to dynamically adjust timeouts without redeployment
- Planner activity to recommend per-task overrides
- Pi 504 handler to learn and persist `StreamTimeout` across `ContinueAsNew` cycles
**No LLM model hardcoded.** Every model ID comes from `ModelSpec.ModelID`, passed as data.
### Concurrency & Locking
Target repo sits on shared filesystem. Git concurrency is safe by construction:
- Task units use isolated worktrees (`git worktree add -b task/T0.x ...`)
- Commits within a worktree don't need a lock; git serializes object writes
- Only `CloneRepoActivity`, `GitPushActivity`, `GitSquashMergeActivity` need the `orchestrator.lock`, since they mutate the main working tree / refs
Monitor for deadlocks: if two workflows try to push simultaneously, the lock will serialize them. Test with concurrent tasks enabled.
## Task Board Format
Each task row in the board specifies:
- **ID**: `T0.1`, `T0.2`, etc.
- **Description**: One-line scope
- **Status**: `[ ]` (todo), `[x]` (done)
- **Branch**: `task/T0.x` (created when task starts, merged to main on submilestone complete)
- **Verification**: Specific criterion that marks it done (e.g. "Unit test passes", "E2E run completes")
A task is NOT marked done until its verification step passes. This mirrors the Judge role's own job: no hallucinated completion.
## Submilestones & Merges
When all tasks in `T0` (`T0.1` through `T0.9`) pass their verification:
1. Orchestrator calls `GitSquashMergeActivity` to merge all `task/T0.*` branches into `main` as a single squashed commit
2. All `task/T0.*` branches and worktrees are cleaned up
3. Main branch is the canonical history; the 9 subtask commits are compacted into one
This demonstrates the very mechanism the system orchestrates: a working software dev pipeline with multiple agents collaborating on a shared codebase, guarded by state checks (Judge), and landing changes via deterministic git workflow.
## Historical Lessons
Lessons live at `tasks/.orchestrator/lessons/<TaskID>.jsonl` — per-task file of failed attempts. When a Judge calls a task failure, the `UpdateLessonsActivity` appends `{Attempt, Critique, FailedApproachSummary, Timestamp}`. On retry, `ReadLessonsActivity` injects the last N entries into the Implementer's prompt as "known errors — do not repeat this time".
Lessons flush to git only as part of the Planner's board commit, not on every retry — keeps history clean.
## Skills & Preparation
Required skill sources are listed in `OrchestratorConfig.Skills` as a list of references (e.g. `["~/.claude/skills/golang-skills", "custom-skill-repo"]`). `PrepareSkillsActivity` runs once per config change, cloning/fetching them onto the shared FS via the `pi` command. All retry/backoff/timeout is delegated to Temporal's retry machinery, with special handling for 504 (stream timeout learning).
## Environment & Secrets
All external system access is via environment variables, loaded at worker startup:
- `TEMPORAL_NAMESPACE`, `TEMPORAL_TLS_CERT`, `TEMPORAL_TLS_KEY` — Temporal cluster connection
- `ANTHROPIC_API_KEY` — LLM API key
- Any target-repo-specific credentials (e.g. git SSH key) are assumed already available on the shared FS (e.g. via Kubernetes secret mount)
Use homelab's `vsource .env` pattern to load from a `.env` file during local development.
## Observability
- Temporal Web UI (`temporal.riotpiao.com:8080` or similar) shows workflow execution, signal delivery, activity retries
- Workflow `current-config` query returns live `OrchestratorConfig` (useful for debugging which tuning values are in effect)
- Activity heartbeats (`activity.RecordHeartbeat`) are sent after each tool-call iteration, visible in Temporal's activity details
- Lessons file grows as retries happen; inspect it on failure to understand what the Implementer is learning
## Deployment & Homelab Integration
This repo is application-layer code against the homelab's Temporal cluster. The orchestrator runs as a Kubernetes pod with:
- Persistent volume (PVC) for the shared FS where target repos are cloned
- Network access to Temporal + Anthropic APIs
- Git SSH key mounted for cloning target repos
The Kubernetes manifests + Helm charts live in the homelab repo under `k8s/` and follow homelab's GitOps workflow (commit → ArgoCD sync). This repo's CI/CD (GitHub Actions or Forgejo Actions) builds and pushes the Docker image; the homelab repo triggers a new pod deploy on image push.
## Questions & Debugging
If a task is marked done but you suspect it's wrong:
1. Re-run its verification step manually
2. Check the unit test against the latest code
3. For e2e tasks, review Temporal Web UI logs + board file + lessons file on the target repo
4. Update the board and task description if the criterion was misunderstood
If code doesn't compile or tests fail:
1. Check Go version and Temporal SDK version match
2. Run `go mod tidy` and `go mod vendor` if dependencies drift
3. Look for hardcoded values or model IDs that should be config instead
+38
View File
@@ -0,0 +1,38 @@
# Task Board — Milestone T0
**Submilestone:** T0 (Multi-Agent Dev Orchestrator Temporal system)
| ID | Scope | Status | Branch | Verification | Notes |
|----|-------|--------|--------|--------------|-------|
| T0.1 | Repo scaffold: go.mod, statemachine/, action/, cmd/, prompts/, internal/, tests/ | [ ] | `task/T0.1` | `go build ./...` succeeds; layout matches PLAN.md | Foundation |
| T0.2 | Shared types: ModelSpec, PromptSpec, OrchestratorConfig, ActivityTuning, PiRetryPolicy | [ ] | `task/T0.2` | Unit test asserts all defaults (5m/2s/30s/2.0/30s stream/2m stream-max) | Config data model |
| T0.3 | Git & locking: CloneRepoActivity, worktrees, squash-merge, orchestrator.lock | [ ] | `task/T0.3` | Test vs local scratch repo: clone-if-empty vs fetch, worktree lifecycle, squash-merge produces 1 commit | Concurrency safety |
| T0.4 | PrepareSkillsActivity, classifyPiErr (4xx/5xx/504), stream timeout learning | [ ] | `task/T0.4` | Unit tests: all 3 error buckets against mocked pi HTTP client | Pi integration |
| T0.5 | Planner/Judge/Implementer activities, LLM client, prompt templates | [ ] | `task/T0.5` | Unit test: PromptSpec renders with system prompt + template override + raw template | LLM orchestration |
| T0.6 | TaskUnitWorkflow: retry loops (timeout/judge-fail split), lessons injection, escalation | [ ] | `task/T0.6` | Testsuite: pass-first-try, fail-then-pass-after-lesson, retries-exhausted, timeout-escalation | Task execution core |
| T0.7 | OrchestratorWorkflow: config state, signals, fan-out/fan-in, continue-as-new, 504 learning | [ ] | `task/T0.7` | Testsuite: fan-out/fan-in, squash-merge on complete, continue-as-new carries config, signals mutate config, 504 doubles StreamTimeout | Orchestration core |
| T0.8 | cmd/worker, cmd/starter, internal/config (env/vsource loading) | [ ] | `task/T0.8` | `go run ./cmd/worker` connects to temporal.riotpiao.com; `go run ./cmd/starter --dry-run` visible in Web UI | CLI integration |
| T0.9 | End-to-end: real temporal.riotpiao.com + disposable forgejo scratch repo, all 7 verification items | [ ] | `task/T0.9` | Clone bootstrap, full cycle, live signal updates, 5xx retry+exhaust, 504 stream-timeout learning, continue-as-new bounded, squash-merge result | System validation |
## Submission Criteria
All T0.1T0.9 marked `[x]` → submilestone complete.
At that point:
1. `git -C /workspace/Poimen/workflows checkout main && git pull`
2. `git merge --squash task/T0.1 task/T0.2 ... task/T0.9`
3. `git commit -m "T0: multi-agent orchestrator initial implementation"`
4. `git push origin main`
5. Delete all `task/T0.*` branches and worktrees
This merge is the first real dogfood of the system's own git workflow: squashing 9 subtask branches into main as a single milestone commit.
---
## Notes
- **Lessons file location:** `tasks/.orchestrator/lessons/<TaskID>.jsonl` (created on first failure, not committed until Planner's board commit)
- **Branch naming:** Strict `task/T0.x` format; Orchestrator expects this pattern
- **Dry-run vs real:** T0.8 tests with `--dry-run` (no real push); T0.9 removes flag (real remote operations)
- **Temporal Web UI:** Monitor at `http://temporal.riotpiao.com:8080` (adjust port/host as needed)
- **E2E fixture:** Disposable forgejo repo (deleted post-run); confirm it's not a production repo before starting T0.9