Add testable task descriptions: T0.1 through T0.9

Each task includes: - Scope: what to build - Implementation: code sketches + details - Verification: concrete test criteria - Done criteria: acceptance checklist
2026-08-20 22:41:54 -07:00
parent 3a8f946105
commit 11a1c1196a
9 changed files with 1027 additions and 0 deletions
@@ -0,0 +1,191 @@
+# T0.9: End-to-End Test
+
+## Scope
+Run against real `temporal.riotpiao.com` cluster + disposable forgejo scratch repo. All 7 verification items from PLAN.md.
+
+## Setup
+
+### Prerequisites
+- `temporal.riotpiao.com` Temporal cluster accessible
+- Forgejo instance running (for scratch repo)
+- Local `git`, `go` 1.21+
+- `ANTHROPIC_API_KEY` env var set
+- `TEMPORAL_NAMESPACE`, `TEMPORAL_TLS_CERT`, `TEMPORAL_TLS_KEY` env vars set if cluster requires them
+
+### Fixture Repo Structure
+Create temporary fixture repo:
+```
+/tmp/fixture/
+  tasks/
+    INDEX.md (copy from this repo)
+    board.md (minimal: 3 trivial tasks for quick run)
+```
+
+Minimal board.md:
+```
+| T0.1 | Create file /tmp/fixture/output.txt with content "hello world" | [ ] |
+| T0.2 | Create file /tmp/fixture/result.json with {"status": "ok"} | [ ] |
+| T0.3 | Create file /tmp/fixture/done.txt with "COMPLETE" | [ ] |
+```
+
+## Run Sequence
+
+### 1. Clone & Fetch Bootstrap Test (T0.3 foundational)
+```bash
+cd /tmp
+mkdir -p test-clone
+go run ./cmd/starter \
+  --repo /tmp/test-clone \
+  --remote /tmp/fixture \
+  --dry-run
+# Check: /tmp/test-clone/.git exists after first run
+# Check: Verify it's a valid git repo
+```
+
+### 2. Full Cycle with Dry-Run (no real push)
+```bash
+export FIXTURE_REMOTE=file:///tmp/fixture-remote-src
+export FIXTURE_WORKTREE=/tmp/fixture-worktree
+
+# Start worker
+go run ./cmd/worker &
+WORKER_PID=$!
+
+# Start orchestrator workflow
+go run ./cmd/starter \
+  --repo /tmp/fixture \
+  --remote file:///tmp/fixture-remote-src \
+  --milestone T0 \
+  --dry-run
+
+# Monitor Temporal Web UI: http://temporal.riotpiao.com:8080
+# Workflow ID: orch-tmp-fixture
+# Expected: all 3 subtasks dispatched, Judge passes each, board updated, NO push to origin
+
+# Verify:
+# - Board file shows all tasks marked done
+# - No commits pushed to remote (because --dry-run)
+# - Lessons file exists if any task was induced to fail
+
+kill $WORKER_PID
+```
+
+### 3. Live Signal Update Mid-Run
+```bash
+# Start same workflow again (different workflow ID)
+go run ./cmd/starter \
+  --repo /tmp/fixture \
+  --remote file:///tmp/fixture-remote-src \
+  --milestone T0.1 \
+  --dry-run &
+WF_ID=$!
+
+# While running, send update signal:
+temporal workflow signal \
+  --workflow-id <orch-id-from-run> \
+  --name update-role-prompt \
+  --input '{"role":"implementer","spec":{"template_ref":"implementer/default.tmpl","variables":{"marker":"from-signal"},...}}'
+
+# Check: next dispatched task includes "from-signal" in Variables
+# Verify via board file or task output
+```
+
+### 4. 5xx Fault Injection (retry-then-succeed)
+```bash
+# Setup: Mock pi command to return 5xx first N times, then succeed
+# (Use a local wrapper script or fault-injection proxy)
+
+# Run workflow:
+go run ./cmd/starter \
+  --repo /tmp/fixture \
+  --remote file:///tmp/fixture-remote-src \
+  --dry-run
+
+# Expected:
+# - PrepareSkillsActivity retries with exponential backoff
+# - Eventually succeeds after N retries
+# - Workflow continues normally
+```
+
+### 5. 5xx Exhaustion (always-5xx, fail at 5m)
+```bash
+# Setup: Mock pi command to always return 503
+
+# Run workflow (must have Implementer call PrepareSkillsActivity or similar pi-dependent step)
+
+# Expected:
+# - PrepareSkillsActivity retries for ~5 minutes (ScheduleToCloseTimeout)
+# - After 5m, activity fails
+# - Workflow marks task as failed
+# - Board reflects failure
+```
+
+### 6. 504 Stream Timeout Learning
+```bash
+# Setup: Mock pi command to return 504
+
+# Run workflow:
+go run ./cmd/starter --repo /tmp/fixture --remote file:///tmp/fixture-remote-src --dry-run
+
+# While running, query workflow state:
+temporal workflow query \
+  --workflow-id <orch-id> \
+  --query-type current-config
+
+# Expected output includes config.Tuning.PiRetry.StreamTimeout (should be doubled from default 30s)
+# After 504, next query shows it as 60s
+# If 504 repeats, doubles again to 120s, capped at StreamTimeoutMax (2m)
+```
+
+### 7. Continue-as-New History Bound
+```bash
+# Run multiple cycles (manually via CLI or workflow logic)
+
+# Check Temporal Web UI: Workflow → History tab
+# Expected:
+# - History is compact (not unbounded growth)
+# - No duplication of events
+# - Cycle count resets per continue-as-new
+```
+
+## Verification Checklist
+- [ ] Fixture repo clones fresh when path empty
+- [ ] Fetch-instead-of-clone on second run
+- [ ] All 3 subtasks dispatched and complete
+- [ ] Judge passes each task
+- [ ] Board file updated with completion marks
+- [ ] No push to origin when --dry-run
+- [ ] Live signal (update-role-prompt) changes next dispatch
+- [ ] Live signal (update-skills) re-preps skills exactly once
+- [ ] 5xx retry-then-succeed: activity retries and eventually succeeds
+- [ ] 5xx exhaustion: activity fails at ~5m mark, task marked failed
+- [ ] 504 learning: StreamTimeout doubled and actually used on next attempt
+- [ ] 504 learning: Stops doubling at StreamTimeoutMax (2m)
+- [ ] Continue-as-new: History bounded, no unbounded growth
+- [ ] Squash-merge result: One commit on main per submilestone (not yet, waiting for T0.1-T0.8 to pass first)
+
+## Done Criteria (All Must Pass)
+1. All 7 checks in Verification Checklist marked `[x]`
+2. No panics or unhandled errors in workflow execution
+3. Temporal Web UI shows clean workflow execution with retries visible
+4. Board file reflects accurate task completion state
+5. Lessons file demonstrates learning across retries (if any failure induced)
+6. Workflow completes within reasonable time (~10-30min for 3 subtasks + fault injection)
+
+## Cleanup
+```bash
+# Delete fixture remote and worktrees
+rm -rf /tmp/fixture-remote-src /tmp/fixture-worktree
+
+# Kill any lingering worker processes
+pkill -f "go run ./cmd/worker"
+
+# Optionally delete workflow from Temporal (if testing repeatedly)
+temporal workflow delete --workflow-id orch-tmp-fixture
+```
+
+## Notes
+- **Real forgejo remote:** The "disposable" remote can be on actual forgejo instance (`[email protected]:test/workflows-e2e.git`), or a file:// URL locally
+- **Anthropic API calls:** Use actual API (not mock) for real e2e; costs will be minimal if test tasks are simple
+- **Temporal Web UI:** Set timezone to match your local time for easier log reading
+- **Fault injection:** Can use `PATH` manipulation (wrapper scripts) or a local HTTP proxy (e.g., mitmproxy, Burp Suite) to inject 5xx/504 responses