poimen-workflows/tasks/T0.9.md

T0.9: End-to-End Test

Scope

Run against real temporal.riotpiao.com cluster + disposable forgejo scratch repo. All 7 verification items from PLAN.md.

Setup

Prerequisites

• temporal.riotpiao.com Temporal cluster accessible
• Forgejo instance running (for scratch repo)
• Local git, go 1.21+
• ANTHROPIC_API_KEY env var set
• TEMPORAL_NAMESPACE, TEMPORAL_TLS_CERT, TEMPORAL_TLS_KEY env vars set if cluster requires them

Fixture Repo Structure

Create temporary fixture repo:

/tmp/fixture/
  tasks/
    INDEX.md (copy from this repo)
    board.md (minimal: 3 trivial tasks for quick run)

Minimal board.md:

| T0.1 | Create file /tmp/fixture/output.txt with content "hello world" | [ ] |
| T0.2 | Create file /tmp/fixture/result.json with {"status": "ok"} | [ ] |
| T0.3 | Create file /tmp/fixture/done.txt with "COMPLETE" | [ ] |

Run Sequence

1. Clone & Fetch Bootstrap Test (T0.3 foundational)

cd /tmp
mkdir -p test-clone
go run ./cmd/starter \
  --repo /tmp/test-clone \
  --remote /tmp/fixture \
  --dry-run
# Check: /tmp/test-clone/.git exists after first run
# Check: Verify it's a valid git repo

2. Full Cycle with Dry-Run (no real push)

export FIXTURE_REMOTE=file:///tmp/fixture-remote-src
export FIXTURE_WORKTREE=/tmp/fixture-worktree

# Start worker
go run ./cmd/worker &
WORKER_PID=$!

# Start orchestrator workflow
go run ./cmd/starter \
  --repo /tmp/fixture \
  --remote file:///tmp/fixture-remote-src \
  --milestone T0 \
  --dry-run

# Monitor Temporal Web UI: http://temporal.riotpiao.com:8080
# Workflow ID: orch-tmp-fixture
# Expected: all 3 subtasks dispatched, Judge passes each, board updated, NO push to origin

# Verify:
# - Board file shows all tasks marked done
# - No commits pushed to remote (because --dry-run)
# - Lessons file exists if any task was induced to fail

kill $WORKER_PID

3. Live Signal Update Mid-Run

# Start same workflow again (different workflow ID)
go run ./cmd/starter \
  --repo /tmp/fixture \
  --remote file:///tmp/fixture-remote-src \
  --milestone T0.1 \
  --dry-run &
WF_ID=$!

# While running, send update signal:
temporal workflow signal \
  --workflow-id <orch-id-from-run> \
  --name update-role-prompt \
  --input '{"role":"implementer","spec":{"template_ref":"implementer/default.tmpl","variables":{"marker":"from-signal"},...}}'

# Check: next dispatched task includes "from-signal" in Variables
# Verify via board file or task output

4. 5xx Fault Injection (retry-then-succeed)

# Setup: Mock pi command to return 5xx first N times, then succeed
# (Use a local wrapper script or fault-injection proxy)

# Run workflow:
go run ./cmd/starter \
  --repo /tmp/fixture \
  --remote file:///tmp/fixture-remote-src \
  --dry-run

# Expected:
# - PrepareSkillsActivity retries with exponential backoff
# - Eventually succeeds after N retries
# - Workflow continues normally

5. 5xx Exhaustion (always-5xx, fail at 5m)

# Setup: Mock pi command to always return 503

# Run workflow (must have Implementer call PrepareSkillsActivity or similar pi-dependent step)

# Expected:
# - PrepareSkillsActivity retries for ~5 minutes (ScheduleToCloseTimeout)
# - After 5m, activity fails
# - Workflow marks task as failed
# - Board reflects failure

6. 504 Stream Timeout Learning

# Setup: Mock pi command to return 504

# Run workflow:
go run ./cmd/starter --repo /tmp/fixture --remote file:///tmp/fixture-remote-src --dry-run

# While running, query workflow state:
temporal workflow query \
  --workflow-id <orch-id> \
  --query-type current-config

# Expected output includes config.Tuning.PiRetry.StreamTimeout (should be doubled from default 30s)
# After 504, next query shows it as 60s
# If 504 repeats, doubles again to 120s, capped at StreamTimeoutMax (2m)

7. Continue-as-New History Bound

# Run multiple cycles (manually via CLI or workflow logic)

# Check Temporal Web UI: Workflow → History tab
# Expected:
# - History is compact (not unbounded growth)
# - No duplication of events
# - Cycle count resets per continue-as-new

Verification Checklist

• Fixture repo clones fresh when path empty
• Fetch-instead-of-clone on second run
• All 3 subtasks dispatched and complete
• Judge passes each task
• Board file updated with completion marks
• No push to origin when --dry-run
• Live signal (update-role-prompt) changes next dispatch
• Live signal (update-skills) re-preps skills exactly once
• 5xx retry-then-succeed: activity retries and eventually succeeds
• 5xx exhaustion: activity fails at ~5m mark, task marked failed
• 504 learning: StreamTimeout doubled and actually used on next attempt
• 504 learning: Stops doubling at StreamTimeoutMax (2m)
• Continue-as-new: History bounded, no unbounded growth
• Squash-merge result: One commit on main per submilestone (not yet, waiting for T0.1-T0.8 to pass first)

Done Criteria (All Must Pass)

1. All 7 checks in Verification Checklist marked [x]
2. No panics or unhandled errors in workflow execution
3. Temporal Web UI shows clean workflow execution with retries visible
4. Board file reflects accurate task completion state
5. Lessons file demonstrates learning across retries (if any failure induced)
6. Workflow completes within reasonable time (~10-30min for 3 subtasks + fault injection)

Cleanup

# Delete fixture remote and worktrees
rm -rf /tmp/fixture-remote-src /tmp/fixture-worktree

# Kill any lingering worker processes
pkill -f "go run ./cmd/worker"

# Optionally delete workflow from Temporal (if testing repeatedly)
temporal workflow delete --workflow-id orch-tmp-fixture

Notes

• Real forgejo remote: The "disposable" remote can be on actual forgejo instance ([email protected]:test/workflows-e2e.git), or a file:// URL locally
• Anthropic API calls: Use actual API (not mock) for real e2e; costs will be minimal if test tasks are simple
• Temporal Web UI: Set timezone to match your local time for easier log reading
• Fault injection: Can use PATH manipulation (wrapper scripts) or a local HTTP proxy (e.g., mitmproxy, Burp Suite) to inject 5xx/504 responses