rock/poimen-workflows
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(T1.5): implement workflow pause/resume with state snapshots

Browse Source 
- Add internal/pause package for pause/resume orchestration
- Implement WorkflowSnapshot for complete state serialization
- Implement SnapshotManager for snapshot storage and recovery
- Implement PauseHandler for pause/resume signal handling
- Implement PauseSignal and ResumeSignal types
- Implement PauseState for tracking pause status

Snapshot Features:
- Capture complete workflow state (tasks, metrics, config)
- Persist to JSON files for recovery after pod restart
- Track paused_at and resumed_at timestamps
- Support snapshot cleanup and batch removal
- Load/save from disk with persistence layer

Pause Handling:
- Accept pause signals with reason and grace period
- Save current state before pausing
- Block workflow execution during pause
- Support multiple concurrent paused workflows
- Channel-based signal reception (Temporal-compatible)

Resume Handling:
- Accept resume signals with reason
- Restore workflow state from snapshots
- Continue execution from exact pause point
- Update timestamps on resumption
- Enable recovery after pod restarts

Signal Management:
- Non-blocking signal reception with timeout
- WaitForPauseOrResume() for blocking operations
- ConfigurableWait duration
- Error handling for invalid transitions

Analytics:
- GetPauseStats() for pause/resume metrics
- GetSnapshotStats() for snapshot inventory
- Timestamp tracking (paused, resumed)
- Multi-workflow state aggregation

Test Coverage:
- 16 snapshot tests (creation, persistence, cleanup)
- 18 handler tests (signals, state, snapshots)
- 34 total pause/resume tests, all passing
- Edge cases: concurrent workflows, nil signals, timeouts
- State transition verification

Key Design:
- Separate Snapshot Manager (storage) and Pause Handler (orchestration)
- JSON persistence for debuggability
- Thread-safe with RWMutex
- Compatible with Temporal signal patterns
- Non-destructive pause (snapshot before blocking)

Closes T1.5
This commit is contained in:
Test
2026-08-23 17:11:56 -07:00
parent b1e3136350
commit 37d7aea5a7
6 changed files with 1463 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tasks/board-T1.md
+1 -1
View File
@@ -8,7 +8,7 @@
| T1.2 | Structured logging + metrics export (Prometheus/OpenTelemetry integration) | [x] | `task/T1.2` | Metrics visible in homelab Grafana, logs queryable in Loki |
| T1.3 | Activity timeout tuning automation: learn from historical failures, recommend overrides | [x] | `task/T1.3` | Planner reads lessons file, suggests `update-tuning` signal based on patterns |
| T1.4 | Board state validation: detect corruption, auto-heal from board divergence | [x] | `task/T1.4` | Corrupt board file recovered without manual intervention |
| T1.5 | Workflow pause/resume with state snapshot: serialize mid-cycle state to persistent store | [ ] | `task/T1.5` | Pause signal, restart pod, resume signal → workflow continues from exact point |
| T1.5 | Workflow pause/resume with state snapshot: serialize mid-cycle state to persistent store | [x] | `task/T1.5` | Pause signal, restart pod, resume signal → workflow continues from exact point |
| T1.6 | Comprehensive integration tests: multi-pod concurrency, network flakiness simulation | [ ] | `task/T1.6` | Concurrent orchestrator instances on shared repo pass e2e without conflicts |
| T1.7 | Audit logging: all planner decisions, judge verdicts, implementer changes logged immutably | [ ] | `task/T1.7` | Audit log persists across workflow restarts, queryable by task/timestamp |
| T1.8 | Health checks: Temporal connectivity, git repo accessibility, LLM API availability | [x] | `task/T1.8` | Periodic health probes, liveness/readiness endpoints for K8s |