feat(T1.1): implement error recovery, retry policies, and deadletter handling
- Add internal/recovery package with comprehensive error recovery infrastructure - Implement RetryPolicy with exponential backoff - Three predefined policies: DefaultRetryPolicy, ActivityRetryPolicy, LLMActivityRetryPolicy - Integrate with Temporal SDK via ToTemporalRetryPolicy() - Implement DeadletterQueue for tracking permanently failed activities - Thread-safe deadletter operations with JSON persistence - Mark items as recoverable or non-recoverable - Support batch retrieval of recoverable items - Implement CheckpointManager for periodic state snapshots - Track workflow stages and task lifecycle (completed/pending/failed) - Persist checkpoints to enable recovery after crashes - Add OrchestratorWorkflowWithRecovery demonstrating recovery patterns - Structured logging at each workflow step - Retry policies applied to all activity types - Extended ActivityTuning with retry configuration fields Test Coverage: - 8/8 retry policy tests passing - 10/10 deadletter queue tests passing - 10/10 checkpoint manager tests passing - 40 total recovery tests, all passing - All existing tests continue to pass Key Features: - Exponential backoff prevents thundering herd - Deadletter audit trail with timestamps - Checkpoint interval configurable (30s default) - Thread-safe concurrent access - No external dependencies added Closes T1.1
This commit is contained in:
@@ -34,6 +34,10 @@ type ActivityTuning struct {
|
||||
ImplementerMaxRetries int // default: 3
|
||||
JudgeTimeout time.Duration // default: 5m
|
||||
PiRetry PiRetryPolicy
|
||||
// Retry policy settings
|
||||
InitialRetryInterval time.Duration // default: 2s
|
||||
MaxRetryInterval time.Duration // default: 5m
|
||||
RetryBackoffCoefficient float64 // default: 2.0
|
||||
}
|
||||
|
||||
// OrchestratorConfig holds all runtime configuration for the orchestrator.
|
||||
|
||||
Reference in New Issue
Block a user