[Observer P1] ObserverWorkflow — long-running event evaluator in Temporal #24

New Issue

2026-09-09T23:42:23Z

poimen commented

2026-09-09 23:42:23 +00:00

What

A long-running Temporal workflow in poimen-workflows that receives cluster events via signals and evaluates them against operational goals.

Why

The api-gateway handles commodity routing (Gotify push, SQS queue). Intelligent observation — "is this PR stale?", "did deploys degrade after this merge?", "are error rates spiking?" — needs durable state and goal tracking. Temporal gives us that for free.

Architecture

api-gateway /v1/events
  └── Temporal signal → ObserverWorkflow (long-running)
        ├── Evaluate event against goals (activity)
        ├── Pattern detection (LLM activity)
        └── Action dispatch
              ├── Start child workflow (remediation)
              ├── Signal running orchestrator (inject lesson)
              └── Gotify/SQS notification (activity)

Files (poimen-workflows repo)

statemachine/observer.go — ObserverWorkflow
statemachine/goals.go — Goal types + built-in goals
action/observer_eval.go — evaluate event against goals
action/observer_notify.go — dispatch notifications
tests/observer_workflow_test.go — unit tests

Acceptance

Workflow Lifecycle

ObserverWorkflow starts with WorkflowID: observer-main and stays running
Receives events via workflow.GetSignalChannel("event") — send signal, workflow processes it
After processing 1000 events, calls continue-as-new with carried state
Goal state (counters, timestamps, active violations) survives continue-as-new: verify by sending events before and after CAN, goal evaluation is consistent
Workflow recoverable after worker crash: kill worker pod, restart, workflow resumes from last checkpoint

Goal Evaluation

Goal pr_review_sla: pr.created event starts a timer; if no pr.review within 2h for same subject → triggers notification action
Goal deploy_health: sync.failed event → immediately triggers alert (priority 10)
Goal build_success_rate: tracks build.success/build.failure ratio over sliding window; drops below 95% → triggers alert
Goal cooldown: same goal violation within cooldown period → suppressed (no duplicate alerts)
Goal with no matching event type → skipped (no error)

Action Dispatch

Violated goal with action notify → calls Gotify activity with correct priority
Violated goal with action start_workflow → starts child workflow with event as input
Violated goal with action signal_orchestrator → signals running orchestrator workflow (e.g., inject lesson)
Action failure → logged as warning, does not crash observer workflow

Tests

go test ./tests/observer_workflow_test.go — receives 3 events via signal, evaluates goals, dispatches actions (mocked activities)
go test ./tests/observer_workflow_test.go — continue-as-new after N events, state carried
go test ./tests/observer_workflow_test.go — goal cooldown: duplicate violation suppressed
go test ./tests/observer_workflow_test.go — no goals registered: events accepted, no actions dispatched
go test ./tests/observer_workflow_test.go — action failure: logged, workflow continues

## What A long-running Temporal workflow in `poimen-workflows` that receives cluster events via signals and evaluates them against operational goals. ## Why The api-gateway handles commodity routing (Gotify push, SQS queue). Intelligent observation — "is this PR stale?", "did deploys degrade after this merge?", "are error rates spiking?" — needs durable state and goal tracking. Temporal gives us that for free. ## Architecture ``` api-gateway /v1/events └── Temporal signal → ObserverWorkflow (long-running) ├── Evaluate event against goals (activity) ├── Pattern detection (LLM activity) └── Action dispatch ├── Start child workflow (remediation) ├── Signal running orchestrator (inject lesson) └── Gotify/SQS notification (activity) ``` ## Files (poimen-workflows repo) - `statemachine/observer.go` — ObserverWorkflow - `statemachine/goals.go` — Goal types + built-in goals - `action/observer_eval.go` — evaluate event against goals - `action/observer_notify.go` — dispatch notifications - `tests/observer_workflow_test.go` — unit tests ## Acceptance ### Workflow Lifecycle - [ ] `ObserverWorkflow` starts with `WorkflowID: observer-main` and stays running - [ ] Receives events via `workflow.GetSignalChannel("event")` — send signal, workflow processes it - [ ] After processing 1000 events, calls `continue-as-new` with carried state - [ ] Goal state (counters, timestamps, active violations) survives `continue-as-new`: verify by sending events before and after CAN, goal evaluation is consistent - [ ] Workflow recoverable after worker crash: kill worker pod, restart, workflow resumes from last checkpoint ### Goal Evaluation - [ ] Goal `pr_review_sla`: `pr.created` event starts a timer; if no `pr.review` within 2h for same subject → triggers notification action - [ ] Goal `deploy_health`: `sync.failed` event → immediately triggers alert (priority 10) - [ ] Goal `build_success_rate`: tracks `build.success`/`build.failure` ratio over sliding window; drops below 95% → triggers alert - [ ] Goal cooldown: same goal violation within cooldown period → suppressed (no duplicate alerts) - [ ] Goal with no matching event type → skipped (no error) ### Action Dispatch - [ ] Violated goal with action `notify` → calls Gotify activity with correct priority - [ ] Violated goal with action `start_workflow` → starts child workflow with event as input - [ ] Violated goal with action `signal_orchestrator` → signals running orchestrator workflow (e.g., inject lesson) - [ ] Action failure → logged as warning, does not crash observer workflow ### Tests - [ ] `go test ./tests/observer_workflow_test.go` — receives 3 events via signal, evaluates goals, dispatches actions (mocked activities) - [ ] `go test ./tests/observer_workflow_test.go` — continue-as-new after N events, state carried - [ ] `go test ./tests/observer_workflow_test.go` — goal cooldown: duplicate violation suppressed - [ ] `go test ./tests/observer_workflow_test.go` — no goals registered: events accepted, no actions dispatched - [ ] `go test ./tests/observer_workflow_test.go` — action failure: logged, workflow continues

poimen added this to the Event-Driven Platform milestone 2026-09-09 23:42:23 +00:00