poimen/tasks/T10.8-orchestrator-reference-service.md

T10.8 — Orchestrator reference service

Field	Value
Phase	P10 — Orchestration
Size	M — 1 to 3 days
Status	Not started
Flags	—
Spec	inlined below
Blocks	—

Goal

Reference process that consumes .events, invokes the registered Reconciler for goal-tagged terminal transitions, and publishes decisions to .dispatch — never spawns a run itself.

Facts (inlined — no spec read needed)

• The orchestrator embeds poimen only via the HTTP API (T10.4) / poimen-sdk, exactly like any other external embedder — no special kernel access.
• Its entire job: consume .events, filter for goal-tagged Verified/Graded/Ungraded, call Reconciler (T10.2), publish the resulting SpawnNext decision to .dispatch. Stop/Retry decisions are recorded but do not publish a dispatch message.
• The default Reconciler implementation may itself be another poimen run (a Model step + a Tool step) — the orchestrator's job is identical regardless of whether the bound Reconciler is LLM-backed or a hand-written Rust impl.
• Never publishes a run directly, never calls POST /v1/runs itself — that would reintroduce the bypass T10.7's inbox relay exists to prevent. Its only write is to .dispatch.

Steps

1. Consume .events, filter to records with a GoalId and a terminal run state (Verified/Graded/Ungraded).
2. For each, build ReconcileCtx (T10.2) and call the Reconciler bound to that goal.
3. On SpawnNext: publish {workflow_ref, input, goal_id, dispatch_key} to .dispatch — dispatch_key derived from the decision id so redelivery is naturally deduped downstream (T10.3).
4. On Retry/Stop: record the decision (WorkEvent provenance per T10.2), publish nothing.
5. Own failure domain: separate process, own consumer group, own retry/backoff on Reconciler-call failure — a Reconciler error does not crash the orchestrator, it skips and retries on next poll or leaves the goal stalled with a visible metric.
6. Emit reconcile-latency and goal-stall metrics.
7. Test the full loop against a stub Reconciler: events in, dispatch out, verified end to end including a two-hop goal chain (run A → reconcile → run B → reconcile → stop).

Acceptance

• A two-run goal chain (A completes → reconcile → B spawns → B completes → reconcile → Stop) completes end to end through real Kafka topics, with the orchestrator never calling spawn directly.
• Reconciler failure on one goal does not block reconciliation of other goals.

Verify

Harness: real Kafka (or the board's chosen fault-injection substrate), stub Reconciler with scripted decisions, T10.7's inbox relay running alongside.

Integration test — tests/it_orchestrator_e2e.rs:

1. Spawn run A with goal_id=G, no dispatch_key. Drive to Verified{pass}.
2. Assert orchestrator calls Reconciler exactly once for A; stub returns SpawnNext for run B.
3. Assert a message lands on .dispatch, and via T10.7's relay, run B is spawned with goal_id=G.
4. Drive run B to Graded. Assert Reconciler called again; stub returns Stop.
5. Assert no further .dispatch message is published; goal G's chain shows exactly [A, B] via T10.1's GoalView.
6. Instrument POST /v1/runs; assert zero calls originate from the orchestrator process — only from the inbox relay.
7. Second goal G2 with a Reconciler that always errors; assert G2 stalls (visible via goal-stall metric) while G's chain, run concurrently, proceeds unaffected.

Command: cargo test -p orchestration orchestrator_e2e -- --test-threads=1

False pass:

• Step 6 omitted: an orchestrator that takes a shortcut and spawns directly under test-only conditions would still pass steps 1-5.
• Testing only a single-hop chain (A → Stop) — the two-hop case is what proves goal-chain provenance actually threads through GoalView correctly across a full reconcile-dispatch-relay-spawn cycle.

Traps

• Orchestrator holding an admission-layer shortcut "to save a network hop." Collapses the exact boundary T10.7 was built to enforce.
• Coupling Reconciler failure handling to a single global retry policy. One broken goal's Reconciler should not starve reconciliation for every other goal in the tenant.

---

Background (not required to do this task): T10.1-goalid-and-goal-scoped-query.md · T10.2-reconciler-port.md · T10.7-kafka-topics-and-inbox-relay.md