poimen/tasks/T10.8-orchestrator-reference-service.md

# T10.8 — Orchestrator reference service

| Field | Value |
|---|---|
| Phase | P10 — Orchestration |
| Size | M — 1 to 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |

## Goal

Reference process that consumes `.events`, invokes the registered
`Reconciler` for goal-tagged terminal transitions, and publishes decisions
to `.dispatch` — never spawns a run itself.

## Facts (inlined — no spec read needed)

- The orchestrator embeds poimen only via the HTTP API (T10.4) /
  `poimen-sdk`, exactly like any other external embedder — no special
  kernel access.
- Its entire job: consume `.events`, filter for goal-tagged
  `Verified`/`Graded`/`Ungraded`, call `Reconciler` (T10.2), publish the
  resulting `SpawnNext` decision to `.dispatch`. `Stop`/`Retry` decisions
  are recorded but do not publish a dispatch message.
- The default `Reconciler` implementation may itself be another poimen run
  (a `Model` step + a `Tool` step) — the orchestrator's job is identical
  regardless of whether the bound `Reconciler` is LLM-backed or a
  hand-written Rust impl.
- Never publishes a run directly, never calls `POST /v1/runs` itself — that
  would reintroduce the bypass T10.7's inbox relay exists to prevent. Its
  only write is to `.dispatch`.

## Steps

1. Consume `.events`, filter to records with a `GoalId` and a terminal run
   state (`Verified`/`Graded`/`Ungraded`).
2. For each, build `ReconcileCtx` (T10.2) and call the `Reconciler` bound to
   that goal.
3. On `SpawnNext`: publish `{workflow_ref, input, goal_id, dispatch_key}` to
   `.dispatch` — `dispatch_key` derived from the decision id so redelivery
   is naturally deduped downstream (T10.3).
4. On `Retry`/`Stop`: record the decision (`WorkEvent` provenance per
   T10.2), publish nothing.
5. Own failure domain: separate process, own consumer group, own
   retry/backoff on `Reconciler`-call failure — a `Reconciler` error does
   not crash the orchestrator, it skips and retries on next poll or leaves
   the goal stalled with a visible metric.
6. Emit reconcile-latency and goal-stall metrics.
7. Test the full loop against a stub `Reconciler`: events in, dispatch out,
   verified end to end including a two-hop goal chain (run A → reconcile →
   run B → reconcile → stop).

## Acceptance

- A two-run goal chain (A completes → reconcile → B spawns → B completes →
  reconcile → `Stop`) completes end to end through real Kafka topics, with
  the orchestrator never calling spawn directly.
- `Reconciler` failure on one goal does not block reconciliation of other
  goals.

## Verify

**Harness:** real Kafka (or the board's chosen fault-injection substrate),
stub `Reconciler` with scripted decisions, T10.7's inbox relay running
alongside.

**Integration test** — `tests/it_orchestrator_e2e.rs`:
1. Spawn run A with `goal_id=G`, no `dispatch_key`. Drive to
   `Verified{pass}`.
2. Assert orchestrator calls `Reconciler` exactly once for A; stub returns
   `SpawnNext` for run B.
3. Assert a message lands on `.dispatch`, and via T10.7's relay, run B is
   spawned with `goal_id=G`.
4. Drive run B to `Graded`. Assert `Reconciler` called again; stub returns
   `Stop`.
5. Assert no further `.dispatch` message is published; goal G's chain shows
   exactly `[A, B]` via T10.1's `GoalView`.
6. Instrument `POST /v1/runs`; assert zero calls originate from the
   orchestrator process — only from the inbox relay.
7. Second goal G2 with a `Reconciler` that always errors; assert G2 stalls
   (visible via goal-stall metric) while G's chain, run concurrently,
   proceeds unaffected.

**Command:** `cargo test -p orchestration orchestrator_e2e -- --test-threads=1`

**False pass:**
- Step 6 omitted: an orchestrator that takes a shortcut and spawns directly
  under test-only conditions would still pass steps 1-5.
- Testing only a single-hop chain (A → `Stop`) — the two-hop case is what
  proves goal-chain provenance actually threads through `GoalView`
  correctly across a full reconcile-dispatch-relay-spawn cycle.

## Traps

- Orchestrator holding an admission-layer shortcut "to save a network hop."
  Collapses the exact boundary T10.7 was built to enforce.
- Coupling `Reconciler` failure handling to a single global retry policy.
  One broken goal's `Reconciler` should not starve reconciliation for every
  other goal in the tenant.

---

Background (not required to do this task):
[T10.1-goalid-and-goal-scoped-query.md](T10.1-goalid-and-goal-scoped-query.md) ·
[T10.2-reconciler-port.md](T10.2-reconciler-port.md) ·
[T10.7-kafka-topics-and-inbox-relay.md](T10.7-kafka-topics-and-inbox-relay.md)