(plan) update the plan to captures more

2026-08-18 20:04:29 -07:00
parent 5a30d0ffc6
commit 139f8419bb
11 changed files with 993 additions and 1 deletions
@@ -0,0 +1,108 @@
+# T10.8 — Orchestrator reference service
+
+| Field | Value |
+|---|---|
+| Phase | P10 — Orchestration |
+| Size | M — 1 to 3 days |
+| Status | Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | — |
+
+## Goal
+
+Reference process that consumes `.events`, invokes the registered
+`Reconciler` for goal-tagged terminal transitions, and publishes decisions
+to `.dispatch` — never spawns a run itself.
+
+## Facts (inlined — no spec read needed)
+
+- The orchestrator embeds poimen only via the HTTP API (T10.4) /
+  `poimen-sdk`, exactly like any other external embedder — no special
+  kernel access.
+- Its entire job: consume `.events`, filter for goal-tagged
+  `Verified`/`Graded`/`Ungraded`, call `Reconciler` (T10.2), publish the
+  resulting `SpawnNext` decision to `.dispatch`. `Stop`/`Retry` decisions
+  are recorded but do not publish a dispatch message.
+- The default `Reconciler` implementation may itself be another poimen run
+  (a `Model` step + a `Tool` step) — the orchestrator's job is identical
+  regardless of whether the bound `Reconciler` is LLM-backed or a
+  hand-written Rust impl.
+- Never publishes a run directly, never calls `POST /v1/runs` itself — that
+  would reintroduce the bypass T10.7's inbox relay exists to prevent. Its
+  only write is to `.dispatch`.
+
+## Steps
+
+1. Consume `.events`, filter to records with a `GoalId` and a terminal run
+   state (`Verified`/`Graded`/`Ungraded`).
+2. For each, build `ReconcileCtx` (T10.2) and call the `Reconciler` bound to
+   that goal.
+3. On `SpawnNext`: publish `{workflow_ref, input, goal_id, dispatch_key}` to
+   `.dispatch` — `dispatch_key` derived from the decision id so redelivery
+   is naturally deduped downstream (T10.3).
+4. On `Retry`/`Stop`: record the decision (`WorkEvent` provenance per
+   T10.2), publish nothing.
+5. Own failure domain: separate process, own consumer group, own
+   retry/backoff on `Reconciler`-call failure — a `Reconciler` error does
+   not crash the orchestrator, it skips and retries on next poll or leaves
+   the goal stalled with a visible metric.
+6. Emit reconcile-latency and goal-stall metrics.
+7. Test the full loop against a stub `Reconciler`: events in, dispatch out,
+   verified end to end including a two-hop goal chain (run A → reconcile →
+   run B → reconcile → stop).
+
+## Acceptance
+
+- A two-run goal chain (A completes → reconcile → B spawns → B completes →
+  reconcile → `Stop`) completes end to end through real Kafka topics, with
+  the orchestrator never calling spawn directly.
+- `Reconciler` failure on one goal does not block reconciliation of other
+  goals.
+
+## Verify
+
+**Harness:** real Kafka (or the board's chosen fault-injection substrate),
+stub `Reconciler` with scripted decisions, T10.7's inbox relay running
+alongside.
+
+**Integration test** — `tests/it_orchestrator_e2e.rs`:
+1. Spawn run A with `goal_id=G`, no `dispatch_key`. Drive to
+   `Verified{pass}`.
+2. Assert orchestrator calls `Reconciler` exactly once for A; stub returns
+   `SpawnNext` for run B.
+3. Assert a message lands on `.dispatch`, and via T10.7's relay, run B is
+   spawned with `goal_id=G`.
+4. Drive run B to `Graded`. Assert `Reconciler` called again; stub returns
+   `Stop`.
+5. Assert no further `.dispatch` message is published; goal G's chain shows
+   exactly `[A, B]` via T10.1's `GoalView`.
+6. Instrument `POST /v1/runs`; assert zero calls originate from the
+   orchestrator process — only from the inbox relay.
+7. Second goal G2 with a `Reconciler` that always errors; assert G2 stalls
+   (visible via goal-stall metric) while G's chain, run concurrently,
+   proceeds unaffected.
+
+**Command:** `cargo test -p orchestration orchestrator_e2e -- --test-threads=1`
+
+**False pass:**
+- Step 6 omitted: an orchestrator that takes a shortcut and spawns directly
+  under test-only conditions would still pass steps 1-5.
+- Testing only a single-hop chain (A → `Stop`) — the two-hop case is what
+  proves goal-chain provenance actually threads through `GoalView`
+  correctly across a full reconcile-dispatch-relay-spawn cycle.
+
+## Traps
+
+- Orchestrator holding an admission-layer shortcut "to save a network hop."
+  Collapses the exact boundary T10.7 was built to enforce.
+- Coupling `Reconciler` failure handling to a single global retry policy.
+  One broken goal's `Reconciler` should not starve reconciliation for every
+  other goal in the tenant.
+
+---
+
+Background (not required to do this task):
+[T10.1-goalid-and-goal-scoped-query.md](T10.1-goalid-and-goal-scoped-query.md) ·
+[T10.2-reconciler-port.md](T10.2-reconciler-port.md) ·
+[T10.7-kafka-topics-and-inbox-relay.md](T10.7-kafka-topics-and-inbox-relay.md)