109 lines
4.6 KiB
Markdown
109 lines
4.6 KiB
Markdown
# T10.8 — Orchestrator reference service
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | P10 — Orchestration |
|
|
| Size | M — 1 to 3 days |
|
|
| Status | Not started |
|
|
| Flags | — |
|
|
| Spec | inlined below |
|
|
| Blocks | — |
|
|
|
|
## Goal
|
|
|
|
Reference process that consumes `.events`, invokes the registered
|
|
`Reconciler` for goal-tagged terminal transitions, and publishes decisions
|
|
to `.dispatch` — never spawns a run itself.
|
|
|
|
## Facts (inlined — no spec read needed)
|
|
|
|
- The orchestrator embeds poimen only via the HTTP API (T10.4) /
|
|
`poimen-sdk`, exactly like any other external embedder — no special
|
|
kernel access.
|
|
- Its entire job: consume `.events`, filter for goal-tagged
|
|
`Verified`/`Graded`/`Ungraded`, call `Reconciler` (T10.2), publish the
|
|
resulting `SpawnNext` decision to `.dispatch`. `Stop`/`Retry` decisions
|
|
are recorded but do not publish a dispatch message.
|
|
- The default `Reconciler` implementation may itself be another poimen run
|
|
(a `Model` step + a `Tool` step) — the orchestrator's job is identical
|
|
regardless of whether the bound `Reconciler` is LLM-backed or a
|
|
hand-written Rust impl.
|
|
- Never publishes a run directly, never calls `POST /v1/runs` itself — that
|
|
would reintroduce the bypass T10.7's inbox relay exists to prevent. Its
|
|
only write is to `.dispatch`.
|
|
|
|
## Steps
|
|
|
|
1. Consume `.events`, filter to records with a `GoalId` and a terminal run
|
|
state (`Verified`/`Graded`/`Ungraded`).
|
|
2. For each, build `ReconcileCtx` (T10.2) and call the `Reconciler` bound to
|
|
that goal.
|
|
3. On `SpawnNext`: publish `{workflow_ref, input, goal_id, dispatch_key}` to
|
|
`.dispatch` — `dispatch_key` derived from the decision id so redelivery
|
|
is naturally deduped downstream (T10.3).
|
|
4. On `Retry`/`Stop`: record the decision (`WorkEvent` provenance per
|
|
T10.2), publish nothing.
|
|
5. Own failure domain: separate process, own consumer group, own
|
|
retry/backoff on `Reconciler`-call failure — a `Reconciler` error does
|
|
not crash the orchestrator, it skips and retries on next poll or leaves
|
|
the goal stalled with a visible metric.
|
|
6. Emit reconcile-latency and goal-stall metrics.
|
|
7. Test the full loop against a stub `Reconciler`: events in, dispatch out,
|
|
verified end to end including a two-hop goal chain (run A → reconcile →
|
|
run B → reconcile → stop).
|
|
|
|
## Acceptance
|
|
|
|
- A two-run goal chain (A completes → reconcile → B spawns → B completes →
|
|
reconcile → `Stop`) completes end to end through real Kafka topics, with
|
|
the orchestrator never calling spawn directly.
|
|
- `Reconciler` failure on one goal does not block reconciliation of other
|
|
goals.
|
|
|
|
## Verify
|
|
|
|
**Harness:** real Kafka (or the board's chosen fault-injection substrate),
|
|
stub `Reconciler` with scripted decisions, T10.7's inbox relay running
|
|
alongside.
|
|
|
|
**Integration test** — `tests/it_orchestrator_e2e.rs`:
|
|
1. Spawn run A with `goal_id=G`, no `dispatch_key`. Drive to
|
|
`Verified{pass}`.
|
|
2. Assert orchestrator calls `Reconciler` exactly once for A; stub returns
|
|
`SpawnNext` for run B.
|
|
3. Assert a message lands on `.dispatch`, and via T10.7's relay, run B is
|
|
spawned with `goal_id=G`.
|
|
4. Drive run B to `Graded`. Assert `Reconciler` called again; stub returns
|
|
`Stop`.
|
|
5. Assert no further `.dispatch` message is published; goal G's chain shows
|
|
exactly `[A, B]` via T10.1's `GoalView`.
|
|
6. Instrument `POST /v1/runs`; assert zero calls originate from the
|
|
orchestrator process — only from the inbox relay.
|
|
7. Second goal G2 with a `Reconciler` that always errors; assert G2 stalls
|
|
(visible via goal-stall metric) while G's chain, run concurrently,
|
|
proceeds unaffected.
|
|
|
|
**Command:** `cargo test -p orchestration orchestrator_e2e -- --test-threads=1`
|
|
|
|
**False pass:**
|
|
- Step 6 omitted: an orchestrator that takes a shortcut and spawns directly
|
|
under test-only conditions would still pass steps 1-5.
|
|
- Testing only a single-hop chain (A → `Stop`) — the two-hop case is what
|
|
proves goal-chain provenance actually threads through `GoalView`
|
|
correctly across a full reconcile-dispatch-relay-spawn cycle.
|
|
|
|
## Traps
|
|
|
|
- Orchestrator holding an admission-layer shortcut "to save a network hop."
|
|
Collapses the exact boundary T10.7 was built to enforce.
|
|
- Coupling `Reconciler` failure handling to a single global retry policy.
|
|
One broken goal's `Reconciler` should not starve reconciliation for every
|
|
other goal in the tenant.
|
|
|
|
---
|
|
|
|
Background (not required to do this task):
|
|
[T10.1-goalid-and-goal-scoped-query.md](T10.1-goalid-and-goal-scoped-query.md) ·
|
|
[T10.2-reconciler-port.md](T10.2-reconciler-port.md) ·
|
|
[T10.7-kafka-topics-and-inbox-relay.md](T10.7-kafka-topics-and-inbox-relay.md)
|