(plan) update the plan to captures more

This commit is contained in:
Story Crater Bot
2026-08-18 20:04:29 -07:00
parent 5a30d0ffc6
commit 139f8419bb
11 changed files with 993 additions and 1 deletions
+31 -1
View File
@@ -211,7 +211,8 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
| 8 | Learning loop | T6.x | 8 | 0 | 0 | 8 | ⬜ T6.8 | — | — |
| 9 | Distribution | T7.x | 8 | 0 | 0 | 8 | ⬜ T7.8 | — | — |
| 10 | Operability | T8.x | 7 | 0 | 0 | 7 | ⬜ T8.7 | — | — |
| | **Total** | | **80** | **3** | **0** | **77** | 0/10 green | **313.0k** | **—** |
| 11 | Orchestration | T10.x | 10 | 0 | 0 | 10 | ⬜ T10.10 | — | — |
| | **Total** | | **90** | **3** | **0** | **87** | 0/11 green | **313.0k** | **—** |
**What changed in this revision.** The public surface moved ahead of the
durability hard parts, and an embedding phase was added after verification.
@@ -361,3 +362,32 @@ whose architecture already expects capabilities to arrive as swappable providers
| [T8.5](T8.5-embedded-mode-smoke.md) | Embedded-mode smoke | S | parallel-ok | ⬜ |
| [T8.6](T8.6-capacity-admission-control.md) | Capacity admission control | M | — | ⬜ |
| [T8.7](T8.7-p8-composition-gate.md) | **Operability composition gate** | L | gate | ⬜ |
## 11 — Orchestration · T10.x
Fills §17's deferred line — multi-agent orchestration, revisited now that
cross-agent coordination is real. A `Reconciler` port picks up a run at its
existing `Verified`/`Graded`/`Ungraded` resting states, decides what runs
next toward a `GoalId`-scoped objective, and dispatches through Kafka via
the same outbox/inbox discipline T7.4 established — never a direct spawn
call. An HTTP surface binds one method table shared with T9.2's eventual
stdio transport. Two front doors, one IR: `tasks/*.md` parsed directly
(door a), or a free-text problem statement drafted and parked in
`Suspended` for human approval before it runs (door b).
Depends on T1.x, T2.1/T2.3, T3.x, T7.3/T7.4, T9.1 landing first; not blocked
on T9.2T9.6 (the stdio binding develops in parallel against the same
method table T10.4 authors).
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [T10.1](T10.1-goalid-and-goal-scoped-query.md) | `GoalId` and goal-scoped query | S | — | ⬜ |
| [T10.2](T10.2-reconciler-port.md) | `Reconciler` port | M | — | ⬜ |
| [T10.3](T10.3-idempotent-run-admission.md) | Idempotent run admission | S | — | ⬜ |
| [T10.4](T10.4-http-api-surface.md) | HTTP API surface | L | — | ⬜ |
| [T10.5](T10.5-taskboardformat-plugin.md) | `TaskBoardFormat` plugin | M | — | ⬜ |
| [T10.6](T10.6-plan-draft-and-suspended-approval.md) | `plan-draft` workflow + `Suspended`-approval flow | M | — | ⬜ |
| [T10.7](T10.7-kafka-topics-and-inbox-relay.md) | Kafka topics and inbox relay | L | — | ⬜ |
| [T10.8](T10.8-orchestrator-reference-service.md) | Orchestrator reference service | M | — | ⬜ |
| [T10.9](T10.9-cli-transcript-and-metrics-ux.md) | CLI: runs list/episode/transcript | S | — | ⬜ |
| [T10.10](T10.10-p10-composition-gate.md) | **Orchestration composition gate** | L | gate | ⬜ |
@@ -0,0 +1,82 @@
# T10.1 — `GoalId` and goal-scoped query
| Field | Value |
|---|---|
| Phase | P10 — Orchestration |
| Size | S — under 1 day |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | T10.2, T10.3, T10.4, T10.8 |
## Goal
New identity `GoalId(Ulid)` grouping runs that share an objective, optional at
spawn. Extend `T1.7`'s query surface to read across `RunId`s sharing one
`GoalId`.
## Facts (inlined — no spec read needed)
- `GoalId` is tenant-scoped like `RunId`, **not** `TaskId`. `TaskId` is already
the grading comparison-group key (§11.5) — reusing it corrupts
group-epoch/quorum semantics the learning loop depends on.
- Attaching `goal_id` at spawn is **optional**. A run with no `GoalId` behaves
exactly as today — this is additive, not a required field.
- `GoalView` is a new, separate query. It does not replace per-run
`EpisodeView`; it lists member `RunId`s and references their existing
metadata, no duplication.
- Same parity rule as T1.7: `GoalView` from materialized state must equal
`GoalView` from a cold re-fold (T0.8's property applied here too).
## Steps
1. Declare `GoalId(Ulid)` in the `ids` crate, alongside `RunId`.
2. Add optional `goal_id: Option<GoalId>` to run-spawn input and the run
record.
3. Index runs by `(TenantId, GoalId)` in materialized state.
4. Define `GoalView { goal: GoalId, runs: Vec<RunSummary> }`, `RunSummary`
referencing existing run metadata — do not duplicate `EpisodeView` fields.
5. Implement the query against materialized state and against cold re-fold.
6. Test: two runs sharing a `GoalId`, one plain run with none; assert
`GoalView` returns exactly the two, in spawn order.
## Acceptance
- Two runs sharing a `GoalId` are both returned by `GoalView`; a run with no
`GoalId` never appears in any `GoalView`.
- `GoalView` from materialized state equals `GoalView` from cold re-fold.
## Verify
**Harness:** two runs sharing a `GoalId`, one run with no `GoalId`, one
branch/rewind case reused from `T1.7`'s harness.
**Integration test**`tests/it_goal_view_equivalence.rs`:
1. Spawn run A and run B both with `goal_id=G`; spawn run C with none.
2. Query `GoalView(G)` from materialized state → `view_a`.
3. Drop state tables; query again via cold re-fold → `view_b`.
4. Assert `serialize(view_a) == serialize(view_b)`.
5. Assert `view_a` contains exactly `{A, B}`, never `C`.
6. Assert order is spawn order, stable.
**Command:** `cargo test -p query goal_view_equivalence`
**False pass:**
- Testing only one run per goal — a grouping bug is invisible with a single
member.
- Skipping the "run without `goal_id`" exclusion check. A filter that
defaults to "all runs" passes silently without it.
## Traps
- Deriving `GoalId` from `TaskId` "since they're both groupings." Different
lifecycle, different owner — grading vs orchestration.
- Making `goal_id` required at spawn. Breaks every existing plain-workflow
spawn call.
---
Background (not required to do this task):
[rust-agentic-sys.md](../rust-agentic-sys.md) §11.5 ·
[T1.7-episode-query-surface.md](T1.7-episode-query-surface.md) ·
[T0.8-fold-and-re-derive.md](T0.8-fold-and-re-derive.md)
+88
View File
@@ -0,0 +1,88 @@
# T10.10 — Orchestration composition gate
| Field | Value |
|---|---|
| Phase | P10 — Orchestration |
| Size | L — over 3 days |
| Status | Not started |
| Flags | gate |
| Spec | inlined below |
| Blocks | — |
## Goal
Prove the properties no single T10.x task owns: a full goal chain survives a
real Kafka outage, dispatch is idempotent under forced redelivery,
`Suspended`-approval produces byte-identical IR to hand-authored YAML, and
the HTTP method table stands as a published artifact.
## Facts (inlined — no spec read needed)
- Mirrors T7.4's fault-injection shape — broker down for the whole test —
extended across the full events → reconcile → dispatch → inbox-relay →
spawn loop, not just outbox delivery.
- Front-door equivalence for door (b) extends T3.10's existing hash-equality
gate — same discipline, new door.
- Gate script follows board convention: one script, diffed against a
committed expected output, empty diff is the only pass.
## Steps
1. Compose a fixture: one goal, a 2-run chain (A → reconcile → B → reconcile
`Stop`), driven through real HTTP calls and real Kafka topics.
2. Kill the broker mid-chain — after A completes, before the dispatch
message for B is consumed; assert the chain still completes once the
broker returns, with B eventually spawned.
3. Force redelivery of the `.dispatch` message for B; assert exactly one
run B exists.
4. Submit a problem statement through door (b), edit the draft, approve;
separately hand-author the identical edited content as YAML and load it
through door (a); assert both `WorkflowVersion` hashes match.
5. Snapshot the HTTP method table (paths + request/response shapes) to a
committed fixture; assert no path removed and no field removed relative
to the fixture — additions are fine, removals fail the gate.
6. Assert a goal-stall metric fires for a deliberately broken `Reconciler`
on one goal while a healthy goal in the same tenant proceeds unaffected
(T10.8's concurrency claim, at gate scope).
## Acceptance
- All six checks above pass with an empty diff against
`verify/expected/T10.10.txt`.
## Verify
**Harness:** full stack — embedded `redb`, real Kafka (or the board's
chosen fault-injection substrate), `axum` API instance, stub `Reconciler`
with scripted multi-step decisions.
**Integration test**`tests/it_p10_composition_gate.rs`, one numbered
assertion per Steps item above (`a1`..`a6`), reported by name per the
board's script convention.
**Command:** `verify/T10.10.sh`, diffed against `verify/expected/T10.10.txt`.
**False pass:**
- Running the chain with the broker never actually taken down — the
"survives an outage" claim needs the outage to really happen mid-chain,
not just be configured as theoretically tolerable.
- Skipping the method-table snapshot diff — a route or field silently
removed here breaks every embedder bound to the table (HTTP today, stdio
later) without any single T10.x task's own tests catching it, since none
of them own the whole table.
## Traps
- Treating this gate as "run all T10.x tests together." It must assert
properties none of them individually assert, per the board's own gate
discipline (INDEX.md, "Verification practice," point 5).
- Committing the method-table fixture as freeform prose instead of a
diffable, structured snapshot. An unreviewable diff defeats the point of
having a fixture at all.
---
Background (not required to do this task):
[../INDEX.md](../INDEX.md) (verification practice) ·
[T7.4-outbox-relay.md](T7.4-outbox-relay.md) ·
[T3.10-front-door-equivalence.md](T3.10-front-door-equivalence.md)
+112
View File
@@ -0,0 +1,112 @@
# T10.2 — `Reconciler` port
| Field | Value |
|---|---|
| Phase | P10 — Orchestration |
| Size | M — 1 to 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | T10.6, T10.8 |
## Goal
New port answering "how far toward the goal, what next" from a run's terminal
resting state — distinct from `Verifier` (fail-closed pass/fail) and `Judge`
(relative-only comparison).
## Facts (inlined — no spec read needed)
- `Verifier::verify` is fail-closed pass/fail (rust-agentic-sys.md §10).
`Judge::compare` is deliberately relative-only, "cannot return an absolute
score" (rust-agentic-sys.md §11.1). "How far toward goal, what next" is a
third question — forcing it through either port blurs a firewall built on
purpose. `Reconciler` is new, sized like `Verifier` (one method, S/M).
- Fires **only** off `Verified{pass|fail}`, `Graded`, `Ungraded` — states
`T1.3` already designed to rest in, never resolved inline.
- Reconciler failure degrades the record, never the run — same principle as
§1: "the framework cannot break the agent it runs." A `Reconciler` error or
timeout leaves the run at its terminal state; the goal simply does not
advance until retried.
- The decision does **not** spawn anything itself. Spawning happens only
through the admission path (T10.3) via the inbox relay (T10.7) — never a
direct call from the `Reconciler` or its caller.
- One new `#[non_exhaustive]` `WorkEvent` variant for provenance:
`SpawnedFromReconciliation{goal_id, parent_run, decision_ref}`. Added, not
replacing any existing variant.
## Steps
1. Define `ReconcileCtx``GoalId`, `GoalView` (T10.1), the just-terminal
run's `EpisodeView`, `WorkflowDef`.
2. Define `ReconcileDecision``SpawnNext{workflow_ref, input, dispatch_key}`,
`Retry{step}`, `Stop{reason}`.
3. Define the trait:
```rust
#[async_trait]
pub trait Reconciler: Send + Sync {
fn id(&self) -> ReconcilerId;
async fn reconcile(&self, cx: &ReconcileCtx) -> Result<ReconcileDecision>;
}
```
4. Wire the trigger: on a run reaching `Verified`/`Graded`/`Ungraded` with a
`GoalId` set, call the `Reconciler` bound to that goal.
5. On `Reconciler` error or timeout: log, leave the run at its terminal state,
do not auto-retry — matches `Verifier`'s explicit-only-retry discipline.
6. Add `WorkEvent::SpawnedFromReconciliation`, written on the **next**
spawned run's first log record, referencing `parent_run` and the decision.
7. Test with a stub `Reconciler` against all three trigger states.
## Acceptance
- A run reaching `Verified{pass}` with a `GoalId` triggers exactly one
reconcile call.
- The decision is durably recorded and traceable from the next spawned run
back to the parent run and goal.
- A `Reconciler` error leaves the run at its terminal state and does not
crash the executor.
## Verify
**Harness:** stub `Reconciler` returning each decision variant in turn; one
goal with a 2-run chain.
**Integration test** — `tests/it_reconciler_trigger.rs`:
1. Run A (goal G) reaches `Verified{pass}`; assert `reconcile()` called
exactly once with the correct `ReconcileCtx`.
2. Stub returns `SpawnNext`; assert the decision is persisted, referencing
run A.
3. Assert the next spawned run's first log record carries
`SpawnedFromReconciliation{goal_id: G, parent_run: A, decision_ref}`.
4. Repeat for `Graded` and `Ungraded` trigger states.
5. Stub `Reconciler` panics or times out; assert run A stays at its terminal
state, no crash, no infinite retry.
6. Run with no `GoalId` reaches `Verified{pass}`; assert `reconcile()` is
never called.
**Command:** `cargo test -p kernel reconciler_trigger`
**False pass:**
- Asserting reconcile "ran" via a boolean flag instead of asserting the
`WorkEvent` provenance record exists — passes even if a crash lost the
decision before persistence.
- Testing only `Verified{pass}`, skipping `Graded`/`Ungraded` — those are the
states most likely miswired, being newer additions to the FSM.
## Traps
- `Reconciler` spawning the next run directly. Violates "never bypass the
admission path" — see T10.3/T10.7.
- Overloading `Judge::compare` to also answer "what next." Collapses a
boundary `Judge`'s signature deliberately enforces (rust-agentic-sys.md
§11.1, "deliberately cannot return an absolute score").
- Auto-retrying a failed `Reconciler` call with no cap — a broken impl then
spins forever against every terminal run in the tenant.
---
Background (not required to do this task):
[rust-agentic-sys.md](../rust-agentic-sys.md) §1, §10, §11.1, §17 ·
[T1.3-run-executor.md](T1.3-run-executor.md) ·
[T4.1-verifier-port.md](T4.1-verifier-port.md) ·
[T10.1-goalid-and-goal-scoped-query.md](T10.1-goalid-and-goal-scoped-query.md)
+78
View File
@@ -0,0 +1,78 @@
# T10.3 — Idempotent run admission
| Field | Value |
|---|---|
| Phase | P10 — Orchestration |
| Size | S — under 1 day |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | T10.4, T10.7 |
## Goal
`dispatch_key` on spawn requests: a redelivered spawn request (from Kafka
at-least-once delivery, T10.7) becomes a no-op insert instead of a duplicate
run.
## Facts (inlined — no spec read needed)
- Same principle as the outbox's `(BranchKey, Lsn)` fold-key idempotency
(T7.4) — exactly-once is achieved at the admission boundary, not in
transport.
- `dispatch_key` is caller-supplied, opaque, unique per intended spawn — e.g.
a `Reconciler` decision id.
- Admission with no `dispatch_key` behaves exactly as today. This is
additive, not a required field.
## Steps
1. Add optional `dispatch_key: Option<String>` to spawn-run input.
2. Add a unique index on `(TenantId, dispatch_key)` where `dispatch_key` is
set.
3. On spawn: if `dispatch_key` is set and already seen, return the existing
`RunId` instead of creating a new run — no error, no duplicate.
4. Test redelivery: same `dispatch_key` submitted twice, concurrently and
sequentially.
## Acceptance
- Two spawn calls with the same `dispatch_key` produce exactly one run; the
second call returns the first run's `RunId`.
- Spawn calls with no `dispatch_key` are unaffected.
## Verify
**Harness:** spawn endpoint (T10.4) or direct kernel call, two concurrent
callers.
**Integration test**`tests/it_idempotent_admission.rs`:
1. Spawn with `dispatch_key=K`; assert one run created, `RunId=R`.
2. Spawn again with `dispatch_key=K`; assert no new run created, returned
`RunId == R`.
3. Fire 20 concurrent spawn calls with the same `dispatch_key`; assert
exactly one run exists after all resolve.
4. Spawn with no `dispatch_key` twice; assert two distinct runs (unaffected
baseline).
**Command:** `cargo test -p kernel idempotent_admission -- --test-threads=1`
**False pass:**
- Testing only sequential redelivery. Concurrent redelivery is the actual
failure mode under Kafka rebalance/retry and needs the unique-index race
caught, not a check-then-insert race.
## Traps
- Making `dispatch_key` required. Breaks every direct/manual spawn call that
isn't part of a goal chain.
- Deduping by a hash of `(workflow_ref, input)` instead of the caller-supplied
key. Two legitimately identical requests would then silently collapse into
one run.
---
Background (not required to do this task):
[rust-agentic-sys.md](../rust-agentic-sys.md) §8.3, §9.3 ·
[T7.4-outbox-relay.md](T7.4-outbox-relay.md) ·
[T3.6-version-pinning-at-spawn.md](T3.6-version-pinning-at-spawn.md)
+101
View File
@@ -0,0 +1,101 @@
# T10.4 — HTTP API surface
| Field | Value |
|---|---|
| Phase | P10 — Orchestration |
| Size | L — over 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | T10.6, T10.7, T10.9 |
## Goal
`axum` HTTP service binding one method table — spawn / status / stream /
episode / transcript / rewind / cancel / retry / goal / metrics — to
`poimen-sdk` ports. One method set: HTTP today, the same table T9.2's stdio
JSON-RPC binds to later.
## Facts (inlined — no spec read needed)
- **The protocol is a published artifact** — INDEX.md's own rule for the
sidecar applies equally here: methods added, never repurposed; fields
added, never removed, from the first commit.
- HTTP is a transport binding onto `poimen-sdk` (T9.1), not a bespoke access
layer with its own business logic. Handlers translate, they do not decide.
- `/v1/problems` and `/v1/plans/{id}/approve|reject` route to T10.6's
workflow, not implemented here — this task defines the routes and
request/response contracts; T10.6 fills in plan-draft semantics.
- `/v1/runs/{id}/stream` tails the same committed log the outbox relay reads
— SSE, not a second read path.
## Steps
1. Stand up the `axum` service, one route module per resource: runs, plans,
goals, metrics.
2. `POST /v1/runs` → spawn_run. Request `{tenant, workflow_ref, input,
goal_id?, dispatch_key?}`. Response `{run_id, status}`.
3. `GET /v1/runs/{id}`, `/episode` (T1.7), `/attempts/{step}/{n}/transcript`
(T1.6 `BlobRef` dereference), `/stream` (SSE).
4. `POST /v1/runs/{id}/rewind` (T2.3), `/cancel`, `/attempts/{step}/retry`
(T1.4).
5. `GET /v1/goals/{id}` (T10.1 `GoalView`).
6. `GET /v1/metrics`, `/v1/runs/{id}/metrics` (T8.2's existing list, scoped).
7. Auth/tenant resolution: reject any request without a resolvable
`TenantId` before touching the kernel — no implicit tenant.
8. Version the route set from commit one: additive only, document the rule
inline in the router module.
## Acceptance
- Full request/response cycle for spawn → poll → episode → transcript
against an embedded `redb` backend.
- Rewind, cancel, retry each produce the same log-level effect as calling
the underlying kernel function directly — no HTTP-layer divergence.
## Verify
**Harness:** embedded `redb`, stub model, `axum` test client
(`tower::ServiceExt::oneshot`).
**Integration test** — `tests/it_http_api_surface.rs`:
1. `POST /v1/runs`; assert 200 with `run_id`; assert kernel shows
`Scheduled`.
2. `GET /v1/runs/{id}` until terminal; assert status matches kernel state
exactly at each poll.
3. `GET /v1/runs/{id}/episode`; assert shape matches T1.7's `EpisodeView`,
includes failed attempts.
4. `GET /v1/runs/{id}/attempts/{step}/{n}/transcript`; assert returned bytes
match `BlobStore` content directly (T1.6).
5. `POST .../rewind`; assert new `BranchId` created, parent branch still
readable via all-branches.
6. `POST .../retry`; assert new `AttemptNo`, prior attempt record
unmodified.
7. `POST .../cancel` mid-run; assert `Cancelled` reached (matches T1.3's
cancel matrix).
8. Request with no resolvable tenant; assert rejected before any kernel call
— instrument kernel entrypoints, assert zero calls.
**Command:** `cargo test -p api http_api_surface`
**False pass:**
- Asserting HTTP 200 without checking underlying kernel state — a route that
accepts and silently drops a request passes this.
- Testing transcript retrieval only for a successful attempt — failed-attempt
transcripts are the debugging case that matters most.
## Traps
- Putting business logic (retry policy, plan validation) in the HTTP
handler instead of the kernel/port layer. The handler is a transport, not
a second brain.
- Resolving `BlobRef`s eagerly inside `/episode` "for convenience" — T4.3's
regression, one layer up.
---
Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §8.5, §10.1 ·
[../../INDEX.md](../../INDEX.md) (embedding contract, ~line 60-86) ·
[T1.7-episode-query-surface.md](T1.7-episode-query-surface.md) ·
[T1.6-prompt-and-output-blob-capture.md](T1.6-prompt-and-output-blob-capture.md)
+96
View File
@@ -0,0 +1,96 @@
# T10.5 — `TaskBoardFormat` plugin
| Field | Value |
|---|---|
| Phase | P10 — Orchestration |
| Size | M — 1 to 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |
## Goal
Front door (a): parse `tasks/*.md` — this board's own format — into a
`WorkflowDef` via the existing `WorkflowFormat` plugin trait. Zero kernel
change.
## Facts (inlined — no spec read needed)
- `WorkflowFormat` is already designed as a plugin point — this adds a third
format implementation, not a new mechanism. YAML and the Rust builder are
the other two; both canonicalize to the same IR (INDEX.md's customization
contract).
- Step order comes from **phase + row order in INDEX.md**, not
filename/id order. Id order is explicitly not execution order (INDEX.md's
own "Ordering" section).
- Each task file's Acceptance/Verify/Command sections map to a `StepDef`'s
verifier binding and completion criteria. Capability ids referenced
(verifier/judge/model) must already be registered (T3.9) or load fails
naming the id and file — same contract as YAML.
- Read-only relative to the board: this parses `tasks/*.md`, it does not
write `Status` back. That stays out of scope here.
## Steps
1. Implement `TaskBoardFormat: WorkflowFormat`, `load(path: &Path) ->
Result<WorkflowDef>` reading a directory of `T*.md` + `INDEX.md`.
2. Parse `INDEX.md`'s phase tables for execution order; parse each task
file's Goal/Acceptance/Verify sections into one `StepDef` per task.
3. Unresolved verifier/judge/model ids referenced in a task file fail at
load, naming the id and file — same contract as the other two front
doors.
4. Canonicalize through the same Blake3 hashing as the other two front doors
(T3.1) — same `WorkflowDef` IR.
5. Test: load this board's own `tasks/` directory, assert the canonicalized
IR is stable across two loads (determinism), and that step order matches
`INDEX.md`'s declared phase order, not directory-listing order.
## Acceptance
- `agent-rust/tasks/` loads into a `WorkflowDef` with step order matching
`INDEX.md`'s declared phase order.
- A task file referencing an unregistered verifier id fails at load, naming
the id and file — not at spawn.
## Verify
**Harness:** this repo's own `agent-rust/tasks/` directory as the test
fixture — self-hosting, no synthetic fixture needed.
**Integration test** — `tests/it_taskboard_format.rs`:
1. Load `agent-rust/tasks/`; assert no error.
2. Assert step count matches `INDEX.md`'s task count.
3. Assert step order matches `INDEX.md`'s phase-table order, not `ls`/
filename order — deliberately scramble the directory-read order in the
test harness to catch an implementation that trusts `readdir` order.
4. Load twice; assert canonicalized `WorkflowVersion` hash identical both
times.
5. Inject one task file referencing a made-up verifier id; assert load
fails, error names the id and file path.
**Command:** `cargo test -p formats taskboard_format`
**False pass:**
- Step 3 without deliberately scrambling directory-read order — most
filesystems return sorted-ish order by accident, hiding a bug that trusts
`readdir`.
- Skipping the unresolved-id failure test — a format that silently drops
unparseable tasks looks like it "works" until one goes missing at spawn.
## Traps
- Deriving step order from task-id string sort. `"T10.1" < "T2.1"` as
strings — `INDEX.md`'s declared order is the only authority, exactly the
bug the board's own "Ordering" section warns about at board scope.
- Making `TaskBoardFormat` also write `Status` back into task files. Turns a
read-only parser into a mutator with its own concurrency/conflict
problems, out of scope.
---
Background (not required to do this task):
[../INDEX.md](../INDEX.md) (customization contract ~line 19-58, ordering
~line 88-103) ·
[T3.1-workflowdef-ir-canonicalization.md](T3.1-workflowdef-ir-canonicalization.md) ·
[T3.9-capability-registry.md](T3.9-capability-registry.md)
@@ -0,0 +1,101 @@
# T10.6 — `plan-draft` workflow + `Suspended`-approval flow
| Field | Value |
|---|---|
| Phase | P10 — Orchestration |
| Size | M — 1 to 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |
## Goal
Front door (b): a free-text problem statement produces a draft
`WorkflowDef`, parked in `Suspended` for human review, resumed through the
identical loader front door (a) uses.
## Facts (inlined — no spec read needed)
- `Suspended` is an exact fit, verified against source: "a run awaiting
human approval or a webhook must not hold an executor slot" (T1.3), lease
released outright with no heartbeat while parked (T7.3).
- `plan-draft` is itself an ordinary poimen workflow: a `Model` step
producing a `WorkflowDef` draft, no new kernel machinery.
- The draft is validated against T3.9's capability registry **before** a
human ever sees it — an unresolved verifier/judge/model id in the draft
fails here, at draft time, not after approval.
- Approval feeds the (possibly human-edited) draft through the **same**
`WorkflowFormat`/builder + T3.1 canonicalization as door (a) — this is
what keeps both doors on one IR without new proof machinery.
## Steps
1. Define the built-in `plan-draft` workflow: one `Model` step (problem
statement → draft `WorkflowDef`), one validation step (registry check via
T3.9).
2. On draft ready, transition the `plan-draft` run to `Suspended`, releasing
its lease (T7.3).
3. `/v1/plans/{id}/approve` (T10.4 route): load the (possibly edited) draft,
run it through the exact loader door (a)'s `WorkflowFormat`/builder uses,
canonicalize (T3.1), resume — matching T1.3's resume-from-`Suspended`
semantics.
4. `/v1/plans/{id}/reject`: cancel from `Suspended` (legal transition per
T1.3's cancel-from-any-non-terminal rule).
5. Test: submit a problem statement, assert `Suspended`, edit the draft,
approve, assert the resulting `WorkflowVersion` hash matches
hand-authoring the same edited YAML directly.
## Acceptance
- A submitted problem statement produces a run parked in `Suspended` with no
held executor slot — lease released, per T7.3's own test shape.
- Approving a draft (edited or not) produces a `WorkflowVersion`
byte-identical to loading the same content by hand through door (a).
## Verify
**Harness:** stub model for the `plan-draft` step, embedded `redb`, T7.3's
lease-release assertion pattern reused.
**Integration test**`tests/it_plan_draft_approval.rs`:
1. `POST /v1/problems` with a free-text statement; assert the run reaches
`Suspended`.
2. Assert the lease is released and no heartbeat is emitted while
`Suspended` — reuse T7.3's assertion (its step 8).
3. Fetch the draft; edit one step; `POST /v1/plans/{id}/approve` with the
edited draft.
4. Assert the resulting `WorkflowVersion` hash equals hashing the same
edited content loaded directly via `WorkflowFormat`/builder — front-door
equivalence, extends T3.10.
5. `POST /v1/plans/{id2}/reject` on a second draft; assert it reaches
`Cancelled`, not stuck in `Suspended`.
6. Draft referencing an unregistered verifier id; assert `plan-draft`'s
validation step fails **before** ever reaching `Suspended` — the human
never sees an unloadable plan.
**Command:** `cargo test -p orchestration plan_draft_approval`
**False pass:**
- Skipping step 4's hash comparison and only checking "run resumed" — a
subtly different canonicalization from door (a) would still resume, just
diverge from what T3.1's guarantee promises.
- Not testing reject — an approval-only test suite misses that `Suspended`
must also support "no."
## Traps
- Having `/approve` construct a `WorkflowDef` directly in the HTTP handler
instead of routing through the same loader as door (a) — this is exactly
the two-IR-drift bug T3.10 exists to prevent, reintroduced one layer up.
- Skipping the registry validation step "since a human will catch it." The
whole point of validating before `Suspended` is that load errors are cheap
pre-human, expensive post-human.
---
Background (not required to do this task):
[../INDEX.md](../INDEX.md) (customization contract ~line 19-58) ·
[T1.3-run-executor.md](T1.3-run-executor.md) ·
[T7.3-leases-and-fencing.md](T7.3-leases-and-fencing.md) ·
[T3.10-front-door-equivalence.md](T3.10-front-door-equivalence.md)
+108
View File
@@ -0,0 +1,108 @@
# T10.7 — Kafka topics and inbox relay
| Field | Value |
|---|---|
| Phase | P10 — Orchestration |
| Size | L — over 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | T10.8 |
## Goal
`poimen.<tenant>.events` (outbound, existing outbox unchanged) and
`poimen.<tenant>.dispatch` (inbound, new) — an inbox relay symmetric to
T7.4's outbox, turning dispatch messages into spawn calls through the same
admission path, never a raw kernel write.
## Facts (inlined — no spec read needed)
- Outbound is unchanged: T7.4's relay ships goal-tagged `WorkEvent`s exactly
as it ships every other event. No new producer logic on this side.
- Inbound is genuinely new: the orchestrator (T10.8) publishes to
`.dispatch` after a `Reconciler` decision; this task builds only the relay
that drains it, not the orchestrator itself.
- The inbox relay's failure-domain rule mirrors T7.4's exactly: own
process/task, own retry, own supervision, not co-located with the
executor's lifecycle.
- Every dispatch message becomes a call to the **same** `POST /v1/runs`
(T10.4) the HTTP API exposes — the relay is an HTTP client to the local
API surface, not a kernel-internal spawn path. This is what makes the
"no bypass" guarantee mechanical, not a code-review convention.
- Redelivery on `.dispatch` is handled entirely by T10.3's `dispatch_key`
admission dedup — the relay passes `dispatch_key` through unchanged, no
separate idempotency layer.
- Partition key on both topics: `(TenantId, RunId)` per T7.5, consistent
with the outbox.
## Steps
1. Declare topics `poimen.<tenant>.events`, `poimen.<tenant>.dispatch`
partition key `(TenantId, RunId)` on both (T7.5).
2. No change to T7.4's relay beyond including goal-tagged records (already
covered since `GoalId` is a field on `WorkEvent`, T10.1).
3. Build the inbox relay: consume `.dispatch`, for each message call
`POST /v1/runs` with `{workflow_ref, input, goal_id, dispatch_key}`
verbatim.
4. Own retry/backoff on relay-to-API-call failure; never crash the relay on
a single bad message — dead-letter it, keep draining.
5. Own failure domain: separate process/task, no shared lifecycle with the
executor or the HTTP API service.
6. Emit inbox lag as a metric, alongside T7.4's existing outbox-lag metric.
7. Test with the broker down for a stretch, and with forced redelivery of
the same dispatch message.
## Acceptance
- A message published to `.dispatch` results in exactly one run, even under
forced redelivery.
- Broker down for the inbox side does not affect runs already in flight;
queued dispatch messages are processed once the broker returns.
## Verify
**Harness:** same broker-down/broker-up harness as T7.4, plus a second
consumer/producer pair for the `.dispatch` direction.
**Integration test**`tests/it_inbox_relay.rs`:
1. Publish one message to `.dispatch`; assert exactly one run created via
the same `POST /v1/runs` path — instrument the API, assert the call, not
a direct kernel spawn.
2. Force redelivery of the same message; assert still exactly one run
(T10.3's admission dedup catches it) — no relay-side special-casing
needed.
3. Take the broker down; publish a message; bring the broker up; assert the
run is eventually created, no message lost.
4. Publish a malformed message; assert it's dead-lettered, relay keeps
draining subsequent valid messages.
5. Assert the relay never calls kernel spawn directly — instrument the
kernel's internal admission entrypoint, assert all calls originate from
the HTTP layer, none from relay threads.
6. Assert inbox lag is emitted as a metric.
**Command:** `cargo test -p distribution inbox_relay -- --test-threads=1`
**False pass:**
- Step 2 without step 1's exact-call-path instrumentation: a relay that
spawns directly and also happens to dedupe correctly would still pass a
coarser "one run" check while violating the no-bypass rule.
- Skipping the malformed-message case: one bad message halting the whole
relay is a silent single point of failure discovered only in production.
## Traps
- Relay calling an in-process spawn function "for performance" instead of
going through `POST /v1/runs` — same trap T7.4 names for the outbound
side, mirrored here.
- Building custom dedup logic in the relay instead of trusting T10.3's
admission-layer dedup. Two idempotency layers can disagree under partial
failure.
---
Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §8.3, §9.2, §9.3, §13.3 ·
[T7.4-outbox-relay.md](T7.4-outbox-relay.md) ·
[T7.5-partition-keys-on-adapters.md](T7.5-partition-keys-on-adapters.md) ·
[T10.3-idempotent-run-admission.md](T10.3-idempotent-run-admission.md)
@@ -0,0 +1,108 @@
# T10.8 — Orchestrator reference service
| Field | Value |
|---|---|
| Phase | P10 — Orchestration |
| Size | M — 1 to 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |
## Goal
Reference process that consumes `.events`, invokes the registered
`Reconciler` for goal-tagged terminal transitions, and publishes decisions
to `.dispatch` — never spawns a run itself.
## Facts (inlined — no spec read needed)
- The orchestrator embeds poimen only via the HTTP API (T10.4) /
`poimen-sdk`, exactly like any other external embedder — no special
kernel access.
- Its entire job: consume `.events`, filter for goal-tagged
`Verified`/`Graded`/`Ungraded`, call `Reconciler` (T10.2), publish the
resulting `SpawnNext` decision to `.dispatch`. `Stop`/`Retry` decisions
are recorded but do not publish a dispatch message.
- The default `Reconciler` implementation may itself be another poimen run
(a `Model` step + a `Tool` step) — the orchestrator's job is identical
regardless of whether the bound `Reconciler` is LLM-backed or a
hand-written Rust impl.
- Never publishes a run directly, never calls `POST /v1/runs` itself — that
would reintroduce the bypass T10.7's inbox relay exists to prevent. Its
only write is to `.dispatch`.
## Steps
1. Consume `.events`, filter to records with a `GoalId` and a terminal run
state (`Verified`/`Graded`/`Ungraded`).
2. For each, build `ReconcileCtx` (T10.2) and call the `Reconciler` bound to
that goal.
3. On `SpawnNext`: publish `{workflow_ref, input, goal_id, dispatch_key}` to
`.dispatch``dispatch_key` derived from the decision id so redelivery
is naturally deduped downstream (T10.3).
4. On `Retry`/`Stop`: record the decision (`WorkEvent` provenance per
T10.2), publish nothing.
5. Own failure domain: separate process, own consumer group, own
retry/backoff on `Reconciler`-call failure — a `Reconciler` error does
not crash the orchestrator, it skips and retries on next poll or leaves
the goal stalled with a visible metric.
6. Emit reconcile-latency and goal-stall metrics.
7. Test the full loop against a stub `Reconciler`: events in, dispatch out,
verified end to end including a two-hop goal chain (run A → reconcile →
run B → reconcile → stop).
## Acceptance
- A two-run goal chain (A completes → reconcile → B spawns → B completes →
reconcile → `Stop`) completes end to end through real Kafka topics, with
the orchestrator never calling spawn directly.
- `Reconciler` failure on one goal does not block reconciliation of other
goals.
## Verify
**Harness:** real Kafka (or the board's chosen fault-injection substrate),
stub `Reconciler` with scripted decisions, T10.7's inbox relay running
alongside.
**Integration test**`tests/it_orchestrator_e2e.rs`:
1. Spawn run A with `goal_id=G`, no `dispatch_key`. Drive to
`Verified{pass}`.
2. Assert orchestrator calls `Reconciler` exactly once for A; stub returns
`SpawnNext` for run B.
3. Assert a message lands on `.dispatch`, and via T10.7's relay, run B is
spawned with `goal_id=G`.
4. Drive run B to `Graded`. Assert `Reconciler` called again; stub returns
`Stop`.
5. Assert no further `.dispatch` message is published; goal G's chain shows
exactly `[A, B]` via T10.1's `GoalView`.
6. Instrument `POST /v1/runs`; assert zero calls originate from the
orchestrator process — only from the inbox relay.
7. Second goal G2 with a `Reconciler` that always errors; assert G2 stalls
(visible via goal-stall metric) while G's chain, run concurrently,
proceeds unaffected.
**Command:** `cargo test -p orchestration orchestrator_e2e -- --test-threads=1`
**False pass:**
- Step 6 omitted: an orchestrator that takes a shortcut and spawns directly
under test-only conditions would still pass steps 1-5.
- Testing only a single-hop chain (A → `Stop`) — the two-hop case is what
proves goal-chain provenance actually threads through `GoalView`
correctly across a full reconcile-dispatch-relay-spawn cycle.
## Traps
- Orchestrator holding an admission-layer shortcut "to save a network hop."
Collapses the exact boundary T10.7 was built to enforce.
- Coupling `Reconciler` failure handling to a single global retry policy.
One broken goal's `Reconciler` should not starve reconciliation for every
other goal in the tenant.
---
Background (not required to do this task):
[T10.1-goalid-and-goal-scoped-query.md](T10.1-goalid-and-goal-scoped-query.md) ·
[T10.2-reconciler-port.md](T10.2-reconciler-port.md) ·
[T10.7-kafka-topics-and-inbox-relay.md](T10.7-kafka-topics-and-inbox-relay.md)
@@ -0,0 +1,88 @@
# T10.9 — CLI: runs list/episode/transcript
| Field | Value |
|---|---|
| Phase | P10 — Orchestration |
| Size | S — under 1 day |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |
## Goal
Three thin CLI wrappers over the HTTP API — `poimen runs list --goal <id>`,
`poimen runs episode <run_id>`, `poimen runs transcript <run_id> <step>
<attempt>` — the whole troubleshooting UX, no dashboard.
## Facts (inlined — no spec read needed)
- Thin HTTP clients, not a second implementation of the query logic already
in T10.1/T1.7/T1.6 — the CLI calls the same routes T10.4 exposes.
- Rides the eventual shared `poimen` binary (T9.3); workable as a standalone
binary if T9.3 hasn't landed yet — soft dependency, not a hard block.
- `episode` output must include failed attempts by default — T1.7's own
rule. A CLI that hides them defeats the purpose of the command.
- `transcript` dereferences the `BlobRef` server-side, via the HTTP route
(T10.4) — the CLI never talks to `BlobStore` directly.
## Steps
1. `poimen runs list [--goal <id>] [--status <state>]``GET /v1/runs`,
table output (`run_id`, `status`, `goal_id`, age).
2. `poimen runs episode <run_id>``GET /v1/runs/{id}/episode`,
human-readable rendering of `EpisodeView` including failed attempts,
clearly marked.
3. `poimen runs transcript <run_id> <step> <attempt>``GET .../transcript`,
prints prompt/output text, dereferenced server-side.
4. Exit codes: non-zero on API error, distinct code for "run not found" vs
"transport error" — operator scripting depends on this distinction.
5. Test against a live embedded API instance (T10.4's test harness) with a
run that has a retry and a rewind.
## Acceptance
- `episode` on a run with two failed attempts and one success shows all
three, visually distinguishable.
- `transcript` returns the same text content as reading the `BlobRef`
directly through `BlobStore` in a test assertion.
## Verify
**Harness:** T10.4's `axum` test instance, one run with a retry (two
attempts, one failed) and a rewind.
**Integration test**`tests/it_cli_runs_ux.rs`:
1. `runs list --goal G`; assert output contains exactly the runs tagged G,
none untagged.
2. `runs episode <id>`; assert both the failed and successful attempt
appear — count them explicitly, not just non-empty output.
3. `runs transcript <id> <step> 1` (the failed attempt) and `... 2` (the
retry); assert distinct content for each, matching their respective
`BlobRef`s byte-for-byte.
4. Run-not-found case; assert a distinct exit code from a transport-down
case.
**Command:** `cargo test -p cli runs_ux`
**False pass:**
- Step 2 checking only that output is non-empty — a CLI silently filtering
to the winning attempt (T1.7's own named trap) passes a non-empty check
while hiding exactly the debugging signal requested.
## Traps
- Filtering episode output to "just the successful attempt for clarity" —
T1.7's trap restated at the CLI layer; the failed attempts are the reason
this command exists.
- Fetching transcripts client-side by resolving `BlobRef` directly instead
of through the HTTP route. Bypasses the tenant auth checks the API
enforces.
---
Background (not required to do this task):
[T10.4-http-api-surface.md](T10.4-http-api-surface.md) ·
[T1.7-episode-query-surface.md](T1.7-episode-query-surface.md) ·
[T1.6-prompt-and-output-blob-capture.md](T1.6-prompt-and-output-blob-capture.md) ·
[T8.2-metrics.md](T8.2-metrics.md)