diff --git a/tasks/INDEX.md b/tasks/INDEX.md index da7962a..05fc113 100644 --- a/tasks/INDEX.md +++ b/tasks/INDEX.md @@ -211,7 +211,8 @@ Legend: โฌœ not started ยท ๐ŸŸก in progress ยท โœ… done ยท โ›” blocked | 8 | Learning loop | T6.x | 8 | 0 | 0 | 8 | โฌœ T6.8 | โ€” | โ€” | | 9 | Distribution | T7.x | 8 | 0 | 0 | 8 | โฌœ T7.8 | โ€” | โ€” | | 10 | Operability | T8.x | 7 | 0 | 0 | 7 | โฌœ T8.7 | โ€” | โ€” | -| | **Total** | | **80** | **3** | **0** | **77** | 0/10 green | **313.0k** | **โ€”** | +| 11 | Orchestration | T10.x | 10 | 0 | 0 | 10 | โฌœ T10.10 | โ€” | โ€” | +| | **Total** | | **90** | **3** | **0** | **87** | 0/11 green | **313.0k** | **โ€”** | **What changed in this revision.** The public surface moved ahead of the durability hard parts, and an embedding phase was added after verification. @@ -361,3 +362,32 @@ whose architecture already expects capabilities to arrive as swappable providers | [T8.5](T8.5-embedded-mode-smoke.md) | Embedded-mode smoke | S | parallel-ok | โฌœ | | [T8.6](T8.6-capacity-admission-control.md) | Capacity admission control | M | โ€” | โฌœ | | [T8.7](T8.7-p8-composition-gate.md) | **Operability composition gate** | L | gate | โฌœ | + +## 11 โ€” Orchestration ยท T10.x + +Fills ยง17's deferred line โ€” multi-agent orchestration, revisited now that +cross-agent coordination is real. A `Reconciler` port picks up a run at its +existing `Verified`/`Graded`/`Ungraded` resting states, decides what runs +next toward a `GoalId`-scoped objective, and dispatches through Kafka via +the same outbox/inbox discipline T7.4 established โ€” never a direct spawn +call. An HTTP surface binds one method table shared with T9.2's eventual +stdio transport. Two front doors, one IR: `tasks/*.md` parsed directly +(door a), or a free-text problem statement drafted and parked in +`Suspended` for human approval before it runs (door b). + +Depends on T1.x, T2.1/T2.3, T3.x, T7.3/T7.4, T9.1 landing first; not blocked +on T9.2โ€“T9.6 (the stdio binding develops in parallel against the same +method table T10.4 authors). + +| Task | Title | Size | Flags | Status | +|---|---|---|---|---| +| [T10.1](T10.1-goalid-and-goal-scoped-query.md) | `GoalId` and goal-scoped query | S | โ€” | โฌœ | +| [T10.2](T10.2-reconciler-port.md) | `Reconciler` port | M | โ€” | โฌœ | +| [T10.3](T10.3-idempotent-run-admission.md) | Idempotent run admission | S | โ€” | โฌœ | +| [T10.4](T10.4-http-api-surface.md) | HTTP API surface | L | โ€” | โฌœ | +| [T10.5](T10.5-taskboardformat-plugin.md) | `TaskBoardFormat` plugin | M | โ€” | โฌœ | +| [T10.6](T10.6-plan-draft-and-suspended-approval.md) | `plan-draft` workflow + `Suspended`-approval flow | M | โ€” | โฌœ | +| [T10.7](T10.7-kafka-topics-and-inbox-relay.md) | Kafka topics and inbox relay | L | โ€” | โฌœ | +| [T10.8](T10.8-orchestrator-reference-service.md) | Orchestrator reference service | M | โ€” | โฌœ | +| [T10.9](T10.9-cli-transcript-and-metrics-ux.md) | CLI: runs list/episode/transcript | S | โ€” | โฌœ | +| [T10.10](T10.10-p10-composition-gate.md) | **Orchestration composition gate** | L | gate | โฌœ | diff --git a/tasks/T10.1-goalid-and-goal-scoped-query.md b/tasks/T10.1-goalid-and-goal-scoped-query.md new file mode 100644 index 0000000..cab3ace --- /dev/null +++ b/tasks/T10.1-goalid-and-goal-scoped-query.md @@ -0,0 +1,82 @@ +# T10.1 โ€” `GoalId` and goal-scoped query + +| Field | Value | +|---|---| +| Phase | P10 โ€” Orchestration | +| Size | S โ€” under 1 day | +| Status | Not started | +| Flags | โ€” | +| Spec | inlined below | +| Blocks | T10.2, T10.3, T10.4, T10.8 | + +## Goal + +New identity `GoalId(Ulid)` grouping runs that share an objective, optional at +spawn. Extend `T1.7`'s query surface to read across `RunId`s sharing one +`GoalId`. + +## Facts (inlined โ€” no spec read needed) + +- `GoalId` is tenant-scoped like `RunId`, **not** `TaskId`. `TaskId` is already + the grading comparison-group key (ยง11.5) โ€” reusing it corrupts + group-epoch/quorum semantics the learning loop depends on. +- Attaching `goal_id` at spawn is **optional**. A run with no `GoalId` behaves + exactly as today โ€” this is additive, not a required field. +- `GoalView` is a new, separate query. It does not replace per-run + `EpisodeView`; it lists member `RunId`s and references their existing + metadata, no duplication. +- Same parity rule as T1.7: `GoalView` from materialized state must equal + `GoalView` from a cold re-fold (T0.8's property applied here too). + +## Steps + +1. Declare `GoalId(Ulid)` in the `ids` crate, alongside `RunId`. +2. Add optional `goal_id: Option` to run-spawn input and the run + record. +3. Index runs by `(TenantId, GoalId)` in materialized state. +4. Define `GoalView { goal: GoalId, runs: Vec }`, `RunSummary` + referencing existing run metadata โ€” do not duplicate `EpisodeView` fields. +5. Implement the query against materialized state and against cold re-fold. +6. Test: two runs sharing a `GoalId`, one plain run with none; assert + `GoalView` returns exactly the two, in spawn order. + +## Acceptance + +- Two runs sharing a `GoalId` are both returned by `GoalView`; a run with no + `GoalId` never appears in any `GoalView`. +- `GoalView` from materialized state equals `GoalView` from cold re-fold. + +## Verify + +**Harness:** two runs sharing a `GoalId`, one run with no `GoalId`, one +branch/rewind case reused from `T1.7`'s harness. + +**Integration test** โ€” `tests/it_goal_view_equivalence.rs`: +1. Spawn run A and run B both with `goal_id=G`; spawn run C with none. +2. Query `GoalView(G)` from materialized state โ†’ `view_a`. +3. Drop state tables; query again via cold re-fold โ†’ `view_b`. +4. Assert `serialize(view_a) == serialize(view_b)`. +5. Assert `view_a` contains exactly `{A, B}`, never `C`. +6. Assert order is spawn order, stable. + +**Command:** `cargo test -p query goal_view_equivalence` + +**False pass:** +- Testing only one run per goal โ€” a grouping bug is invisible with a single + member. +- Skipping the "run without `goal_id`" exclusion check. A filter that + defaults to "all runs" passes silently without it. + +## Traps + +- Deriving `GoalId` from `TaskId` "since they're both groupings." Different + lifecycle, different owner โ€” grading vs orchestration. +- Making `goal_id` required at spawn. Breaks every existing plain-workflow + spawn call. + +--- + +Background (not required to do this task): +[rust-agentic-sys.md](../rust-agentic-sys.md) ยง11.5 ยท +[T1.7-episode-query-surface.md](T1.7-episode-query-surface.md) ยท +[T0.8-fold-and-re-derive.md](T0.8-fold-and-re-derive.md) diff --git a/tasks/T10.10-p10-composition-gate.md b/tasks/T10.10-p10-composition-gate.md new file mode 100644 index 0000000..40ad444 --- /dev/null +++ b/tasks/T10.10-p10-composition-gate.md @@ -0,0 +1,88 @@ +# T10.10 โ€” Orchestration composition gate + +| Field | Value | +|---|---| +| Phase | P10 โ€” Orchestration | +| Size | L โ€” over 3 days | +| Status | Not started | +| Flags | gate | +| Spec | inlined below | +| Blocks | โ€” | + +## Goal + +Prove the properties no single T10.x task owns: a full goal chain survives a +real Kafka outage, dispatch is idempotent under forced redelivery, +`Suspended`-approval produces byte-identical IR to hand-authored YAML, and +the HTTP method table stands as a published artifact. + +## Facts (inlined โ€” no spec read needed) + +- Mirrors T7.4's fault-injection shape โ€” broker down for the whole test โ€” + extended across the full events โ†’ reconcile โ†’ dispatch โ†’ inbox-relay โ†’ + spawn loop, not just outbox delivery. +- Front-door equivalence for door (b) extends T3.10's existing hash-equality + gate โ€” same discipline, new door. +- Gate script follows board convention: one script, diffed against a + committed expected output, empty diff is the only pass. + +## Steps + +1. Compose a fixture: one goal, a 2-run chain (A โ†’ reconcile โ†’ B โ†’ reconcile + โ†’ `Stop`), driven through real HTTP calls and real Kafka topics. +2. Kill the broker mid-chain โ€” after A completes, before the dispatch + message for B is consumed; assert the chain still completes once the + broker returns, with B eventually spawned. +3. Force redelivery of the `.dispatch` message for B; assert exactly one + run B exists. +4. Submit a problem statement through door (b), edit the draft, approve; + separately hand-author the identical edited content as YAML and load it + through door (a); assert both `WorkflowVersion` hashes match. +5. Snapshot the HTTP method table (paths + request/response shapes) to a + committed fixture; assert no path removed and no field removed relative + to the fixture โ€” additions are fine, removals fail the gate. +6. Assert a goal-stall metric fires for a deliberately broken `Reconciler` + on one goal while a healthy goal in the same tenant proceeds unaffected + (T10.8's concurrency claim, at gate scope). + +## Acceptance + +- All six checks above pass with an empty diff against + `verify/expected/T10.10.txt`. + +## Verify + +**Harness:** full stack โ€” embedded `redb`, real Kafka (or the board's +chosen fault-injection substrate), `axum` API instance, stub `Reconciler` +with scripted multi-step decisions. + +**Integration test** โ€” `tests/it_p10_composition_gate.rs`, one numbered +assertion per Steps item above (`a1`..`a6`), reported by name per the +board's script convention. + +**Command:** `verify/T10.10.sh`, diffed against `verify/expected/T10.10.txt`. + +**False pass:** +- Running the chain with the broker never actually taken down โ€” the + "survives an outage" claim needs the outage to really happen mid-chain, + not just be configured as theoretically tolerable. +- Skipping the method-table snapshot diff โ€” a route or field silently + removed here breaks every embedder bound to the table (HTTP today, stdio + later) without any single T10.x task's own tests catching it, since none + of them own the whole table. + +## Traps + +- Treating this gate as "run all T10.x tests together." It must assert + properties none of them individually assert, per the board's own gate + discipline (INDEX.md, "Verification practice," point 5). +- Committing the method-table fixture as freeform prose instead of a + diffable, structured snapshot. An unreviewable diff defeats the point of + having a fixture at all. + +--- + +Background (not required to do this task): +[../INDEX.md](../INDEX.md) (verification practice) ยท +[T7.4-outbox-relay.md](T7.4-outbox-relay.md) ยท +[T3.10-front-door-equivalence.md](T3.10-front-door-equivalence.md) diff --git a/tasks/T10.2-reconciler-port.md b/tasks/T10.2-reconciler-port.md new file mode 100644 index 0000000..f10dcb5 --- /dev/null +++ b/tasks/T10.2-reconciler-port.md @@ -0,0 +1,112 @@ +# T10.2 โ€” `Reconciler` port + +| Field | Value | +|---|---| +| Phase | P10 โ€” Orchestration | +| Size | M โ€” 1 to 3 days | +| Status | Not started | +| Flags | โ€” | +| Spec | inlined below | +| Blocks | T10.6, T10.8 | + +## Goal + +New port answering "how far toward the goal, what next" from a run's terminal +resting state โ€” distinct from `Verifier` (fail-closed pass/fail) and `Judge` +(relative-only comparison). + +## Facts (inlined โ€” no spec read needed) + +- `Verifier::verify` is fail-closed pass/fail (rust-agentic-sys.md ยง10). + `Judge::compare` is deliberately relative-only, "cannot return an absolute + score" (rust-agentic-sys.md ยง11.1). "How far toward goal, what next" is a + third question โ€” forcing it through either port blurs a firewall built on + purpose. `Reconciler` is new, sized like `Verifier` (one method, S/M). +- Fires **only** off `Verified{pass|fail}`, `Graded`, `Ungraded` โ€” states + `T1.3` already designed to rest in, never resolved inline. +- Reconciler failure degrades the record, never the run โ€” same principle as + ยง1: "the framework cannot break the agent it runs." A `Reconciler` error or + timeout leaves the run at its terminal state; the goal simply does not + advance until retried. +- The decision does **not** spawn anything itself. Spawning happens only + through the admission path (T10.3) via the inbox relay (T10.7) โ€” never a + direct call from the `Reconciler` or its caller. +- One new `#[non_exhaustive]` `WorkEvent` variant for provenance: + `SpawnedFromReconciliation{goal_id, parent_run, decision_ref}`. Added, not + replacing any existing variant. + +## Steps + +1. Define `ReconcileCtx` โ€” `GoalId`, `GoalView` (T10.1), the just-terminal + run's `EpisodeView`, `WorkflowDef`. +2. Define `ReconcileDecision` โ€” `SpawnNext{workflow_ref, input, dispatch_key}`, + `Retry{step}`, `Stop{reason}`. +3. Define the trait: + ```rust + #[async_trait] + pub trait Reconciler: Send + Sync { + fn id(&self) -> ReconcilerId; + async fn reconcile(&self, cx: &ReconcileCtx) -> Result; + } + ``` +4. Wire the trigger: on a run reaching `Verified`/`Graded`/`Ungraded` with a + `GoalId` set, call the `Reconciler` bound to that goal. +5. On `Reconciler` error or timeout: log, leave the run at its terminal state, + do not auto-retry โ€” matches `Verifier`'s explicit-only-retry discipline. +6. Add `WorkEvent::SpawnedFromReconciliation`, written on the **next** + spawned run's first log record, referencing `parent_run` and the decision. +7. Test with a stub `Reconciler` against all three trigger states. + +## Acceptance + +- A run reaching `Verified{pass}` with a `GoalId` triggers exactly one + reconcile call. +- The decision is durably recorded and traceable from the next spawned run + back to the parent run and goal. +- A `Reconciler` error leaves the run at its terminal state and does not + crash the executor. + +## Verify + +**Harness:** stub `Reconciler` returning each decision variant in turn; one +goal with a 2-run chain. + +**Integration test** โ€” `tests/it_reconciler_trigger.rs`: +1. Run A (goal G) reaches `Verified{pass}`; assert `reconcile()` called + exactly once with the correct `ReconcileCtx`. +2. Stub returns `SpawnNext`; assert the decision is persisted, referencing + run A. +3. Assert the next spawned run's first log record carries + `SpawnedFromReconciliation{goal_id: G, parent_run: A, decision_ref}`. +4. Repeat for `Graded` and `Ungraded` trigger states. +5. Stub `Reconciler` panics or times out; assert run A stays at its terminal + state, no crash, no infinite retry. +6. Run with no `GoalId` reaches `Verified{pass}`; assert `reconcile()` is + never called. + +**Command:** `cargo test -p kernel reconciler_trigger` + +**False pass:** +- Asserting reconcile "ran" via a boolean flag instead of asserting the + `WorkEvent` provenance record exists โ€” passes even if a crash lost the + decision before persistence. +- Testing only `Verified{pass}`, skipping `Graded`/`Ungraded` โ€” those are the + states most likely miswired, being newer additions to the FSM. + +## Traps + +- `Reconciler` spawning the next run directly. Violates "never bypass the + admission path" โ€” see T10.3/T10.7. +- Overloading `Judge::compare` to also answer "what next." Collapses a + boundary `Judge`'s signature deliberately enforces (rust-agentic-sys.md + ยง11.1, "deliberately cannot return an absolute score"). +- Auto-retrying a failed `Reconciler` call with no cap โ€” a broken impl then + spins forever against every terminal run in the tenant. + +--- + +Background (not required to do this task): +[rust-agentic-sys.md](../rust-agentic-sys.md) ยง1, ยง10, ยง11.1, ยง17 ยท +[T1.3-run-executor.md](T1.3-run-executor.md) ยท +[T4.1-verifier-port.md](T4.1-verifier-port.md) ยท +[T10.1-goalid-and-goal-scoped-query.md](T10.1-goalid-and-goal-scoped-query.md) diff --git a/tasks/T10.3-idempotent-run-admission.md b/tasks/T10.3-idempotent-run-admission.md new file mode 100644 index 0000000..068638a --- /dev/null +++ b/tasks/T10.3-idempotent-run-admission.md @@ -0,0 +1,78 @@ +# T10.3 โ€” Idempotent run admission + +| Field | Value | +|---|---| +| Phase | P10 โ€” Orchestration | +| Size | S โ€” under 1 day | +| Status | Not started | +| Flags | โ€” | +| Spec | inlined below | +| Blocks | T10.4, T10.7 | + +## Goal + +`dispatch_key` on spawn requests: a redelivered spawn request (from Kafka +at-least-once delivery, T10.7) becomes a no-op insert instead of a duplicate +run. + +## Facts (inlined โ€” no spec read needed) + +- Same principle as the outbox's `(BranchKey, Lsn)` fold-key idempotency + (T7.4) โ€” exactly-once is achieved at the admission boundary, not in + transport. +- `dispatch_key` is caller-supplied, opaque, unique per intended spawn โ€” e.g. + a `Reconciler` decision id. +- Admission with no `dispatch_key` behaves exactly as today. This is + additive, not a required field. + +## Steps + +1. Add optional `dispatch_key: Option` to spawn-run input. +2. Add a unique index on `(TenantId, dispatch_key)` where `dispatch_key` is + set. +3. On spawn: if `dispatch_key` is set and already seen, return the existing + `RunId` instead of creating a new run โ€” no error, no duplicate. +4. Test redelivery: same `dispatch_key` submitted twice, concurrently and + sequentially. + +## Acceptance + +- Two spawn calls with the same `dispatch_key` produce exactly one run; the + second call returns the first run's `RunId`. +- Spawn calls with no `dispatch_key` are unaffected. + +## Verify + +**Harness:** spawn endpoint (T10.4) or direct kernel call, two concurrent +callers. + +**Integration test** โ€” `tests/it_idempotent_admission.rs`: +1. Spawn with `dispatch_key=K`; assert one run created, `RunId=R`. +2. Spawn again with `dispatch_key=K`; assert no new run created, returned + `RunId == R`. +3. Fire 20 concurrent spawn calls with the same `dispatch_key`; assert + exactly one run exists after all resolve. +4. Spawn with no `dispatch_key` twice; assert two distinct runs (unaffected + baseline). + +**Command:** `cargo test -p kernel idempotent_admission -- --test-threads=1` + +**False pass:** +- Testing only sequential redelivery. Concurrent redelivery is the actual + failure mode under Kafka rebalance/retry and needs the unique-index race + caught, not a check-then-insert race. + +## Traps + +- Making `dispatch_key` required. Breaks every direct/manual spawn call that + isn't part of a goal chain. +- Deduping by a hash of `(workflow_ref, input)` instead of the caller-supplied + key. Two legitimately identical requests would then silently collapse into + one run. + +--- + +Background (not required to do this task): +[rust-agentic-sys.md](../rust-agentic-sys.md) ยง8.3, ยง9.3 ยท +[T7.4-outbox-relay.md](T7.4-outbox-relay.md) ยท +[T3.6-version-pinning-at-spawn.md](T3.6-version-pinning-at-spawn.md) diff --git a/tasks/T10.4-http-api-surface.md b/tasks/T10.4-http-api-surface.md new file mode 100644 index 0000000..d980348 --- /dev/null +++ b/tasks/T10.4-http-api-surface.md @@ -0,0 +1,101 @@ +# T10.4 โ€” HTTP API surface + +| Field | Value | +|---|---| +| Phase | P10 โ€” Orchestration | +| Size | L โ€” over 3 days | +| Status | Not started | +| Flags | โ€” | +| Spec | inlined below | +| Blocks | T10.6, T10.7, T10.9 | + +## Goal + +`axum` HTTP service binding one method table โ€” spawn / status / stream / +episode / transcript / rewind / cancel / retry / goal / metrics โ€” to +`poimen-sdk` ports. One method set: HTTP today, the same table T9.2's stdio +JSON-RPC binds to later. + +## Facts (inlined โ€” no spec read needed) + +- **The protocol is a published artifact** โ€” INDEX.md's own rule for the + sidecar applies equally here: methods added, never repurposed; fields + added, never removed, from the first commit. +- HTTP is a transport binding onto `poimen-sdk` (T9.1), not a bespoke access + layer with its own business logic. Handlers translate, they do not decide. +- `/v1/problems` and `/v1/plans/{id}/approve|reject` route to T10.6's + workflow, not implemented here โ€” this task defines the routes and + request/response contracts; T10.6 fills in plan-draft semantics. +- `/v1/runs/{id}/stream` tails the same committed log the outbox relay reads + โ€” SSE, not a second read path. + +## Steps + +1. Stand up the `axum` service, one route module per resource: runs, plans, + goals, metrics. +2. `POST /v1/runs` โ†’ spawn_run. Request `{tenant, workflow_ref, input, + goal_id?, dispatch_key?}`. Response `{run_id, status}`. +3. `GET /v1/runs/{id}`, `/episode` (T1.7), `/attempts/{step}/{n}/transcript` + (T1.6 `BlobRef` dereference), `/stream` (SSE). +4. `POST /v1/runs/{id}/rewind` (T2.3), `/cancel`, `/attempts/{step}/retry` + (T1.4). +5. `GET /v1/goals/{id}` (T10.1 `GoalView`). +6. `GET /v1/metrics`, `/v1/runs/{id}/metrics` (T8.2's existing list, scoped). +7. Auth/tenant resolution: reject any request without a resolvable + `TenantId` before touching the kernel โ€” no implicit tenant. +8. Version the route set from commit one: additive only, document the rule + inline in the router module. + +## Acceptance + +- Full request/response cycle for spawn โ†’ poll โ†’ episode โ†’ transcript + against an embedded `redb` backend. +- Rewind, cancel, retry each produce the same log-level effect as calling + the underlying kernel function directly โ€” no HTTP-layer divergence. + +## Verify + +**Harness:** embedded `redb`, stub model, `axum` test client +(`tower::ServiceExt::oneshot`). + +**Integration test** โ€” `tests/it_http_api_surface.rs`: +1. `POST /v1/runs`; assert 200 with `run_id`; assert kernel shows + `Scheduled`. +2. `GET /v1/runs/{id}` until terminal; assert status matches kernel state + exactly at each poll. +3. `GET /v1/runs/{id}/episode`; assert shape matches T1.7's `EpisodeView`, + includes failed attempts. +4. `GET /v1/runs/{id}/attempts/{step}/{n}/transcript`; assert returned bytes + match `BlobStore` content directly (T1.6). +5. `POST .../rewind`; assert new `BranchId` created, parent branch still + readable via all-branches. +6. `POST .../retry`; assert new `AttemptNo`, prior attempt record + unmodified. +7. `POST .../cancel` mid-run; assert `Cancelled` reached (matches T1.3's + cancel matrix). +8. Request with no resolvable tenant; assert rejected before any kernel call + โ€” instrument kernel entrypoints, assert zero calls. + +**Command:** `cargo test -p api http_api_surface` + +**False pass:** +- Asserting HTTP 200 without checking underlying kernel state โ€” a route that + accepts and silently drops a request passes this. +- Testing transcript retrieval only for a successful attempt โ€” failed-attempt + transcripts are the debugging case that matters most. + +## Traps + +- Putting business logic (retry policy, plan validation) in the HTTP + handler instead of the kernel/port layer. The handler is a transport, not + a second brain. +- Resolving `BlobRef`s eagerly inside `/episode` "for convenience" โ€” T4.3's + regression, one layer up. + +--- + +Background (not required to do this task): +[rust-agentic-sys.md](../../../rust-agentic-sys.md) ยง8.5, ยง10.1 ยท +[../../INDEX.md](../../INDEX.md) (embedding contract, ~line 60-86) ยท +[T1.7-episode-query-surface.md](T1.7-episode-query-surface.md) ยท +[T1.6-prompt-and-output-blob-capture.md](T1.6-prompt-and-output-blob-capture.md) diff --git a/tasks/T10.5-taskboardformat-plugin.md b/tasks/T10.5-taskboardformat-plugin.md new file mode 100644 index 0000000..797695e --- /dev/null +++ b/tasks/T10.5-taskboardformat-plugin.md @@ -0,0 +1,96 @@ +# T10.5 โ€” `TaskBoardFormat` plugin + +| Field | Value | +|---|---| +| Phase | P10 โ€” Orchestration | +| Size | M โ€” 1 to 3 days | +| Status | Not started | +| Flags | โ€” | +| Spec | inlined below | +| Blocks | โ€” | + +## Goal + +Front door (a): parse `tasks/*.md` โ€” this board's own format โ€” into a +`WorkflowDef` via the existing `WorkflowFormat` plugin trait. Zero kernel +change. + +## Facts (inlined โ€” no spec read needed) + +- `WorkflowFormat` is already designed as a plugin point โ€” this adds a third + format implementation, not a new mechanism. YAML and the Rust builder are + the other two; both canonicalize to the same IR (INDEX.md's customization + contract). +- Step order comes from **phase + row order in INDEX.md**, not + filename/id order. Id order is explicitly not execution order (INDEX.md's + own "Ordering" section). +- Each task file's Acceptance/Verify/Command sections map to a `StepDef`'s + verifier binding and completion criteria. Capability ids referenced + (verifier/judge/model) must already be registered (T3.9) or load fails + naming the id and file โ€” same contract as YAML. +- Read-only relative to the board: this parses `tasks/*.md`, it does not + write `Status` back. That stays out of scope here. + +## Steps + +1. Implement `TaskBoardFormat: WorkflowFormat`, `load(path: &Path) -> + Result` reading a directory of `T*.md` + `INDEX.md`. +2. Parse `INDEX.md`'s phase tables for execution order; parse each task + file's Goal/Acceptance/Verify sections into one `StepDef` per task. +3. Unresolved verifier/judge/model ids referenced in a task file fail at + load, naming the id and file โ€” same contract as the other two front + doors. +4. Canonicalize through the same Blake3 hashing as the other two front doors + (T3.1) โ€” same `WorkflowDef` IR. +5. Test: load this board's own `tasks/` directory, assert the canonicalized + IR is stable across two loads (determinism), and that step order matches + `INDEX.md`'s declared phase order, not directory-listing order. + +## Acceptance + +- `agent-rust/tasks/` loads into a `WorkflowDef` with step order matching + `INDEX.md`'s declared phase order. +- A task file referencing an unregistered verifier id fails at load, naming + the id and file โ€” not at spawn. + +## Verify + +**Harness:** this repo's own `agent-rust/tasks/` directory as the test +fixture โ€” self-hosting, no synthetic fixture needed. + +**Integration test** โ€” `tests/it_taskboard_format.rs`: +1. Load `agent-rust/tasks/`; assert no error. +2. Assert step count matches `INDEX.md`'s task count. +3. Assert step order matches `INDEX.md`'s phase-table order, not `ls`/ + filename order โ€” deliberately scramble the directory-read order in the + test harness to catch an implementation that trusts `readdir` order. +4. Load twice; assert canonicalized `WorkflowVersion` hash identical both + times. +5. Inject one task file referencing a made-up verifier id; assert load + fails, error names the id and file path. + +**Command:** `cargo test -p formats taskboard_format` + +**False pass:** +- Step 3 without deliberately scrambling directory-read order โ€” most + filesystems return sorted-ish order by accident, hiding a bug that trusts + `readdir`. +- Skipping the unresolved-id failure test โ€” a format that silently drops + unparseable tasks looks like it "works" until one goes missing at spawn. + +## Traps + +- Deriving step order from task-id string sort. `"T10.1" < "T2.1"` as + strings โ€” `INDEX.md`'s declared order is the only authority, exactly the + bug the board's own "Ordering" section warns about at board scope. +- Making `TaskBoardFormat` also write `Status` back into task files. Turns a + read-only parser into a mutator with its own concurrency/conflict + problems, out of scope. + +--- + +Background (not required to do this task): +[../INDEX.md](../INDEX.md) (customization contract ~line 19-58, ordering +~line 88-103) ยท +[T3.1-workflowdef-ir-canonicalization.md](T3.1-workflowdef-ir-canonicalization.md) ยท +[T3.9-capability-registry.md](T3.9-capability-registry.md) diff --git a/tasks/T10.6-plan-draft-and-suspended-approval.md b/tasks/T10.6-plan-draft-and-suspended-approval.md new file mode 100644 index 0000000..e32934d --- /dev/null +++ b/tasks/T10.6-plan-draft-and-suspended-approval.md @@ -0,0 +1,101 @@ +# T10.6 โ€” `plan-draft` workflow + `Suspended`-approval flow + +| Field | Value | +|---|---| +| Phase | P10 โ€” Orchestration | +| Size | M โ€” 1 to 3 days | +| Status | Not started | +| Flags | โ€” | +| Spec | inlined below | +| Blocks | โ€” | + +## Goal + +Front door (b): a free-text problem statement produces a draft +`WorkflowDef`, parked in `Suspended` for human review, resumed through the +identical loader front door (a) uses. + +## Facts (inlined โ€” no spec read needed) + +- `Suspended` is an exact fit, verified against source: "a run awaiting + human approval or a webhook must not hold an executor slot" (T1.3), lease + released outright with no heartbeat while parked (T7.3). +- `plan-draft` is itself an ordinary poimen workflow: a `Model` step + producing a `WorkflowDef` draft, no new kernel machinery. +- The draft is validated against T3.9's capability registry **before** a + human ever sees it โ€” an unresolved verifier/judge/model id in the draft + fails here, at draft time, not after approval. +- Approval feeds the (possibly human-edited) draft through the **same** + `WorkflowFormat`/builder + T3.1 canonicalization as door (a) โ€” this is + what keeps both doors on one IR without new proof machinery. + +## Steps + +1. Define the built-in `plan-draft` workflow: one `Model` step (problem + statement โ†’ draft `WorkflowDef`), one validation step (registry check via + T3.9). +2. On draft ready, transition the `plan-draft` run to `Suspended`, releasing + its lease (T7.3). +3. `/v1/plans/{id}/approve` (T10.4 route): load the (possibly edited) draft, + run it through the exact loader door (a)'s `WorkflowFormat`/builder uses, + canonicalize (T3.1), resume โ€” matching T1.3's resume-from-`Suspended` + semantics. +4. `/v1/plans/{id}/reject`: cancel from `Suspended` (legal transition per + T1.3's cancel-from-any-non-terminal rule). +5. Test: submit a problem statement, assert `Suspended`, edit the draft, + approve, assert the resulting `WorkflowVersion` hash matches + hand-authoring the same edited YAML directly. + +## Acceptance + +- A submitted problem statement produces a run parked in `Suspended` with no + held executor slot โ€” lease released, per T7.3's own test shape. +- Approving a draft (edited or not) produces a `WorkflowVersion` + byte-identical to loading the same content by hand through door (a). + +## Verify + +**Harness:** stub model for the `plan-draft` step, embedded `redb`, T7.3's +lease-release assertion pattern reused. + +**Integration test** โ€” `tests/it_plan_draft_approval.rs`: +1. `POST /v1/problems` with a free-text statement; assert the run reaches + `Suspended`. +2. Assert the lease is released and no heartbeat is emitted while + `Suspended` โ€” reuse T7.3's assertion (its step 8). +3. Fetch the draft; edit one step; `POST /v1/plans/{id}/approve` with the + edited draft. +4. Assert the resulting `WorkflowVersion` hash equals hashing the same + edited content loaded directly via `WorkflowFormat`/builder โ€” front-door + equivalence, extends T3.10. +5. `POST /v1/plans/{id2}/reject` on a second draft; assert it reaches + `Cancelled`, not stuck in `Suspended`. +6. Draft referencing an unregistered verifier id; assert `plan-draft`'s + validation step fails **before** ever reaching `Suspended` โ€” the human + never sees an unloadable plan. + +**Command:** `cargo test -p orchestration plan_draft_approval` + +**False pass:** +- Skipping step 4's hash comparison and only checking "run resumed" โ€” a + subtly different canonicalization from door (a) would still resume, just + diverge from what T3.1's guarantee promises. +- Not testing reject โ€” an approval-only test suite misses that `Suspended` + must also support "no." + +## Traps + +- Having `/approve` construct a `WorkflowDef` directly in the HTTP handler + instead of routing through the same loader as door (a) โ€” this is exactly + the two-IR-drift bug T3.10 exists to prevent, reintroduced one layer up. +- Skipping the registry validation step "since a human will catch it." The + whole point of validating before `Suspended` is that load errors are cheap + pre-human, expensive post-human. + +--- + +Background (not required to do this task): +[../INDEX.md](../INDEX.md) (customization contract ~line 19-58) ยท +[T1.3-run-executor.md](T1.3-run-executor.md) ยท +[T7.3-leases-and-fencing.md](T7.3-leases-and-fencing.md) ยท +[T3.10-front-door-equivalence.md](T3.10-front-door-equivalence.md) diff --git a/tasks/T10.7-kafka-topics-and-inbox-relay.md b/tasks/T10.7-kafka-topics-and-inbox-relay.md new file mode 100644 index 0000000..a4f86bd --- /dev/null +++ b/tasks/T10.7-kafka-topics-and-inbox-relay.md @@ -0,0 +1,108 @@ +# T10.7 โ€” Kafka topics and inbox relay + +| Field | Value | +|---|---| +| Phase | P10 โ€” Orchestration | +| Size | L โ€” over 3 days | +| Status | Not started | +| Flags | โ€” | +| Spec | inlined below | +| Blocks | T10.8 | + +## Goal + +`poimen..events` (outbound, existing outbox unchanged) and +`poimen..dispatch` (inbound, new) โ€” an inbox relay symmetric to +T7.4's outbox, turning dispatch messages into spawn calls through the same +admission path, never a raw kernel write. + +## Facts (inlined โ€” no spec read needed) + +- Outbound is unchanged: T7.4's relay ships goal-tagged `WorkEvent`s exactly + as it ships every other event. No new producer logic on this side. +- Inbound is genuinely new: the orchestrator (T10.8) publishes to + `.dispatch` after a `Reconciler` decision; this task builds only the relay + that drains it, not the orchestrator itself. +- The inbox relay's failure-domain rule mirrors T7.4's exactly: own + process/task, own retry, own supervision, not co-located with the + executor's lifecycle. +- Every dispatch message becomes a call to the **same** `POST /v1/runs` + (T10.4) the HTTP API exposes โ€” the relay is an HTTP client to the local + API surface, not a kernel-internal spawn path. This is what makes the + "no bypass" guarantee mechanical, not a code-review convention. +- Redelivery on `.dispatch` is handled entirely by T10.3's `dispatch_key` + admission dedup โ€” the relay passes `dispatch_key` through unchanged, no + separate idempotency layer. +- Partition key on both topics: `(TenantId, RunId)` per T7.5, consistent + with the outbox. + +## Steps + +1. Declare topics `poimen..events`, `poimen..dispatch` โ€” + partition key `(TenantId, RunId)` on both (T7.5). +2. No change to T7.4's relay beyond including goal-tagged records (already + covered since `GoalId` is a field on `WorkEvent`, T10.1). +3. Build the inbox relay: consume `.dispatch`, for each message call + `POST /v1/runs` with `{workflow_ref, input, goal_id, dispatch_key}` + verbatim. +4. Own retry/backoff on relay-to-API-call failure; never crash the relay on + a single bad message โ€” dead-letter it, keep draining. +5. Own failure domain: separate process/task, no shared lifecycle with the + executor or the HTTP API service. +6. Emit inbox lag as a metric, alongside T7.4's existing outbox-lag metric. +7. Test with the broker down for a stretch, and with forced redelivery of + the same dispatch message. + +## Acceptance + +- A message published to `.dispatch` results in exactly one run, even under + forced redelivery. +- Broker down for the inbox side does not affect runs already in flight; + queued dispatch messages are processed once the broker returns. + +## Verify + +**Harness:** same broker-down/broker-up harness as T7.4, plus a second +consumer/producer pair for the `.dispatch` direction. + +**Integration test** โ€” `tests/it_inbox_relay.rs`: +1. Publish one message to `.dispatch`; assert exactly one run created via + the same `POST /v1/runs` path โ€” instrument the API, assert the call, not + a direct kernel spawn. +2. Force redelivery of the same message; assert still exactly one run + (T10.3's admission dedup catches it) โ€” no relay-side special-casing + needed. +3. Take the broker down; publish a message; bring the broker up; assert the + run is eventually created, no message lost. +4. Publish a malformed message; assert it's dead-lettered, relay keeps + draining subsequent valid messages. +5. Assert the relay never calls kernel spawn directly โ€” instrument the + kernel's internal admission entrypoint, assert all calls originate from + the HTTP layer, none from relay threads. +6. Assert inbox lag is emitted as a metric. + +**Command:** `cargo test -p distribution inbox_relay -- --test-threads=1` + +**False pass:** +- Step 2 without step 1's exact-call-path instrumentation: a relay that + spawns directly and also happens to dedupe correctly would still pass a + coarser "one run" check while violating the no-bypass rule. +- Skipping the malformed-message case: one bad message halting the whole + relay is a silent single point of failure discovered only in production. + +## Traps + +- Relay calling an in-process spawn function "for performance" instead of + going through `POST /v1/runs` โ€” same trap T7.4 names for the outbound + side, mirrored here. +- Building custom dedup logic in the relay instead of trusting T10.3's + admission-layer dedup. Two idempotency layers can disagree under partial + failure. + +--- + +Background (not required to do this task): +[rust-agentic-sys.md](../../../rust-agentic-sys.md) ยง8.3, ยง9.2, ยง9.3, ยง13.3 ยท +[T7.4-outbox-relay.md](T7.4-outbox-relay.md) ยท +[T7.5-partition-keys-on-adapters.md](T7.5-partition-keys-on-adapters.md) ยท +[T10.3-idempotent-run-admission.md](T10.3-idempotent-run-admission.md) diff --git a/tasks/T10.8-orchestrator-reference-service.md b/tasks/T10.8-orchestrator-reference-service.md new file mode 100644 index 0000000..a835862 --- /dev/null +++ b/tasks/T10.8-orchestrator-reference-service.md @@ -0,0 +1,108 @@ +# T10.8 โ€” Orchestrator reference service + +| Field | Value | +|---|---| +| Phase | P10 โ€” Orchestration | +| Size | M โ€” 1 to 3 days | +| Status | Not started | +| Flags | โ€” | +| Spec | inlined below | +| Blocks | โ€” | + +## Goal + +Reference process that consumes `.events`, invokes the registered +`Reconciler` for goal-tagged terminal transitions, and publishes decisions +to `.dispatch` โ€” never spawns a run itself. + +## Facts (inlined โ€” no spec read needed) + +- The orchestrator embeds poimen only via the HTTP API (T10.4) / + `poimen-sdk`, exactly like any other external embedder โ€” no special + kernel access. +- Its entire job: consume `.events`, filter for goal-tagged + `Verified`/`Graded`/`Ungraded`, call `Reconciler` (T10.2), publish the + resulting `SpawnNext` decision to `.dispatch`. `Stop`/`Retry` decisions + are recorded but do not publish a dispatch message. +- The default `Reconciler` implementation may itself be another poimen run + (a `Model` step + a `Tool` step) โ€” the orchestrator's job is identical + regardless of whether the bound `Reconciler` is LLM-backed or a + hand-written Rust impl. +- Never publishes a run directly, never calls `POST /v1/runs` itself โ€” that + would reintroduce the bypass T10.7's inbox relay exists to prevent. Its + only write is to `.dispatch`. + +## Steps + +1. Consume `.events`, filter to records with a `GoalId` and a terminal run + state (`Verified`/`Graded`/`Ungraded`). +2. For each, build `ReconcileCtx` (T10.2) and call the `Reconciler` bound to + that goal. +3. On `SpawnNext`: publish `{workflow_ref, input, goal_id, dispatch_key}` to + `.dispatch` โ€” `dispatch_key` derived from the decision id so redelivery + is naturally deduped downstream (T10.3). +4. On `Retry`/`Stop`: record the decision (`WorkEvent` provenance per + T10.2), publish nothing. +5. Own failure domain: separate process, own consumer group, own + retry/backoff on `Reconciler`-call failure โ€” a `Reconciler` error does + not crash the orchestrator, it skips and retries on next poll or leaves + the goal stalled with a visible metric. +6. Emit reconcile-latency and goal-stall metrics. +7. Test the full loop against a stub `Reconciler`: events in, dispatch out, + verified end to end including a two-hop goal chain (run A โ†’ reconcile โ†’ + run B โ†’ reconcile โ†’ stop). + +## Acceptance + +- A two-run goal chain (A completes โ†’ reconcile โ†’ B spawns โ†’ B completes โ†’ + reconcile โ†’ `Stop`) completes end to end through real Kafka topics, with + the orchestrator never calling spawn directly. +- `Reconciler` failure on one goal does not block reconciliation of other + goals. + +## Verify + +**Harness:** real Kafka (or the board's chosen fault-injection substrate), +stub `Reconciler` with scripted decisions, T10.7's inbox relay running +alongside. + +**Integration test** โ€” `tests/it_orchestrator_e2e.rs`: +1. Spawn run A with `goal_id=G`, no `dispatch_key`. Drive to + `Verified{pass}`. +2. Assert orchestrator calls `Reconciler` exactly once for A; stub returns + `SpawnNext` for run B. +3. Assert a message lands on `.dispatch`, and via T10.7's relay, run B is + spawned with `goal_id=G`. +4. Drive run B to `Graded`. Assert `Reconciler` called again; stub returns + `Stop`. +5. Assert no further `.dispatch` message is published; goal G's chain shows + exactly `[A, B]` via T10.1's `GoalView`. +6. Instrument `POST /v1/runs`; assert zero calls originate from the + orchestrator process โ€” only from the inbox relay. +7. Second goal G2 with a `Reconciler` that always errors; assert G2 stalls + (visible via goal-stall metric) while G's chain, run concurrently, + proceeds unaffected. + +**Command:** `cargo test -p orchestration orchestrator_e2e -- --test-threads=1` + +**False pass:** +- Step 6 omitted: an orchestrator that takes a shortcut and spawns directly + under test-only conditions would still pass steps 1-5. +- Testing only a single-hop chain (A โ†’ `Stop`) โ€” the two-hop case is what + proves goal-chain provenance actually threads through `GoalView` + correctly across a full reconcile-dispatch-relay-spawn cycle. + +## Traps + +- Orchestrator holding an admission-layer shortcut "to save a network hop." + Collapses the exact boundary T10.7 was built to enforce. +- Coupling `Reconciler` failure handling to a single global retry policy. + One broken goal's `Reconciler` should not starve reconciliation for every + other goal in the tenant. + +--- + +Background (not required to do this task): +[T10.1-goalid-and-goal-scoped-query.md](T10.1-goalid-and-goal-scoped-query.md) ยท +[T10.2-reconciler-port.md](T10.2-reconciler-port.md) ยท +[T10.7-kafka-topics-and-inbox-relay.md](T10.7-kafka-topics-and-inbox-relay.md) diff --git a/tasks/T10.9-cli-transcript-and-metrics-ux.md b/tasks/T10.9-cli-transcript-and-metrics-ux.md new file mode 100644 index 0000000..33aaf6d --- /dev/null +++ b/tasks/T10.9-cli-transcript-and-metrics-ux.md @@ -0,0 +1,88 @@ +# T10.9 โ€” CLI: runs list/episode/transcript + +| Field | Value | +|---|---| +| Phase | P10 โ€” Orchestration | +| Size | S โ€” under 1 day | +| Status | Not started | +| Flags | โ€” | +| Spec | inlined below | +| Blocks | โ€” | + +## Goal + +Three thin CLI wrappers over the HTTP API โ€” `poimen runs list --goal `, +`poimen runs episode `, `poimen runs transcript +` โ€” the whole troubleshooting UX, no dashboard. + +## Facts (inlined โ€” no spec read needed) + +- Thin HTTP clients, not a second implementation of the query logic already + in T10.1/T1.7/T1.6 โ€” the CLI calls the same routes T10.4 exposes. +- Rides the eventual shared `poimen` binary (T9.3); workable as a standalone + binary if T9.3 hasn't landed yet โ€” soft dependency, not a hard block. +- `episode` output must include failed attempts by default โ€” T1.7's own + rule. A CLI that hides them defeats the purpose of the command. +- `transcript` dereferences the `BlobRef` server-side, via the HTTP route + (T10.4) โ€” the CLI never talks to `BlobStore` directly. + +## Steps + +1. `poimen runs list [--goal ] [--status ]` โ†’ `GET /v1/runs`, + table output (`run_id`, `status`, `goal_id`, age). +2. `poimen runs episode ` โ†’ `GET /v1/runs/{id}/episode`, + human-readable rendering of `EpisodeView` including failed attempts, + clearly marked. +3. `poimen runs transcript ` โ†’ `GET .../transcript`, + prints prompt/output text, dereferenced server-side. +4. Exit codes: non-zero on API error, distinct code for "run not found" vs + "transport error" โ€” operator scripting depends on this distinction. +5. Test against a live embedded API instance (T10.4's test harness) with a + run that has a retry and a rewind. + +## Acceptance + +- `episode` on a run with two failed attempts and one success shows all + three, visually distinguishable. +- `transcript` returns the same text content as reading the `BlobRef` + directly through `BlobStore` in a test assertion. + +## Verify + +**Harness:** T10.4's `axum` test instance, one run with a retry (two +attempts, one failed) and a rewind. + +**Integration test** โ€” `tests/it_cli_runs_ux.rs`: +1. `runs list --goal G`; assert output contains exactly the runs tagged G, + none untagged. +2. `runs episode `; assert both the failed and successful attempt + appear โ€” count them explicitly, not just non-empty output. +3. `runs transcript 1` (the failed attempt) and `... 2` (the + retry); assert distinct content for each, matching their respective + `BlobRef`s byte-for-byte. +4. Run-not-found case; assert a distinct exit code from a transport-down + case. + +**Command:** `cargo test -p cli runs_ux` + +**False pass:** +- Step 2 checking only that output is non-empty โ€” a CLI silently filtering + to the winning attempt (T1.7's own named trap) passes a non-empty check + while hiding exactly the debugging signal requested. + +## Traps + +- Filtering episode output to "just the successful attempt for clarity" โ€” + T1.7's trap restated at the CLI layer; the failed attempts are the reason + this command exists. +- Fetching transcripts client-side by resolving `BlobRef` directly instead + of through the HTTP route. Bypasses the tenant auth checks the API + enforces. + +--- + +Background (not required to do this task): +[T10.4-http-api-surface.md](T10.4-http-api-surface.md) ยท +[T1.7-episode-query-surface.md](T1.7-episode-query-surface.md) ยท +[T1.6-prompt-and-output-blob-capture.md](T1.6-prompt-and-output-blob-capture.md) ยท +[T8.2-metrics.md](T8.2-metrics.md)