(plan) update the plan to captures more

2026-08-18 20:04:29 -07:00
parent 5a30d0ffc6
commit 139f8419bb
11 changed files with 993 additions and 1 deletions
@@ -0,0 +1,108 @@
+# T10.7 — Kafka topics and inbox relay
+
+| Field | Value |
+|---|---|
+| Phase | P10 — Orchestration |
+| Size | L — over 3 days |
+| Status | Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | T10.8 |
+
+## Goal
+
+`poimen.<tenant>.events` (outbound, existing outbox unchanged) and
+`poimen.<tenant>.dispatch` (inbound, new) — an inbox relay symmetric to
+T7.4's outbox, turning dispatch messages into spawn calls through the same
+admission path, never a raw kernel write.
+
+## Facts (inlined — no spec read needed)
+
+- Outbound is unchanged: T7.4's relay ships goal-tagged `WorkEvent`s exactly
+  as it ships every other event. No new producer logic on this side.
+- Inbound is genuinely new: the orchestrator (T10.8) publishes to
+  `.dispatch` after a `Reconciler` decision; this task builds only the relay
+  that drains it, not the orchestrator itself.
+- The inbox relay's failure-domain rule mirrors T7.4's exactly: own
+  process/task, own retry, own supervision, not co-located with the
+  executor's lifecycle.
+- Every dispatch message becomes a call to the **same** `POST /v1/runs`
+  (T10.4) the HTTP API exposes — the relay is an HTTP client to the local
+  API surface, not a kernel-internal spawn path. This is what makes the
+  "no bypass" guarantee mechanical, not a code-review convention.
+- Redelivery on `.dispatch` is handled entirely by T10.3's `dispatch_key`
+  admission dedup — the relay passes `dispatch_key` through unchanged, no
+  separate idempotency layer.
+- Partition key on both topics: `(TenantId, RunId)` per T7.5, consistent
+  with the outbox.
+
+## Steps
+
+1. Declare topics `poimen.<tenant>.events`, `poimen.<tenant>.dispatch` —
+   partition key `(TenantId, RunId)` on both (T7.5).
+2. No change to T7.4's relay beyond including goal-tagged records (already
+   covered since `GoalId` is a field on `WorkEvent`, T10.1).
+3. Build the inbox relay: consume `.dispatch`, for each message call
+   `POST /v1/runs` with `{workflow_ref, input, goal_id, dispatch_key}`
+   verbatim.
+4. Own retry/backoff on relay-to-API-call failure; never crash the relay on
+   a single bad message — dead-letter it, keep draining.
+5. Own failure domain: separate process/task, no shared lifecycle with the
+   executor or the HTTP API service.
+6. Emit inbox lag as a metric, alongside T7.4's existing outbox-lag metric.
+7. Test with the broker down for a stretch, and with forced redelivery of
+   the same dispatch message.
+
+## Acceptance
+
+- A message published to `.dispatch` results in exactly one run, even under
+  forced redelivery.
+- Broker down for the inbox side does not affect runs already in flight;
+  queued dispatch messages are processed once the broker returns.
+
+## Verify
+
+**Harness:** same broker-down/broker-up harness as T7.4, plus a second
+consumer/producer pair for the `.dispatch` direction.
+
+**Integration test** — `tests/it_inbox_relay.rs`:
+1. Publish one message to `.dispatch`; assert exactly one run created via
+   the same `POST /v1/runs` path — instrument the API, assert the call, not
+   a direct kernel spawn.
+2. Force redelivery of the same message; assert still exactly one run
+   (T10.3's admission dedup catches it) — no relay-side special-casing
+   needed.
+3. Take the broker down; publish a message; bring the broker up; assert the
+   run is eventually created, no message lost.
+4. Publish a malformed message; assert it's dead-lettered, relay keeps
+   draining subsequent valid messages.
+5. Assert the relay never calls kernel spawn directly — instrument the
+   kernel's internal admission entrypoint, assert all calls originate from
+   the HTTP layer, none from relay threads.
+6. Assert inbox lag is emitted as a metric.
+
+**Command:** `cargo test -p distribution inbox_relay -- --test-threads=1`
+
+**False pass:**
+- Step 2 without step 1's exact-call-path instrumentation: a relay that
+  spawns directly and also happens to dedupe correctly would still pass a
+  coarser "one run" check while violating the no-bypass rule.
+- Skipping the malformed-message case: one bad message halting the whole
+  relay is a silent single point of failure discovered only in production.
+
+## Traps
+
+- Relay calling an in-process spawn function "for performance" instead of
+  going through `POST /v1/runs` — same trap T7.4 names for the outbound
+  side, mirrored here.
+- Building custom dedup logic in the relay instead of trusting T10.3's
+  admission-layer dedup. Two idempotency layers can disagree under partial
+  failure.
+
+---
+
+Background (not required to do this task):
+[rust-agentic-sys.md](../../../rust-agentic-sys.md) §8.3, §9.2, §9.3, §13.3 ·
+[T7.4-outbox-relay.md](T7.4-outbox-relay.md) ·
+[T7.5-partition-keys-on-adapters.md](T7.5-partition-keys-on-adapters.md) ·
+[T10.3-idempotent-run-admission.md](T10.3-idempotent-run-admission.md)