Files
poimen/tasks/T7.4-outbox-relay.md
T

4.2 KiB

T7.4 — Outbox relay

Field Value
Phase P7 — Distribution
Size M — 1 to 3 days
Status Not started
Flags
Spec inlined below
Blocks

Goal

Ship committed export intents out of process, with its own retry and its own failure domain. The execution path never calls a broker.

Facts (inlined — no spec read needed)

  • The framework never calls a broker from the execution path. Export intent is written in the same transaction as the state (T0.6); a separate relay reads committed intents and ships them.
  • This makes export restartable, keeps a broker outage from stalling a run, and is the only pattern that survives a crash between "state committed" and "event published".
  • A user's broker being down is not an agent outage.
  • The relay reads through drain_outbox / ack_outbox. Ordering is per BranchKey, ascending Lsn — that is the only ordering the system promises, and it falls out of the outbox key shape.
  • Delivery is at-least-once. Exactly-once is achieved at the fold, not in transport: (BranchKey, Lsn) is the natural idempotency key, so a redelivered record is a no-op insert.
  • Only the live branch is exported (T2.3).

Steps

  1. Run the relay as its own process or task with its own supervision — separate failure domain is the point, so do not co-locate it with the executor's lifecycle.
  2. Loop: drain_outbox(tenant, limit) → publish in (BranchKey, Lsn) order → ack_outbox(shipped). Ack only what the broker accepted.
  3. Retry with backoff inside the relay. Never propagate a broker error into run execution — there is no path back by construction, so verify none is added.
  4. Set the broker partition key from (TenantId, RunId) (T7.5).
  5. Make the consumer-side fold idempotent on (BranchKey, Lsn) so redelivery is a no-op insert.
  6. Emit relay lag as a metric — outbox backlog is one of the two that grow silently.
  7. Test with the broker down for an entire run.

Acceptance

  • Broker down for the whole run; the run completes; every event ships after the broker returns, in order, exactly once at the fold.

Verify

Harness: a broker that can be held down for the whole test, plus a consumer that folds on (BranchKey, Lsn) so redelivery is observable.

Integration testtests/it_outbox_relay.rs:

  1. Take the broker down. Run a full run to completion.
  2. Assert the run completed normally — no stall, no error, and its latency is comparable to a broker-up baseline. A user's broker being down is not an agent outage.
  3. Bring the broker up. Assert every event ships.
  4. Assert ordering per BranchKey, ascending Lsn at the consumer.
  5. Force redelivery (ack loss): assert the consumer's folded state is identical to the single-delivery case — exactly-once at the fold, not in transport.
  6. Kill the relay mid-batch; restart; assert no event is lost and none is duplicated at the fold.
  7. Assert the execution path never calls the broker: instrument the broker client and assert zero calls originate from executor threads.
  8. Assert only the live branch is exported after a rewind.
  9. Assert relay lag is emitted as a metric.

Command: cargo test -p distribution outbox -- --test-threads=1

False pass:

  • Step 2 asserting only that the run completed. If the executor retries the broker inline, the run completes — slowly. The latency comparison is what catches it; step 7 is what proves it.
  • Acking before the broker confirms: steps 3 and 4 still pass in a happy-path test, and events vanish silently under failure. Step 6 is the guard.
  • Testing ordering within one branch only, where any implementation is ordered. Use several branches and several runs interleaved.

Traps

  • Publishing from the execution path "just for low latency". A broker outage then stalls agent work.
  • Acking before the broker confirms. Events are lost with no trace.
  • Attempting transport-level exactly-once. The fold is where it is achievable.

Background (not required to do this task): rust-agentic-sys.md §8.3, §9.2, §9.3, §13.3, §15 · rust-agentic-task.md