4.2 KiB
4.2 KiB
T7.4 — Outbox relay
| Field | Value |
|---|---|
| Phase | P7 — Distribution |
| Size | M — 1 to 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |
Goal
Ship committed export intents out of process, with its own retry and its own failure domain. The execution path never calls a broker.
Facts (inlined — no spec read needed)
- The framework never calls a broker from the execution path. Export intent is written in the same transaction as the state (T0.6); a separate relay reads committed intents and ships them.
- This makes export restartable, keeps a broker outage from stalling a run, and is the only pattern that survives a crash between "state committed" and "event published".
- A user's broker being down is not an agent outage.
- The relay reads through
drain_outbox/ack_outbox. Ordering is perBranchKey, ascendingLsn— that is the only ordering the system promises, and it falls out of the outbox key shape. - Delivery is at-least-once. Exactly-once is achieved at the fold, not in
transport:
(BranchKey, Lsn)is the natural idempotency key, so a redelivered record is a no-op insert. - Only the live branch is exported (T2.3).
Steps
- Run the relay as its own process or task with its own supervision — separate failure domain is the point, so do not co-locate it with the executor's lifecycle.
- Loop:
drain_outbox(tenant, limit)→ publish in(BranchKey, Lsn)order →ack_outbox(shipped). Ack only what the broker accepted. - Retry with backoff inside the relay. Never propagate a broker error into run execution — there is no path back by construction, so verify none is added.
- Set the broker partition key from
(TenantId, RunId)(T7.5). - Make the consumer-side fold idempotent on
(BranchKey, Lsn)so redelivery is a no-op insert. - Emit relay lag as a metric — outbox backlog is one of the two that grow silently.
- Test with the broker down for an entire run.
Acceptance
- Broker down for the whole run; the run completes; every event ships after the broker returns, in order, exactly once at the fold.
Verify
Harness: a broker that can be held down for the whole test, plus a consumer
that folds on (BranchKey, Lsn) so redelivery is observable.
Integration test — tests/it_outbox_relay.rs:
- Take the broker down. Run a full run to completion.
- Assert the run completed normally — no stall, no error, and its latency is comparable to a broker-up baseline. A user's broker being down is not an agent outage.
- Bring the broker up. Assert every event ships.
- Assert ordering per
BranchKey, ascendingLsnat the consumer. - Force redelivery (ack loss): assert the consumer's folded state is identical to the single-delivery case — exactly-once at the fold, not in transport.
- Kill the relay mid-batch; restart; assert no event is lost and none is duplicated at the fold.
- Assert the execution path never calls the broker: instrument the broker client and assert zero calls originate from executor threads.
- Assert only the live branch is exported after a rewind.
- Assert relay lag is emitted as a metric.
Command: cargo test -p distribution outbox -- --test-threads=1
False pass:
- Step 2 asserting only that the run completed. If the executor retries the broker inline, the run completes — slowly. The latency comparison is what catches it; step 7 is what proves it.
- Acking before the broker confirms: steps 3 and 4 still pass in a happy-path test, and events vanish silently under failure. Step 6 is the guard.
- Testing ordering within one branch only, where any implementation is ordered. Use several branches and several runs interleaved.
Traps
- Publishing from the execution path "just for low latency". A broker outage then stalls agent work.
- Acking before the broker confirms. Events are lost with no trace.
- Attempting transport-level exactly-once. The fold is where it is achievable.
Background (not required to do this task): rust-agentic-sys.md §8.3, §9.2, §9.3, §13.3, §15 · rust-agentic-task.md