99 lines
4.2 KiB
Markdown
99 lines
4.2 KiB
Markdown
# T7.4 — Outbox relay
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | P7 — Distribution |
|
|
| Size | M — 1 to 3 days |
|
|
| Status | Not started |
|
|
| Flags | — |
|
|
| Spec | inlined below |
|
|
| Blocks | — |
|
|
|
|
## Goal
|
|
|
|
Ship committed export intents out of process, with its own retry and its own
|
|
failure domain. The execution path never calls a broker.
|
|
|
|
## Facts (inlined — no spec read needed)
|
|
|
|
- **The framework never calls a broker from the execution path.** Export intent
|
|
is written in the same transaction as the state (T0.6); a separate relay reads
|
|
committed intents and ships them.
|
|
- This makes export restartable, keeps a broker outage from stalling a run, and
|
|
is **the only pattern that survives a crash between "state committed" and
|
|
"event published"**.
|
|
- **A user's broker being down is not an agent outage.**
|
|
- The relay reads through `drain_outbox` / `ack_outbox`. Ordering is per
|
|
`BranchKey`, ascending `Lsn` — that is the only ordering the system promises,
|
|
and it falls out of the outbox key shape.
|
|
- **Delivery is at-least-once. Exactly-once is achieved at the fold, not in
|
|
transport**: `(BranchKey, Lsn)` is the natural idempotency key, so a redelivered
|
|
record is a no-op insert.
|
|
- Only the live branch is exported (T2.3).
|
|
|
|
## Steps
|
|
|
|
1. Run the relay as its own process or task with its own supervision — separate
|
|
failure domain is the point, so do not co-locate it with the executor's
|
|
lifecycle.
|
|
2. Loop: `drain_outbox(tenant, limit)` → publish in `(BranchKey, Lsn)` order →
|
|
`ack_outbox(shipped)`. Ack only what the broker accepted.
|
|
3. Retry with backoff inside the relay. Never propagate a broker error into run
|
|
execution — there is no path back by construction, so verify none is added.
|
|
4. Set the broker partition key from `(TenantId, RunId)` (T7.5).
|
|
5. Make the consumer-side fold idempotent on `(BranchKey, Lsn)` so redelivery is
|
|
a no-op insert.
|
|
6. Emit relay lag as a metric — outbox backlog is one of the two that grow
|
|
silently.
|
|
7. Test with the broker down for an entire run.
|
|
|
|
## Acceptance
|
|
|
|
- Broker down for the whole run; **the run completes**; every event ships after
|
|
the broker returns, **in order, exactly once at the fold**.
|
|
|
|
## Verify
|
|
|
|
**Harness:** a broker that can be held down for the whole test, plus a consumer
|
|
that folds on `(BranchKey, Lsn)` so redelivery is observable.
|
|
|
|
**Integration test** — `tests/it_outbox_relay.rs`:
|
|
1. Take the broker **down**. Run a full run to completion.
|
|
2. Assert the run **completed normally** — no stall, no error, and its latency is
|
|
comparable to a broker-up baseline. A user's broker being down is not an agent
|
|
outage.
|
|
3. Bring the broker up. Assert every event ships.
|
|
4. Assert ordering **per `BranchKey`, ascending `Lsn`** at the consumer.
|
|
5. Force redelivery (ack loss): assert the consumer's folded state is identical
|
|
to the single-delivery case — exactly-once **at the fold**, not in transport.
|
|
6. Kill the relay mid-batch; restart; assert no event is lost and none is
|
|
duplicated at the fold.
|
|
7. Assert the execution path never calls the broker: instrument the broker client
|
|
and assert zero calls originate from executor threads.
|
|
8. Assert only the **live branch** is exported after a rewind.
|
|
9. Assert relay lag is emitted as a metric.
|
|
|
|
**Command:** `cargo test -p distribution outbox -- --test-threads=1`
|
|
|
|
**False pass:**
|
|
- Step 2 asserting only that the run completed. If the executor retries the
|
|
broker inline, the run completes — slowly. The latency comparison is what
|
|
catches it; step 7 is what proves it.
|
|
- Acking before the broker confirms: steps 3 and 4 still pass in a happy-path
|
|
test, and events vanish silently under failure. Step 6 is the guard.
|
|
- Testing ordering within one branch only, where any implementation is ordered.
|
|
Use several branches and several runs interleaved.
|
|
|
|
## Traps
|
|
|
|
- Publishing from the execution path "just for low latency". A broker outage then
|
|
stalls agent work.
|
|
- Acking before the broker confirms. Events are lost with no trace.
|
|
- Attempting transport-level exactly-once. The fold is where it is achievable.
|
|
|
|
---
|
|
|
|
Background (not required to do this task):
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §8.3, §9.2, §9.3, §13.3, §15 ·
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|