(tasks) add tasks for harness

2026-08-17 23:05:20 -07:00
parent e2d678b118
commit 5a30d0ffc6
74 changed files with 8204 additions and 1 deletions
@@ -0,0 +1,98 @@
+# T7.4 — Outbox relay
+
+| Field | Value |
+|---|---|
+| Phase | P7 — Distribution |
+| Size | M — 1 to 3 days |
+| Status | Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | — |
+
+## Goal
+
+Ship committed export intents out of process, with its own retry and its own
+failure domain. The execution path never calls a broker.
+
+## Facts (inlined — no spec read needed)
+
+- **The framework never calls a broker from the execution path.** Export intent
+  is written in the same transaction as the state (T0.6); a separate relay reads
+  committed intents and ships them.
+- This makes export restartable, keeps a broker outage from stalling a run, and
+  is **the only pattern that survives a crash between "state committed" and
+  "event published"**.
+- **A user's broker being down is not an agent outage.**
+- The relay reads through `drain_outbox` / `ack_outbox`. Ordering is per
+  `BranchKey`, ascending `Lsn` — that is the only ordering the system promises,
+  and it falls out of the outbox key shape.
+- **Delivery is at-least-once. Exactly-once is achieved at the fold, not in
+  transport**: `(BranchKey, Lsn)` is the natural idempotency key, so a redelivered
+  record is a no-op insert.
+- Only the live branch is exported (T2.3).
+
+## Steps
+
+1. Run the relay as its own process or task with its own supervision — separate
+   failure domain is the point, so do not co-locate it with the executor's
+   lifecycle.
+2. Loop: `drain_outbox(tenant, limit)` → publish in `(BranchKey, Lsn)` order →
+   `ack_outbox(shipped)`. Ack only what the broker accepted.
+3. Retry with backoff inside the relay. Never propagate a broker error into run
+   execution — there is no path back by construction, so verify none is added.
+4. Set the broker partition key from `(TenantId, RunId)` (T7.5).
+5. Make the consumer-side fold idempotent on `(BranchKey, Lsn)` so redelivery is
+   a no-op insert.
+6. Emit relay lag as a metric — outbox backlog is one of the two that grow
+   silently.
+7. Test with the broker down for an entire run.
+
+## Acceptance
+
+- Broker down for the whole run; **the run completes**; every event ships after
+  the broker returns, **in order, exactly once at the fold**.
+
+## Verify
+
+**Harness:** a broker that can be held down for the whole test, plus a consumer
+that folds on `(BranchKey, Lsn)` so redelivery is observable.
+
+**Integration test** — `tests/it_outbox_relay.rs`:
+1. Take the broker **down**. Run a full run to completion.
+2. Assert the run **completed normally** — no stall, no error, and its latency is
+   comparable to a broker-up baseline. A user's broker being down is not an agent
+   outage.
+3. Bring the broker up. Assert every event ships.
+4. Assert ordering **per `BranchKey`, ascending `Lsn`** at the consumer.
+5. Force redelivery (ack loss): assert the consumer's folded state is identical
+   to the single-delivery case — exactly-once **at the fold**, not in transport.
+6. Kill the relay mid-batch; restart; assert no event is lost and none is
+   duplicated at the fold.
+7. Assert the execution path never calls the broker: instrument the broker client
+   and assert zero calls originate from executor threads.
+8. Assert only the **live branch** is exported after a rewind.
+9. Assert relay lag is emitted as a metric.
+
+**Command:** `cargo test -p distribution outbox -- --test-threads=1`
+
+**False pass:**
+- Step 2 asserting only that the run completed. If the executor retries the
+  broker inline, the run completes — slowly. The latency comparison is what
+  catches it; step 7 is what proves it.
+- Acking before the broker confirms: steps 3 and 4 still pass in a happy-path
+  test, and events vanish silently under failure. Step 6 is the guard.
+- Testing ordering within one branch only, where any implementation is ordered.
+  Use several branches and several runs interleaved.
+
+## Traps
+
+- Publishing from the execution path "just for low latency". A broker outage then
+  stalls agent work.
+- Acking before the broker confirms. Events are lost with no trace.
+- Attempting transport-level exactly-once. The fold is where it is achievable.
+
+---
+
+Background (not required to do this task):
+[rust-agentic-sys.md](../../../rust-agentic-sys.md) §8.3, §9.2, §9.3, §13.3, §15 ·
+[rust-agentic-task.md](../../../rust-agentic-task.md)