poimen/tasks/T7.7-turmoil-suite.md

# T7.7 — `turmoil` suite

| Field | Value |
|---|---|
| Phase | P7 — Distribution |
| Size | L — over 3 days |
| Status | Not started |
| Flags | parallel-ok |
| Spec | inlined below |
| Blocks | — |

## Goal

Network partition, latency and node loss simulated deterministically. This is the
P7 gate.

## Facts (inlined — no spec read needed)

- Test tooling and what each covers:
  - **`turmoil`** — network partition and latency simulation.
  - **`loom`** — the lock-free bits.
  - **`tokio::time::pause`** — time.
- Together these are **weaker than a seeded scheduler and sufficient with
  discipline**. The residual risk is stated honestly rather than engineered
  around: tokio's cancellation is cooperative, so a `select!`-dropped future
  stops at its next await and not before.
- What the suite must prove: **no double-execution and no lost run under
  partition, with automatic recovery**.
- Double-execution is prevented by fencing (T7.3), not by the partition never
  happening. This suite is what demonstrates the fence actually fires.
- `parallel-ok`: this task does not gate others and can be built alongside the
  rest of P7.
- Open: whether a lint, a wrapper type, or a `loom` harness is the right
  enforcement for cancellation rigour under tokio. Unresolved — do not invent a
  fourth mechanism here.

## Steps

1. Stand up a two-node topology under `turmoil`, both nodes claiming runs from
   the same Postgres backend.
2. Partition scenario: isolate node A mid-run, let its lease expire, watch node B
   reclaim, then heal the partition and assert node A's writes are fenced off.
3. Latency scenario: inject delays that push heartbeats close to the TTL, and
   assert renewals either succeed or cancel the local scope — never a silent
   continue.
4. Node-loss scenario: kill a node outright; assert the run is reclaimed and
   completes exactly once.
5. Add `loom` coverage for any lock-free structure introduced in P7.
6. Use `tokio::time::pause` for all TTL and deadline arithmetic so the suite runs
   in seconds.
7. Print per-scenario progress and enforce per-scenario timeouts — a suite that
   prints nothing cannot distinguish slow from hung.

## Acceptance

- **No double-execution and no lost run under partition; recovery is automatic.**

## Phase gate

P7 closes on two nodes surviving a partition with no double-execution.

## Verify

**Harness:** `turmoil` two-node topology; `loom` for any lock-free structure;
`tokio::time::pause` for TTL arithmetic; the external side-effect ledger as the
double-execution witness.

**Integration test** — `tests/it_turmoil.rs`:
1. **Partition:** isolate node A mid-run. Assert node B reclaims after lease
   expiry, the run completes, and the ledger shows **exactly one** effect per
   intended effect.
2. Heal the partition; assert A's writes are fenced (cross-check T7.3) and that A
   does not resume executing.
3. **Latency:** inject delays that push heartbeats close to the TTL. Assert
   renewals either succeed or **cancel the local scope** — assert no case where a
   worker continues past a failed renewal.
4. **Node loss:** kill a node outright. Assert the run is reclaimed and completes
   exactly once.
5. Assert **no lost run**: every spawned run reaches a terminal state across all
   scenarios; count spawned vs terminal.
6. Recovery is **automatic** — assert no manual intervention hook was invoked.
7. `loom` test for each lock-free structure added in P7.
8. Per-scenario progress output and per-scenario timeouts.
9. Print the seed; a failure that cannot be replayed is not a finding.

**Command:** `cargo test -p distribution --test it_turmoil -- --nocapture`

**False pass:**
- Asserting the run completed. **Completing twice also completes** — the ledger
  count is the only assertion that distinguishes them, and double-execution is
  the primary risk this suite exists to retire.
- Real `sleep` for TTL expiry, making the suite slow enough that it stops being
  run — which is the same as not having it.
- Step 5 omitted: a run that is silently dropped during a partition leaves no
  error, and every other assertion passes.
- Testing only the partition scenario. Node loss and slow-network cases exercise
  different reclaim paths.

## Traps

- Asserting only that the run completes. Completing twice also completes.
- Real sleeps for TTL expiry, which makes the suite too slow to run and therefore
  unrun.

---

Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §6, §9.4, §18, §19 ·
[rust-agentic-task.md](../../../rust-agentic-task.md)