(tasks) add tasks for harness

tasks/T7.7-turmoil-suite.md

@@ -0,0 +1,109 @@
+# T7.7 — `turmoil` suite
+
+| Field | Value |
+|---|---|
+| Phase | P7 — Distribution |
+| Size | L — over 3 days |
+| Status | Not started |
+| Flags | parallel-ok |
+| Spec | inlined below |
+| Blocks | — |
+
+## Goal
+
+Network partition, latency and node loss simulated deterministically. This is the
+P7 gate.
+
+## Facts (inlined — no spec read needed)
+
+- Test tooling and what each covers:
+  - **`turmoil`** — network partition and latency simulation.
+  - **`loom`** — the lock-free bits.
+  - **`tokio::time::pause`** — time.
+- Together these are **weaker than a seeded scheduler and sufficient with
+  discipline**. The residual risk is stated honestly rather than engineered
+  around: tokio's cancellation is cooperative, so a `select!`-dropped future
+  stops at its next await and not before.
+- What the suite must prove: **no double-execution and no lost run under
+  partition, with automatic recovery**.
+- Double-execution is prevented by fencing (T7.3), not by the partition never
+  happening. This suite is what demonstrates the fence actually fires.
+- `parallel-ok`: this task does not gate others and can be built alongside the
+  rest of P7.
+- Open: whether a lint, a wrapper type, or a `loom` harness is the right
+  enforcement for cancellation rigour under tokio. Unresolved — do not invent a
+  fourth mechanism here.
+
+## Steps
+
+1. Stand up a two-node topology under `turmoil`, both nodes claiming runs from
+   the same Postgres backend.
+2. Partition scenario: isolate node A mid-run, let its lease expire, watch node B
+   reclaim, then heal the partition and assert node A's writes are fenced off.
+3. Latency scenario: inject delays that push heartbeats close to the TTL, and
+   assert renewals either succeed or cancel the local scope — never a silent
+   continue.
+4. Node-loss scenario: kill a node outright; assert the run is reclaimed and
+   completes exactly once.
+5. Add `loom` coverage for any lock-free structure introduced in P7.
+6. Use `tokio::time::pause` for all TTL and deadline arithmetic so the suite runs
+   in seconds.
+7. Print per-scenario progress and enforce per-scenario timeouts — a suite that
+   prints nothing cannot distinguish slow from hung.
+
+## Acceptance
+
+- **No double-execution and no lost run under partition; recovery is automatic.**
+
+## Phase gate
+
+P7 closes on two nodes surviving a partition with no double-execution.
+
+## Verify
+
+**Harness:** `turmoil` two-node topology; `loom` for any lock-free structure;
+`tokio::time::pause` for TTL arithmetic; the external side-effect ledger as the
+double-execution witness.
+
+**Integration test** — `tests/it_turmoil.rs`:
+1. **Partition:** isolate node A mid-run. Assert node B reclaims after lease
+   expiry, the run completes, and the ledger shows **exactly one** effect per
+   intended effect.
+2. Heal the partition; assert A's writes are fenced (cross-check T7.3) and that A
+   does not resume executing.
+3. **Latency:** inject delays that push heartbeats close to the TTL. Assert
+   renewals either succeed or **cancel the local scope** — assert no case where a
+   worker continues past a failed renewal.
+4. **Node loss:** kill a node outright. Assert the run is reclaimed and completes
+   exactly once.
+5. Assert **no lost run**: every spawned run reaches a terminal state across all
+   scenarios; count spawned vs terminal.
+6. Recovery is **automatic** — assert no manual intervention hook was invoked.
+7. `loom` test for each lock-free structure added in P7.
+8. Per-scenario progress output and per-scenario timeouts.
+9. Print the seed; a failure that cannot be replayed is not a finding.
+
+**Command:** `cargo test -p distribution --test it_turmoil -- --nocapture`
+
+**False pass:**
+- Asserting the run completed. **Completing twice also completes** — the ledger
+  count is the only assertion that distinguishes them, and double-execution is
+  the primary risk this suite exists to retire.
+- Real `sleep` for TTL expiry, making the suite slow enough that it stops being
+  run — which is the same as not having it.
+- Step 5 omitted: a run that is silently dropped during a partition leaves no
+  error, and every other assertion passes.
+- Testing only the partition scenario. Node loss and slow-network cases exercise
+  different reclaim paths.
+
+## Traps
+
+- Asserting only that the run completes. Completing twice also completes.
+- Real sleeps for TTL expiry, which makes the suite too slow to run and therefore
+  unrun.
+
+---
+
+Background (not required to do this task):
+[rust-agentic-sys.md](../../../rust-agentic-sys.md) §6, §9.4, §18, §19 ·
+[rust-agentic-task.md](../../../rust-agentic-task.md)