poimen/tasks/T7.7-turmoil-suite.md

T7.7 — turmoil suite

Field	Value
Phase	P7 — Distribution
Size	L — over 3 days
Status	Not started
Flags	parallel-ok
Spec	inlined below
Blocks	—

Goal

Network partition, latency and node loss simulated deterministically. This is the P7 gate.

Facts (inlined — no spec read needed)

• Test tooling and what each covers:
  • turmoil — network partition and latency simulation.
  • loom — the lock-free bits.
  • tokio::time::pause — time.
• Together these are weaker than a seeded scheduler and sufficient with discipline. The residual risk is stated honestly rather than engineered around: tokio's cancellation is cooperative, so a select!-dropped future stops at its next await and not before.
• What the suite must prove: no double-execution and no lost run under partition, with automatic recovery.
• Double-execution is prevented by fencing (T7.3), not by the partition never happening. This suite is what demonstrates the fence actually fires.
• parallel-ok: this task does not gate others and can be built alongside the rest of P7.
• Open: whether a lint, a wrapper type, or a loom harness is the right enforcement for cancellation rigour under tokio. Unresolved — do not invent a fourth mechanism here.

Steps

1. Stand up a two-node topology under turmoil, both nodes claiming runs from the same Postgres backend.
2. Partition scenario: isolate node A mid-run, let its lease expire, watch node B reclaim, then heal the partition and assert node A's writes are fenced off.
3. Latency scenario: inject delays that push heartbeats close to the TTL, and assert renewals either succeed or cancel the local scope — never a silent continue.
4. Node-loss scenario: kill a node outright; assert the run is reclaimed and completes exactly once.
5. Add loom coverage for any lock-free structure introduced in P7.
6. Use tokio::time::pause for all TTL and deadline arithmetic so the suite runs in seconds.
7. Print per-scenario progress and enforce per-scenario timeouts — a suite that prints nothing cannot distinguish slow from hung.

Acceptance

• No double-execution and no lost run under partition; recovery is automatic.

Phase gate

P7 closes on two nodes surviving a partition with no double-execution.

Verify

Harness: turmoil two-node topology; loom for any lock-free structure; tokio::time::pause for TTL arithmetic; the external side-effect ledger as the double-execution witness.

Integration test — tests/it_turmoil.rs:

1. Partition: isolate node A mid-run. Assert node B reclaims after lease expiry, the run completes, and the ledger shows exactly one effect per intended effect.
2. Heal the partition; assert A's writes are fenced (cross-check T7.3) and that A does not resume executing.
3. Latency: inject delays that push heartbeats close to the TTL. Assert renewals either succeed or cancel the local scope — assert no case where a worker continues past a failed renewal.
4. Node loss: kill a node outright. Assert the run is reclaimed and completes exactly once.
5. Assert no lost run: every spawned run reaches a terminal state across all scenarios; count spawned vs terminal.
6. Recovery is automatic — assert no manual intervention hook was invoked.
7. loom test for each lock-free structure added in P7.
8. Per-scenario progress output and per-scenario timeouts.
9. Print the seed; a failure that cannot be replayed is not a finding.

Command: cargo test -p distribution --test it_turmoil -- --nocapture

False pass:

• Asserting the run completed. Completing twice also completes — the ledger count is the only assertion that distinguishes them, and double-execution is the primary risk this suite exists to retire.
• Real sleep for TTL expiry, making the suite slow enough that it stops being run — which is the same as not having it.
• Step 5 omitted: a run that is silently dropped during a partition leaves no error, and every other assertion passes.
• Testing only the partition scenario. Node loss and slow-network cases exercise different reclaim paths.

Traps

• Asserting only that the run completes. Completing twice also completes.
• Real sleeps for TTL expiry, which makes the suite too slow to run and therefore unrun.

---

Background (not required to do this task): rust-agentic-sys.md §6, §9.4, §18, §19 · rust-agentic-task.md