(tasks) add tasks for harness

2026-08-17 23:05:20 -07:00
parent e2d678b118
commit 5a30d0ffc6
74 changed files with 8204 additions and 1 deletions
@@ -0,0 +1,101 @@
+# T7.3 — Leases and fencing
+
+| Field | Value |
+|---|---|
+| Phase | P7 — Distribution |
+| Size | L — over 3 days |
+| Status | Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | — |
+
+## Goal
+
+TTL leases with heartbeat renewal and monotonic fence tokens, so a partitioned
+worker cannot write under an expired claim.
+
+## Facts (inlined — no spec read needed)
+
+- Runs are claimed by workers under a lease with a TTL. **A worker that dies has
+  its runs reclaimed after expiry.**
+- Lease renewal is a heartbeat on the run record.
+- **`Suspended` runs release their lease entirely** rather than heartbeating
+  through a human's lunch break. A run awaiting human approval or a webhook must
+  release its worker; holding an executor slot across a human decision does not
+  scale past a handful of concurrent runs.
+- **Fencing tokens on every lease.** A partitioned worker that resumes must not
+  write under an expired claim, and a **monotonic fence in the run record makes
+  that a rejected write rather than a silent double-execution**.
+- Coordination backend: Postgres advisory locks, or Redis if lease churn
+  dominates. Redis is deferred until that churn is real.
+
+## Steps
+
+1. Add to the run record: `lease_owner`, `lease_expires_at`, `fence: u64`.
+2. Claim: atomic conditional update — claim only if unowned or expired.
+   Increment `fence` on every successful claim. The fence is monotonic per run.
+3. The worker carries its fence value in memory and includes it on **every**
+   write. Writes compare against the stored fence and are rejected when lower.
+4. Heartbeat renews `lease_expires_at` on an interval well inside the TTL. A
+   failed renewal cancels the local `RunScope` (T1.1) rather than continuing
+   optimistically.
+5. On transition to `Suspended`, release the lease outright — clear the owner and
+   stop heartbeating. On resume, re-claim, which takes a new fence.
+6. Reclaim path: a run whose lease expired is claimable by any worker.
+7. Test the full sequence: partition, expiry, reclaim elsewhere, reconnect the
+   original, assert its writes are **rejected by fence**.
+
+## Acceptance
+
+- Partition a worker, let its lease expire, reclaim the run elsewhere, then
+  reconnect the original — its writes are **rejected by fence, not merely late**.
+
+## Verify
+
+**Harness:** two worker processes against one Postgres, plus `turmoil` or an
+iptables-style partition for the network half. `tokio::time::pause` cannot be
+used across processes — use a short real TTL instead.
+
+**Integration test** — `tests/it_leases_fencing.rs`:
+1. Worker A claims run R (fence = 1) and begins executing.
+2. Partition A from the database. Let the lease expire.
+3. Worker B reclaims R; assert its fence is **2**.
+4. Heal the partition. Worker A attempts a write under fence 1.
+5. Assert the write is **rejected by the fence check** — and assert the rejection
+   reason is the fence, not a timestamp comparison and not a generic conflict.
+6. Assert R completed **exactly once**: check the external side-effect ledger for
+   a single entry.
+7. Assert A's failed heartbeat **cancelled its local `RunScope`**, so it stopped
+   working rather than continuing optimistically.
+8. `Suspended`: transition a run to `Suspended`; assert the lease is **released**
+   (owner cleared) and that no heartbeat is emitted while suspended. Resume and
+   assert a **new, higher** fence.
+9. Monotonicity: 100 claim/expire cycles; assert the fence never decreases or
+   repeats.
+
+**Command:** `cargo test -p distribution leases -- --test-threads=1`
+
+**False pass:**
+- Step 5 satisfied by a timestamp check ("your lease expired"). Under clock skew
+  that check can pass for a stale worker; the **fence comparison** is what makes
+  it clock-independent. Assert the reason.
+- Step 6 omitted: both workers can execute, both can be "correct", and the
+  double-execution is only visible in the external ledger.
+- A per-worker fence counter rather than per-run monotonic — step 9 with two
+  workers alternating catches it.
+- Step 8 verified by reading a flag rather than asserting heartbeats stop.
+
+## Traps
+
+- Checking only `lease_expires_at` at write time. Clock skew makes "not expired
+  yet" a lie, and the fence is what makes the check independent of clocks.
+- Heartbeating through `Suspended`. It burns a worker slot for the duration of a
+  human decision.
+- A per-worker fence rather than per-run monotonic. Two workers then produce
+  incomparable tokens.
+
+---
+
+Background (not required to do this task):
+[rust-agentic-sys.md](../../../rust-agentic-sys.md) §5.2, §7, §9.4 ·
+[rust-agentic-task.md](../../../rust-agentic-task.md)