poimen/tasks/T7.3-leases-and-fencing.md

# T7.3 — Leases and fencing

| Field | Value |
|---|---|
| Phase | P7 — Distribution |
| Size | L — over 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |

## Goal

TTL leases with heartbeat renewal and monotonic fence tokens, so a partitioned
worker cannot write under an expired claim.

## Facts (inlined — no spec read needed)

- Runs are claimed by workers under a lease with a TTL. **A worker that dies has
  its runs reclaimed after expiry.**
- Lease renewal is a heartbeat on the run record.
- **`Suspended` runs release their lease entirely** rather than heartbeating
  through a human's lunch break. A run awaiting human approval or a webhook must
  release its worker; holding an executor slot across a human decision does not
  scale past a handful of concurrent runs.
- **Fencing tokens on every lease.** A partitioned worker that resumes must not
  write under an expired claim, and a **monotonic fence in the run record makes
  that a rejected write rather than a silent double-execution**.
- Coordination backend: Postgres advisory locks, or Redis if lease churn
  dominates. Redis is deferred until that churn is real.

## Steps

1. Add to the run record: `lease_owner`, `lease_expires_at`, `fence: u64`.
2. Claim: atomic conditional update — claim only if unowned or expired.
   Increment `fence` on every successful claim. The fence is monotonic per run.
3. The worker carries its fence value in memory and includes it on **every**
   write. Writes compare against the stored fence and are rejected when lower.
4. Heartbeat renews `lease_expires_at` on an interval well inside the TTL. A
   failed renewal cancels the local `RunScope` (T1.1) rather than continuing
   optimistically.
5. On transition to `Suspended`, release the lease outright — clear the owner and
   stop heartbeating. On resume, re-claim, which takes a new fence.
6. Reclaim path: a run whose lease expired is claimable by any worker.
7. Test the full sequence: partition, expiry, reclaim elsewhere, reconnect the
   original, assert its writes are **rejected by fence**.

## Acceptance

- Partition a worker, let its lease expire, reclaim the run elsewhere, then
  reconnect the original — its writes are **rejected by fence, not merely late**.

## Verify

**Harness:** two worker processes against one Postgres, plus `turmoil` or an
iptables-style partition for the network half. `tokio::time::pause` cannot be
used across processes — use a short real TTL instead.

**Integration test** — `tests/it_leases_fencing.rs`:
1. Worker A claims run R (fence = 1) and begins executing.
2. Partition A from the database. Let the lease expire.
3. Worker B reclaims R; assert its fence is **2**.
4. Heal the partition. Worker A attempts a write under fence 1.
5. Assert the write is **rejected by the fence check** — and assert the rejection
   reason is the fence, not a timestamp comparison and not a generic conflict.
6. Assert R completed **exactly once**: check the external side-effect ledger for
   a single entry.
7. Assert A's failed heartbeat **cancelled its local `RunScope`**, so it stopped
   working rather than continuing optimistically.
8. `Suspended`: transition a run to `Suspended`; assert the lease is **released**
   (owner cleared) and that no heartbeat is emitted while suspended. Resume and
   assert a **new, higher** fence.
9. Monotonicity: 100 claim/expire cycles; assert the fence never decreases or
   repeats.

**Command:** `cargo test -p distribution leases -- --test-threads=1`

**False pass:**
- Step 5 satisfied by a timestamp check ("your lease expired"). Under clock skew
  that check can pass for a stale worker; the **fence comparison** is what makes
  it clock-independent. Assert the reason.
- Step 6 omitted: both workers can execute, both can be "correct", and the
  double-execution is only visible in the external ledger.
- A per-worker fence counter rather than per-run monotonic — step 9 with two
  workers alternating catches it.
- Step 8 verified by reading a flag rather than asserting heartbeats stop.

## Traps

- Checking only `lease_expires_at` at write time. Clock skew makes "not expired
  yet" a lie, and the fence is what makes the check independent of clocks.
- Heartbeating through `Suspended`. It burns a worker slot for the duration of a
  human decision.
- A per-worker fence rather than per-run monotonic. Two workers then produce
  incomparable tokens.

---

Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §5.2, §7, §9.4 ·
[rust-agentic-task.md](../../../rust-agentic-task.md)