poimen/tasks/T7.3-leases-and-fencing.md

T7.3 — Leases and fencing

Field	Value
Phase	P7 — Distribution
Size	L — over 3 days
Status	Not started
Flags	—
Spec	inlined below
Blocks	—

Goal

TTL leases with heartbeat renewal and monotonic fence tokens, so a partitioned worker cannot write under an expired claim.

Facts (inlined — no spec read needed)

• Runs are claimed by workers under a lease with a TTL. A worker that dies has its runs reclaimed after expiry.
• Lease renewal is a heartbeat on the run record.
• Suspended runs release their lease entirely rather than heartbeating through a human's lunch break. A run awaiting human approval or a webhook must release its worker; holding an executor slot across a human decision does not scale past a handful of concurrent runs.
• Fencing tokens on every lease. A partitioned worker that resumes must not write under an expired claim, and a monotonic fence in the run record makes that a rejected write rather than a silent double-execution.
• Coordination backend: Postgres advisory locks, or Redis if lease churn dominates. Redis is deferred until that churn is real.

Steps

1. Add to the run record: lease_owner, lease_expires_at, fence: u64.
2. Claim: atomic conditional update — claim only if unowned or expired. Increment fence on every successful claim. The fence is monotonic per run.
3. The worker carries its fence value in memory and includes it on every write. Writes compare against the stored fence and are rejected when lower.
4. Heartbeat renews lease_expires_at on an interval well inside the TTL. A failed renewal cancels the local RunScope (T1.1) rather than continuing optimistically.
5. On transition to Suspended, release the lease outright — clear the owner and stop heartbeating. On resume, re-claim, which takes a new fence.
6. Reclaim path: a run whose lease expired is claimable by any worker.
7. Test the full sequence: partition, expiry, reclaim elsewhere, reconnect the original, assert its writes are rejected by fence.

Acceptance

• Partition a worker, let its lease expire, reclaim the run elsewhere, then reconnect the original — its writes are rejected by fence, not merely late.

Verify

Harness: two worker processes against one Postgres, plus turmoil or an iptables-style partition for the network half. tokio::time::pause cannot be used across processes — use a short real TTL instead.

Integration test — tests/it_leases_fencing.rs:

1. Worker A claims run R (fence = 1) and begins executing.
2. Partition A from the database. Let the lease expire.
3. Worker B reclaims R; assert its fence is 2.
4. Heal the partition. Worker A attempts a write under fence 1.
5. Assert the write is rejected by the fence check — and assert the rejection reason is the fence, not a timestamp comparison and not a generic conflict.
6. Assert R completed exactly once: check the external side-effect ledger for a single entry.
7. Assert A's failed heartbeat cancelled its local RunScope, so it stopped working rather than continuing optimistically.
8. Suspended: transition a run to Suspended; assert the lease is released (owner cleared) and that no heartbeat is emitted while suspended. Resume and assert a new, higher fence.
9. Monotonicity: 100 claim/expire cycles; assert the fence never decreases or repeats.

Command: cargo test -p distribution leases -- --test-threads=1

False pass:

• Step 5 satisfied by a timestamp check ("your lease expired"). Under clock skew that check can pass for a stale worker; the fence comparison is what makes it clock-independent. Assert the reason.
• Step 6 omitted: both workers can execute, both can be "correct", and the double-execution is only visible in the external ledger.
• A per-worker fence counter rather than per-run monotonic — step 9 with two workers alternating catches it.
• Step 8 verified by reading a flag rather than asserting heartbeats stop.

Traps

• Checking only lease_expires_at at write time. Clock skew makes "not expired yet" a lie, and the fence is what makes the check independent of clocks.
• Heartbeating through Suspended. It burns a worker slot for the duration of a human decision.
• A per-worker fence rather than per-run monotonic. Two workers then produce incomparable tokens.

---

Background (not required to do this task): rust-agentic-sys.md §5.2, §7, §9.4 · rust-agentic-task.md