(tasks) add tasks for harness

2026-08-17 23:05:20 -07:00
parent e2d678b118
commit 5a30d0ffc6
74 changed files with 8204 additions and 1 deletions
@@ -0,0 +1,103 @@
+# T6.1 — Challenger allocation
+
+| Field | Value |
+|---|---|
+| Phase | P6 — Learning loop |
+| Size | M — 1 to 3 days |
+| Status | Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | T6.3 |
+
+## Goal
+
+One current version, at most one challenger, 95/5 traffic split, guarded by a
+generation counter with compare-and-swap.
+
+## Facts (inlined — no spec read needed)
+
+- **One current version, at most one challenger.** The loop converges toward a
+  single state rather than maintaining a population. That is the change that
+  makes everything else affordable: no N-way traffic split, no per-variant
+  aggregation across groups, no allocation state to contend over, one comparison
+  per episode instead of a bracket.
+- The cost is exploration. A single challenger at a time is hill-climbing — it
+  finds improvements more slowly than a population would, and it can sit in a
+  local optimum indefinitely with nothing in the loop able to say so. The
+  held-out report (T6.6) is the only instrument that will notice, which makes it
+  **more** important here, not less.
+- **The decide stage is single-writer per `(TenantId, WorkflowId)`.** Two
+  schedulers adjusting traffic allocation concurrently produce an allocation
+  neither holds. A compare-and-swap on a generation counter is sufficient; no
+  lock service is needed at this size.
+- The slow generate loop fires when the fast loop **rejects a challenger without
+  finding a replacement** — a trigger, not a timer.
+- Multi-variant selection stays available for deployments that enable the
+  tournament and allow N challengers; the machinery is the same.
+
+## Steps
+
+1. Define the allocation record: `(TenantId, WorkflowId) → { current: WorkflowVersion,
+   challenger: Option<WorkflowVersion>, split, generation: u64 }`.
+2. Registration of a challenger is rejected when one is already live — the error
+   **names the incumbent challenger**, so the operator knows what to retire.
+3. Every update is a CAS on `generation`. The loser retries against the winner's
+   generation rather than overwriting.
+4. Route spawns by the split: 95% current, 5% challenger. Record which version a
+   run was allocated to on the spawn event.
+5. Emit the "rejected without replacement" trigger as an event so the generate
+   loop can subscribe. Do not add a timer.
+6. Test two concurrent schedulers committing allocations.
+
+## Acceptance
+
+- Registering a second challenger while one is live is **rejected, naming the
+  incumbent challenger**.
+- Two concurrent schedulers cannot both commit an allocation; the loser retries
+  and observes the winner's generation.
+
+## Verify
+
+**Harness:** two scheduler instances against one backend — real concurrency, not
+a simulated race.
+
+**Integration test** — `tests/it_challenger_allocation.rs`:
+1. Register challenger A. Register challenger B while A is live; assert
+   **rejected**, and that the error **names A**.
+2. **Concurrent CAS:** two schedulers commit different allocations
+   simultaneously, in a loop of 100 rounds. Assert exactly one wins per round,
+   the loser retries, and the final generation equals the number of successful
+   commits — no lost updates.
+3. Traffic split: spawn 1000 runs; assert the challenger share is within
+   tolerance of 5% and that each run's spawn event **records which version it was
+   allocated to**.
+4. Assert control appears in every allocation, so T6.4's anchor exists.
+5. Trigger: reject a challenger with no replacement; assert the generate-loop
+   trigger event fires **once**. Assert no timer path exists that would fire it
+   otherwise — run 60s of simulated time with no rejection and assert zero
+   triggers.
+6. Restart mid-decision; assert the allocation and generation survive.
+
+**Command:** `cargo test -p loop allocation -- --test-threads=1`
+
+**False pass:**
+- Step 2 simulated by calling the update function twice sequentially. Last-write-
+  wins passes that and loses an allocation under real concurrency. Use two live
+  schedulers.
+- Step 3 asserting the split ratio without asserting the per-run recorded
+  version. Runs allocated correctly but recorded wrongly are unattributable, and
+  the ratio still looks right.
+- Step 5's negative half omitted, so a timer-driven generate loop passes.
+
+## Traps
+
+- Last-write-wins on the allocation record. The result is a split neither
+  scheduler intended and no error anywhere.
+- A timer-driven generate loop. It proposes challengers when nothing has been
+  learned.
+
+---
+
+Background (not required to do this task):
+[rust-agentic-sys.md](../../../rust-agentic-sys.md) §9.1, §12.1 ·
+[rust-agentic-task.md](../../../rust-agentic-task.md)