Files
poimen/tasks/T6.1-challenger-allocation.md
T

4.5 KiB

T6.1 — Challenger allocation

Field Value
Phase P6 — Learning loop
Size M — 1 to 3 days
Status Not started
Flags
Spec inlined below
Blocks T6.3

Goal

One current version, at most one challenger, 95/5 traffic split, guarded by a generation counter with compare-and-swap.

Facts (inlined — no spec read needed)

  • One current version, at most one challenger. The loop converges toward a single state rather than maintaining a population. That is the change that makes everything else affordable: no N-way traffic split, no per-variant aggregation across groups, no allocation state to contend over, one comparison per episode instead of a bracket.
  • The cost is exploration. A single challenger at a time is hill-climbing — it finds improvements more slowly than a population would, and it can sit in a local optimum indefinitely with nothing in the loop able to say so. The held-out report (T6.6) is the only instrument that will notice, which makes it more important here, not less.
  • The decide stage is single-writer per (TenantId, WorkflowId). Two schedulers adjusting traffic allocation concurrently produce an allocation neither holds. A compare-and-swap on a generation counter is sufficient; no lock service is needed at this size.
  • The slow generate loop fires when the fast loop rejects a challenger without finding a replacement — a trigger, not a timer.
  • Multi-variant selection stays available for deployments that enable the tournament and allow N challengers; the machinery is the same.

Steps

  1. Define the allocation record: (TenantId, WorkflowId) → { current: WorkflowVersion, challenger: Option<WorkflowVersion>, split, generation: u64 }.
  2. Registration of a challenger is rejected when one is already live — the error names the incumbent challenger, so the operator knows what to retire.
  3. Every update is a CAS on generation. The loser retries against the winner's generation rather than overwriting.
  4. Route spawns by the split: 95% current, 5% challenger. Record which version a run was allocated to on the spawn event.
  5. Emit the "rejected without replacement" trigger as an event so the generate loop can subscribe. Do not add a timer.
  6. Test two concurrent schedulers committing allocations.

Acceptance

  • Registering a second challenger while one is live is rejected, naming the incumbent challenger.
  • Two concurrent schedulers cannot both commit an allocation; the loser retries and observes the winner's generation.

Verify

Harness: two scheduler instances against one backend — real concurrency, not a simulated race.

Integration testtests/it_challenger_allocation.rs:

  1. Register challenger A. Register challenger B while A is live; assert rejected, and that the error names A.
  2. Concurrent CAS: two schedulers commit different allocations simultaneously, in a loop of 100 rounds. Assert exactly one wins per round, the loser retries, and the final generation equals the number of successful commits — no lost updates.
  3. Traffic split: spawn 1000 runs; assert the challenger share is within tolerance of 5% and that each run's spawn event records which version it was allocated to.
  4. Assert control appears in every allocation, so T6.4's anchor exists.
  5. Trigger: reject a challenger with no replacement; assert the generate-loop trigger event fires once. Assert no timer path exists that would fire it otherwise — run 60s of simulated time with no rejection and assert zero triggers.
  6. Restart mid-decision; assert the allocation and generation survive.

Command: cargo test -p loop allocation -- --test-threads=1

False pass:

  • Step 2 simulated by calling the update function twice sequentially. Last-write- wins passes that and loses an allocation under real concurrency. Use two live schedulers.
  • Step 3 asserting the split ratio without asserting the per-run recorded version. Runs allocated correctly but recorded wrongly are unattributable, and the ratio still looks right.
  • Step 5's negative half omitted, so a timer-driven generate loop passes.

Traps

  • Last-write-wins on the allocation record. The result is a split neither scheduler intended and no error anywhere.
  • A timer-driven generate loop. It proposes challengers when nothing has been learned.

Background (not required to do this task): rust-agentic-sys.md §9.1, §12.1 · rust-agentic-task.md