104 lines
4.5 KiB
Markdown
104 lines
4.5 KiB
Markdown
# T6.1 — Challenger allocation
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | P6 — Learning loop |
|
|
| Size | M — 1 to 3 days |
|
|
| Status | Not started |
|
|
| Flags | — |
|
|
| Spec | inlined below |
|
|
| Blocks | T6.3 |
|
|
|
|
## Goal
|
|
|
|
One current version, at most one challenger, 95/5 traffic split, guarded by a
|
|
generation counter with compare-and-swap.
|
|
|
|
## Facts (inlined — no spec read needed)
|
|
|
|
- **One current version, at most one challenger.** The loop converges toward a
|
|
single state rather than maintaining a population. That is the change that
|
|
makes everything else affordable: no N-way traffic split, no per-variant
|
|
aggregation across groups, no allocation state to contend over, one comparison
|
|
per episode instead of a bracket.
|
|
- The cost is exploration. A single challenger at a time is hill-climbing — it
|
|
finds improvements more slowly than a population would, and it can sit in a
|
|
local optimum indefinitely with nothing in the loop able to say so. The
|
|
held-out report (T6.6) is the only instrument that will notice, which makes it
|
|
**more** important here, not less.
|
|
- **The decide stage is single-writer per `(TenantId, WorkflowId)`.** Two
|
|
schedulers adjusting traffic allocation concurrently produce an allocation
|
|
neither holds. A compare-and-swap on a generation counter is sufficient; no
|
|
lock service is needed at this size.
|
|
- The slow generate loop fires when the fast loop **rejects a challenger without
|
|
finding a replacement** — a trigger, not a timer.
|
|
- Multi-variant selection stays available for deployments that enable the
|
|
tournament and allow N challengers; the machinery is the same.
|
|
|
|
## Steps
|
|
|
|
1. Define the allocation record: `(TenantId, WorkflowId) → { current: WorkflowVersion,
|
|
challenger: Option<WorkflowVersion>, split, generation: u64 }`.
|
|
2. Registration of a challenger is rejected when one is already live — the error
|
|
**names the incumbent challenger**, so the operator knows what to retire.
|
|
3. Every update is a CAS on `generation`. The loser retries against the winner's
|
|
generation rather than overwriting.
|
|
4. Route spawns by the split: 95% current, 5% challenger. Record which version a
|
|
run was allocated to on the spawn event.
|
|
5. Emit the "rejected without replacement" trigger as an event so the generate
|
|
loop can subscribe. Do not add a timer.
|
|
6. Test two concurrent schedulers committing allocations.
|
|
|
|
## Acceptance
|
|
|
|
- Registering a second challenger while one is live is **rejected, naming the
|
|
incumbent challenger**.
|
|
- Two concurrent schedulers cannot both commit an allocation; the loser retries
|
|
and observes the winner's generation.
|
|
|
|
## Verify
|
|
|
|
**Harness:** two scheduler instances against one backend — real concurrency, not
|
|
a simulated race.
|
|
|
|
**Integration test** — `tests/it_challenger_allocation.rs`:
|
|
1. Register challenger A. Register challenger B while A is live; assert
|
|
**rejected**, and that the error **names A**.
|
|
2. **Concurrent CAS:** two schedulers commit different allocations
|
|
simultaneously, in a loop of 100 rounds. Assert exactly one wins per round,
|
|
the loser retries, and the final generation equals the number of successful
|
|
commits — no lost updates.
|
|
3. Traffic split: spawn 1000 runs; assert the challenger share is within
|
|
tolerance of 5% and that each run's spawn event **records which version it was
|
|
allocated to**.
|
|
4. Assert control appears in every allocation, so T6.4's anchor exists.
|
|
5. Trigger: reject a challenger with no replacement; assert the generate-loop
|
|
trigger event fires **once**. Assert no timer path exists that would fire it
|
|
otherwise — run 60s of simulated time with no rejection and assert zero
|
|
triggers.
|
|
6. Restart mid-decision; assert the allocation and generation survive.
|
|
|
|
**Command:** `cargo test -p loop allocation -- --test-threads=1`
|
|
|
|
**False pass:**
|
|
- Step 2 simulated by calling the update function twice sequentially. Last-write-
|
|
wins passes that and loses an allocation under real concurrency. Use two live
|
|
schedulers.
|
|
- Step 3 asserting the split ratio without asserting the per-run recorded
|
|
version. Runs allocated correctly but recorded wrongly are unattributable, and
|
|
the ratio still looks right.
|
|
- Step 5's negative half omitted, so a timer-driven generate loop passes.
|
|
|
|
## Traps
|
|
|
|
- Last-write-wins on the allocation record. The result is a split neither
|
|
scheduler intended and no error anywhere.
|
|
- A timer-driven generate loop. It proposes challengers when nothing has been
|
|
learned.
|
|
|
|
---
|
|
|
|
Background (not required to do this task):
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §9.1, §12.1 ·
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|