# T7.6 — Tournament as a join stage | Field | Value | |---|---| | Phase | P7 — Distribution | | Size | L — over 3 days | | Status | Not started | | Flags | — | | Spec | inlined below | | Blocks | — | ## Goal Repartition from the run key to the group key, close groups on a completeness trigger, and cap burst concurrency so grading cannot starve execution. ## Facts (inlined — no spec read needed) ``` ingest ──► execute ──► verify ──► tournament ──► aggregate ──► decide │ │ │ │ │ │ (T,Run) (T,Run) (T,Run) (T,Task,Class, (T,Variant) (T,Workflow) Epoch) ▲ ▲ ▲ shuffle 1 shuffle 2 single writer ``` - Three keys, two shuffles, one single-writer stage. Everything up to verification keys on `(TenantId, RunId)` and is embarrassingly parallel. - **Grading is a join** — a comparison group must be co-located — keyed on `(TenantId, TaskId, VerifierOutcome, GroupEpoch)`: the **outcome class** because brackets exist only within one, and the **epoch** because a closed group never reopens for a late arrival. - The decide stage is single-writer per `(TenantId, WorkflowId)`; a CAS on a generation counter is sufficient (T6.1). - A group closes on **quorum or timeout**, grading whatever arrived, with group size attached to the confidence interval (T5.11). - **Grading yields to agent work.** When both contend, agent inference wins and grading queues — a framework that lets a judge call delay the work it is judging has inverted its first principle. - Tournament and reduction backlogs are the two lags that grow silently, so both are metered. ## Steps 1. Implement the repartition from `(TenantId, RunId)` to `(TenantId, TaskId, VerifierOutcome, GroupEpoch)` after verification. 2. Maintain group state per key: members, open/closed, quorum target, deadline. 3. Completeness trigger: close on quorum reached **or** deadline passed. Increment `GroupEpoch` at closure; route later arrivals to the next epoch. 4. Cap concurrent tournament work with a semaphore sized independently of the executor's. Make the cap and its saturation **observable in metrics**. 5. Give agent inference priority over grading when both want the same resident model (enforced fully in T8.6). 6. Emit stage-boundary lag for the tournament stage. 7. Test with a 64-episode group while runs are executing, asserting execution throughput is not degraded past a bound. ## Acceptance - A 64-episode group **does not starve run execution**. - The cap is **observable in metrics**. ## Verify **Harness:** a load generator producing runs continuously while a large group fills, so starvation is observable rather than theoretical. **Integration test** — `tests/it_tournament_join.rs`: 1. Establish a baseline: run-execution throughput and agent-call latency with no grading load. 2. Trigger a **64-episode group**. Assert execution throughput and agent latency stay within a configured bound of baseline — assert on the **numbers**, not on the absence of an error. 3. Assert the concurrency cap is **observable in metrics**: read the saturation gauge and assert it reflects the burst. 4. Group key: assert repartition uses all four components. Feed episodes differing only in `VerifierOutcome`; assert they land in **different** groups. Repeat for `GroupEpoch`. 5. Closure: a group that never reaches quorum closes on **timeout**, with size recorded, and `GroupEpoch` increments. 6. Late arrival after closure lands in epoch N+1; epoch N's stored strengths are unchanged. 7. Single-writer decide stage: two schedulers attempt to adjust allocation for one `(TenantId, WorkflowId)`; assert CAS behaviour (cross-check T6.1). 8. Emit and assert stage-boundary lag for the tournament stage. **Command:** `cargo test -p distribution tournament_join -- --test-threads=1` **False pass:** - Step 2 asserting only that both finished. Grading that delays agent work still finishes — the latency and throughput comparison against baseline is the test, because the principle being enforced is a priority, not a liveness property. - Sharing the executor's concurrency limiter with grading: step 2 may still pass under light load and the cap becomes invisible. Step 3 is the guard. - Step 4 omitted: dropping `VerifierOutcome` from the key puts passes and failures in one bracket, which produces plausible results and violates the one rule with no exemptions. ## Traps - Omitting `VerifierOutcome` from the group key, which puts passes and failures in one bracket — the one rule with no exemptions. - Omitting `GroupEpoch`, which lets a late arrival reopen a published fit. - Sharing the executor's concurrency limiter with grading. Then a burst of grading is indistinguishable from a burst of work. --- Background (not required to do this task): [rust-agentic-sys.md](../../../rust-agentic-sys.md) §9.1, §11.6, §11.8, §14.2, §15 · [rust-agentic-task.md](../../../rust-agentic-task.md)