poimen/tasks/T7.6-tournament-as-a-join-stage.md

T7.6 — Tournament as a join stage

Field	Value
Phase	P7 — Distribution
Size	L — over 3 days
Status	Not started
Flags	—
Spec	inlined below
Blocks	—

Goal

Repartition from the run key to the group key, close groups on a completeness trigger, and cap burst concurrency so grading cannot starve execution.

Facts (inlined — no spec read needed)

 ingest ──► execute ──► verify ──► tournament ──► aggregate ──► decide
   │           │           │           │              │            │
 (T,Run)    (T,Run)     (T,Run)  (T,Task,Class,   (T,Variant)  (T,Workflow)
                                     Epoch)
                                       ▲               ▲            ▲
                                   shuffle 1       shuffle 2   single writer

• Three keys, two shuffles, one single-writer stage. Everything up to verification keys on (TenantId, RunId) and is embarrassingly parallel.
• Grading is a join — a comparison group must be co-located — keyed on (TenantId, TaskId, VerifierOutcome, GroupEpoch): the outcome class because brackets exist only within one, and the epoch because a closed group never reopens for a late arrival.
• The decide stage is single-writer per (TenantId, WorkflowId); a CAS on a generation counter is sufficient (T6.1).
• A group closes on quorum or timeout, grading whatever arrived, with group size attached to the confidence interval (T5.11).
• Grading yields to agent work. When both contend, agent inference wins and grading queues — a framework that lets a judge call delay the work it is judging has inverted its first principle.
• Tournament and reduction backlogs are the two lags that grow silently, so both are metered.

Steps

1. Implement the repartition from (TenantId, RunId) to (TenantId, TaskId, VerifierOutcome, GroupEpoch) after verification.
2. Maintain group state per key: members, open/closed, quorum target, deadline.
3. Completeness trigger: close on quorum reached or deadline passed. Increment GroupEpoch at closure; route later arrivals to the next epoch.
4. Cap concurrent tournament work with a semaphore sized independently of the executor's. Make the cap and its saturation observable in metrics.
5. Give agent inference priority over grading when both want the same resident model (enforced fully in T8.6).
6. Emit stage-boundary lag for the tournament stage.
7. Test with a 64-episode group while runs are executing, asserting execution throughput is not degraded past a bound.

Acceptance

• A 64-episode group does not starve run execution.
• The cap is observable in metrics.

Verify

Harness: a load generator producing runs continuously while a large group fills, so starvation is observable rather than theoretical.

Integration test — tests/it_tournament_join.rs:

1. Establish a baseline: run-execution throughput and agent-call latency with no grading load.
2. Trigger a 64-episode group. Assert execution throughput and agent latency stay within a configured bound of baseline — assert on the numbers, not on the absence of an error.
3. Assert the concurrency cap is observable in metrics: read the saturation gauge and assert it reflects the burst.
4. Group key: assert repartition uses all four components. Feed episodes differing only in VerifierOutcome; assert they land in different groups. Repeat for GroupEpoch.
5. Closure: a group that never reaches quorum closes on timeout, with size recorded, and GroupEpoch increments.
6. Late arrival after closure lands in epoch N+1; epoch N's stored strengths are unchanged.
7. Single-writer decide stage: two schedulers attempt to adjust allocation for one (TenantId, WorkflowId); assert CAS behaviour (cross-check T6.1).
8. Emit and assert stage-boundary lag for the tournament stage.

Command: cargo test -p distribution tournament_join -- --test-threads=1

False pass:

• Step 2 asserting only that both finished. Grading that delays agent work still finishes — the latency and throughput comparison against baseline is the test, because the principle being enforced is a priority, not a liveness property.
• Sharing the executor's concurrency limiter with grading: step 2 may still pass under light load and the cap becomes invisible. Step 3 is the guard.
• Step 4 omitted: dropping VerifierOutcome from the key puts passes and failures in one bracket, which produces plausible results and violates the one rule with no exemptions.

Traps

• Omitting VerifierOutcome from the group key, which puts passes and failures in one bracket — the one rule with no exemptions.
• Omitting GroupEpoch, which lets a late arrival reopen a published fit.
• Sharing the executor's concurrency limiter with grading. Then a burst of grading is indistinguishable from a burst of work.

---

Background (not required to do this task): rust-agentic-sys.md §9.1, §11.6, §11.8, §14.2, §15 · rust-agentic-task.md