poimen/tasks/T7.6-tournament-as-a-join-stage.md

# T7.6 — Tournament as a join stage

| Field | Value |
|---|---|
| Phase | P7 — Distribution |
| Size | L — over 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |

## Goal

Repartition from the run key to the group key, close groups on a completeness
trigger, and cap burst concurrency so grading cannot starve execution.

## Facts (inlined — no spec read needed)

```
 ingest ──► execute ──► verify ──► tournament ──► aggregate ──► decide
   │           │           │           │              │            │
 (T,Run)    (T,Run)     (T,Run)  (T,Task,Class,   (T,Variant)  (T,Workflow)
                                     Epoch)
                                       ▲               ▲            ▲
                                   shuffle 1       shuffle 2   single writer
```

- Three keys, two shuffles, one single-writer stage. Everything up to
  verification keys on `(TenantId, RunId)` and is embarrassingly parallel.
- **Grading is a join** — a comparison group must be co-located — keyed on
  `(TenantId, TaskId, VerifierOutcome, GroupEpoch)`: the **outcome class**
  because brackets exist only within one, and the **epoch** because a closed
  group never reopens for a late arrival.
- The decide stage is single-writer per `(TenantId, WorkflowId)`; a CAS on a
  generation counter is sufficient (T6.1).
- A group closes on **quorum or timeout**, grading whatever arrived, with group
  size attached to the confidence interval (T5.11).
- **Grading yields to agent work.** When both contend, agent inference wins and
  grading queues — a framework that lets a judge call delay the work it is
  judging has inverted its first principle.
- Tournament and reduction backlogs are the two lags that grow silently, so both
  are metered.

## Steps

1. Implement the repartition from `(TenantId, RunId)` to
   `(TenantId, TaskId, VerifierOutcome, GroupEpoch)` after verification.
2. Maintain group state per key: members, open/closed, quorum target, deadline.
3. Completeness trigger: close on quorum reached **or** deadline passed.
   Increment `GroupEpoch` at closure; route later arrivals to the next epoch.
4. Cap concurrent tournament work with a semaphore sized independently of the
   executor's. Make the cap and its saturation **observable in metrics**.
5. Give agent inference priority over grading when both want the same resident
   model (enforced fully in T8.6).
6. Emit stage-boundary lag for the tournament stage.
7. Test with a 64-episode group while runs are executing, asserting execution
   throughput is not degraded past a bound.

## Acceptance

- A 64-episode group **does not starve run execution**.
- The cap is **observable in metrics**.

## Verify

**Harness:** a load generator producing runs continuously while a large group
fills, so starvation is observable rather than theoretical.

**Integration test** — `tests/it_tournament_join.rs`:
1. Establish a baseline: run-execution throughput and agent-call latency with no
   grading load.
2. Trigger a **64-episode group**. Assert execution throughput and agent latency
   stay within a configured bound of baseline — assert on the **numbers**, not on
   the absence of an error.
3. Assert the concurrency cap is **observable in metrics**: read the saturation
   gauge and assert it reflects the burst.
4. Group key: assert repartition uses all four components. Feed episodes
   differing only in `VerifierOutcome`; assert they land in **different** groups.
   Repeat for `GroupEpoch`.
5. Closure: a group that never reaches quorum closes on **timeout**, with size
   recorded, and `GroupEpoch` increments.
6. Late arrival after closure lands in epoch N+1; epoch N's stored strengths are
   unchanged.
7. Single-writer decide stage: two schedulers attempt to adjust allocation for
   one `(TenantId, WorkflowId)`; assert CAS behaviour (cross-check T6.1).
8. Emit and assert stage-boundary lag for the tournament stage.

**Command:** `cargo test -p distribution tournament_join -- --test-threads=1`

**False pass:**
- Step 2 asserting only that both finished. Grading that delays agent work still
  finishes — the latency and throughput comparison against baseline is the test,
  because the principle being enforced is a priority, not a liveness property.
- Sharing the executor's concurrency limiter with grading: step 2 may still pass
  under light load and the cap becomes invisible. Step 3 is the guard.
- Step 4 omitted: dropping `VerifierOutcome` from the key puts passes and
  failures in one bracket, which produces plausible results and violates the one
  rule with no exemptions.

## Traps

- Omitting `VerifierOutcome` from the group key, which puts passes and failures
  in one bracket — the one rule with no exemptions.
- Omitting `GroupEpoch`, which lets a late arrival reopen a published fit.
- Sharing the executor's concurrency limiter with grading. Then a burst of
  grading is indistinguishable from a burst of work.

---

Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §9.1, §11.6, §11.8, §14.2, §15 ·
[rust-agentic-task.md](../../../rust-agentic-task.md)