# T5.10 — Attempt tournaments

| Field | Value |
|---|---|
| Phase | P5 — Grading |
| Size | M — 1 to 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |

## Goal

Grade the attempts of one step as a comparison group. On by default, even though
T5.8/T5.9 are not, because it costs no extra agent runs.

## Facts (inlined — no spec read needed)

Where comparison groups come from:

| Source | Group | Cost | Needs |
|---|---|---|---|
| **Attempts** | attempts 1..N of one step, identical context | free, already recorded | nothing |
| **Recurring tasks** | runs sharing a `TaskId` over time | free, slow to fill | `TaskId` |
| **Replay** | one task re-executed under N variants | N full runs | blobs, sandbox |

- **Attempt tournaments first.** Retries are already on disk and attempt N+1
  never mutates attempt N, so the attempts of one step are a group on identical
  context — the cheapest per-step credit signal available, and the **only** source
  of per-step credit the system has.
- **Two paths, not one:**
  - **Same-outcome attempts** go to the judge. The question there is "which
    failure got further" — which no verifier can answer.
  - **A fail→pass pair is never judged.** The verifier has already ordered it;
    asking a judge which is better asks it to re-decide what verification
    decided. The pair is consumed **structurally**: what differed between attempt
    N and N+1 — context partition, tool selection, prompt — is attributed to the
    `StepId` as the change that turned a fail into a pass.
- The bracketing rule has **no exemptions**. If a pairing crosses an outcome
  class, it is evidence for attribution, never input to a judge. This is the
  place the rule is easiest to break, because the two attempts sit side by side
  on disk and look like a free comparison.
- Per-step credit attributes to `StepId`, which is why the `StepId` stability
  contract (T3.4) is load-bearing rather than cosmetic.

## Steps

1. Build the group: all attempts of one `(BranchKey, StepId)`, partitioned by
   verifier outcome class.
2. **Route by class before any judge call.** Same-class subsets go to the
   comparison path; cross-class pairs go to the attribution path. Make this a
   single branch point so it cannot be bypassed.
3. Comparison path: hand the same-class subset to the configured strategy.
4. Attribution path: diff attempt N against N+1 across context partition, tool
   selection and prompt ref; emit the delta attributed to `StepId`. No judge call.
5. Persist per-step credit keyed by `(TenantId, WorkflowVersion, StepId)`.
6. Instrument the `Judge::compare` counter and assert **zero** cross-class calls —
   count calls rather than inspecting pairs, since a pair-inspection test passes
   against an implementation that builds the pair and then filters it late.

## Acceptance

- A run with three attempts produces per-step credit attributed to `StepId`.
- A group containing both a failed and a passed attempt issues **zero
  `Judge::compare` calls across the outcome boundary** — asserted by counting
  calls, not by inspecting pairs.

## Verify

**Harness:** a run with three attempts, plus a counting mock judge. The counter
is the instrument — pair inspection is not.

**Integration test** — `tests/it_attempt_tournament.rs`:
1. Three attempts on one step, **all failing**. Assert `compare` is called on the
   same-outcome subset and per-step credit is attributed to the `StepId`.
2. **Mixed group — the guard:** attempts 1 and 2 fail, attempt 3 passes. Assert
   **zero** `Judge::compare` calls cross the outcome boundary, counted at the
   judge. Assert the fail→pass pair produced a **structural attribution** record
   instead.
3. Assert the attribution names the actual delta — plant a context-partition
   difference between attempts 2 and 3 and assert it appears in the record.
4. Assert credit is keyed `(TenantId, WorkflowVersion, StepId)` and survives a
   workflow version bump that retains the `StepId` (T3.4).
5. Assert this path runs **with the tournament flags off** — it is on by default
   while T5.8/T5.9 are not.
6. Zero-cost assertion: no agent runs are spawned by grading; assert the run
   counter is unchanged.
7. Single-attempt step: assert no comparison and no spurious credit record.

**Command:** `cargo test -p grading attempt_tournament`

**False pass:**
- **Inspecting the pair list rather than counting judge calls.** An
  implementation that builds the cross-class pair and filters it just before
  dispatch passes pair inspection and still sends it under a later refactor.
  Count at the judge boundary.
- Step 2 written with all-failing attempts, where no boundary exists to cross.
  The mixed group is the whole test.
- Step 3 omitted: emitting an empty attribution record satisfies "credit
  produced" while carrying no signal.

## Traps

- Pairing the failure with its successful retry. It is free, it looks like
  signal, and it makes the judge re-decide what the verifier already decided.
- Attributing credit to step index rather than `StepId`, which breaks at the next
  workflow version.

---

Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §11.4, §11.5, §11.7, §11.8 ·
[rust-agentic-task.md](../../../rust-agentic-task.md)