114 lines
5.2 KiB
Markdown
114 lines
5.2 KiB
Markdown
# T5.10 — Attempt tournaments
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | P5 — Grading |
|
|
| Size | M — 1 to 3 days |
|
|
| Status | Not started |
|
|
| Flags | — |
|
|
| Spec | inlined below |
|
|
| Blocks | — |
|
|
|
|
## Goal
|
|
|
|
Grade the attempts of one step as a comparison group. On by default, even though
|
|
T5.8/T5.9 are not, because it costs no extra agent runs.
|
|
|
|
## Facts (inlined — no spec read needed)
|
|
|
|
Where comparison groups come from:
|
|
|
|
| Source | Group | Cost | Needs |
|
|
|---|---|---|---|
|
|
| **Attempts** | attempts 1..N of one step, identical context | free, already recorded | nothing |
|
|
| **Recurring tasks** | runs sharing a `TaskId` over time | free, slow to fill | `TaskId` |
|
|
| **Replay** | one task re-executed under N variants | N full runs | blobs, sandbox |
|
|
|
|
- **Attempt tournaments first.** Retries are already on disk and attempt N+1
|
|
never mutates attempt N, so the attempts of one step are a group on identical
|
|
context — the cheapest per-step credit signal available, and the **only** source
|
|
of per-step credit the system has.
|
|
- **Two paths, not one:**
|
|
- **Same-outcome attempts** go to the judge. The question there is "which
|
|
failure got further" — which no verifier can answer.
|
|
- **A fail→pass pair is never judged.** The verifier has already ordered it;
|
|
asking a judge which is better asks it to re-decide what verification
|
|
decided. The pair is consumed **structurally**: what differed between attempt
|
|
N and N+1 — context partition, tool selection, prompt — is attributed to the
|
|
`StepId` as the change that turned a fail into a pass.
|
|
- The bracketing rule has **no exemptions**. If a pairing crosses an outcome
|
|
class, it is evidence for attribution, never input to a judge. This is the
|
|
place the rule is easiest to break, because the two attempts sit side by side
|
|
on disk and look like a free comparison.
|
|
- Per-step credit attributes to `StepId`, which is why the `StepId` stability
|
|
contract (T3.4) is load-bearing rather than cosmetic.
|
|
|
|
## Steps
|
|
|
|
1. Build the group: all attempts of one `(BranchKey, StepId)`, partitioned by
|
|
verifier outcome class.
|
|
2. **Route by class before any judge call.** Same-class subsets go to the
|
|
comparison path; cross-class pairs go to the attribution path. Make this a
|
|
single branch point so it cannot be bypassed.
|
|
3. Comparison path: hand the same-class subset to the configured strategy.
|
|
4. Attribution path: diff attempt N against N+1 across context partition, tool
|
|
selection and prompt ref; emit the delta attributed to `StepId`. No judge call.
|
|
5. Persist per-step credit keyed by `(TenantId, WorkflowVersion, StepId)`.
|
|
6. Instrument the `Judge::compare` counter and assert **zero** cross-class calls —
|
|
count calls rather than inspecting pairs, since a pair-inspection test passes
|
|
against an implementation that builds the pair and then filters it late.
|
|
|
|
## Acceptance
|
|
|
|
- A run with three attempts produces per-step credit attributed to `StepId`.
|
|
- A group containing both a failed and a passed attempt issues **zero
|
|
`Judge::compare` calls across the outcome boundary** — asserted by counting
|
|
calls, not by inspecting pairs.
|
|
|
|
## Verify
|
|
|
|
**Harness:** a run with three attempts, plus a counting mock judge. The counter
|
|
is the instrument — pair inspection is not.
|
|
|
|
**Integration test** — `tests/it_attempt_tournament.rs`:
|
|
1. Three attempts on one step, **all failing**. Assert `compare` is called on the
|
|
same-outcome subset and per-step credit is attributed to the `StepId`.
|
|
2. **Mixed group — the guard:** attempts 1 and 2 fail, attempt 3 passes. Assert
|
|
**zero** `Judge::compare` calls cross the outcome boundary, counted at the
|
|
judge. Assert the fail→pass pair produced a **structural attribution** record
|
|
instead.
|
|
3. Assert the attribution names the actual delta — plant a context-partition
|
|
difference between attempts 2 and 3 and assert it appears in the record.
|
|
4. Assert credit is keyed `(TenantId, WorkflowVersion, StepId)` and survives a
|
|
workflow version bump that retains the `StepId` (T3.4).
|
|
5. Assert this path runs **with the tournament flags off** — it is on by default
|
|
while T5.8/T5.9 are not.
|
|
6. Zero-cost assertion: no agent runs are spawned by grading; assert the run
|
|
counter is unchanged.
|
|
7. Single-attempt step: assert no comparison and no spurious credit record.
|
|
|
|
**Command:** `cargo test -p grading attempt_tournament`
|
|
|
|
**False pass:**
|
|
- **Inspecting the pair list rather than counting judge calls.** An
|
|
implementation that builds the cross-class pair and filters it just before
|
|
dispatch passes pair inspection and still sends it under a later refactor.
|
|
Count at the judge boundary.
|
|
- Step 2 written with all-failing attempts, where no boundary exists to cross.
|
|
The mixed group is the whole test.
|
|
- Step 3 omitted: emitting an empty attribution record satisfies "credit
|
|
produced" while carrying no signal.
|
|
|
|
## Traps
|
|
|
|
- Pairing the failure with its successful retry. It is free, it looks like
|
|
signal, and it makes the judge re-decide what the verifier already decided.
|
|
- Attributing credit to step index rather than `StepId`, which breaks at the next
|
|
workflow version.
|
|
|
|
---
|
|
|
|
Background (not required to do this task):
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §11.4, §11.5, §11.7, §11.8 ·
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|