poimen/tasks/T5.11-degradation-reasons.md

# T5.11 — Degradation reasons

| Field | Value |
|---|---|
| Phase | P5 — Grading |
| Size | M — 1 to 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |

## Goal

When no comparison is possible, emit `Score::Ungraded { reason }` — a stated
reason, never a neutral score — and reach terminal `Ungraded`.

## Facts (inlined — no spec read needed)

Three reasons, each a distinct condition:

1. **`NoReference`** — a novel `TaskId`. Nothing else has done this task, so
   there is nothing to compare against. That is not a degradation to engineer
   around; it is the first observation of a new task. Verifier outcome and
   deterministic dimensions (cost, latency, tool efficiency) are still recorded,
   and **the episode is retained as the reference for that `TaskId`** — so the
   second run on that task grades normally.
2. **`BudgetExhausted`** — the tenant is over its grading ceiling. Not a quality
   signal; a spend signal.
3. **`InsufficientGroup`** — **tournament only.** A group closes on quorum *or*
   on a timeout, grading whatever arrived, with group size attached to the
   confidence interval.

- **A closed group is immutable, and the next episode starts a new one.** A group
  closed on timeout at G = 3, then a fourth episode with the same `TaskId`
  arrives an hour later: re-opening and re-fitting is the wrong answer, because
  strengths from that group have already been published, aggregated and possibly
  acted on. A fit that silently changes underneath a decision already made is
  worse than a small group.
- So the group key carries a generation: `(TenantId, TaskId, VerifierOutcome,
  GroupEpoch)`. **Closure increments the epoch**; late arrivals accumulate into
  the next one.
- The cost is honest and should be stated rather than discovered: a low-volume
  tenant with a long inter-arrival time gets a run of G = 1 epochs, each
  `InsufficientGroup`. That is a real signal about their volume; the fix is a
  longer timeout, which is a tenant-level tradeoff between waiting and grading.
- **A tenant whose loop never engages must see that as a stated reason.** Silent
  no-op is the worst outcome: it looks like a working loop that finds no
  improvements.
- Every one of these runs reaches terminal `Ungraded` and is therefore reducible
  (T4.4). Without that, they rest in `Grading` forever and strand storage.
- Group timeout default is unsettled — it trades grading latency against group
  size and depends on tenant arrival rate. Only bites when the tournament is on.

## Steps

1. Define `UngradedReason { NoReference, BudgetExhausted, InsufficientGroup { group_size } }`.
2. `NoReference` path: on a novel `TaskId`, emit `Ungraded`, record the
   deterministic dimensions, and **register the episode as that task's
   reference**. The registration is the half that is easy to omit.
3. `BudgetExhausted` path: check the tenant ceiling at the group boundary, before
   any model call. Emit the reason; never a default score.
4. `InsufficientGroup` path (tournament only): implement quorum-or-timeout
   closure, attach `group_size`, and **increment `GroupEpoch` on close**.
5. Route a late arrival into epoch N+1. Epoch N's stored strengths are never
   recomputed.
6. Transition the run to terminal `Ungraded` in every case, carrying the reason.
7. Emit `Ungraded` counts **by reason** as a metric, separately from `Graded`.

## Acceptance

- A novel `TaskId` yields `NoReference` **and** the episode is retained as that
  task's reference, so the **second** run on the same task produces `Relative` —
  assert the second run, since retaining the reference is the half easily omitted.
- A tenant over its ceiling yields `BudgetExhausted`, never a default score.
- Tournament: a group that never fills closes on timeout with size recorded; a
  late episode lands in epoch N+1 leaving epoch N's fitted strengths
  **byte-identical** — assert on the stored strengths, not on the absence of a
  re-fit call.
- In every case the run reaches terminal `Ungraded` and is therefore reducible
  (T4.4).

## Phase gate

P5 closes when `PairwiseSequential` decides an accept and a reject against a mock
judge, on one resident model with zero swaps.

## Verify

**Harness:** a fresh tenant with no history (for `NoReference`), a tenant with a
zero grading ceiling (for `BudgetExhausted`), and a tournament group that never
fills (for `InsufficientGroup`).

**Integration test** — `tests/it_degradation_reasons.rs`:
1. **Novel `TaskId`:** assert `Score::Ungraded { NoReference }`, and that
   deterministic dimensions (cost, latency) are still recorded.
2. **The second half, which is the one that gets omitted:** run a **second**
   episode on the same `TaskId`; assert it produces `Score::Relative`. That only
   works if run 1's episode was retained as the reference.
3. **Budget:** tenant over its grading ceiling → `BudgetExhausted`. Assert
   **zero** model calls were made — the check must precede the spend, not follow
   it. Assert no default score anywhere in the output.
4. **Tournament group:** a group that never reaches quorum closes on timeout.
   Assert `InsufficientGroup { group_size }` with the real size recorded.
5. **Epoch immutability:** snapshot epoch N's fitted strengths. Deliver a late
   episode on the same `TaskId`. Assert it lands in epoch **N+1** and that epoch
   N's **stored strengths are byte-identical** to the snapshot.
6. **Terminal state:** for every one of the three reasons, assert the run reaches
   terminal `Ungraded` and that T4.4's `is_reducible` returns true.
7. Assert `Ungraded` counts are emitted **by reason** as separate metric series.

**Command:** `cargo test -p grading degradation`

**False pass:**
- Step 5 asserting "no re-fit call was made". A refactor that recomputes lazily
  on read passes that and still changes a published number. Assert on the
  **stored strengths**.
- Step 2 omitted: emitting `NoReference` is trivial; **retaining the reference**
  is the half that makes the loop ever engage, and nothing else detects its
  absence.
- Step 6 omitted: a run left in `Grading` looks correct in the score output and
  strands storage forever for exactly the tenants this section exists to
  accommodate.
- Step 3 checking only the reason. If the ceiling is enforced after the calls,
  the reason is right and the money is gone.

## Traps

- Emitting a neutral 0.5 for an ungradable episode. It averages into promotion
  gates and looks like data.
- Re-opening a closed group for a late arrival.
- Leaving the run in `Grading`, which strands storage for exactly the low-volume
  and cost-capped tenants least able to absorb it.

---

Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §5.2, §11.6, §14, §15, §18 ·
[rust-agentic-task.md](../../../rust-agentic-task.md)