poimen/tasks/T5.11-degradation-reasons.md

T5.11 — Degradation reasons

Field	Value
Phase	P5 — Grading
Size	M — 1 to 3 days
Status	Not started
Flags	—
Spec	inlined below
Blocks	—

Goal

When no comparison is possible, emit Score::Ungraded { reason } — a stated reason, never a neutral score — and reach terminal Ungraded.

Facts (inlined — no spec read needed)

Three reasons, each a distinct condition:

1. NoReference — a novel TaskId. Nothing else has done this task, so there is nothing to compare against. That is not a degradation to engineer around; it is the first observation of a new task. Verifier outcome and deterministic dimensions (cost, latency, tool efficiency) are still recorded, and the episode is retained as the reference for that TaskId — so the second run on that task grades normally.
2. BudgetExhausted — the tenant is over its grading ceiling. Not a quality signal; a spend signal.
3. InsufficientGroup — tournament only. A group closes on quorum or on a timeout, grading whatever arrived, with group size attached to the confidence interval.

• A closed group is immutable, and the next episode starts a new one. A group closed on timeout at G = 3, then a fourth episode with the same TaskId arrives an hour later: re-opening and re-fitting is the wrong answer, because strengths from that group have already been published, aggregated and possibly acted on. A fit that silently changes underneath a decision already made is worse than a small group.
• So the group key carries a generation: (TenantId, TaskId, VerifierOutcome, GroupEpoch). Closure increments the epoch; late arrivals accumulate into the next one.
• The cost is honest and should be stated rather than discovered: a low-volume tenant with a long inter-arrival time gets a run of G = 1 epochs, each InsufficientGroup. That is a real signal about their volume; the fix is a longer timeout, which is a tenant-level tradeoff between waiting and grading.
• A tenant whose loop never engages must see that as a stated reason. Silent no-op is the worst outcome: it looks like a working loop that finds no improvements.
• Every one of these runs reaches terminal Ungraded and is therefore reducible (T4.4). Without that, they rest in Grading forever and strand storage.
• Group timeout default is unsettled — it trades grading latency against group size and depends on tenant arrival rate. Only bites when the tournament is on.

Steps

1. Define UngradedReason { NoReference, BudgetExhausted, InsufficientGroup { group_size } }.
2. NoReference path: on a novel TaskId, emit Ungraded, record the deterministic dimensions, and register the episode as that task's reference. The registration is the half that is easy to omit.
3. BudgetExhausted path: check the tenant ceiling at the group boundary, before any model call. Emit the reason; never a default score.
4. InsufficientGroup path (tournament only): implement quorum-or-timeout closure, attach group_size, and increment GroupEpoch on close.
5. Route a late arrival into epoch N+1. Epoch N's stored strengths are never recomputed.
6. Transition the run to terminal Ungraded in every case, carrying the reason.
7. Emit Ungraded counts by reason as a metric, separately from Graded.

Acceptance

• A novel TaskId yields NoReference and the episode is retained as that task's reference, so the second run on the same task produces Relative — assert the second run, since retaining the reference is the half easily omitted.
• A tenant over its ceiling yields BudgetExhausted, never a default score.
• Tournament: a group that never fills closes on timeout with size recorded; a late episode lands in epoch N+1 leaving epoch N's fitted strengths byte-identical — assert on the stored strengths, not on the absence of a re-fit call.
• In every case the run reaches terminal Ungraded and is therefore reducible (T4.4).

Phase gate

P5 closes when PairwiseSequential decides an accept and a reject against a mock judge, on one resident model with zero swaps.

Verify

Harness: a fresh tenant with no history (for NoReference), a tenant with a zero grading ceiling (for BudgetExhausted), and a tournament group that never fills (for InsufficientGroup).

Integration test — tests/it_degradation_reasons.rs:

1. Novel TaskId: assert Score::Ungraded { NoReference }, and that deterministic dimensions (cost, latency) are still recorded.
2. The second half, which is the one that gets omitted: run a second episode on the same TaskId; assert it produces Score::Relative. That only works if run 1's episode was retained as the reference.
3. Budget: tenant over its grading ceiling → BudgetExhausted. Assert zero model calls were made — the check must precede the spend, not follow it. Assert no default score anywhere in the output.
4. Tournament group: a group that never reaches quorum closes on timeout. Assert InsufficientGroup { group_size } with the real size recorded.
5. Epoch immutability: snapshot epoch N's fitted strengths. Deliver a late episode on the same TaskId. Assert it lands in epoch N+1 and that epoch N's stored strengths are byte-identical to the snapshot.
6. Terminal state: for every one of the three reasons, assert the run reaches terminal Ungraded and that T4.4's is_reducible returns true.
7. Assert Ungraded counts are emitted by reason as separate metric series.

Command: cargo test -p grading degradation

False pass:

• Step 5 asserting "no re-fit call was made". A refactor that recomputes lazily on read passes that and still changes a published number. Assert on the stored strengths.
• Step 2 omitted: emitting NoReference is trivial; retaining the reference is the half that makes the loop ever engage, and nothing else detects its absence.
• Step 6 omitted: a run left in Grading looks correct in the score output and strands storage forever for exactly the tenants this section exists to accommodate.
• Step 3 checking only the reason. If the ceiling is enforced after the calls, the reason is right and the money is gone.

Traps

• Emitting a neutral 0.5 for an ungradable episode. It averages into promotion gates and looks like data.
• Re-opening a closed group for a late arrival.
• Leaving the run in Grading, which strands storage for exactly the low-volume and cost-capped tenants least able to absorb it.

---

Background (not required to do this task): rust-agentic-sys.md §5.2, §11.6, §14, §15, §18 · rust-agentic-task.md