89 lines
3.5 KiB
Markdown
89 lines
3.5 KiB
Markdown
|
# T5.4 — `DeterministicGrader`
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | P5 — Grading |
|
|||
|
|
| Size | S — under 1 day |
|
|||
|
|
| Status | Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | — |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Grade on a number the user already computes — latency, cost, test pass count —
|
|||
|
|
with zero model calls.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
- It exists so that **adopting the framework does not require adopting
|
|||
|
|
LLM-as-judge at all.**
|
|||
|
|
- Cost profile: 0 model calls per episode, 0 models resident, produces `Ranked`
|
|||
|
|
on a computed number.
|
|||
|
|
- Its `ResourceProfile` declares an **empty** `models` vector. That is the whole
|
|||
|
|
point — it forces no residency and passes the capacity check trivially.
|
|||
|
|
- It still produces `Score::Ranked`, so the aggregation path downstream is
|
|||
|
|
unchanged.
|
|||
|
|
- Never let a rubric judge what a verifier can check: criteria that can be made
|
|||
|
|
mechanical belong here or in a `Verifier`, not in a rubric line.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Implement `EvaluationStrategy` with `resources()` returning
|
|||
|
|
`ResourceProfile { models: vec![], max_context_tokens: 0, calls_per_episode: 0.0 }`.
|
|||
|
|
2. Define the metric extractor as a user-supplied function over `EpisodeView` —
|
|||
|
|
the framework supplies latency, cost and usage; the user supplies anything
|
|||
|
|
domain-specific such as test pass count.
|
|||
|
|
3. Rank within the group by the computed number and emit `Score::Ranked` with
|
|||
|
|
`group_size`.
|
|||
|
|
4. Ensure the interval is meaningful or explicitly degenerate — do not fabricate
|
|||
|
|
a confidence interval around a deterministic measurement.
|
|||
|
|
5. Test with a workflow graded purely on test-pass-count and latency; assert the
|
|||
|
|
model-call counter is exactly zero.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- A workflow graded purely on test-pass-count and latency, **no model calls**.
|
|||
|
|
- Its `ResourceProfile` declares **zero models**.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** a model-call counter installed at the provider boundary, counting
|
|||
|
|
**all** purposes.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_deterministic_grader.rs`:
|
|||
|
|
1. Run a workflow graded on test-pass-count and latency.
|
|||
|
|
2. Assert the model-call counter is **exactly 0** across the whole run's grading
|
|||
|
|
phase — including any tiebreak path.
|
|||
|
|
3. Assert `resources().models.is_empty()` and `calls_per_episode == 0.0`.
|
|||
|
|
4. Assert registration passes the capacity check under a `CapacityLimits` with
|
|||
|
|
**zero** devices — nothing about this strategy should need a GPU.
|
|||
|
|
5. Determinism: grade the same group twice; assert identical `Score::Ranked`
|
|||
|
|
values including ordering of ties.
|
|||
|
|
6. Assert the emitted score is `Ranked`, so downstream aggregation is unchanged
|
|||
|
|
from the tournament path.
|
|||
|
|
7. Assert no fabricated confidence interval — either a real one or an explicitly
|
|||
|
|
degenerate one, asserted as such.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p grading deterministic`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Counting only judge-purpose model calls. A tiebreak issued under
|
|||
|
|
`Purpose::Work` would slip through — count all purposes.
|
|||
|
|
- Step 4 omitted: declaring the agent's model in `models` "since it is resident
|
|||
|
|
anyway" passes steps 1–3, then fails a capacity check it should pass and
|
|||
|
|
misreports the spend projection.
|
|||
|
|
- Step 5 with a group of one, where ordering is trivially stable.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Declaring the agent's model in `models` "since it is resident anyway". It then
|
|||
|
|
fails a capacity check it should pass, and it lies in the spend projection.
|
|||
|
|
- Calling a model for a tiebreak.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background (not required to do this task):
|
|||
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §11.1, §11.7 ·
|
|||
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|