106 lines
4.7 KiB
Markdown
106 lines
4.7 KiB
Markdown
|
# T8.1 — Metering
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | P8 — Operability |
|
|||
|
|
| Size | M — 1 to 3 days |
|
|||
|
|
| Status | Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | — |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Attribute every model call to `(TenantId, RunId, Purpose)` and enforce separate
|
|||
|
|
ceilings for judging and replay.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
- **A framework that spends users' model budget on grading must account for it.**
|
|||
|
|
- Every model call — agent, judge, or proposer — is attributed to
|
|||
|
|
`(TenantId, RunId, Purpose)`, where `Purpose` distinguishes **work** from
|
|||
|
|
**grading** from **replay**.
|
|||
|
|
- Per-tenant ceilings on grading spend are enforced **at the group boundary**,
|
|||
|
|
where group size is known and a tournament can be skipped or downsampled before
|
|||
|
|
it starts — not mid-tournament.
|
|||
|
|
- **Replay is the expensive one and needs its own ceiling separate from
|
|||
|
|
judging.** N variants × M tasks is N·M full agent runs plus judge calls.
|
|||
|
|
- Grading cost is reported **next to grading value**. A loop that costs more than
|
|||
|
|
the work it grades may still be worth running; it should never be an unpleasant
|
|||
|
|
discovery.
|
|||
|
|
- A skipped run must reach terminal `Ungraded` with `BudgetExhausted`, not rest
|
|||
|
|
in `Grading`. **A cost ceiling that also strands storage is a bill that arrives
|
|||
|
|
twice**, landing on the tenants least able to absorb it.
|
|||
|
|
- The grading spend ratio ceiling itself is unset — metered here, policy elsewhere.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Define `Purpose { Work, Grading, Replay }` and thread it through every model
|
|||
|
|
call site. No default — the call site knows.
|
|||
|
|
2. Record usage per call keyed `(TenantId, RunId, Purpose)`: tokens in/out, model,
|
|||
|
|
and cost if a price table is configured.
|
|||
|
|
3. Define per-tenant ceilings: one for grading, a **separate** one for replay.
|
|||
|
|
4. Check the grading ceiling at the group boundary. Over the ceiling: skip or
|
|||
|
|
downsample, record `UngradedReason::BudgetExhausted`, and transition the run to
|
|||
|
|
terminal `Ungraded` (T5.11).
|
|||
|
|
5. Never let a ceiling check touch the agent execution path. Work continues
|
|||
|
|
regardless of grading spend.
|
|||
|
|
6. Emit a report pairing grading cost with grading value per tenant.
|
|||
|
|
7. Test: drive a tenant over its grading ceiling and assert agent throughput is
|
|||
|
|
unchanged.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- A tenant exceeding its grading ceiling has tournaments **skipped with a
|
|||
|
|
recorded reason**, and **agent work continues unaffected**.
|
|||
|
|
- The skipped runs reach terminal `Ungraded` rather than resting in `Grading`.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** a model provider wrapper recording `(TenantId, RunId, Purpose,
|
|||
|
|
tokens)` for every call, plus a tenant configured with a **zero** grading ceiling
|
|||
|
|
and a generous replay ceiling (and the reverse).
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_metering.rs`:
|
|||
|
|
1. Run a full cycle: agent work, grading, replay. Assert every recorded call
|
|||
|
|
carries a `Purpose`, and that the three purposes are all present. Assert no
|
|||
|
|
call is attributed to a default or unknown purpose.
|
|||
|
|
2. **Ceiling separation:** with grading ceiling = 0 and replay ceiling high,
|
|||
|
|
assert grading is skipped and replay still runs. Then invert and assert the
|
|||
|
|
opposite. One shared ceiling fails one of these.
|
|||
|
|
3. **Skip happens before spend:** assert model calls under `Purpose::Grading` is
|
|||
|
|
**exactly 0** when over the ceiling — the check is at the group boundary,
|
|||
|
|
before any call.
|
|||
|
|
4. Assert agent throughput and latency are **unchanged** while the tenant is over
|
|||
|
|
its grading ceiling; compare against a baseline run.
|
|||
|
|
5. Assert skipped runs reach **terminal `Ungraded { BudgetExhausted }`** and that
|
|||
|
|
T4.4's `is_reducible` returns true for them.
|
|||
|
|
6. Assert the cost-versus-value report is produced per tenant.
|
|||
|
|
7. Assert attribution survives a cold re-fold — the usage lives in the log, not
|
|||
|
|
only in state.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p operability metering`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Step 3 asserting only the recorded reason. Enforcing the ceiling **after** the
|
|||
|
|
calls produces the correct reason and spends the money anyway.
|
|||
|
|
- Step 5 omitted: this is the "bill arrives twice" case — the tenant is capped on
|
|||
|
|
spend and then also accumulates storage forever because the run never leaves
|
|||
|
|
`Grading`.
|
|||
|
|
- Step 2 omitted: a single ceiling passes every other assertion here, then replay
|
|||
|
|
consumes the whole budget in production.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- One shared ceiling for judging and replay. Replay dominates and consumes it.
|
|||
|
|
- Checking the ceiling mid-tournament, which pays for half a group and produces
|
|||
|
|
nothing.
|
|||
|
|
- Leaving budget-skipped runs in `Grading`. The cost ceiling then also strands
|
|||
|
|
storage.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background (not required to do this task):
|
|||
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §10.3, §11.6, §12.4, §14.1 ·
|
|||
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|