119 lines
5.3 KiB
Markdown
119 lines
5.3 KiB
Markdown
# T8.2 — Metrics
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | P8 — Operability |
|
|
| Size | M — 1 to 3 days |
|
|
| Status | Not started |
|
|
| Flags | — |
|
|
| Spec | inlined below |
|
|
| Blocks | — |
|
|
|
|
## Goal
|
|
|
|
The full observability list. The framework observes agents; it must also be
|
|
observable.
|
|
|
|
## Facts (inlined — no spec read needed)
|
|
|
|
The list, with the reason each one exists:
|
|
|
|
- **Kernel state transitions**, tagged by tenant and workflow version.
|
|
- **Lag on every stage boundary** of `ingest → execute → verify → tournament →
|
|
aggregate → decide`. **Tournament and reduction backlogs are the two that grow
|
|
silently.**
|
|
- **`Indeterminate` attempt count as a first-class alert.** It should be near
|
|
zero; a nonzero rate means either a crash loop or a misdeclared effect class.
|
|
**This threshold only holds because `Cancelled` is a separate state** — route
|
|
cancellations here and the alert has a noisy floor, which is the same as not
|
|
having it.
|
|
- **`Ungraded` run count by reason**, separately from `Graded`. A tenant whose
|
|
runs are mostly `InsufficientGroup` has a loop that is not engaging, and that
|
|
reads as healthy on any dashboard that only counts failures.
|
|
- **Judge order-inconsistency rate**, measured on the sampled fraction under the
|
|
alternating-order scheme (T5.7).
|
|
- **Judge-versus-verifier agreement on the calibration set** (T6.2). Under one
|
|
resident model the judge is the agent's model, and this is the only instrument
|
|
that can see the grader drifting toward its own bias.
|
|
- **Model residency and swap count.** A nonzero swap rate on a single-model
|
|
deployment means something is requesting a non-resident model, and the load
|
|
cost will dominate everything else in the trace.
|
|
- **Admission refusals by reason, separating "capacity" from "budget".** They
|
|
look identical in a queue-depth graph and have opposite fixes.
|
|
- **Held-out versus selection gap** as the overfitting alarm (T6.6).
|
|
- **Trace context propagated through `Ctx`, never through task-locals.**
|
|
|
|
## Steps
|
|
|
|
1. Pick the metrics facade once and use it everywhere. Define the tag set —
|
|
tenant, workflow version, purpose — as a shared helper so tags stay consistent.
|
|
2. Emit a counter per kernel transition from the transition function itself, so
|
|
no call site can forget.
|
|
3. Instrument each stage boundary with a lag gauge measured at dequeue: now minus
|
|
the record's commit timestamp.
|
|
4. Emit `Indeterminate` and `Ungraded`-by-reason as separate series. Do not
|
|
collapse `Ungraded` reasons into one counter.
|
|
5. Wire the grading-health metrics from T5.7 and T6.2.
|
|
6. Emit residency and swap counters from the capacity layer (T5.2/T8.6).
|
|
7. Split admission refusals into `capacity` and `budget` reasons at the refusal
|
|
site.
|
|
8. Propagate trace context through `Ctx`. Audit for `task_local!` and remove.
|
|
9. Test: stall the reducer deliberately and assert it is visible within one
|
|
scrape interval.
|
|
|
|
## Acceptance
|
|
|
|
- **Every stage boundary emits lag.**
|
|
- **A stalled compactor is visible within one scrape interval.**
|
|
|
|
## Verify
|
|
|
|
**Harness:** an in-process metrics recorder the test can scrape, plus a
|
|
deliberately stalled reducer.
|
|
|
|
**Integration test** — `tests/it_metrics.rs`:
|
|
1. **Completeness audit:** assert one metric exists per item in this file's list.
|
|
Drive it from an enumerated list of expected metric names so a missing one
|
|
fails rather than going unnoticed.
|
|
2. **Stage lag:** run work through all six stages; assert **each** boundary emits
|
|
a lag value. A stage emitting nothing must fail.
|
|
3. **Stalled reducer:** stop the reducer, keep producing work, scrape once.
|
|
Assert the backlog is visible **within one scrape interval** — assert on the
|
|
scraped value, not on the internal counter.
|
|
4. Assert `Indeterminate` count is its own series and that a cancelled attempt
|
|
increments **`Cancelled`, not `Indeterminate`** — the alert's near-zero
|
|
threshold depends entirely on this.
|
|
5. Assert `Ungraded` is emitted **by reason** as separate series; a single
|
|
`ungraded_total` fails.
|
|
6. Assert admission refusals split into `capacity` and `budget`; produce one of
|
|
each and assert two distinct series move.
|
|
7. Assert every metric carries tenant and workflow-version tags where the list
|
|
says so.
|
|
8. Assert trace context propagates across a spawn boundary through `Ctx`; grep
|
|
for `task_local!` in kernel crates and assert none.
|
|
|
|
**Command:** `cargo test -p operability metrics`
|
|
|
|
**False pass:**
|
|
- Asserting metrics exist without asserting they **move**. A registered-but-never-
|
|
incremented counter passes existence checks and reads as a healthy zero.
|
|
- Step 3 measured on the internal counter rather than a scrape. The stall may be
|
|
visible internally and never exported.
|
|
- Step 4 omitted: routing cancellations into `Indeterminate` produces a perfectly
|
|
functional metric with a permanent noisy floor, which quietly retires the
|
|
alert.
|
|
- Step 6 omitted: capacity and budget refusals look identical in a queue-depth
|
|
graph and have opposite fixes.
|
|
|
|
## Traps
|
|
|
|
- One `ungraded_total` counter. The reason is the whole signal.
|
|
- Merging capacity and budget refusals. Opposite fixes, identical graph.
|
|
- Trace context in a task-local. It survives until the first spawn boundary.
|
|
|
|
---
|
|
|
|
Background (not required to do this task):
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §5.1, §9.1, §15 ·
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|