poimen/tasks/T8.2-metrics.md

T8.2 — Metrics

Field	Value
Phase	P8 — Operability
Size	M — 1 to 3 days
Status	Not started
Flags	—
Spec	inlined below
Blocks	—

Goal

The full observability list. The framework observes agents; it must also be observable.

Facts (inlined — no spec read needed)

The list, with the reason each one exists:

• Kernel state transitions, tagged by tenant and workflow version.
• Lag on every stage boundary of ingest → execute → verify → tournament → aggregate → decide. Tournament and reduction backlogs are the two that grow silently.
• Indeterminate attempt count as a first-class alert. It should be near zero; a nonzero rate means either a crash loop or a misdeclared effect class. This threshold only holds because Cancelled is a separate state — route cancellations here and the alert has a noisy floor, which is the same as not having it.
• Ungraded run count by reason, separately from Graded. A tenant whose runs are mostly InsufficientGroup has a loop that is not engaging, and that reads as healthy on any dashboard that only counts failures.
• Judge order-inconsistency rate, measured on the sampled fraction under the alternating-order scheme (T5.7).
• Judge-versus-verifier agreement on the calibration set (T6.2). Under one resident model the judge is the agent's model, and this is the only instrument that can see the grader drifting toward its own bias.
• Model residency and swap count. A nonzero swap rate on a single-model deployment means something is requesting a non-resident model, and the load cost will dominate everything else in the trace.
• Admission refusals by reason, separating "capacity" from "budget". They look identical in a queue-depth graph and have opposite fixes.
• Held-out versus selection gap as the overfitting alarm (T6.6).
• Trace context propagated through Ctx, never through task-locals.

Steps

1. Pick the metrics facade once and use it everywhere. Define the tag set — tenant, workflow version, purpose — as a shared helper so tags stay consistent.
2. Emit a counter per kernel transition from the transition function itself, so no call site can forget.
3. Instrument each stage boundary with a lag gauge measured at dequeue: now minus the record's commit timestamp.
4. Emit Indeterminate and Ungraded-by-reason as separate series. Do not collapse Ungraded reasons into one counter.
5. Wire the grading-health metrics from T5.7 and T6.2.
6. Emit residency and swap counters from the capacity layer (T5.2/T8.6).
7. Split admission refusals into capacity and budget reasons at the refusal site.
8. Propagate trace context through Ctx. Audit for task_local! and remove.
9. Test: stall the reducer deliberately and assert it is visible within one scrape interval.

Acceptance

• Every stage boundary emits lag.
• A stalled compactor is visible within one scrape interval.

Verify

Harness: an in-process metrics recorder the test can scrape, plus a deliberately stalled reducer.

Integration test — tests/it_metrics.rs:

1. Completeness audit: assert one metric exists per item in this file's list. Drive it from an enumerated list of expected metric names so a missing one fails rather than going unnoticed.
2. Stage lag: run work through all six stages; assert each boundary emits a lag value. A stage emitting nothing must fail.
3. Stalled reducer: stop the reducer, keep producing work, scrape once. Assert the backlog is visible within one scrape interval — assert on the scraped value, not on the internal counter.
4. Assert Indeterminate count is its own series and that a cancelled attempt increments Cancelled, not Indeterminate — the alert's near-zero threshold depends entirely on this.
5. Assert Ungraded is emitted by reason as separate series; a single ungraded_total fails.
6. Assert admission refusals split into capacity and budget; produce one of each and assert two distinct series move.
7. Assert every metric carries tenant and workflow-version tags where the list says so.
8. Assert trace context propagates across a spawn boundary through Ctx; grep for task_local! in kernel crates and assert none.

Command: cargo test -p operability metrics

False pass:

• Asserting metrics exist without asserting they move. A registered-but-never- incremented counter passes existence checks and reads as a healthy zero.
• Step 3 measured on the internal counter rather than a scrape. The stall may be visible internally and never exported.
• Step 4 omitted: routing cancellations into Indeterminate produces a perfectly functional metric with a permanent noisy floor, which quietly retires the alert.
• Step 6 omitted: capacity and budget refusals look identical in a queue-depth graph and have opposite fixes.

Traps

• One ungraded_total counter. The reason is the whole signal.
• Merging capacity and budget refusals. Opposite fixes, identical graph.
• Trace context in a task-local. It survives until the first spawn boundary.

---

Background (not required to do this task): rust-agentic-sys.md §5.1, §9.1, §15 · rust-agentic-task.md