poimen/tasks/T8.2-metrics.md

# T8.2 — Metrics

| Field | Value |
|---|---|
| Phase | P8 — Operability |
| Size | M — 1 to 3 days |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |

## Goal

The full observability list. The framework observes agents; it must also be
observable.

## Facts (inlined — no spec read needed)

The list, with the reason each one exists:

- **Kernel state transitions**, tagged by tenant and workflow version.
- **Lag on every stage boundary** of `ingest → execute → verify → tournament →
  aggregate → decide`. **Tournament and reduction backlogs are the two that grow
  silently.**
- **`Indeterminate` attempt count as a first-class alert.** It should be near
  zero; a nonzero rate means either a crash loop or a misdeclared effect class.
  **This threshold only holds because `Cancelled` is a separate state** — route
  cancellations here and the alert has a noisy floor, which is the same as not
  having it.
- **`Ungraded` run count by reason**, separately from `Graded`. A tenant whose
  runs are mostly `InsufficientGroup` has a loop that is not engaging, and that
  reads as healthy on any dashboard that only counts failures.
- **Judge order-inconsistency rate**, measured on the sampled fraction under the
  alternating-order scheme (T5.7).
- **Judge-versus-verifier agreement on the calibration set** (T6.2). Under one
  resident model the judge is the agent's model, and this is the only instrument
  that can see the grader drifting toward its own bias.
- **Model residency and swap count.** A nonzero swap rate on a single-model
  deployment means something is requesting a non-resident model, and the load
  cost will dominate everything else in the trace.
- **Admission refusals by reason, separating "capacity" from "budget".** They
  look identical in a queue-depth graph and have opposite fixes.
- **Held-out versus selection gap** as the overfitting alarm (T6.6).
- **Trace context propagated through `Ctx`, never through task-locals.**

## Steps

1. Pick the metrics facade once and use it everywhere. Define the tag set —
   tenant, workflow version, purpose — as a shared helper so tags stay consistent.
2. Emit a counter per kernel transition from the transition function itself, so
   no call site can forget.
3. Instrument each stage boundary with a lag gauge measured at dequeue: now minus
   the record's commit timestamp.
4. Emit `Indeterminate` and `Ungraded`-by-reason as separate series. Do not
   collapse `Ungraded` reasons into one counter.
5. Wire the grading-health metrics from T5.7 and T6.2.
6. Emit residency and swap counters from the capacity layer (T5.2/T8.6).
7. Split admission refusals into `capacity` and `budget` reasons at the refusal
   site.
8. Propagate trace context through `Ctx`. Audit for `task_local!` and remove.
9. Test: stall the reducer deliberately and assert it is visible within one
   scrape interval.

## Acceptance

- **Every stage boundary emits lag.**
- **A stalled compactor is visible within one scrape interval.**

## Verify

**Harness:** an in-process metrics recorder the test can scrape, plus a
deliberately stalled reducer.

**Integration test** — `tests/it_metrics.rs`:
1. **Completeness audit:** assert one metric exists per item in this file's list.
   Drive it from an enumerated list of expected metric names so a missing one
   fails rather than going unnoticed.
2. **Stage lag:** run work through all six stages; assert **each** boundary emits
   a lag value. A stage emitting nothing must fail.
3. **Stalled reducer:** stop the reducer, keep producing work, scrape once.
   Assert the backlog is visible **within one scrape interval** — assert on the
   scraped value, not on the internal counter.
4. Assert `Indeterminate` count is its own series and that a cancelled attempt
   increments **`Cancelled`, not `Indeterminate`** — the alert's near-zero
   threshold depends entirely on this.
5. Assert `Ungraded` is emitted **by reason** as separate series; a single
   `ungraded_total` fails.
6. Assert admission refusals split into `capacity` and `budget`; produce one of
   each and assert two distinct series move.
7. Assert every metric carries tenant and workflow-version tags where the list
   says so.
8. Assert trace context propagates across a spawn boundary through `Ctx`; grep
   for `task_local!` in kernel crates and assert none.

**Command:** `cargo test -p operability metrics`

**False pass:**
- Asserting metrics exist without asserting they **move**. A registered-but-never-
  incremented counter passes existence checks and reads as a healthy zero.
- Step 3 measured on the internal counter rather than a scrape. The stall may be
  visible internally and never exported.
- Step 4 omitted: routing cancellations into `Indeterminate` produces a perfectly
  functional metric with a permanent noisy floor, which quietly retires the
  alert.
- Step 6 omitted: capacity and budget refusals look identical in a queue-depth
  graph and have opposite fixes.

## Traps

- One `ungraded_total` counter. The reason is the whole signal.
- Merging capacity and budget refusals. Opposite fixes, identical graph.
- Trace context in a task-local. It survives until the first spawn boundary.

---

Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §5.1, §9.1, §15 ·
[rust-agentic-task.md](../../../rust-agentic-task.md)