(tasks) add tasks for harness

tasks/T8.2-metrics.md

@@ -0,0 +1,118 @@
+# T8.2 — Metrics
+
+| Field | Value |
+|---|---|
+| Phase | P8 — Operability |
+| Size | M — 1 to 3 days |
+| Status | Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | — |
+
+## Goal
+
+The full observability list. The framework observes agents; it must also be
+observable.
+
+## Facts (inlined — no spec read needed)
+
+The list, with the reason each one exists:
+
+- **Kernel state transitions**, tagged by tenant and workflow version.
+- **Lag on every stage boundary** of `ingest → execute → verify → tournament →
+  aggregate → decide`. **Tournament and reduction backlogs are the two that grow
+  silently.**
+- **`Indeterminate` attempt count as a first-class alert.** It should be near
+  zero; a nonzero rate means either a crash loop or a misdeclared effect class.
+  **This threshold only holds because `Cancelled` is a separate state** — route
+  cancellations here and the alert has a noisy floor, which is the same as not
+  having it.
+- **`Ungraded` run count by reason**, separately from `Graded`. A tenant whose
+  runs are mostly `InsufficientGroup` has a loop that is not engaging, and that
+  reads as healthy on any dashboard that only counts failures.
+- **Judge order-inconsistency rate**, measured on the sampled fraction under the
+  alternating-order scheme (T5.7).
+- **Judge-versus-verifier agreement on the calibration set** (T6.2). Under one
+  resident model the judge is the agent's model, and this is the only instrument
+  that can see the grader drifting toward its own bias.
+- **Model residency and swap count.** A nonzero swap rate on a single-model
+  deployment means something is requesting a non-resident model, and the load
+  cost will dominate everything else in the trace.
+- **Admission refusals by reason, separating "capacity" from "budget".** They
+  look identical in a queue-depth graph and have opposite fixes.
+- **Held-out versus selection gap** as the overfitting alarm (T6.6).
+- **Trace context propagated through `Ctx`, never through task-locals.**
+
+## Steps
+
+1. Pick the metrics facade once and use it everywhere. Define the tag set —
+   tenant, workflow version, purpose — as a shared helper so tags stay consistent.
+2. Emit a counter per kernel transition from the transition function itself, so
+   no call site can forget.
+3. Instrument each stage boundary with a lag gauge measured at dequeue: now minus
+   the record's commit timestamp.
+4. Emit `Indeterminate` and `Ungraded`-by-reason as separate series. Do not
+   collapse `Ungraded` reasons into one counter.
+5. Wire the grading-health metrics from T5.7 and T6.2.
+6. Emit residency and swap counters from the capacity layer (T5.2/T8.6).
+7. Split admission refusals into `capacity` and `budget` reasons at the refusal
+   site.
+8. Propagate trace context through `Ctx`. Audit for `task_local!` and remove.
+9. Test: stall the reducer deliberately and assert it is visible within one
+   scrape interval.
+
+## Acceptance
+
+- **Every stage boundary emits lag.**
+- **A stalled compactor is visible within one scrape interval.**
+
+## Verify
+
+**Harness:** an in-process metrics recorder the test can scrape, plus a
+deliberately stalled reducer.
+
+**Integration test** — `tests/it_metrics.rs`:
+1. **Completeness audit:** assert one metric exists per item in this file's list.
+   Drive it from an enumerated list of expected metric names so a missing one
+   fails rather than going unnoticed.
+2. **Stage lag:** run work through all six stages; assert **each** boundary emits
+   a lag value. A stage emitting nothing must fail.
+3. **Stalled reducer:** stop the reducer, keep producing work, scrape once.
+   Assert the backlog is visible **within one scrape interval** — assert on the
+   scraped value, not on the internal counter.
+4. Assert `Indeterminate` count is its own series and that a cancelled attempt
+   increments **`Cancelled`, not `Indeterminate`** — the alert's near-zero
+   threshold depends entirely on this.
+5. Assert `Ungraded` is emitted **by reason** as separate series; a single
+   `ungraded_total` fails.
+6. Assert admission refusals split into `capacity` and `budget`; produce one of
+   each and assert two distinct series move.
+7. Assert every metric carries tenant and workflow-version tags where the list
+   says so.
+8. Assert trace context propagates across a spawn boundary through `Ctx`; grep
+   for `task_local!` in kernel crates and assert none.
+
+**Command:** `cargo test -p operability metrics`
+
+**False pass:**
+- Asserting metrics exist without asserting they **move**. A registered-but-never-
+  incremented counter passes existence checks and reads as a healthy zero.
+- Step 3 measured on the internal counter rather than a scrape. The stall may be
+  visible internally and never exported.
+- Step 4 omitted: routing cancellations into `Indeterminate` produces a perfectly
+  functional metric with a permanent noisy floor, which quietly retires the
+  alert.
+- Step 6 omitted: capacity and budget refusals look identical in a queue-depth
+  graph and have opposite fixes.
+
+## Traps
+
+- One `ungraded_total` counter. The reason is the whole signal.
+- Merging capacity and budget refusals. Opposite fixes, identical graph.
+- Trace context in a task-local. It survives until the first spawn boundary.
+
+---
+
+Background (not required to do this task):
+[rust-agentic-sys.md](../../../rust-agentic-sys.md) §5.1, §9.1, §15 ·
+[rust-agentic-task.md](../../../rust-agentic-task.md)