poimen/tasks/T6.7-drift-check.md

# T6.7 — Drift check

| Field | Value |
|---|---|
| Phase | P6 — Learning loop |
| Size | S — under 1 day |
| Status | Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |

## Goal

Track the distribution of incoming `TaskId` characteristics over time and alarm
on a shift.

## Facts (inlined — no spec read needed)

- Held-out (T6.6) catches overfitting to **episodes**. It does **not** catch
  drift in the **task mix**. That needs a separate distribution check on incoming
  `TaskId` characteristics over time.
- The failure it prevents: the workflow genuinely improves on the task mix it was
  selected against, while the incoming mix moves elsewhere. Every internal
  instrument reads healthy.
- It is a promotion gate input: the `current` rung requires "drift check clean"
  alongside the sequential test crossing its accept boundary (T6.3).
- `TaskId` is an opaque hash, so the characteristics tracked are the metadata
  recorded beside it — input size, declared category, source, whatever the
  deployment's `TaskId` hasher was fed.

## Steps

1. Define the characteristic vector extracted per incoming task at spawn. Keep it
   small and explicitly listed; this is not a feature store.
2. Maintain a reference distribution over a trailing window, and a current
   window.
3. Compare with a distribution distance appropriate to the feature types
   (population stability index or a KS test per numeric feature; chi-square for
   categorical). Pick one, name the threshold as a constant.
4. Emit the distance as a metric and raise the alarm past the threshold, naming
   which characteristic moved.
5. Expose `is_clean()` for T6.3's `current` rung.
6. Test with a synthetic shift and with a stable mix.

## Acceptance

- A synthetic shift in task mix raises the alarm.
- A stable mix does not.

## Phase gate

P6 closes on an automatic challenger accept and an automatic rollback, both
unattended, on one resident model.

## Verify

**Harness:** synthetic task streams with a controllable characteristic
distribution.

**Integration test** — `tests/it_drift_check.rs`:
1. Feed a stable mix for the full window. Assert **no alarm** — the false-positive
   half matters as much as detection.
2. Shift one characteristic sharply mid-stream. Assert the alarm fires **and
   names the characteristic that moved**.
3. Shift a different characteristic; assert the named one changes accordingly.
4. Gradual drift over many windows: assert it is eventually detected, so the
   check is not tuned only for step changes.
5. Assert `is_clean()` gates T6.3's `current` rung — promote with drift present
   and assert the promotion is blocked.
6. Assert the threshold reads its **own named constant**, not one shared with
   another limit.
7. Assert the distance metric is emitted continuously, not only on alarm, so the
   trend is visible before the threshold.

**Command:** `cargo test -p loop drift`

**False pass:**
- Step 1 omitted. A check that alarms on everything passes step 2 perfectly and
  is useless.
- Step 2 asserting the alarm boolean only. Knowing *which* characteristic moved
  is what makes it actionable; an aggregate distance alone sends the operator
  hunting.
- Testing only a step change, where any distance measure works. Step 4 is where a
  badly chosen window length shows.

## Traps

- Alarming on the aggregate distance only. Knowing *which* characteristic moved
  is what makes the alarm actionable.
- A threshold shared with an unrelated limit. Give it its own named constant.

---

Background (not required to do this task):
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §12.3, §12.5 ·
[rust-agentic-task.md](../../../rust-agentic-task.md)