97 lines
3.6 KiB
Markdown
97 lines
3.6 KiB
Markdown
# T6.7 — Drift check
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | P6 — Learning loop |
|
|
| Size | S — under 1 day |
|
|
| Status | Not started |
|
|
| Flags | — |
|
|
| Spec | inlined below |
|
|
| Blocks | — |
|
|
|
|
## Goal
|
|
|
|
Track the distribution of incoming `TaskId` characteristics over time and alarm
|
|
on a shift.
|
|
|
|
## Facts (inlined — no spec read needed)
|
|
|
|
- Held-out (T6.6) catches overfitting to **episodes**. It does **not** catch
|
|
drift in the **task mix**. That needs a separate distribution check on incoming
|
|
`TaskId` characteristics over time.
|
|
- The failure it prevents: the workflow genuinely improves on the task mix it was
|
|
selected against, while the incoming mix moves elsewhere. Every internal
|
|
instrument reads healthy.
|
|
- It is a promotion gate input: the `current` rung requires "drift check clean"
|
|
alongside the sequential test crossing its accept boundary (T6.3).
|
|
- `TaskId` is an opaque hash, so the characteristics tracked are the metadata
|
|
recorded beside it — input size, declared category, source, whatever the
|
|
deployment's `TaskId` hasher was fed.
|
|
|
|
## Steps
|
|
|
|
1. Define the characteristic vector extracted per incoming task at spawn. Keep it
|
|
small and explicitly listed; this is not a feature store.
|
|
2. Maintain a reference distribution over a trailing window, and a current
|
|
window.
|
|
3. Compare with a distribution distance appropriate to the feature types
|
|
(population stability index or a KS test per numeric feature; chi-square for
|
|
categorical). Pick one, name the threshold as a constant.
|
|
4. Emit the distance as a metric and raise the alarm past the threshold, naming
|
|
which characteristic moved.
|
|
5. Expose `is_clean()` for T6.3's `current` rung.
|
|
6. Test with a synthetic shift and with a stable mix.
|
|
|
|
## Acceptance
|
|
|
|
- A synthetic shift in task mix raises the alarm.
|
|
- A stable mix does not.
|
|
|
|
## Phase gate
|
|
|
|
P6 closes on an automatic challenger accept and an automatic rollback, both
|
|
unattended, on one resident model.
|
|
|
|
## Verify
|
|
|
|
**Harness:** synthetic task streams with a controllable characteristic
|
|
distribution.
|
|
|
|
**Integration test** — `tests/it_drift_check.rs`:
|
|
1. Feed a stable mix for the full window. Assert **no alarm** — the false-positive
|
|
half matters as much as detection.
|
|
2. Shift one characteristic sharply mid-stream. Assert the alarm fires **and
|
|
names the characteristic that moved**.
|
|
3. Shift a different characteristic; assert the named one changes accordingly.
|
|
4. Gradual drift over many windows: assert it is eventually detected, so the
|
|
check is not tuned only for step changes.
|
|
5. Assert `is_clean()` gates T6.3's `current` rung — promote with drift present
|
|
and assert the promotion is blocked.
|
|
6. Assert the threshold reads its **own named constant**, not one shared with
|
|
another limit.
|
|
7. Assert the distance metric is emitted continuously, not only on alarm, so the
|
|
trend is visible before the threshold.
|
|
|
|
**Command:** `cargo test -p loop drift`
|
|
|
|
**False pass:**
|
|
- Step 1 omitted. A check that alarms on everything passes step 2 perfectly and
|
|
is useless.
|
|
- Step 2 asserting the alarm boolean only. Knowing *which* characteristic moved
|
|
is what makes it actionable; an aggregate distance alone sends the operator
|
|
hunting.
|
|
- Testing only a step change, where any distance measure works. Step 4 is where a
|
|
badly chosen window length shows.
|
|
|
|
## Traps
|
|
|
|
- Alarming on the aggregate distance only. Knowing *which* characteristic moved
|
|
is what makes the alarm actionable.
|
|
- A threshold shared with an unrelated limit. Give it its own named constant.
|
|
|
|
---
|
|
|
|
Background (not required to do this task):
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §12.3, §12.5 ·
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|