poimen/tasks/T6.7-drift-check.md

T6.7 — Drift check

Field	Value
Phase	P6 — Learning loop
Size	S — under 1 day
Status	Not started
Flags	—
Spec	inlined below
Blocks	—

Goal

Track the distribution of incoming TaskId characteristics over time and alarm on a shift.

Facts (inlined — no spec read needed)

• Held-out (T6.6) catches overfitting to episodes. It does not catch drift in the task mix. That needs a separate distribution check on incoming TaskId characteristics over time.
• The failure it prevents: the workflow genuinely improves on the task mix it was selected against, while the incoming mix moves elsewhere. Every internal instrument reads healthy.
• It is a promotion gate input: the current rung requires "drift check clean" alongside the sequential test crossing its accept boundary (T6.3).
• TaskId is an opaque hash, so the characteristics tracked are the metadata recorded beside it — input size, declared category, source, whatever the deployment's TaskId hasher was fed.

Steps

1. Define the characteristic vector extracted per incoming task at spawn. Keep it small and explicitly listed; this is not a feature store.
2. Maintain a reference distribution over a trailing window, and a current window.
3. Compare with a distribution distance appropriate to the feature types (population stability index or a KS test per numeric feature; chi-square for categorical). Pick one, name the threshold as a constant.
4. Emit the distance as a metric and raise the alarm past the threshold, naming which characteristic moved.
5. Expose is_clean() for T6.3's current rung.
6. Test with a synthetic shift and with a stable mix.

Acceptance

• A synthetic shift in task mix raises the alarm.
• A stable mix does not.

Phase gate

P6 closes on an automatic challenger accept and an automatic rollback, both unattended, on one resident model.

Verify

Harness: synthetic task streams with a controllable characteristic distribution.

Integration test — tests/it_drift_check.rs:

1. Feed a stable mix for the full window. Assert no alarm — the false-positive half matters as much as detection.
2. Shift one characteristic sharply mid-stream. Assert the alarm fires and names the characteristic that moved.
3. Shift a different characteristic; assert the named one changes accordingly.
4. Gradual drift over many windows: assert it is eventually detected, so the check is not tuned only for step changes.
5. Assert is_clean() gates T6.3's current rung — promote with drift present and assert the promotion is blocked.
6. Assert the threshold reads its own named constant, not one shared with another limit.
7. Assert the distance metric is emitted continuously, not only on alarm, so the trend is visible before the threshold.

Command: cargo test -p loop drift

False pass:

• Step 1 omitted. A check that alarms on everything passes step 2 perfectly and is useless.
• Step 2 asserting the alarm boolean only. Knowing which characteristic moved is what makes it actionable; an aggregate distance alone sends the operator hunting.
• Testing only a step change, where any distance measure works. Step 4 is where a badly chosen window length shows.

Traps

• Alarming on the aggregate distance only. Knowing which characteristic moved is what makes the alarm actionable.
• A threshold shared with an unrelated limit. Give it its own named constant.

---

Background (not required to do this task): rust-agentic-sys.md §12.3, §12.5 · rust-agentic-task.md