(tasks) add tasks for harness

2026-08-17 23:05:20 -07:00
parent e2d678b118
commit 5a30d0ffc6
74 changed files with 8204 additions and 1 deletions
@@ -0,0 +1,96 @@
+# T6.7 — Drift check
+
+| Field | Value |
+|---|---|
+| Phase | P6 — Learning loop |
+| Size | S — under 1 day |
+| Status | Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | — |
+
+## Goal
+
+Track the distribution of incoming `TaskId` characteristics over time and alarm
+on a shift.
+
+## Facts (inlined — no spec read needed)
+
+- Held-out (T6.6) catches overfitting to **episodes**. It does **not** catch
+  drift in the **task mix**. That needs a separate distribution check on incoming
+  `TaskId` characteristics over time.
+- The failure it prevents: the workflow genuinely improves on the task mix it was
+  selected against, while the incoming mix moves elsewhere. Every internal
+  instrument reads healthy.
+- It is a promotion gate input: the `current` rung requires "drift check clean"
+  alongside the sequential test crossing its accept boundary (T6.3).
+- `TaskId` is an opaque hash, so the characteristics tracked are the metadata
+  recorded beside it — input size, declared category, source, whatever the
+  deployment's `TaskId` hasher was fed.
+
+## Steps
+
+1. Define the characteristic vector extracted per incoming task at spawn. Keep it
+   small and explicitly listed; this is not a feature store.
+2. Maintain a reference distribution over a trailing window, and a current
+   window.
+3. Compare with a distribution distance appropriate to the feature types
+   (population stability index or a KS test per numeric feature; chi-square for
+   categorical). Pick one, name the threshold as a constant.
+4. Emit the distance as a metric and raise the alarm past the threshold, naming
+   which characteristic moved.
+5. Expose `is_clean()` for T6.3's `current` rung.
+6. Test with a synthetic shift and with a stable mix.
+
+## Acceptance
+
+- A synthetic shift in task mix raises the alarm.
+- A stable mix does not.
+
+## Phase gate
+
+P6 closes on an automatic challenger accept and an automatic rollback, both
+unattended, on one resident model.
+
+## Verify
+
+**Harness:** synthetic task streams with a controllable characteristic
+distribution.
+
+**Integration test** — `tests/it_drift_check.rs`:
+1. Feed a stable mix for the full window. Assert **no alarm** — the false-positive
+   half matters as much as detection.
+2. Shift one characteristic sharply mid-stream. Assert the alarm fires **and
+   names the characteristic that moved**.
+3. Shift a different characteristic; assert the named one changes accordingly.
+4. Gradual drift over many windows: assert it is eventually detected, so the
+   check is not tuned only for step changes.
+5. Assert `is_clean()` gates T6.3's `current` rung — promote with drift present
+   and assert the promotion is blocked.
+6. Assert the threshold reads its **own named constant**, not one shared with
+   another limit.
+7. Assert the distance metric is emitted continuously, not only on alarm, so the
+   trend is visible before the threshold.
+
+**Command:** `cargo test -p loop drift`
+
+**False pass:**
+- Step 1 omitted. A check that alarms on everything passes step 2 perfectly and
+  is useless.
+- Step 2 asserting the alarm boolean only. Knowing *which* characteristic moved
+  is what makes it actionable; an aggregate distance alone sends the operator
+  hunting.
+- Testing only a step change, where any distance measure works. Step 4 is where a
+  badly chosen window length shows.
+
+## Traps
+
+- Alarming on the aggregate distance only. Knowing *which* characteristic moved
+  is what makes the alarm actionable.
+- A threshold shared with an unrelated limit. Give it its own named constant.
+
+---
+
+Background (not required to do this task):
+[rust-agentic-sys.md](../../../rust-agentic-sys.md) §12.3, §12.5 ·
+[rust-agentic-task.md](../../../rust-agentic-task.md)