(tasks) add tasks for harness

2026-08-17 23:05:20 -07:00
parent e2d678b118
commit 5a30d0ffc6
74 changed files with 8204 additions and 1 deletions
@@ -0,0 +1,83 @@
+# T1.5 — Context partition capture
+
+| Field | Value |
+|---|---|
+| Phase | P1 — Walking skeleton |
+| Size | S — under 1 day |
+| Status | Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | — |
+
+## Goal
+
+Record, per model step, which context was packed, which was available but not
+packed, and which was dropped — as identifiers.
+
+## Facts (inlined — no spec read needed)
+
+- Three buckets: **packed**, **available-not-packed**, **dropped**.
+- They hold **identifiers**, not text. That keeps them small enough to inline in
+  the attempt view rather than going by blob reference, which is what lets a
+  verifier read them without paying for prompt text.
+- This is the signal behind "retried three times because context was missing X" —
+  the thing a pass rate cannot see and a learning loop needs.
+- It is **not derivable after the fact** from the conversation. If it is not
+  captured at pack time it is gone, which is why its absence is a test failure
+  rather than a gap.
+
+## Steps
+
+1. Define `ContextPartition { packed: Vec<ContextItemId>, available: Vec<ContextItemId>,
+   dropped: Vec<ContextItemId> }`. Ordered collections, so serialization is stable.
+2. Capture at the point the prompt is assembled — the packer already knows all
+   three sets; recording them costs a clone of id vectors.
+3. Attach the partition to the attempt record and emit it in the same
+   `EventLog::commit` as the rest of the attempt transition.
+4. Surface it on `AttemptView.context: Option<ContextPartition>` for verifiers
+   (T4.2) and graders.
+5. Test: assert every model-step attempt in a completed run carries a partition;
+   a missing one fails the test rather than being tolerated as `None`.
+
+## Acceptance
+
+- Partition present on **every** model step.
+- Not derivable from the conversation, so its absence is a test failure.
+
+## Verify
+
+**Harness:** a workflow with a packer configured to drop known items, so the
+expected partition is known in advance.
+
+**Integration test** — `tests/it_context_partition.rs`:
+1. Configure the packer with 10 candidate context items and a budget admitting 6.
+2. Run a model step.
+3. Assert the recorded partition has `packed.len() == 6`, `available` +
+   `dropped` covering the remaining 4, and that the three sets are **disjoint**
+   and their union is the full candidate set.
+4. Assert `dropped` is non-empty — a capture that only ever records `packed` is
+   the common half-implementation and passes any "partition present" check.
+5. Iterate every model-step attempt in a completed multi-step run; assert
+   `context.is_some()` for **each**. A `None` fails the test.
+6. Assert the partition survives `assert_refold_identical`.
+
+**Command:** `cargo test -p executor context_partition`
+
+**False pass:**
+- Asserting only `context.is_some()`. An empty partition is `Some`. Step 3's set
+  arithmetic is the real check.
+- Testing on a workflow whose budget admits everything, so `dropped` is
+  legitimately empty and step 4 cannot fail.
+
+## Traps
+
+- Storing the packed context text here. It belongs in the prompt blob (T1.6);
+  inlining text blows broker payload limits and penalizes verifiers that need
+  none of it.
+- Capturing only `packed`. The interesting signal is usually in `dropped`.
+
+---
+
+Background (not required to do this task):
+[rust-agentic-sys.md](../../../rust-agentic-sys.md) §10.1 ·
+[rust-agentic-task.md](../../../rust-agentic-task.md)