78 lines
3.1 KiB
Markdown
78 lines
3.1 KiB
Markdown
|
# T1.2 — Stub model provider
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | P1 — Walking skeleton |
|
|||
|
|
| Size | S — under 1 day |
|
|||
|
|
| Status | Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | — |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
A deterministic scripted model provider so the walking skeleton runs end to end
|
|||
|
|
with no network and no spend.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
- Nothing is built before the record is trustworthy: no dashboards and no
|
|||
|
|
learning loop until one full run works end to end **against a stub model**.
|
|||
|
|
- The stub is not a test fixture that gets deleted — it is the fixed point every
|
|||
|
|
later determinism assertion is measured against, including T0.8's re-fold
|
|||
|
|
property and T5.x's grading tests.
|
|||
|
|
- Failure injection here is what T1.4 (retry), T2.1 (crash windows) and T2.6
|
|||
|
|
(crash matrix) drive their scenarios with.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Define the model port the executor calls: request in, response plus `Usage`
|
|||
|
|
out, `async`, cancellation-aware.
|
|||
|
|
2. Implement the stub over a script: an ordered list of responses, optionally
|
|||
|
|
keyed by step id so a workflow with branches stays readable.
|
|||
|
|
3. Add configurable latency per response, driven by `tokio::time::sleep` so
|
|||
|
|
`tokio::time::pause` in tests makes it instant.
|
|||
|
|
4. Add failure injection: error responses, timeouts, and a hang that outlives the
|
|||
|
|
deadline. Each selectable per script entry.
|
|||
|
|
5. Make every field of the emitted response deterministic — no timestamps from
|
|||
|
|
the wall clock, no generated ids that are not seeded.
|
|||
|
|
6. Test: run the same script twice, serialize both episodes, assert equal bytes.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- The same script yields byte-identical episodes across runs.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** the stub itself plus T0.8's `assert_refold_identical`.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_stub_determinism.rs`:
|
|||
|
|
1. Run the same script twice, in two fresh databases, in the same process.
|
|||
|
|
2. Serialize both episodes; assert **byte equality**.
|
|||
|
|
3. Run a third time in a **separate process** and compare against a committed
|
|||
|
|
expected serialization — catches anything seeded from process state.
|
|||
|
|
4. With `tokio::time::pause`, assert a script declaring 30s latency completes in
|
|||
|
|
near-zero wall time. If it does not, the stub is sleeping off the tokio clock.
|
|||
|
|
5. Failure injection: assert an error entry produces a failed attempt, a timeout
|
|||
|
|
entry produces `TimedOut`, and a hang entry is cut off by the deadline.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p testkit stub`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Steps 1–2 in one process with a lazily-initialized global seed: both runs share
|
|||
|
|
it and agree, while a fresh process differs. Step 3 is the guard.
|
|||
|
|
- Comparing episodes with a `PartialEq` that skips timestamps. That hides exactly
|
|||
|
|
the nondeterminism being hunted — compare the full serialized bytes.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- A wall-clock timestamp or a random id in the stub response. It defeats every
|
|||
|
|
byte-identity assertion downstream and looks like a fold bug.
|
|||
|
|
- Sleeping with `std::thread::sleep`, which `tokio::time::pause` cannot skip.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background (not required to do this task):
|
|||
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §16 ·
|
|||
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|