(tasks) add tasks for harness

2026-08-17 23:05:20 -07:00
parent e2d678b118
commit 5a30d0ffc6
74 changed files with 8204 additions and 1 deletions
@@ -0,0 +1,94 @@
+# T8.5 — Embedded-mode smoke
+
+| Field | Value |
+|---|---|
+| Phase | P8 — Operability |
+| Size | S — under 1 day |
+| Status | Not started |
+| Flags | parallel-ok |
+| Spec | inlined below |
+| Blocks | — |
+
+## Goal
+
+A fresh crate does `cargo add`, runs one workflow, and gets durability and
+grading with zero infrastructure.
+
+## Facts (inlined — no spec read needed)
+
+- **Embedded mode is a first-class product, not a test harness.** A user must be
+  able to `cargo add` this, run an agent, and get durability and grading with
+  zero infrastructure.
+- **That constraint is what keeps the ports honest.** If this breaks, the ports
+  have leaked — something in the default path now requires Postgres, an object
+  store, or a broker.
+- Embedded mode is `redb` for log and state, a `redb` table for blobs, and
+  in-process coordination.
+- Defaults ship working: a user who wants the built-in behaviour writes no code.
+  The smoke test is the proof of that claim.
+- `parallel-ok`: this gates nothing and can be built alongside the rest of P8.
+
+## Steps
+
+1. Create the smoke test as a **separate crate outside the workspace**, depending
+   on the published crate surface only. Inside the workspace it can reach
+   internals and prove nothing.
+2. Depend on the default feature set. No `features = ["postgres", ...]`.
+3. Define one small workflow in YAML, register a stub model provider and one
+   verifier.
+4. Run it: spawn with a `TaskId`, execute, verify, grade with the default
+   `PairwiseSequential` (first run yields `Ungraded { NoReference }`; a second run
+   on the same `TaskId` grades).
+5. Assert no network listener is opened and no external service is contacted.
+6. Run it in CI on every change to the default path.
+
+## Acceptance
+
+- A fresh crate with **no infrastructure** records, verifies and grades a run.
+- If this breaks, the ports have leaked.
+
+## Verify
+
+**Harness:** a crate **outside the workspace**, depending on the published crate
+by version or path with `default-features = true` and nothing added. That
+location is the test — inside the workspace it can reach internals and proves
+nothing.
+
+**Integration test** — `smoke-crate/tests/it_embedded_smoke.rs`:
+1. `cargo add` the framework (or a path dependency with default features only).
+2. Define one small YAML workflow, register a stub model and one verifier.
+3. Spawn with a `TaskId`, execute, verify, grade. Assert the run reaches a
+   terminal state and produces a `Score` — the first run legitimately yields
+   `Ungraded { NoReference }`.
+4. Second run on the same `TaskId`: assert `Score::Relative`, proving the whole
+   default grading path works with no infrastructure.
+5. Assert **no network listener is opened** and no external host is contacted —
+   run under a network-denied sandbox if the CI supports it, otherwise assert on
+   an instrumented socket layer.
+6. Assert no Postgres, object store or broker dependency is pulled in: check
+   `cargo tree` for the distributed-only crates and assert absent.
+7. Assert the process exits cleanly and the `redb` file is readable afterwards.
+8. Run this in CI on **every change to the default path**, not nightly.
+
+**Command:** `cargo test --manifest-path smoke-crate/Cargo.toml`
+
+**False pass:**
+- The smoke crate living inside the workspace. It compiles against internals and
+  stays green while the published API is unusable — which is the exact leak this
+  test exists to detect.
+- Enabling a non-default feature to make it build. That is the leak, papered
+  over; the failure is the finding.
+- Step 4 omitted: a first run that returns `Ungraded` is trivially reachable
+  without a working grading path.
+
+## Traps
+
+- Writing the smoke test inside the workspace with access to `pub(crate)` items.
+  It then passes while the public API is unusable.
+- Enabling a non-default feature to make it pass. That is the leak, papered over.
+
+---
+
+Background (not required to do this task):
+[rust-agentic-sys.md](../../../rust-agentic-sys.md) §1, §7 ·
+[rust-agentic-task.md](../../../rust-agentic-task.md)