95 lines
3.9 KiB
Markdown
95 lines
3.9 KiB
Markdown
# T8.5 — Embedded-mode smoke
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | P8 — Operability |
|
|
| Size | S — under 1 day |
|
|
| Status | Not started |
|
|
| Flags | parallel-ok |
|
|
| Spec | inlined below |
|
|
| Blocks | — |
|
|
|
|
## Goal
|
|
|
|
A fresh crate does `cargo add`, runs one workflow, and gets durability and
|
|
grading with zero infrastructure.
|
|
|
|
## Facts (inlined — no spec read needed)
|
|
|
|
- **Embedded mode is a first-class product, not a test harness.** A user must be
|
|
able to `cargo add` this, run an agent, and get durability and grading with
|
|
zero infrastructure.
|
|
- **That constraint is what keeps the ports honest.** If this breaks, the ports
|
|
have leaked — something in the default path now requires Postgres, an object
|
|
store, or a broker.
|
|
- Embedded mode is `redb` for log and state, a `redb` table for blobs, and
|
|
in-process coordination.
|
|
- Defaults ship working: a user who wants the built-in behaviour writes no code.
|
|
The smoke test is the proof of that claim.
|
|
- `parallel-ok`: this gates nothing and can be built alongside the rest of P8.
|
|
|
|
## Steps
|
|
|
|
1. Create the smoke test as a **separate crate outside the workspace**, depending
|
|
on the published crate surface only. Inside the workspace it can reach
|
|
internals and prove nothing.
|
|
2. Depend on the default feature set. No `features = ["postgres", ...]`.
|
|
3. Define one small workflow in YAML, register a stub model provider and one
|
|
verifier.
|
|
4. Run it: spawn with a `TaskId`, execute, verify, grade with the default
|
|
`PairwiseSequential` (first run yields `Ungraded { NoReference }`; a second run
|
|
on the same `TaskId` grades).
|
|
5. Assert no network listener is opened and no external service is contacted.
|
|
6. Run it in CI on every change to the default path.
|
|
|
|
## Acceptance
|
|
|
|
- A fresh crate with **no infrastructure** records, verifies and grades a run.
|
|
- If this breaks, the ports have leaked.
|
|
|
|
## Verify
|
|
|
|
**Harness:** a crate **outside the workspace**, depending on the published crate
|
|
by version or path with `default-features = true` and nothing added. That
|
|
location is the test — inside the workspace it can reach internals and proves
|
|
nothing.
|
|
|
|
**Integration test** — `smoke-crate/tests/it_embedded_smoke.rs`:
|
|
1. `cargo add` the framework (or a path dependency with default features only).
|
|
2. Define one small YAML workflow, register a stub model and one verifier.
|
|
3. Spawn with a `TaskId`, execute, verify, grade. Assert the run reaches a
|
|
terminal state and produces a `Score` — the first run legitimately yields
|
|
`Ungraded { NoReference }`.
|
|
4. Second run on the same `TaskId`: assert `Score::Relative`, proving the whole
|
|
default grading path works with no infrastructure.
|
|
5. Assert **no network listener is opened** and no external host is contacted —
|
|
run under a network-denied sandbox if the CI supports it, otherwise assert on
|
|
an instrumented socket layer.
|
|
6. Assert no Postgres, object store or broker dependency is pulled in: check
|
|
`cargo tree` for the distributed-only crates and assert absent.
|
|
7. Assert the process exits cleanly and the `redb` file is readable afterwards.
|
|
8. Run this in CI on **every change to the default path**, not nightly.
|
|
|
|
**Command:** `cargo test --manifest-path smoke-crate/Cargo.toml`
|
|
|
|
**False pass:**
|
|
- The smoke crate living inside the workspace. It compiles against internals and
|
|
stays green while the published API is unusable — which is the exact leak this
|
|
test exists to detect.
|
|
- Enabling a non-default feature to make it build. That is the leak, papered
|
|
over; the failure is the finding.
|
|
- Step 4 omitted: a first run that returns `Ungraded` is trivially reachable
|
|
without a working grading path.
|
|
|
|
## Traps
|
|
|
|
- Writing the smoke test inside the workspace with access to `pub(crate)` items.
|
|
It then passes while the public API is unusable.
|
|
- Enabling a non-default feature to make it pass. That is the leak, papered over.
|
|
|
|
---
|
|
|
|
Background (not required to do this task):
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §1, §7 ·
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|