104 lines
4.2 KiB
Markdown
104 lines
4.2 KiB
Markdown
# T4.1 — `Verifier` port
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | P4 — Verification |
|
||
| Size | S — under 1 day |
|
||
| Status | Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | T4.2 |
|
||
|
||
## Goal
|
||
|
||
The verification port with a fail-closed contract enforced by the framework, not
|
||
by the verifier author.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
```rust
|
||
#[async_trait]
|
||
pub trait Verifier: Send + Sync {
|
||
fn id(&self) -> VerifierId;
|
||
/// Any error, panic or timeout MUST resolve to `Fail`. A verifier that
|
||
/// throws or hangs can never report `Pass`.
|
||
async fn verify(&self, cx: &VerifierCtx) -> VerifierOutcome;
|
||
}
|
||
```
|
||
|
||
- **Verification returns ground truth. Grading attributes cause and ranks.**
|
||
Collapse them and the system grades its own homework.
|
||
- **The framework cannot break the agent it runs.** Observation, verification and
|
||
grading failures degrade the record, never the work. Any code path where a
|
||
verifier or grader can fail into an agent's execution is a defect.
|
||
- Fail-closed means the *framework* converts panic, error and timeout into
|
||
`Fail`. A contract documented in a doc comment and implemented by each author
|
||
is not a contract.
|
||
- Never let a rubric judge what a verifier can check. Every criterion that can be
|
||
made mechanical should be a `Verifier` — deterministic, cheap, and not subject
|
||
to judge drift.
|
||
|
||
## Steps
|
||
|
||
1. Define `VerifierId`, `VerifierOutcome` (pass/fail plus detail) and the trait
|
||
above.
|
||
2. Write the framework-side runner that wraps every `verify` call:
|
||
- `tokio::time::timeout` at `cx.deadline` → `Fail`
|
||
- `FutureExt::catch_unwind` (or a spawn-and-join boundary) for panics → `Fail`
|
||
- any `Err` → `Fail`
|
||
3. Make the runner the only call path. `Verifier::verify` is never invoked
|
||
directly from the executor.
|
||
4. Record which failure mode produced a `Fail` — timeout, panic and error are the
|
||
same outcome but different operational signals.
|
||
5. Run verifiers for one attempt as a fan-out/join; they are independent checks.
|
||
6. Fault-inject all three modes in tests with real verifiers that panic, hang and
|
||
error.
|
||
|
||
## Acceptance
|
||
|
||
- Verifiers that panic, hang past deadline, and return an error **all resolve to
|
||
`Fail`** — fault-injected, not asserted by comment.
|
||
|
||
## Verify
|
||
|
||
**Harness:** three real misbehaving verifiers — one that panics, one that loops
|
||
past the deadline, one that returns `Err`. Fault-injected, not mocked.
|
||
|
||
**Integration test** — `tests/it_verifier_fail_closed.rs`:
|
||
1. Run a real run with each bad verifier in turn.
|
||
2. Assert each resolves to `Fail`.
|
||
3. Assert **the run itself completes** and reaches a terminal state — the
|
||
framework must not break the agent it runs. A panicking verifier taking the
|
||
run down is the primary failure being tested for.
|
||
4. Assert the recorded failure mode distinguishes timeout / panic / error, since
|
||
they are the same outcome but different operational signals.
|
||
5. Assert a verifier that panics does **not** poison shared state: run a second,
|
||
healthy verifier afterwards in the same process and assert it returns `Pass`.
|
||
6. Fan-out: three verifiers on one attempt, one of them panicking; assert the
|
||
other two still return their own results.
|
||
7. Assert no code path can produce `Pass` from an `Err` — a `trybuild` case if
|
||
`verify` is only reachable through the framework runner.
|
||
|
||
**Command:** `cargo test -p verify fail_closed`
|
||
|
||
**False pass:**
|
||
- Testing the runner in isolation with a mock verifier rather than in a real run.
|
||
Step 3 — the run surviving — only means something end to end.
|
||
- Catching the panic at the test boundary with `catch_unwind` in the *test*,
|
||
which makes a framework that does not catch it look fine.
|
||
- Step 5 omitted: a panic that leaves a mutex poisoned passes steps 1–4 and
|
||
breaks every subsequent verification in the process.
|
||
|
||
## Traps
|
||
|
||
- Letting a panic unwind into the executor task. It takes the run with it, which
|
||
is the exact inversion the first principle forbids.
|
||
- A `Result<VerifierOutcome, E>` signature that pushes the decision onto callers.
|
||
One caller will map `Err` to `Pass` by accident.
|
||
|
||
---
|
||
|
||
Background (not required to do this task):
|
||
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §1, §10, §11.7 ·
|
||
[rust-agentic-task.md](../../../rust-agentic-task.md)
|