104 lines
4.2 KiB
Markdown
104 lines
4.2 KiB
Markdown
|
# T4.1 — `Verifier` port
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | P4 — Verification |
|
|||
|
|
| Size | S — under 1 day |
|
|||
|
|
| Status | Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | T4.2 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
The verification port with a fail-closed contract enforced by the framework, not
|
|||
|
|
by the verifier author.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
```rust
|
|||
|
|
#[async_trait]
|
|||
|
|
pub trait Verifier: Send + Sync {
|
|||
|
|
fn id(&self) -> VerifierId;
|
|||
|
|
/// Any error, panic or timeout MUST resolve to `Fail`. A verifier that
|
|||
|
|
/// throws or hangs can never report `Pass`.
|
|||
|
|
async fn verify(&self, cx: &VerifierCtx) -> VerifierOutcome;
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- **Verification returns ground truth. Grading attributes cause and ranks.**
|
|||
|
|
Collapse them and the system grades its own homework.
|
|||
|
|
- **The framework cannot break the agent it runs.** Observation, verification and
|
|||
|
|
grading failures degrade the record, never the work. Any code path where a
|
|||
|
|
verifier or grader can fail into an agent's execution is a defect.
|
|||
|
|
- Fail-closed means the *framework* converts panic, error and timeout into
|
|||
|
|
`Fail`. A contract documented in a doc comment and implemented by each author
|
|||
|
|
is not a contract.
|
|||
|
|
- Never let a rubric judge what a verifier can check. Every criterion that can be
|
|||
|
|
made mechanical should be a `Verifier` — deterministic, cheap, and not subject
|
|||
|
|
to judge drift.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Define `VerifierId`, `VerifierOutcome` (pass/fail plus detail) and the trait
|
|||
|
|
above.
|
|||
|
|
2. Write the framework-side runner that wraps every `verify` call:
|
|||
|
|
- `tokio::time::timeout` at `cx.deadline` → `Fail`
|
|||
|
|
- `FutureExt::catch_unwind` (or a spawn-and-join boundary) for panics → `Fail`
|
|||
|
|
- any `Err` → `Fail`
|
|||
|
|
3. Make the runner the only call path. `Verifier::verify` is never invoked
|
|||
|
|
directly from the executor.
|
|||
|
|
4. Record which failure mode produced a `Fail` — timeout, panic and error are the
|
|||
|
|
same outcome but different operational signals.
|
|||
|
|
5. Run verifiers for one attempt as a fan-out/join; they are independent checks.
|
|||
|
|
6. Fault-inject all three modes in tests with real verifiers that panic, hang and
|
|||
|
|
error.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- Verifiers that panic, hang past deadline, and return an error **all resolve to
|
|||
|
|
`Fail`** — fault-injected, not asserted by comment.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** three real misbehaving verifiers — one that panics, one that loops
|
|||
|
|
past the deadline, one that returns `Err`. Fault-injected, not mocked.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_verifier_fail_closed.rs`:
|
|||
|
|
1. Run a real run with each bad verifier in turn.
|
|||
|
|
2. Assert each resolves to `Fail`.
|
|||
|
|
3. Assert **the run itself completes** and reaches a terminal state — the
|
|||
|
|
framework must not break the agent it runs. A panicking verifier taking the
|
|||
|
|
run down is the primary failure being tested for.
|
|||
|
|
4. Assert the recorded failure mode distinguishes timeout / panic / error, since
|
|||
|
|
they are the same outcome but different operational signals.
|
|||
|
|
5. Assert a verifier that panics does **not** poison shared state: run a second,
|
|||
|
|
healthy verifier afterwards in the same process and assert it returns `Pass`.
|
|||
|
|
6. Fan-out: three verifiers on one attempt, one of them panicking; assert the
|
|||
|
|
other two still return their own results.
|
|||
|
|
7. Assert no code path can produce `Pass` from an `Err` — a `trybuild` case if
|
|||
|
|
`verify` is only reachable through the framework runner.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p verify fail_closed`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Testing the runner in isolation with a mock verifier rather than in a real run.
|
|||
|
|
Step 3 — the run surviving — only means something end to end.
|
|||
|
|
- Catching the panic at the test boundary with `catch_unwind` in the *test*,
|
|||
|
|
which makes a framework that does not catch it look fine.
|
|||
|
|
- Step 5 omitted: a panic that leaves a mutex poisoned passes steps 1–4 and
|
|||
|
|
breaks every subsequent verification in the process.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Letting a panic unwind into the executor task. It takes the run with it, which
|
|||
|
|
is the exact inversion the first principle forbids.
|
|||
|
|
- A `Result<VerifierOutcome, E>` signature that pushes the decision onto callers.
|
|||
|
|
One caller will map `Err` to `Pass` by accident.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background (not required to do this task):
|
|||
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §1, §10, §11.7 ·
|
|||
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|