109 lines
4.9 KiB
Markdown
109 lines
4.9 KiB
Markdown
|
# T4.5 — P4 composition gate
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | P4 — Verification |
|
|||
|
|
| Size | M — 1 to 3 days |
|
|||
|
|
| Status | Not started |
|
|||
|
|
| Flags | gate |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | P5 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Prove the verifier port, the snapshot barrier, lazy blob access and the retention
|
|||
|
|
guard hold **simultaneously**, and that any verifier implementation is
|
|||
|
|
interchangeable.
|
|||
|
|
|
|||
|
|
**Phase gate criterion:** fail-closed proven by fault injection.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
- **The framework cannot break the agent it runs.** Verification failures degrade
|
|||
|
|
the record, never the work. Any path where a verifier can fail into an agent's
|
|||
|
|
execution is a defect — and the composition is where such a path appears, since
|
|||
|
|
no single verifier test runs a full fleet.
|
|||
|
|
- The properties no single P4 task owns:
|
|||
|
|
- a **panicking** verifier must not disturb a **concurrent** healthy verifier's
|
|||
|
|
snapshot, nor the run;
|
|||
|
|
- the barrier must hold while **all three** external mutators fire, including
|
|||
|
|
at once;
|
|||
|
|
- laziness must survive fan-out — one text-reading verifier must not cause the
|
|||
|
|
exit-code verifiers beside it to fetch anything;
|
|||
|
|
- retention must refuse while **any** verifier is outstanding, which is the
|
|||
|
|
`Verifying` case T4.4 checks and the composition makes real.
|
|||
|
|
- Verification decides, grading explains. A gate that lets a verifier outcome be
|
|||
|
|
influenced by grading state has collapsed the two.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Assemble a verifier fleet on one attempt: exit-code, text-reading, panicking,
|
|||
|
|
hanging, erroring — five implementations, one fan-out.
|
|||
|
|
2. Fire all three snapshot mutators concurrently while the fleet is mid-flight.
|
|||
|
|
3. Assert every fail-closed outcome, every unaffected outcome, the blob call
|
|||
|
|
count, and the retention refusal, in one test.
|
|||
|
|
4. Swap the fleet for a completely different set of verifier implementations;
|
|||
|
|
assert the run's outcome shape is unchanged.
|
|||
|
|
5. Make this the required CI job gating P5.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- With the five-verifier fleet on one attempt: the three bad verifiers resolve to
|
|||
|
|
`Fail`, the two good ones return their own results, and **the run completes**.
|
|||
|
|
- The snapshot holds across all three mutators fired concurrently.
|
|||
|
|
- Exit-code verifiers record **zero** blob reads even while a sibling reads text.
|
|||
|
|
- Reduction is refused for the whole duration of `Verifying`.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** five verifier implementations; a counting `BlobStore` wrapper per
|
|||
|
|
verifier, not one shared counter; rendezvous channels so mutators fire only once
|
|||
|
|
every verifier has demonstrably parked.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_p4_composition.rs`:
|
|||
|
|
1. **Fleet fail-closed:** assert panic → `Fail`, hang → `Fail`, error → `Fail`,
|
|||
|
|
and that the exit-code and text verifiers return their **own** correct
|
|||
|
|
results. A fleet where one bad verifier poisons the others is the failure.
|
|||
|
|
2. **Run survives:** assert the run reaches a terminal state and that agent work
|
|||
|
|
was never blocked — compare run duration against a no-verifier baseline.
|
|||
|
|
3. **Concurrent mutators:** with the fleet parked, fire a rewind, a cancel and an
|
|||
|
|
intent recovery **at once**. Assert every held `EpisodeView` is byte-identical
|
|||
|
|
to its entry snapshot.
|
|||
|
|
4. **Per-verifier laziness:** assert the exit-code verifiers' counters are
|
|||
|
|
**exactly 0** while the text verifier's is exactly the number of refs it
|
|||
|
|
requested. A single shared counter hides this.
|
|||
|
|
5. **Retention interlock:** attempt reduction at three points — during
|
|||
|
|
`Verifying`, after `Verifying` but during `Grading`, and after terminal
|
|||
|
|
`Ungraded`. Assert refuse, refuse, permit.
|
|||
|
|
6. **Verifier interchange:** re-run the entire test with a second, unrelated set
|
|||
|
|
of five verifier implementations. Assert identical run-level outcomes.
|
|||
|
|
7. **Isolation after panic:** run a healthy verifier in the same process **after**
|
|||
|
|
the panicking one; assert `Pass`. A poisoned mutex passes every earlier step.
|
|||
|
|
8. **Regression:** re-run P0–P3 gates and all P4 suites.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p verify --test it_p4_composition -- --test-threads=1`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Running the five verifiers **sequentially**. Fan-out is the declared
|
|||
|
|
concurrency shape, and cross-contamination between verifiers only exists
|
|||
|
|
concurrently.
|
|||
|
|
- Step 3 firing the mutators one at a time. Each is already covered by T4.2; the
|
|||
|
|
gate is about them arriving together.
|
|||
|
|
- One shared blob counter in step 4 — the sibling's reads mask the exit-code
|
|||
|
|
verifiers' zero.
|
|||
|
|
- Step 5 testing only the `Grading` refusal, leaving the `Ungraded`-is-permitted
|
|||
|
|
half untested. That is the half that strands storage in production.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Catching the verifier panic in the test rather than asserting the framework
|
|||
|
|
catches it.
|
|||
|
|
- Asserting the run "did not error" instead of asserting it reached a terminal
|
|||
|
|
state with a complete log.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background (not required to do this task):
|
|||
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §1, §10 ·
|
|||
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|