109 lines
4.9 KiB
Markdown
109 lines
4.9 KiB
Markdown
# T4.5 — P4 composition gate
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | P4 — Verification |
|
||
| Size | M — 1 to 3 days |
|
||
| Status | Not started |
|
||
| Flags | gate |
|
||
| Spec | inlined below |
|
||
| Blocks | P5 |
|
||
|
||
## Goal
|
||
|
||
Prove the verifier port, the snapshot barrier, lazy blob access and the retention
|
||
guard hold **simultaneously**, and that any verifier implementation is
|
||
interchangeable.
|
||
|
||
**Phase gate criterion:** fail-closed proven by fault injection.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
- **The framework cannot break the agent it runs.** Verification failures degrade
|
||
the record, never the work. Any path where a verifier can fail into an agent's
|
||
execution is a defect — and the composition is where such a path appears, since
|
||
no single verifier test runs a full fleet.
|
||
- The properties no single P4 task owns:
|
||
- a **panicking** verifier must not disturb a **concurrent** healthy verifier's
|
||
snapshot, nor the run;
|
||
- the barrier must hold while **all three** external mutators fire, including
|
||
at once;
|
||
- laziness must survive fan-out — one text-reading verifier must not cause the
|
||
exit-code verifiers beside it to fetch anything;
|
||
- retention must refuse while **any** verifier is outstanding, which is the
|
||
`Verifying` case T4.4 checks and the composition makes real.
|
||
- Verification decides, grading explains. A gate that lets a verifier outcome be
|
||
influenced by grading state has collapsed the two.
|
||
|
||
## Steps
|
||
|
||
1. Assemble a verifier fleet on one attempt: exit-code, text-reading, panicking,
|
||
hanging, erroring — five implementations, one fan-out.
|
||
2. Fire all three snapshot mutators concurrently while the fleet is mid-flight.
|
||
3. Assert every fail-closed outcome, every unaffected outcome, the blob call
|
||
count, and the retention refusal, in one test.
|
||
4. Swap the fleet for a completely different set of verifier implementations;
|
||
assert the run's outcome shape is unchanged.
|
||
5. Make this the required CI job gating P5.
|
||
|
||
## Acceptance
|
||
|
||
- With the five-verifier fleet on one attempt: the three bad verifiers resolve to
|
||
`Fail`, the two good ones return their own results, and **the run completes**.
|
||
- The snapshot holds across all three mutators fired concurrently.
|
||
- Exit-code verifiers record **zero** blob reads even while a sibling reads text.
|
||
- Reduction is refused for the whole duration of `Verifying`.
|
||
|
||
## Verify
|
||
|
||
**Harness:** five verifier implementations; a counting `BlobStore` wrapper per
|
||
verifier, not one shared counter; rendezvous channels so mutators fire only once
|
||
every verifier has demonstrably parked.
|
||
|
||
**Integration test** — `tests/it_p4_composition.rs`:
|
||
1. **Fleet fail-closed:** assert panic → `Fail`, hang → `Fail`, error → `Fail`,
|
||
and that the exit-code and text verifiers return their **own** correct
|
||
results. A fleet where one bad verifier poisons the others is the failure.
|
||
2. **Run survives:** assert the run reaches a terminal state and that agent work
|
||
was never blocked — compare run duration against a no-verifier baseline.
|
||
3. **Concurrent mutators:** with the fleet parked, fire a rewind, a cancel and an
|
||
intent recovery **at once**. Assert every held `EpisodeView` is byte-identical
|
||
to its entry snapshot.
|
||
4. **Per-verifier laziness:** assert the exit-code verifiers' counters are
|
||
**exactly 0** while the text verifier's is exactly the number of refs it
|
||
requested. A single shared counter hides this.
|
||
5. **Retention interlock:** attempt reduction at three points — during
|
||
`Verifying`, after `Verifying` but during `Grading`, and after terminal
|
||
`Ungraded`. Assert refuse, refuse, permit.
|
||
6. **Verifier interchange:** re-run the entire test with a second, unrelated set
|
||
of five verifier implementations. Assert identical run-level outcomes.
|
||
7. **Isolation after panic:** run a healthy verifier in the same process **after**
|
||
the panicking one; assert `Pass`. A poisoned mutex passes every earlier step.
|
||
8. **Regression:** re-run P0–P3 gates and all P4 suites.
|
||
|
||
**Command:** `cargo test -p verify --test it_p4_composition -- --test-threads=1`
|
||
|
||
**False pass:**
|
||
- Running the five verifiers **sequentially**. Fan-out is the declared
|
||
concurrency shape, and cross-contamination between verifiers only exists
|
||
concurrently.
|
||
- Step 3 firing the mutators one at a time. Each is already covered by T4.2; the
|
||
gate is about them arriving together.
|
||
- One shared blob counter in step 4 — the sibling's reads mask the exit-code
|
||
verifiers' zero.
|
||
- Step 5 testing only the `Grading` refusal, leaving the `Ungraded`-is-permitted
|
||
half untested. That is the half that strands storage in production.
|
||
|
||
## Traps
|
||
|
||
- Catching the verifier panic in the test rather than asserting the framework
|
||
catches it.
|
||
- Asserting the run "did not error" instead of asserting it reached a terminal
|
||
state with a complete log.
|
||
|
||
---
|
||
|
||
Background (not required to do this task):
|
||
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §1, §10 ·
|
||
[rust-agentic-task.md](../../../rust-agentic-task.md)
|