117 lines
5.4 KiB
Markdown
117 lines
5.4 KiB
Markdown
# T8.7 — P8 composition gate
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | P8 — Operability |
|
||
| Size | L — over 3 days |
|
||
| Status | Not started |
|
||
| Flags | gate |
|
||
| Spec | inlined below |
|
||
| Blocks | — release gate |
|
||
|
||
## Goal
|
||
|
||
Prove the operability layer composes with everything before it — and that turning
|
||
all of P8 on does not break the zero-infrastructure embedded product.
|
||
|
||
**Phase gate criterion:** metering, metrics, sandbox enforcement and capacity
|
||
admission all active, with the embedded smoke crate still green.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
- P8's features all sit **across** the rest of the system rather than beside it,
|
||
so each one's own test is necessarily narrow:
|
||
- metering attributes calls the executor and grader make;
|
||
- metrics observe every stage;
|
||
- keyed capability constrains every tool call;
|
||
- reduction rewrites what grading and verification later read;
|
||
- admission control gates work the scheduler wants to run.
|
||
- The properties no single P8 task owns:
|
||
- **a reduced episode is still gradeable** — reduction and grading are built by
|
||
different tasks and read the same blobs;
|
||
- **budget and capacity refusals are distinguishable** end to end, since they
|
||
look identical in a queue-depth graph and have opposite fixes;
|
||
- **metering totals reconcile** with the model provider's own call count —
|
||
attribution that misses a call path understates spend silently;
|
||
- **the embedded smoke crate still passes with every P8 feature enabled.** If
|
||
it does not, an operability feature has leaked a service dependency into the
|
||
default path.
|
||
- The retention ceiling is derived from hardware, not chosen: where
|
||
`max_context_tokens / 2` is tighter than the configured ceiling, the bound
|
||
wins. That interaction spans T5.2 and T8.4 and belongs to neither.
|
||
|
||
## Steps
|
||
|
||
1. Enable every P8 feature and run the full behavioural suite from T7.8.
|
||
2. Run the reduce-then-grade path: reduce a run, then grade the reduced episode.
|
||
3. Reconcile metering totals against the provider's own counter.
|
||
4. Run the embedded smoke crate with all features on.
|
||
5. Drive both refusal reasons and assert they are separable everywhere.
|
||
6. Make this the release gate.
|
||
|
||
## Acceptance
|
||
|
||
- The full behavioural suite passes with **all P8 features enabled**, in both
|
||
deployment modes.
|
||
- A **reduced** episode is still gradeable and still produces a valid `Score`.
|
||
- Metering totals reconcile exactly with the provider's call count.
|
||
- The embedded smoke crate passes with every feature on.
|
||
- Capacity and budget refusals are distinguishable end to end.
|
||
|
||
## Verify
|
||
|
||
**Harness:** T7.8's mode-parameterized suite; the provider call counter; the
|
||
external side-effect ledger; the out-of-workspace smoke crate; a scrapeable
|
||
metrics recorder.
|
||
|
||
**Integration test** — `tests/it_p8_composition.rs`:
|
||
1. **Everything on:** run the T7.8 behavioural suite with metering, metrics,
|
||
keyed capability, reduction and admission control all enabled, in both modes.
|
||
2. **Reduce then grade:** reduce a run past the ceiling, then grade it. Assert a
|
||
valid `Score`, and assert the grading path handled `get` → `None` on reduced
|
||
originals by following the `Reduced` event. Then drop state, re-fold, and
|
||
assert byte-identical.
|
||
3. **Derived ceiling interaction:** configure a 200k retention ceiling with a
|
||
hardware-derived bound of 80k. Assert reduction targets **80k** and that the
|
||
judge's context request is admitted.
|
||
4. **Metering reconciliation:** assert `sum(metered calls)` equals the provider's
|
||
own counter, and that every call carries a non-default `Purpose`. A mismatch
|
||
means a call path is unattributed.
|
||
5. **Refusal separability:** drive one capacity refusal and one budget refusal.
|
||
Assert two distinct metric series moved, that each error names its limit, and
|
||
that the budget-refused run reaches terminal `Ungraded` while the
|
||
capacity-refused work was never queued.
|
||
6. **Keyed capability under the real interpreter:** assert every tool call in the
|
||
suite carried a kernel-supplied key or was denied and audited — count against
|
||
the intent records, not against the tool's own reporting.
|
||
7. **Zero swaps** across the whole gate.
|
||
8. **Smoke crate:** run it, outside the workspace, default features only, with
|
||
every P8 feature active. Assert no network listener and no distributed-only
|
||
crate in `cargo tree`.
|
||
9. **Regression:** re-run P0–P7 gates.
|
||
|
||
**Command:** `cargo test -p operability --test it_p8_composition && cargo test --manifest-path smoke-crate/Cargo.toml`
|
||
|
||
**False pass:**
|
||
- Running each P8 feature's suite with the others off. Every one is green that
|
||
way; the interactions — reduction versus grading, metering versus purpose
|
||
attribution, admission versus the join stage — are the reason this gate exists.
|
||
- Step 2 asserting the reduced run produced *a* score. Assert the re-fold too:
|
||
a mapping stored outside the log passes the grading half and destroys
|
||
drop-and-re-fold.
|
||
- Step 4 comparing metered totals to themselves. The **provider's** counter is
|
||
the independent witness.
|
||
- Step 8 run with a non-default feature enabled to make it build. That is the
|
||
leak, papered over.
|
||
|
||
## Traps
|
||
|
||
- Treating this as a formality once P0–P7 are green. Every property here spans
|
||
tasks that were each verified in isolation, and P8's features touch all of them.
|
||
|
||
---
|
||
|
||
Background (not required to do this task):
|
||
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §8.6, §13, §14, §15 ·
|
||
[rust-agentic-task.md](../../../rust-agentic-task.md)
|