117 lines
5.4 KiB
Markdown
117 lines
5.4 KiB
Markdown
# T8.7 — P8 composition gate
|
||||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | P8 — Operability |
|
|||
|
|
| Size | L — over 3 days |
|
|||
|
|
| Status | Not started |
|
|||
|
|
| Flags | gate |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | — release gate |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Prove the operability layer composes with everything before it — and that turning
|
|||
|
|
all of P8 on does not break the zero-infrastructure embedded product.
|
|||
|
|
|
|||
|
|
**Phase gate criterion:** metering, metrics, sandbox enforcement and capacity
|
|||
|
|
admission all active, with the embedded smoke crate still green.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
- P8's features all sit **across** the rest of the system rather than beside it,
|
|||
|
|
so each one's own test is necessarily narrow:
|
|||
|
|
- metering attributes calls the executor and grader make;
|
|||
|
|
- metrics observe every stage;
|
|||
|
|
- keyed capability constrains every tool call;
|
|||
|
|
- reduction rewrites what grading and verification later read;
|
|||
|
|
- admission control gates work the scheduler wants to run.
|
|||
|
|
- The properties no single P8 task owns:
|
|||
|
|
- **a reduced episode is still gradeable** — reduction and grading are built by
|
|||
|
|
different tasks and read the same blobs;
|
|||
|
|
- **budget and capacity refusals are distinguishable** end to end, since they
|
|||
|
|
look identical in a queue-depth graph and have opposite fixes;
|
|||
|
|
- **metering totals reconcile** with the model provider's own call count —
|
|||
|
|
attribution that misses a call path understates spend silently;
|
|||
|
|
- **the embedded smoke crate still passes with every P8 feature enabled.** If
|
|||
|
|
it does not, an operability feature has leaked a service dependency into the
|
|||
|
|
default path.
|
|||
|
|
- The retention ceiling is derived from hardware, not chosen: where
|
|||
|
|
`max_context_tokens / 2` is tighter than the configured ceiling, the bound
|
|||
|
|
wins. That interaction spans T5.2 and T8.4 and belongs to neither.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Enable every P8 feature and run the full behavioural suite from T7.8.
|
|||
|
|
2. Run the reduce-then-grade path: reduce a run, then grade the reduced episode.
|
|||
|
|
3. Reconcile metering totals against the provider's own counter.
|
|||
|
|
4. Run the embedded smoke crate with all features on.
|
|||
|
|
5. Drive both refusal reasons and assert they are separable everywhere.
|
|||
|
|
6. Make this the release gate.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- The full behavioural suite passes with **all P8 features enabled**, in both
|
|||
|
|
deployment modes.
|
|||
|
|
- A **reduced** episode is still gradeable and still produces a valid `Score`.
|
|||
|
|
- Metering totals reconcile exactly with the provider's call count.
|
|||
|
|
- The embedded smoke crate passes with every feature on.
|
|||
|
|
- Capacity and budget refusals are distinguishable end to end.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** T7.8's mode-parameterized suite; the provider call counter; the
|
|||
|
|
external side-effect ledger; the out-of-workspace smoke crate; a scrapeable
|
|||
|
|
metrics recorder.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_p8_composition.rs`:
|
|||
|
|
1. **Everything on:** run the T7.8 behavioural suite with metering, metrics,
|
|||
|
|
keyed capability, reduction and admission control all enabled, in both modes.
|
|||
|
|
2. **Reduce then grade:** reduce a run past the ceiling, then grade it. Assert a
|
|||
|
|
valid `Score`, and assert the grading path handled `get` → `None` on reduced
|
|||
|
|
originals by following the `Reduced` event. Then drop state, re-fold, and
|
|||
|
|
assert byte-identical.
|
|||
|
|
3. **Derived ceiling interaction:** configure a 200k retention ceiling with a
|
|||
|
|
hardware-derived bound of 80k. Assert reduction targets **80k** and that the
|
|||
|
|
judge's context request is admitted.
|
|||
|
|
4. **Metering reconciliation:** assert `sum(metered calls)` equals the provider's
|
|||
|
|
own counter, and that every call carries a non-default `Purpose`. A mismatch
|
|||
|
|
means a call path is unattributed.
|
|||
|
|
5. **Refusal separability:** drive one capacity refusal and one budget refusal.
|
|||
|
|
Assert two distinct metric series moved, that each error names its limit, and
|
|||
|
|
that the budget-refused run reaches terminal `Ungraded` while the
|
|||
|
|
capacity-refused work was never queued.
|
|||
|
|
6. **Keyed capability under the real interpreter:** assert every tool call in the
|
|||
|
|
suite carried a kernel-supplied key or was denied and audited — count against
|
|||
|
|
the intent records, not against the tool's own reporting.
|
|||
|
|
7. **Zero swaps** across the whole gate.
|
|||
|
|
8. **Smoke crate:** run it, outside the workspace, default features only, with
|
|||
|
|
every P8 feature active. Assert no network listener and no distributed-only
|
|||
|
|
crate in `cargo tree`.
|
|||
|
|
9. **Regression:** re-run P0–P7 gates.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p operability --test it_p8_composition && cargo test --manifest-path smoke-crate/Cargo.toml`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Running each P8 feature's suite with the others off. Every one is green that
|
|||
|
|
way; the interactions — reduction versus grading, metering versus purpose
|
|||
|
|
attribution, admission versus the join stage — are the reason this gate exists.
|
|||
|
|
- Step 2 asserting the reduced run produced *a* score. Assert the re-fold too:
|
|||
|
|
a mapping stored outside the log passes the grading half and destroys
|
|||
|
|
drop-and-re-fold.
|
|||
|
|
- Step 4 comparing metered totals to themselves. The **provider's** counter is
|
|||
|
|
the independent witness.
|
|||
|
|
- Step 8 run with a non-default feature enabled to make it build. That is the
|
|||
|
|
leak, papered over.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Treating this as a formality once P0–P7 are green. Every property here spans
|
|||
|
|
tasks that were each verified in isolation, and P8's features touch all of them.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background (not required to do this task):
|
|||
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §8.6, §13, §14, §15 ·
|
|||
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|