(tasks) add tasks for harness
This commit is contained in:
@@ -0,0 +1,116 @@
|
||||
# T8.7 — P8 composition gate
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | P8 — Operability |
|
||||
| Size | L — over 3 days |
|
||||
| Status | Not started |
|
||||
| Flags | gate |
|
||||
| Spec | inlined below |
|
||||
| Blocks | — release gate |
|
||||
|
||||
## Goal
|
||||
|
||||
Prove the operability layer composes with everything before it — and that turning
|
||||
all of P8 on does not break the zero-infrastructure embedded product.
|
||||
|
||||
**Phase gate criterion:** metering, metrics, sandbox enforcement and capacity
|
||||
admission all active, with the embedded smoke crate still green.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
- P8's features all sit **across** the rest of the system rather than beside it,
|
||||
so each one's own test is necessarily narrow:
|
||||
- metering attributes calls the executor and grader make;
|
||||
- metrics observe every stage;
|
||||
- keyed capability constrains every tool call;
|
||||
- reduction rewrites what grading and verification later read;
|
||||
- admission control gates work the scheduler wants to run.
|
||||
- The properties no single P8 task owns:
|
||||
- **a reduced episode is still gradeable** — reduction and grading are built by
|
||||
different tasks and read the same blobs;
|
||||
- **budget and capacity refusals are distinguishable** end to end, since they
|
||||
look identical in a queue-depth graph and have opposite fixes;
|
||||
- **metering totals reconcile** with the model provider's own call count —
|
||||
attribution that misses a call path understates spend silently;
|
||||
- **the embedded smoke crate still passes with every P8 feature enabled.** If
|
||||
it does not, an operability feature has leaked a service dependency into the
|
||||
default path.
|
||||
- The retention ceiling is derived from hardware, not chosen: where
|
||||
`max_context_tokens / 2` is tighter than the configured ceiling, the bound
|
||||
wins. That interaction spans T5.2 and T8.4 and belongs to neither.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Enable every P8 feature and run the full behavioural suite from T7.8.
|
||||
2. Run the reduce-then-grade path: reduce a run, then grade the reduced episode.
|
||||
3. Reconcile metering totals against the provider's own counter.
|
||||
4. Run the embedded smoke crate with all features on.
|
||||
5. Drive both refusal reasons and assert they are separable everywhere.
|
||||
6. Make this the release gate.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- The full behavioural suite passes with **all P8 features enabled**, in both
|
||||
deployment modes.
|
||||
- A **reduced** episode is still gradeable and still produces a valid `Score`.
|
||||
- Metering totals reconcile exactly with the provider's call count.
|
||||
- The embedded smoke crate passes with every feature on.
|
||||
- Capacity and budget refusals are distinguishable end to end.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** T7.8's mode-parameterized suite; the provider call counter; the
|
||||
external side-effect ledger; the out-of-workspace smoke crate; a scrapeable
|
||||
metrics recorder.
|
||||
|
||||
**Integration test** — `tests/it_p8_composition.rs`:
|
||||
1. **Everything on:** run the T7.8 behavioural suite with metering, metrics,
|
||||
keyed capability, reduction and admission control all enabled, in both modes.
|
||||
2. **Reduce then grade:** reduce a run past the ceiling, then grade it. Assert a
|
||||
valid `Score`, and assert the grading path handled `get` → `None` on reduced
|
||||
originals by following the `Reduced` event. Then drop state, re-fold, and
|
||||
assert byte-identical.
|
||||
3. **Derived ceiling interaction:** configure a 200k retention ceiling with a
|
||||
hardware-derived bound of 80k. Assert reduction targets **80k** and that the
|
||||
judge's context request is admitted.
|
||||
4. **Metering reconciliation:** assert `sum(metered calls)` equals the provider's
|
||||
own counter, and that every call carries a non-default `Purpose`. A mismatch
|
||||
means a call path is unattributed.
|
||||
5. **Refusal separability:** drive one capacity refusal and one budget refusal.
|
||||
Assert two distinct metric series moved, that each error names its limit, and
|
||||
that the budget-refused run reaches terminal `Ungraded` while the
|
||||
capacity-refused work was never queued.
|
||||
6. **Keyed capability under the real interpreter:** assert every tool call in the
|
||||
suite carried a kernel-supplied key or was denied and audited — count against
|
||||
the intent records, not against the tool's own reporting.
|
||||
7. **Zero swaps** across the whole gate.
|
||||
8. **Smoke crate:** run it, outside the workspace, default features only, with
|
||||
every P8 feature active. Assert no network listener and no distributed-only
|
||||
crate in `cargo tree`.
|
||||
9. **Regression:** re-run P0–P7 gates.
|
||||
|
||||
**Command:** `cargo test -p operability --test it_p8_composition && cargo test --manifest-path smoke-crate/Cargo.toml`
|
||||
|
||||
**False pass:**
|
||||
- Running each P8 feature's suite with the others off. Every one is green that
|
||||
way; the interactions — reduction versus grading, metering versus purpose
|
||||
attribution, admission versus the join stage — are the reason this gate exists.
|
||||
- Step 2 asserting the reduced run produced *a* score. Assert the re-fold too:
|
||||
a mapping stored outside the log passes the grading half and destroys
|
||||
drop-and-re-fold.
|
||||
- Step 4 comparing metered totals to themselves. The **provider's** counter is
|
||||
the independent witness.
|
||||
- Step 8 run with a non-default feature enabled to make it build. That is the
|
||||
leak, papered over.
|
||||
|
||||
## Traps
|
||||
|
||||
- Treating this as a formality once P0–P7 are green. Every property here spans
|
||||
tasks that were each verified in isolation, and P8's features touch all of them.
|
||||
|
||||
---
|
||||
|
||||
Background (not required to do this task):
|
||||
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §8.6, §13, §14, §15 ·
|
||||
[rust-agentic-task.md](../../../rust-agentic-task.md)
|
||||
Reference in New Issue
Block a user