153 lines
6.7 KiB
Markdown
153 lines
6.7 KiB
Markdown
# T5.2 — `CapacityLimits` and the residency invariant
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | P5 — Grading |
|
||
| Size | M — 1 to 3 days |
|
||
| Status | Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | T5.3 |
|
||
|
||
## Goal
|
||
|
||
Declared VRAM limits, the residency invariant, and a `max_context_tokens` that is
|
||
**derived** from them rather than configured beside them.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
```rust
|
||
pub struct CapacityLimits {
|
||
/// Per-device VRAM this framework may use. Not the card's total — leave
|
||
/// headroom for anything else sharing the device.
|
||
pub vram_bytes_per_device: u64,
|
||
pub devices: u32,
|
||
/// Hard cap on simultaneously resident models across all devices.
|
||
pub max_resident_models: u32,
|
||
/// Never evictable. The agent's model belongs here; if it can be evicted,
|
||
/// grading can stall agent work, which the first principle forbids.
|
||
pub pinned: Vec<ModelId>,
|
||
}
|
||
|
||
pub struct ModelProfile {
|
||
pub id: ModelId,
|
||
pub weights_bytes: u64,
|
||
/// KV cache cost per token at the deployed dtype and parallelism.
|
||
pub kv_bytes_per_token: u64,
|
||
/// Devices this model spans under tensor parallelism.
|
||
pub devices_required: u32,
|
||
}
|
||
```
|
||
|
||
Invariant, checked at load **and** before any admission:
|
||
|
||
```
|
||
sum(weights of resident models) + peak_concurrent_kv ≤ vram_bytes_per_device × devices
|
||
resident_model_count ≤ max_resident_models
|
||
```
|
||
|
||
Derived ceiling — never a separate config knob:
|
||
|
||
```
|
||
max_context_tokens = (vram_bytes_per_device × devices − sum(weights)) / kv_bytes_per_token
|
||
```
|
||
|
||
- Metering counts tokens after the fact; on self-hosted weights the binding limit
|
||
arrives earlier and harder. A model that is not resident cannot be inferred
|
||
against, and making it resident means evicting something else and paying a load
|
||
measured in tens of seconds.
|
||
- **One resident model is the default configuration.** The agent's model is
|
||
pinned; the judge and the proposer run on that same model. A strategy naming a
|
||
second model is legal and is rejected at load unless the invariant holds with
|
||
both resident — **never** by swapping between them per call.
|
||
- The worked case, because the result is not marginal: at GQA fp16, per-token KV
|
||
runs roughly 0.13 MB for an 8B-class model and 0.33 MB for a 70B-class one. Two
|
||
episodes at a 200k-token retention ceiling is 400k tokens of context —
|
||
**52 GB of KV cache at 8B, 131 GB at 70B**, before weights. Neither fits an
|
||
80 GB device. A judge reading two full-ceiling episodes is not expensive, it is
|
||
impossible.
|
||
- Therefore the retention ceiling is derived from this, not set beside it: the
|
||
reduction target is `max_context_tokens / 2`, and where that is smaller than
|
||
the configured ceiling, **the bound wins**.
|
||
- Distributed GPUs change the arithmetic, not the rule. `devices_required`
|
||
expresses tensor parallelism; `max_resident_models` is a fleet-wide count, so
|
||
two nodes each holding the agent model are two resident instances, not one.
|
||
- `kv_bytes_per_token` varies with dtype, quantization, attention implementation
|
||
and parallelism. It is measured per deployment or read from an operator-supplied
|
||
profile — **the framework refuses a guess.**
|
||
|
||
## Steps
|
||
|
||
1. Define `CapacityLimits` and `ModelProfile` as above. No `Default` for
|
||
`kv_bytes_per_token` — an absent value is a load error, not a guess.
|
||
2. Write `check_residency(&CapacityLimits, &[ModelProfile]) -> Result<(), CapacityError>`
|
||
implementing both inequalities. `CapacityError` names the model and the
|
||
shortfall in bytes.
|
||
3. Write `derive_max_context_tokens(...)` from the formula above. Expose it as a
|
||
function, not a settable field.
|
||
4. Call the check at load, and again before admission (T8.6 enforces the runtime
|
||
half).
|
||
5. Refuse a context request over the derived ceiling with the computed limit in
|
||
the error message.
|
||
6. Encode the worked case as a test: 70B-class profile, 0.33 MB/token, 80 GB
|
||
device, two 200k-token episodes → refused.
|
||
|
||
## Acceptance
|
||
|
||
- A config whose resident set exceeds VRAM is rejected **at load**, naming the
|
||
model and the shortfall — not at first inference.
|
||
- A judge context request exceeding the derived ceiling is refused with the
|
||
computed limit in the error.
|
||
- The worked case asserts the arithmetic: 70B-class, 0.33 MB/token, 80 GB device,
|
||
two 200k-token episodes must be refused.
|
||
|
||
## Verify
|
||
|
||
**Harness:** pure arithmetic — no GPU required. Model profiles as fixtures with
|
||
declared `weights_bytes` and `kv_bytes_per_token`.
|
||
|
||
**Integration test** — `tests/it_capacity_invariant.rs`:
|
||
1. **The worked case, hardcoded:** 70B-class profile, `kv_bytes_per_token =
|
||
0.33 MB`, one 80 GB device. Two 200k-token episodes = 400k tokens = 131 GB of
|
||
KV before weights. Assert the request is **refused** and that the error
|
||
carries the computed limit.
|
||
2. Same arithmetic at 8B / 0.13 MB per token: 52 GB. Assert also refused on an
|
||
80 GB device once weights are counted.
|
||
3. Over-capacity config at load: resident set exceeding VRAM → rejected **at
|
||
load**, error naming **the model and the byte shortfall**. Assert the message
|
||
contains both.
|
||
4. Assert `max_context_tokens` is only reachable as a **function**, not a
|
||
settable field — `trybuild` if it is a private field with no setter.
|
||
5. Multi-device: `devices_required = 4` under tensor parallelism; assert the
|
||
invariant uses `vram × devices` and that `max_resident_models` is counted
|
||
fleet-wide, so the same model on two nodes counts as two.
|
||
6. Missing `kv_bytes_per_token` → load error, **not** a default. Assert no
|
||
`Default` impl exists.
|
||
7. Pinned model: assert no code path can evict a pinned model to satisfy the
|
||
invariant — the eviction candidate list excludes pinned entries.
|
||
|
||
**Command:** `cargo test -p capacity invariant`
|
||
|
||
**False pass:**
|
||
- Asserting only that "some error" is returned. The acceptance criterion is that
|
||
the **arithmetic** is right — pin the expected byte figures in the test, so an
|
||
off-by-a-factor error in the KV formula is caught rather than rounded away.
|
||
- Step 3 checked at first inference rather than at load. Both produce an error;
|
||
only one produces it before 3am.
|
||
- Testing with a permissive config where everything fits. Every implementation
|
||
passes that.
|
||
|
||
## Traps
|
||
|
||
- A `max_context_tokens` config field. It will be set to something plausible and
|
||
discovered wrong at the first judge call.
|
||
- Treating "call the judge model" as equivalent in cost to "call the agent
|
||
model". On this hardware that is wrong by two orders of magnitude.
|
||
- Allowing an eviction of a pinned model to satisfy the invariant.
|
||
|
||
---
|
||
|
||
Background (not required to do this task):
|
||
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §8.6, §14.2, §18 ·
|
||
[rust-agentic-task.md](../../../rust-agentic-task.md)
|