poimen/tasks/T5.2-capacitylimits-and-the-residency-invariant.md

T5.2 — CapacityLimits and the residency invariant

Field	Value
Phase	P5 — Grading
Size	M — 1 to 3 days
Status	Not started
Flags	—
Spec	inlined below
Blocks	T5.3

Goal

Declared VRAM limits, the residency invariant, and a max_context_tokens that is derived from them rather than configured beside them.

Facts (inlined — no spec read needed)

pub struct CapacityLimits {
    /// Per-device VRAM this framework may use. Not the card's total — leave
    /// headroom for anything else sharing the device.
    pub vram_bytes_per_device: u64,
    pub devices: u32,
    /// Hard cap on simultaneously resident models across all devices.
    pub max_resident_models: u32,
    /// Never evictable. The agent's model belongs here; if it can be evicted,
    /// grading can stall agent work, which the first principle forbids.
    pub pinned: Vec<ModelId>,
}

pub struct ModelProfile {
    pub id: ModelId,
    pub weights_bytes: u64,
    /// KV cache cost per token at the deployed dtype and parallelism.
    pub kv_bytes_per_token: u64,
    /// Devices this model spans under tensor parallelism.
    pub devices_required: u32,
}

Invariant, checked at load and before any admission:

sum(weights of resident models) + peak_concurrent_kv  ≤  vram_bytes_per_device × devices
resident_model_count                                  ≤  max_resident_models

Derived ceiling — never a separate config knob:

max_context_tokens = (vram_bytes_per_device × devices − sum(weights)) / kv_bytes_per_token

• Metering counts tokens after the fact; on self-hosted weights the binding limit arrives earlier and harder. A model that is not resident cannot be inferred against, and making it resident means evicting something else and paying a load measured in tens of seconds.
• One resident model is the default configuration. The agent's model is pinned; the judge and the proposer run on that same model. A strategy naming a second model is legal and is rejected at load unless the invariant holds with both resident — never by swapping between them per call.
• The worked case, because the result is not marginal: at GQA fp16, per-token KV runs roughly 0.13 MB for an 8B-class model and 0.33 MB for a 70B-class one. Two episodes at a 200k-token retention ceiling is 400k tokens of context — 52 GB of KV cache at 8B, 131 GB at 70B, before weights. Neither fits an 80 GB device. A judge reading two full-ceiling episodes is not expensive, it is impossible.
• Therefore the retention ceiling is derived from this, not set beside it: the reduction target is max_context_tokens / 2, and where that is smaller than the configured ceiling, the bound wins.
• Distributed GPUs change the arithmetic, not the rule. devices_required expresses tensor parallelism; max_resident_models is a fleet-wide count, so two nodes each holding the agent model are two resident instances, not one.
• kv_bytes_per_token varies with dtype, quantization, attention implementation and parallelism. It is measured per deployment or read from an operator-supplied profile — the framework refuses a guess.

Steps

1. Define CapacityLimits and ModelProfile as above. No Default for kv_bytes_per_token — an absent value is a load error, not a guess.
2. Write check_residency(&CapacityLimits, &[ModelProfile]) -> Result<(), CapacityError> implementing both inequalities. CapacityError names the model and the shortfall in bytes.
3. Write derive_max_context_tokens(...) from the formula above. Expose it as a function, not a settable field.
4. Call the check at load, and again before admission (T8.6 enforces the runtime half).
5. Refuse a context request over the derived ceiling with the computed limit in the error message.
6. Encode the worked case as a test: 70B-class profile, 0.33 MB/token, 80 GB device, two 200k-token episodes → refused.

Acceptance

• A config whose resident set exceeds VRAM is rejected at load, naming the model and the shortfall — not at first inference.
• A judge context request exceeding the derived ceiling is refused with the computed limit in the error.
• The worked case asserts the arithmetic: 70B-class, 0.33 MB/token, 80 GB device, two 200k-token episodes must be refused.

Verify

Harness: pure arithmetic — no GPU required. Model profiles as fixtures with declared weights_bytes and kv_bytes_per_token.

Integration test — tests/it_capacity_invariant.rs:

1. The worked case, hardcoded: 70B-class profile, kv_bytes_per_token = 0.33 MB, one 80 GB device. Two 200k-token episodes = 400k tokens = 131 GB of KV before weights. Assert the request is refused and that the error carries the computed limit.
2. Same arithmetic at 8B / 0.13 MB per token: 52 GB. Assert also refused on an 80 GB device once weights are counted.
3. Over-capacity config at load: resident set exceeding VRAM → rejected at load, error naming the model and the byte shortfall. Assert the message contains both.
4. Assert max_context_tokens is only reachable as a function, not a settable field — trybuild if it is a private field with no setter.
5. Multi-device: devices_required = 4 under tensor parallelism; assert the invariant uses vram × devices and that max_resident_models is counted fleet-wide, so the same model on two nodes counts as two.
6. Missing kv_bytes_per_token → load error, not a default. Assert no Default impl exists.
7. Pinned model: assert no code path can evict a pinned model to satisfy the invariant — the eviction candidate list excludes pinned entries.

Command: cargo test -p capacity invariant

False pass:

• Asserting only that "some error" is returned. The acceptance criterion is that the arithmetic is right — pin the expected byte figures in the test, so an off-by-a-factor error in the KV formula is caught rather than rounded away.
• Step 3 checked at first inference rather than at load. Both produce an error; only one produces it before 3am.
• Testing with a permissive config where everything fits. Every implementation passes that.

Traps

• A max_context_tokens config field. It will be set to something plausible and discovered wrong at the first judge call.
• Treating "call the judge model" as equivalent in cost to "call the agent model". On this hardware that is wrong by two orders of magnitude.
• Allowing an eviction of a pinned model to satisfy the invariant.

---

Background (not required to do this task): rust-agentic-sys.md §8.6, §14.2, §18 · rust-agentic-task.md