poimen/tasks/T8.4-reduction-and-tiering.md

T8.4 — Reduction and tiering

Field	Value
Phase	P8 — Operability
Size	L — over 3 days
Status	Not started
Flags	—
Spec	inlined below
Blocks	—

Goal

Reduce episodes to a token ceiling, then move terminal reduced runs to cold storage. Never rewrite a blob body; never edit a log record.

Facts (inlined — no spec read needed)

Vocabulary, because the previous revision collided two of these: Archived is a run state; reduction is the token-budget operation; tiering is the move to cold storage.

• Retention is measured in tokens, because tokens are the currency of every downstream consumer: what a replay costs, what fits in a judge's window, what an export is billed at. Default ceiling 200k per run, per-tenant configurable.
• On self-hosted weights that ceiling is not free to choose. It is bounded by max_context_tokens / 2 (T5.2), because a pairwise judge reads two episodes into one KV cache and two 200k episodes do not fit on any single device. Where the derived bound is tighter, the bound wins and the judge reads a further-reduced view.

Loss order is fixed:

Kept	Reduced	Dropped
every transition record	blob bodies → summary blob	raw text on dead branches
context partitions, tool info, usage	dead-branch attempts → summary
verifier results, grades

• Reduction never rewrites a blob and never edits the log. Blobs are content-addressed, so replacing a body under its existing ref makes the ref a lie; repointing the log at a new ref is the history rewrite the append-only rules forbid.
• Instead: write the summary as a new blob, append a Reduced { original: BlobRef, summary: BlobRef } event, and only then delete the original body. The reduction is a later fact about an earlier record, not a change to it. BlobStore::get on the original returns None, and the fold knows why and what stands in its place — so drop-and-re-fold from LSN 0 still yields byte-identical state, which it would not if the mapping lived only in the projection.
• The transition sequence always survives. What reduces is text, because it dominates token count and is the only part with a cheap lossy representation. A reduced episode can still be graded, attributed and structurally rewound — it just cannot be replayed verbatim.
• Tiering: moved, not copied, with local rows deleted only after the remote commit acknowledges.
• Eligibility is Graded/Ungraded/Archived (T4.4) — grading terminated.
• Open: what produces the summary. A model call makes reduction non-deterministic, which interacts badly with replay; extractive or structural reduction may suffice.

Steps

1. Compute the effective ceiling as min(configured_ceiling, max_context_tokens / 2) from T5.2. Never read the configured value alone.
2. Gate on T4.4's is_reducible. Reject, do not skip.
3. Reduce in the fixed loss order: blob bodies to summaries first, then dead-branch attempts, then raw text on dead branches.
4. For each reduction: put the summary blob → commit the Reduced { original, summary } event → then delete the original body. That order survives a crash at any point.
5. Make the fold handle Reduced so a None from get is explained by the log.
6. Tiering: write to cold storage, wait for the remote commit acknowledgement, then delete local rows. Crash between the two leaves a duplicate, not a loss — so make the move idempotent.
7. Verify by re-hashing every surviving BlobRef against its body.

Acceptance

• A run exceeding the ceiling reduces; transitions survive intact; a reduced episode is still gradeable.
• Every surviving BlobRef still hashes to its content — asserted by re-hashing, since a rewritten body type-checks silently.
• Drop the state tables and re-fold a reduced run from LSN 0: state is byte-identical. This fails if the original→summary mapping lives anywhere but the log.
• Tiering is idempotent under a crash between remote commit and local delete.

Verify

Harness: a run built to exceed the token ceiling, T0.8's assert_refold_identical, and a cold-storage stub that can fail between remote commit and local delete.

Integration test — tests/it_reduction_tiering.rs:

1. Reduce a run over the ceiling. Assert the transition sequence is intact — compare the event list before and after; only Reduced events were added, none removed or altered.
2. Assert the reduced episode is still gradeable: run it through the grading path and assert a real Score.
3. Re-hash every surviving BlobRef against its body and assert equality. A rewritten body type-checks silently and this is the only assertion that sees it.
4. The load-bearing test: drop the state tables, re-fold the reduced run from LSN 0, assert byte-identical state. This fails if the original→summary mapping lives anywhere but the log.
5. Assert get on an original ref returns Ok(None) and that the fold explains it via the Reduced event.
6. Ordering under crash: arm faults at each of the three points (after put, after commit, after delete). Assert no state exists where a record points at a deleted body with no Reduced event.
7. Ceiling derivation: configure a 200k ceiling with a hardware-derived bound of 80k; assert the bound wins and reduction targets 80k.
8. Tiering idempotence: crash between remote commit and local delete; re-run. Assert one copy remains, no data lost, operation converges.
9. Assert reduction refuses on a run in Grading (T4.4).

Command: cargo test -p retention --features test-hooks reduction -- --test-threads=1

False pass:

• Step 4 omitted. Storing the original→summary mapping in the projection makes steps 1–3 pass perfectly and silently destroys drop-and-re-fold, which is the property the whole durability design rests on.
• Step 3 replaced by "the ref still resolves". A rewritten body resolves fine.
• Step 7 omitted: reading the configured ceiling alone passes every test on hardware nobody checked, and fails at the first real judge call.
• Step 6 with faults only after delete, which is the safe point.

Traps

• Overwriting the blob body under its existing ref. It type-checks, it saves an event, and it makes the ref a lie.
• Repointing the log record at the summary ref. That is a history rewrite.
• Deleting the original before the Reduced event commits. A crash then leaves a ref with no body and no explanation.
• Reading the configured ceiling without the hardware-derived bound.

---

Background (not required to do this task): rust-agentic-sys.md §8.6, §8.7, §10.3, §14.2, §18 · rust-agentic-task.md