(plan) system review and break down plans

2026-08-19 09:52:07 -07:00
commit 9d28b63ff2
46 changed files with 5620 additions and 0 deletions
@@ -0,0 +1,92 @@
+# M0.4 — Tokenizer-backed chunk sizing
+
+| Field | Value |
+|---|---|
+| Phase | M0 — Read-only spine |
+| Size | M — 1–3 days |
+| Status | ⬜ Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | M0.3 |
+
+## Goal
+
+Count tokens with the tokenizer the serving model actually uses, so a chunk that
+fits locally also fits at the gateway.
+
+## Facts (inlined — no spec read needed)
+
+The controller is `qwen2.5:3b-instruct` — Qwen2.5-3B-Instruct, the GRU-Mem
+paper's exact 3B backbone. Its tokenizer is the Qwen2 BPE.
+
+Budget arithmetic, and why an approximation is not good enough:
+
+```
+Ollama context cap        32768   (OLLAMA_CONTEXT_LENGTH, cluster-side)
+chunk                      5000
+prompt overhead + memory   ~3200   (system + question + M_{t-1} at 1024)
+response                   2048
+```
+
+A chars/4 estimate drifts 20–30% on code and JSON — which is most of this corpus.
+Undercount and the gateway rejects the request; overcount and chunks are smaller
+than they need to be, which multiplies the number of model calls.
+
+`tokenizers` (HuggingFace) loads the Qwen2 tokenizer from a vendored
+`tokenizer.json`. **Vendor the file into the repo** rather than downloading at
+runtime: a build that needs the network is a build that fails offline, and a
+tokenizer that changes under you silently re-chunks the entire corpus.
+
+## Steps
+
+1. Vendor `assets/qwen2-tokenizer.json` and record its sha256 in the repo.
+2. Implement `TokenCounter` for it in `mem-chunk`, loading once and reusing.
+3. Assert at load that the vendored file's hash matches the recorded one.
+4. Make `max_tokens` and the model id configurable, defaulting to 5000 and
+   `qwen2.5:3b-instruct`.
+5. Add a `mem tokens <file>` debug subcommand printing the token count of a file,
+   for cross-checking against the gateway's reported `prompt_tokens`.
+
+## Acceptance
+
+- Counts match the gateway's `usage.prompt_tokens` within ±2% on a sample of
+  real records.
+- Loading with a modified tokenizer file fails loudly, not silently.
+
+## Verify
+
+**Harness:** the vendored tokenizer plus recorded gateway responses. The
+cross-check against the live gateway is a separate, network-gated test.
+
+**Integration test** — `tests/it_tokens.rs`:
+1. `a1_known_strings` — a table of ~20 strings with hand-recorded expected counts
+   (ASCII, CJK, code, JSON, emoji), asserted exactly.
+2. `a2_hash_guard` — corrupt a copy of the tokenizer file, assert load returns an
+   error naming the file.
+3. `a3_gateway_agreement` — `#[ignore]` by default, run with `--ignored`: send 10
+   real records to `/v1/qwen/chat/completions` with `max_tokens: 1`, compare
+   `usage.prompt_tokens` to the local count minus the measured template overhead;
+   assert within 2%.
+4. `a4_budget_holds` — chunk a real pi session at 5000 tokens; assert no chunk's
+   locally-counted size exceeds the budget.
+
+**Command:** `cargo test -p mem-chunk tokens` (add `-- --ignored` for a3)
+
+**False pass:**
+- Testing only ASCII. Qwen2 BPE tokenizes CJK and emoji very differently, and
+  this corpus contains both — the caveman skill text alone is bilingual.
+- Comparing the local count to itself via a helper that calls the same function.
+  Assertion 1's expected values must be recorded from the tokenizer once and
+  written as literals.
+
+## Traps
+
+- Downloading the tokenizer at runtime. Offline builds break, and a silent
+  upstream change re-chunks everything, which changes every chunk hash, which
+  orphans every stored node.
+- Forgetting that the 32768 cap is cluster-side. `models.json` claims 131072 for
+  ornith and is wrong; do not take a client-side config as the source of truth.
+
+---
+
+Background: [DESIGN.md](../DESIGN.md) — Verified facts, `mem-chunk`