93 lines
3.6 KiB
Markdown
93 lines
3.6 KiB
Markdown
# M0.4 — Tokenizer-backed chunk sizing
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M0 — Read-only spine |
|
||
| Size | M — 1–3 days |
|
||
| Status | ✅ Done |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M0.3 |
|
||
|
||
## Goal
|
||
|
||
Count tokens with the tokenizer the serving model actually uses, so a chunk that
|
||
fits locally also fits at the gateway.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
The controller is `qwen2.5:3b-instruct` — Qwen2.5-3B-Instruct, the GRU-Mem
|
||
paper's exact 3B backbone. Its tokenizer is the Qwen2 BPE.
|
||
|
||
Budget arithmetic, and why an approximation is not good enough:
|
||
|
||
```
|
||
Ollama context cap 32768 (OLLAMA_CONTEXT_LENGTH, cluster-side)
|
||
chunk 5000
|
||
prompt overhead + memory ~3200 (system + question + M_{t-1} at 1024)
|
||
response 2048
|
||
```
|
||
|
||
A chars/4 estimate drifts 20–30% on code and JSON — which is most of this corpus.
|
||
Undercount and the gateway rejects the request; overcount and chunks are smaller
|
||
than they need to be, which multiplies the number of model calls.
|
||
|
||
`tokenizers` (HuggingFace) loads the Qwen2 tokenizer from a vendored
|
||
`tokenizer.json`. **Vendor the file into the repo** rather than downloading at
|
||
runtime: a build that needs the network is a build that fails offline, and a
|
||
tokenizer that changes under you silently re-chunks the entire corpus.
|
||
|
||
## Steps
|
||
|
||
1. Vendor `assets/qwen2-tokenizer.json` and record its sha256 in the repo.
|
||
2. Implement `TokenCounter` for it in `mem-chunk`, loading once and reusing.
|
||
3. Assert at load that the vendored file's hash matches the recorded one.
|
||
4. Make `max_tokens` and the model id configurable, defaulting to 5000 and
|
||
`qwen2.5:3b-instruct`.
|
||
5. Add a `mem tokens <file>` debug subcommand printing the token count of a file,
|
||
for cross-checking against the gateway's reported `prompt_tokens`.
|
||
|
||
## Acceptance
|
||
|
||
- Counts match the gateway's `usage.prompt_tokens` within ±2% on a sample of
|
||
real records.
|
||
- Loading with a modified tokenizer file fails loudly, not silently.
|
||
|
||
## Verify
|
||
|
||
**Harness:** the vendored tokenizer plus recorded gateway responses. The
|
||
cross-check against the live gateway is a separate, network-gated test.
|
||
|
||
**Integration test** — `tests/it_tokens.rs`:
|
||
1. `a1_known_strings` — a table of ~20 strings with hand-recorded expected counts
|
||
(ASCII, CJK, code, JSON, emoji), asserted exactly.
|
||
2. `a2_hash_guard` — corrupt a copy of the tokenizer file, assert load returns an
|
||
error naming the file.
|
||
3. `a3_gateway_agreement` — `#[ignore]` by default, run with `--ignored`: send 10
|
||
real records to `/v1/qwen/chat/completions` with `max_tokens: 1`, compare
|
||
`usage.prompt_tokens` to the local count minus the measured template overhead;
|
||
assert within 2%.
|
||
4. `a4_budget_holds` — chunk a real pi session at 5000 tokens; assert no chunk's
|
||
locally-counted size exceeds the budget.
|
||
|
||
**Command:** `cargo test -p mem-chunk tokens` (add `-- --ignored` for a3)
|
||
|
||
**False pass:**
|
||
- Testing only ASCII. Qwen2 BPE tokenizes CJK and emoji very differently, and
|
||
this corpus contains both — the caveman skill text alone is bilingual.
|
||
- Comparing the local count to itself via a helper that calls the same function.
|
||
Assertion 1's expected values must be recorded from the tokenizer once and
|
||
written as literals.
|
||
|
||
## Traps
|
||
|
||
- Downloading the tokenizer at runtime. Offline builds break, and a silent
|
||
upstream change re-chunks everything, which changes every chunk hash, which
|
||
orphans every stored node.
|
||
- Forgetting that the 32768 cap is cluster-side. `models.json` claims 131072 for
|
||
ornith and is wrong; do not take a client-side config as the source of truth.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — Verified facts, `mem-chunk`
|