93 lines
3.6 KiB
Markdown
93 lines
3.6 KiB
Markdown
|
# M0.4 — Tokenizer-backed chunk sizing
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M0 — Read-only spine |
|
|||
|
|
| Size | M — 1–3 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M0.3 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Count tokens with the tokenizer the serving model actually uses, so a chunk that
|
|||
|
|
fits locally also fits at the gateway.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
The controller is `qwen2.5:3b-instruct` — Qwen2.5-3B-Instruct, the GRU-Mem
|
|||
|
|
paper's exact 3B backbone. Its tokenizer is the Qwen2 BPE.
|
|||
|
|
|
|||
|
|
Budget arithmetic, and why an approximation is not good enough:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
Ollama context cap 32768 (OLLAMA_CONTEXT_LENGTH, cluster-side)
|
|||
|
|
chunk 5000
|
|||
|
|
prompt overhead + memory ~3200 (system + question + M_{t-1} at 1024)
|
|||
|
|
response 2048
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
A chars/4 estimate drifts 20–30% on code and JSON — which is most of this corpus.
|
|||
|
|
Undercount and the gateway rejects the request; overcount and chunks are smaller
|
|||
|
|
than they need to be, which multiplies the number of model calls.
|
|||
|
|
|
|||
|
|
`tokenizers` (HuggingFace) loads the Qwen2 tokenizer from a vendored
|
|||
|
|
`tokenizer.json`. **Vendor the file into the repo** rather than downloading at
|
|||
|
|
runtime: a build that needs the network is a build that fails offline, and a
|
|||
|
|
tokenizer that changes under you silently re-chunks the entire corpus.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Vendor `assets/qwen2-tokenizer.json` and record its sha256 in the repo.
|
|||
|
|
2. Implement `TokenCounter` for it in `mem-chunk`, loading once and reusing.
|
|||
|
|
3. Assert at load that the vendored file's hash matches the recorded one.
|
|||
|
|
4. Make `max_tokens` and the model id configurable, defaulting to 5000 and
|
|||
|
|
`qwen2.5:3b-instruct`.
|
|||
|
|
5. Add a `mem tokens <file>` debug subcommand printing the token count of a file,
|
|||
|
|
for cross-checking against the gateway's reported `prompt_tokens`.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- Counts match the gateway's `usage.prompt_tokens` within ±2% on a sample of
|
|||
|
|
real records.
|
|||
|
|
- Loading with a modified tokenizer file fails loudly, not silently.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** the vendored tokenizer plus recorded gateway responses. The
|
|||
|
|
cross-check against the live gateway is a separate, network-gated test.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_tokens.rs`:
|
|||
|
|
1. `a1_known_strings` — a table of ~20 strings with hand-recorded expected counts
|
|||
|
|
(ASCII, CJK, code, JSON, emoji), asserted exactly.
|
|||
|
|
2. `a2_hash_guard` — corrupt a copy of the tokenizer file, assert load returns an
|
|||
|
|
error naming the file.
|
|||
|
|
3. `a3_gateway_agreement` — `#[ignore]` by default, run with `--ignored`: send 10
|
|||
|
|
real records to `/v1/qwen/chat/completions` with `max_tokens: 1`, compare
|
|||
|
|
`usage.prompt_tokens` to the local count minus the measured template overhead;
|
|||
|
|
assert within 2%.
|
|||
|
|
4. `a4_budget_holds` — chunk a real pi session at 5000 tokens; assert no chunk's
|
|||
|
|
locally-counted size exceeds the budget.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p mem-chunk tokens` (add `-- --ignored` for a3)
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Testing only ASCII. Qwen2 BPE tokenizes CJK and emoji very differently, and
|
|||
|
|
this corpus contains both — the caveman skill text alone is bilingual.
|
|||
|
|
- Comparing the local count to itself via a helper that calls the same function.
|
|||
|
|
Assertion 1's expected values must be recorded from the tokenizer once and
|
|||
|
|
written as literals.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Downloading the tokenizer at runtime. Offline builds break, and a silent
|
|||
|
|
upstream change re-chunks everything, which changes every chunk hash, which
|
|||
|
|
orphans every stored node.
|
|||
|
|
- Forgetting that the 32768 cap is cluster-side. `models.json` claims 131072 for
|
|||
|
|
ornith and is wrong; do not take a client-side config as the source of truth.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — Verified facts, `mem-chunk`
|