poimen-memory/tasks/M0.4-tokenizer-sizing.md

M0.4 — Tokenizer-backed chunk sizing

Field	Value
Phase	M0 — Read-only spine
Size	M — 1–3 days
Status	✅ Done
Flags	—
Spec	inlined below
Blocks	M0.3

Goal

Count tokens with the tokenizer the serving model actually uses, so a chunk that fits locally also fits at the gateway.

Facts (inlined — no spec read needed)

The controller is qwen2.5:3b-instruct — Qwen2.5-3B-Instruct, the GRU-Mem paper's exact 3B backbone. Its tokenizer is the Qwen2 BPE.

Budget arithmetic, and why an approximation is not good enough:

Ollama context cap        32768   (OLLAMA_CONTEXT_LENGTH, cluster-side)
chunk                      5000
prompt overhead + memory   ~3200   (system + question + M_{t-1} at 1024)
response                   2048

A chars/4 estimate drifts 20–30% on code and JSON — which is most of this corpus. Undercount and the gateway rejects the request; overcount and chunks are smaller than they need to be, which multiplies the number of model calls.

tokenizers (HuggingFace) loads the Qwen2 tokenizer from a vendored tokenizer.json. Vendor the file into the repo rather than downloading at runtime: a build that needs the network is a build that fails offline, and a tokenizer that changes under you silently re-chunks the entire corpus.

Steps

1. Vendor assets/qwen2-tokenizer.json and record its sha256 in the repo.
2. Implement TokenCounter for it in mem-chunk, loading once and reusing.
3. Assert at load that the vendored file's hash matches the recorded one.
4. Make max_tokens and the model id configurable, defaulting to 5000 and qwen2.5:3b-instruct.
5. Add a mem tokens <file> debug subcommand printing the token count of a file, for cross-checking against the gateway's reported prompt_tokens.

Acceptance

• Counts match the gateway's usage.prompt_tokens within ±2% on a sample of real records.
• Loading with a modified tokenizer file fails loudly, not silently.

Verify

Harness: the vendored tokenizer plus recorded gateway responses. The cross-check against the live gateway is a separate, network-gated test.

Integration test — tests/it_tokens.rs:

1. a1_known_strings — a table of ~20 strings with hand-recorded expected counts (ASCII, CJK, code, JSON, emoji), asserted exactly.
2. a2_hash_guard — corrupt a copy of the tokenizer file, assert load returns an error naming the file.
3. a3_gateway_agreement — #[ignore] by default, run with --ignored: send 10 real records to /v1/qwen/chat/completions with max_tokens: 1, compare usage.prompt_tokens to the local count minus the measured template overhead; assert within 2%.
4. a4_budget_holds — chunk a real pi session at 5000 tokens; assert no chunk's locally-counted size exceeds the budget.

Command: cargo test -p mem-chunk tokens (add -- --ignored for a3)

False pass:

• Testing only ASCII. Qwen2 BPE tokenizes CJK and emoji very differently, and this corpus contains both — the caveman skill text alone is bilingual.
• Comparing the local count to itself via a helper that calls the same function. Assertion 1's expected values must be recorded from the tokenizer once and written as literals.

Traps

• Downloading the tokenizer at runtime. Offline builds break, and a silent upstream change re-chunks everything, which changes every chunk hash, which orphans every stored node.
• Forgetting that the 32768 cap is cluster-side. models.json claims 131072 for ornith and is wrong; do not take a client-side config as the source of truth.

---

Background: DESIGN.md — Verified facts, mem-chunk