# M0.4 — Tokenizer-backed chunk sizing

| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.3 |

## Goal

Count tokens with the tokenizer the serving model actually uses, so a chunk that
fits locally also fits at the gateway.

## Facts (inlined — no spec read needed)

The controller is `qwen2.5:3b-instruct` — Qwen2.5-3B-Instruct, the GRU-Mem
paper's exact 3B backbone. Its tokenizer is the Qwen2 BPE.

Budget arithmetic, and why an approximation is not good enough:

```
Ollama context cap        32768   (OLLAMA_CONTEXT_LENGTH, cluster-side)
chunk                      5000
prompt overhead + memory   ~3200   (system + question + M_{t-1} at 1024)
response                   2048
```

A chars/4 estimate drifts 20–30% on code and JSON — which is most of this corpus.
Undercount and the gateway rejects the request; overcount and chunks are smaller
than they need to be, which multiplies the number of model calls.

`tokenizers` (HuggingFace) loads the Qwen2 tokenizer from a vendored
`tokenizer.json`. **Vendor the file into the repo** rather than downloading at
runtime: a build that needs the network is a build that fails offline, and a
tokenizer that changes under you silently re-chunks the entire corpus.

## Steps

1. Vendor `assets/qwen2-tokenizer.json` and record its sha256 in the repo.
2. Implement `TokenCounter` for it in `mem-chunk`, loading once and reusing.
3. Assert at load that the vendored file's hash matches the recorded one.
4. Make `max_tokens` and the model id configurable, defaulting to 5000 and
   `qwen2.5:3b-instruct`.
5. Add a `mem tokens <file>` debug subcommand printing the token count of a file,
   for cross-checking against the gateway's reported `prompt_tokens`.

## Acceptance

- Counts match the gateway's `usage.prompt_tokens` within ±2% on a sample of
  real records.
- Loading with a modified tokenizer file fails loudly, not silently.

## Verify

**Harness:** the vendored tokenizer plus recorded gateway responses. The
cross-check against the live gateway is a separate, network-gated test.

**Integration test** — `tests/it_tokens.rs`:
1. `a1_known_strings` — a table of ~20 strings with hand-recorded expected counts
   (ASCII, CJK, code, JSON, emoji), asserted exactly.
2. `a2_hash_guard` — corrupt a copy of the tokenizer file, assert load returns an
   error naming the file.
3. `a3_gateway_agreement` — `#[ignore]` by default, run with `--ignored`: send 10
   real records to `/v1/qwen/chat/completions` with `max_tokens: 1`, compare
   `usage.prompt_tokens` to the local count minus the measured template overhead;
   assert within 2%.
4. `a4_budget_holds` — chunk a real pi session at 5000 tokens; assert no chunk's
   locally-counted size exceeds the budget.

**Command:** `cargo test -p mem-chunk tokens` (add `-- --ignored` for a3)

**False pass:**
- Testing only ASCII. Qwen2 BPE tokenizes CJK and emoji very differently, and
  this corpus contains both — the caveman skill text alone is bilingual.
- Comparing the local count to itself via a helper that calls the same function.
  Assertion 1's expected values must be recorded from the tokenizer once and
  written as literals.

## Traps

- Downloading the tokenizer at runtime. Offline builds break, and a silent
  upstream change re-chunks everything, which changes every chunk hash, which
  orphans every stored node.
- Forgetting that the 32768 cap is cluster-side. `models.json` claims 131072 for
  ornith and is wrong; do not take a client-side config as the source of truth.

---

Background: [DESIGN.md](../DESIGN.md) — Verified facts, `mem-chunk`