87 lines
3.0 KiB
Markdown
87 lines
3.0 KiB
Markdown
|
# M2.1 — Embeddings client
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M2 — Projections |
|
|||
|
|
| Size | S — under 1 day |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M1.1 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Turn text into 768-dim vectors, batched, against the gateway's TEI endpoint.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
POST /v1/embeddings
|
|||
|
|
{"model":"nomic-ai/nomic-embed-text-v2-moe","input":["..."]}
|
|||
|
|
-> {"object":"list","data":[{"embedding":[...768 floats...]}],"usage":{...}}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**768 dimensions**, probed and confirmed. It is the `vector(768)` in the schema
|
|||
|
|
and in the HNSW index; a model swap is a schema migration, not a config change.
|
|||
|
|
|
|||
|
|
**Batch limit is 32.** Verified: 1200 inputs returned
|
|||
|
|
`{"message":"batch size 1200 > maximum allowed batch size 32","code":413}`.
|
|||
|
|
Chunk the input list accordingly.
|
|||
|
|
|
|||
|
|
This route currently has **no auth** — no `konghq.com/plugins` annotation, so
|
|||
|
|
`model-key-auth` never attaches. Send the `apikey` header anyway: the route
|
|||
|
|
should be fixed, and a client that only works while auth is broken breaks when
|
|||
|
|
it is fixed.
|
|||
|
|
|
|||
|
|
Bodies here are large (many texts × long strings) and the Kong buffer is 16m, so
|
|||
|
|
batching also keeps requests well inside it.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. `EmbeddingsClient::embed(texts: &[String]) -> Result<Vec<Vec<f32>>>` in `mem-llm`.
|
|||
|
|
2. Split into batches of ≤32, preserving input order in the output.
|
|||
|
|
3. Assert every returned vector is exactly 768 long; a mismatch is an error
|
|||
|
|
naming the model, not a silent pad or truncate.
|
|||
|
|
4. Reuse M1.1's client config: `apikey` header, retry on 5xx only, generous
|
|||
|
|
timeout.
|
|||
|
|
5. `embed_one` convenience wrapper.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- 100 texts return 100 vectors in input order.
|
|||
|
|
- Every vector is 768-dim.
|
|||
|
|
- A dimension mismatch errors loudly.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** `wiremock` offline, one `#[ignore]` live test.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_embeddings.rs`:
|
|||
|
|
1. `a1_batches_at_32` — 100 inputs produce exactly 4 requests.
|
|||
|
|
2. `a2_order_preserved` — mock returns identifiable vectors; assert output order
|
|||
|
|
matches input order across batch boundaries.
|
|||
|
|
3. `a3_dimension_asserted` — mock returns a 512-dim vector; assert an error
|
|||
|
|
naming the model.
|
|||
|
|
4. `a4_apikey_sent` — assert the header is present even though the route does not
|
|||
|
|
require it.
|
|||
|
|
5. `a5_live_dims` — `#[ignore]`; real gateway, assert 768.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p mem-llm embeddings` (add `-- --ignored` for a5)
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Testing with ≤32 inputs. The batching path never runs and order-across-batches
|
|||
|
|
— the thing most likely to be wrong — is never exercised.
|
|||
|
|
- Trusting the response order within a batch without asserting it. Assertion 2
|
|||
|
|
must use distinguishable vectors, not a length check.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Hardcoding 768 in three places. Put it in one constant that the schema
|
|||
|
|
migration also references, so a model change is one edit and one migration.
|
|||
|
|
- Assuming no auth is needed because it currently works without a key. That route
|
|||
|
|
is missing its plugin annotation, which is a bug scheduled to be fixed.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — Verified facts, pgvector
|