108 lines
4.0 KiB
Markdown
108 lines
4.0 KiB
Markdown
# M1.1 — `mem-llm` chat client
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M1 — Gated loop at L1 |
|
||
| Size | M — 1–3 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M0.1 |
|
||
|
||
## Goal
|
||
|
||
Talk to the homelab gateway, with the two non-obvious details that cost a day to
|
||
find already baked in.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
```
|
||
base https://api.riotpiao.com/v1
|
||
route POST /v1/qwen/chat/completions qwen2.5:3b-instruct
|
||
POST /v1/ornith/chat/completions ornith:35b
|
||
POST /v1/reasoning/chat/completions DeepSeek-R1-Distill-32B (no tools)
|
||
```
|
||
|
||
**Auth is `apikey:`, not `Authorization: Bearer`.** Kong's `key-auth` compares
|
||
the whole header value against the stored key, so the OpenAI SDK convention
|
||
returns 401. Verified:
|
||
|
||
```
|
||
-H "apikey: $KEY" -> 200
|
||
-H "Authorization: $KEY" -> 200
|
||
-H "Authorization: Bearer $KEY" -> 401
|
||
```
|
||
|
||
**One provider per route.** `baseUrl` is per-route and the model id is `ornith:35b`
|
||
with the tag, not `ornith`. `/v1/models` advertises `/v1/score` which does not
|
||
work — do not trust that list as a capability probe.
|
||
|
||
Request bodies above ~10.6 KB used to fail with
|
||
`{"error":{"message":"[] is too short - 'messages'"}}`; the Kong body buffer was
|
||
raised to 16m and it is fixed. If that error ever reappears, it is the buffer,
|
||
not the client.
|
||
|
||
Send **no `tools` array**. The controller needs none, and the reasoning route
|
||
rejects any request carrying one.
|
||
|
||
## Steps
|
||
|
||
1. `ChatClient::new(base_url, api_key, model)` in `mem-llm`.
|
||
2. Send the `apikey` header. Read the key from `MEM_API_KEY`, never from a
|
||
committed file.
|
||
3. `complete(system, user, max_tokens) -> Completion { text, usage, latency }`.
|
||
4. Timeout default 300s — local models are slow to first token and a cold load
|
||
can take minutes.
|
||
5. Retry on 5xx and timeout with exponential backoff, max 3. **Do not retry 4xx**
|
||
— a 400 is a malformed request and retrying it just costs three times as much.
|
||
6. On any error, include the response body in the error. The useful information
|
||
is always in the body, never the status.
|
||
7. `MEM_LLM_RECORD=<dir>` writes every request/response pair to disk, for
|
||
building fixtures without hand-writing them.
|
||
|
||
## Acceptance
|
||
|
||
- A real completion round-trips against the gateway.
|
||
- A 401 is reported as an auth error naming the header convention.
|
||
- A 4xx is not retried; a 5xx is.
|
||
|
||
## Verify
|
||
|
||
**Harness:** `wiremock` for the offline tests; one `#[ignore]` test against the
|
||
live gateway.
|
||
|
||
**Integration test** — `tests/it_chat_client.rs`:
|
||
1. `a1_sends_apikey_header` — assert the mock received `apikey` and **no**
|
||
`authorization` header.
|
||
2. `a2_no_tools_field` — assert the serialized body has no `tools` key at all,
|
||
not merely an empty array.
|
||
3. `a3_retries_5xx` — mock 503 twice then 200; assert 3 requests and success.
|
||
4. `a4_does_not_retry_4xx` — mock 400; assert exactly 1 request and an error
|
||
carrying the body text.
|
||
5. `a5_timeout_is_configurable` — mock a 2s delay with a 1s timeout; assert a
|
||
timeout error.
|
||
6. `a6_live_smoke` — `#[ignore]`; real gateway, `qwen2.5:3b-instruct`, prompt
|
||
"reply with exactly: pong", assert the text contains `pong`.
|
||
|
||
**Command:** `cargo test -p mem-llm chat_client` (add `-- --ignored` for a6)
|
||
|
||
**False pass:**
|
||
- Testing only against the mock. The mock accepts whatever header you send it;
|
||
assertion 6 against the live gateway is the only thing that proves the auth
|
||
convention is right.
|
||
- Asserting `tools: []` is absent by checking `body.tools.is_empty()`. An empty
|
||
array serialized into the request is still a `tools` key, and that is what the
|
||
reasoning route rejects. Assert on the raw JSON.
|
||
|
||
## Traps
|
||
|
||
- Using `Authorization: Bearer`. It is what every SDK does and it 401s here.
|
||
- Retrying 400s. The body-buffer bug produced a 400 for a whole day; retrying it
|
||
tripled the load and produced identical failures more slowly.
|
||
- A 60s timeout. That is the value that made `ornith` look broken when it was
|
||
merely cold.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — Verified facts
|