M1.1 — `mem-llm` chat client

Field	Value
Phase	M1 — Gated loop at L1
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M0.1

Goal

Talk to the homelab gateway, with the two non-obvious details that cost a day to find already baked in.

Facts (inlined — no spec read needed)

base   https://api.riotpiao.com/v1
route  POST /v1/qwen/chat/completions      qwen2.5:3b-instruct
       POST /v1/ornith/chat/completions    ornith:35b
       POST /v1/reasoning/chat/completions DeepSeek-R1-Distill-32B (no tools)

Auth is apikey:, not Authorization: Bearer. Kong's key-auth compares the whole header value against the stored key, so the OpenAI SDK convention returns 401. Verified:

-H "apikey: $KEY"                 -> 200
-H "Authorization: $KEY"          -> 200
-H "Authorization: Bearer $KEY"   -> 401

One provider per route. baseUrl is per-route and the model id is ornith:35b with the tag, not ornith. /v1/models advertises /v1/score which does not work — do not trust that list as a capability probe.

Request bodies above ~10.6 KB used to fail with {"error":{"message":"[] is too short - 'messages'"}}; the Kong body buffer was raised to 16m and it is fixed. If that error ever reappears, it is the buffer, not the client.

Send no tools array. The controller needs none, and the reasoning route rejects any request carrying one.

Steps

ChatClient::new(base_url, api_key, model) in mem-llm.
Send the apikey header. Read the key from MEM_API_KEY, never from a committed file.
complete(system, user, max_tokens) -> Completion { text, usage, latency }.
Timeout default 300s — local models are slow to first token and a cold load can take minutes.
Retry on 5xx and timeout with exponential backoff, max 3. Do not retry 4xx — a 400 is a malformed request and retrying it just costs three times as much.
On any error, include the response body in the error. The useful information is always in the body, never the status.
MEM_LLM_RECORD=<dir> writes every request/response pair to disk, for building fixtures without hand-writing them.

Acceptance

A real completion round-trips against the gateway.
A 401 is reported as an auth error naming the header convention.
A 4xx is not retried; a 5xx is.

Verify

Harness: wiremock for the offline tests; one #[ignore] test against the live gateway.

Integration test — tests/it_chat_client.rs:

a1_sends_apikey_header — assert the mock received apikey and no authorization header.
a2_no_tools_field — assert the serialized body has no tools key at all, not merely an empty array.
a3_retries_5xx — mock 503 twice then 200; assert 3 requests and success.
a4_does_not_retry_4xx — mock 400; assert exactly 1 request and an error carrying the body text.
a5_timeout_is_configurable — mock a 2s delay with a 1s timeout; assert a timeout error.
a6_live_smoke — #[ignore]; real gateway, qwen2.5:3b-instruct, prompt "reply with exactly: pong", assert the text contains pong.

Command: cargo test -p mem-llm chat_client (add -- --ignored for a6)

False pass:

Testing only against the mock. The mock accepts whatever header you send it; assertion 6 against the live gateway is the only thing that proves the auth convention is right.
Asserting tools: [] is absent by checking body.tools.is_empty(). An empty array serialized into the request is still a tools key, and that is what the reasoning route rejects. Assert on the raw JSON.

Traps

Using Authorization: Bearer. It is what every SDK does and it 401s here.
Retrying 400s. The body-buffer bug produced a 400 for a whole day; retrying it tripled the load and produced identical failures more slowly.
A 60s timeout. That is the value that made ornith look broken when it was merely cold.

Background: DESIGN.md — Verified facts

4.0 KiB Raw Blame History Unescape Escape

M1.1 — mem-llm chat client