poimen-memory/tasks/M5.4-vllm-lora-serving.md

M5.4 — vLLM + --enable-lora

Field	Value
Phase	M5 — Post-training
Size	L — 3+ days
Status	⬜ Not started
Flags	homelab
Spec	inlined below
Blocks	—

Goal

A serving path that can load the memory adapter — because the current one cannot.

Facts (inlined — no spec read needed)

Ollama cannot hot-swap LoRA adapters. The controller runs on qwen2.5:3b-instruct served by Ollama today, which is fine for prompted-only use and a dead end for post-training. vLLM supports --enable-lora with --lora-modules name=path, serving base plus adapters from one resident model.

The pattern already exists in this cluster: the reasoning predictor is vllm/vllm-openai:v0.11.0 under KServe, adopted into ArgoCD as k8s/apps/llm-serving/reasoning.yaml. Copy its shape.

VRAM is the constraint that makes an adapter the right answer. One GPU, OLLAMA_MAX_LOADED_MODELS=2, currently holding ornith:35b + qwen2.5:3b. A separate full memory model evicts something, and eviction is a weights reload measured in tens of seconds — ornith's cold start already blew a 60s gateway timeout once. A LoRA rides on a resident base for near-zero extra VRAM.

Two hard-won operational facts to carry over:

1. Kong reads timeouts from the Service, not the Ingress. konghq.com/read-timeout on an Ingress is ignored; it must reach the predictor Service, and KServe propagates InferenceService annotations there. Getting this wrong produces a 504 at exactly 60s on the first cold request.
2. Readiness must mean "can serve", not "process is up." vLLM's startup probe needs a long failureThreshold — model load plus torch compile measured 108s
   • 55s on the 32B. Report ready too early and the first request 504s.

Steps

1. k8s/apps/llm-serving/memory.yaml — InferenceService, vLLM, Qwen2.5-3B-Instruct.
2. Args: --enable-lora, --max-lora-rank 32, --max-model-len 32768, --served-model-name memory.
3. Adapter storage: a PVC or an initContainer fetching from object storage; --lora-modules memory-v1=/mnt/adapters/memory-v1.
4. Kong timeout annotations on the InferenceService metadata so KServe propagates them to the Service.
5. Startup probe with a generous failureThreshold, gated on the OpenAI /health endpoint.
6. New Kong route /v1/memory/chat/completions in llm-routes.yaml, with the model-key-auth plugin — in namespace llm-serving, since a KongPlugin reference resolves in the annotated object's own namespace and a dangling one fails open.
7. Commit, push, let ArgoCD sync. No kubectl apply.

Acceptance

• Base model answers through /v1/memory/chat/completions.
• A named adapter is selectable via the model field.
• Unauthenticated requests to the new route return 401.
• Cold start does not 504.

Verify

Harness: kubectl and curl against the live gateway after sync.

Integration test — verify/m5.4.sh diffed against expected/m5.4.txt:

1. a1_isvc_ready — InferenceService reports Ready.
2. a2_base_completion — a completion naming the base model returns 200.
3. a3_adapter_selectable — with a dummy adapter mounted, request model: memory-v1; assert 200 and that it differs from the base response.
4. a4_auth_enforced — no key → 401; apikey header → 200.
5. a5_cold_start_no_504 — delete the pod, wait for Ready, immediately send a request; assert 200, not 504. This is the regression test for the timeout bug.
6. a6_timeouts_on_service — assert konghq.com/read-timeout is present on the predictor Service, not only the Ingress.
7. a7_vram_headroom — with the memory model resident alongside the others, assert nvidia-smi free memory stays above a threshold.

Command: bash verify/m5.4.sh | diff - expected/m5.4.txt

False pass:

• Testing after the model is warm. The 504 bug only appears on the first request after a restart, which is exactly the case assertion 5 forces.
• Checking the timeout annotation on the Ingress. It is ignored there, and its presence is what makes the bug hard to see.

Traps

• Adding a third resident model without checking VRAM. Something gets evicted, and the symptom is a slow unrelated model rather than an obvious failure.
• Putting the KongPlugin in the wrong namespace. It fails open — the route serves unauthenticated and looks healthy, which is how the model routes ran with no auth for a while.

---

Background: DESIGN.md — Separate weights · k8s/apps/llm-serving/