103 lines
4.5 KiB
Markdown
103 lines
4.5 KiB
Markdown
# M5.4 — vLLM + `--enable-lora`
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | M5 — Post-training |
|
|
| Size | L — 3+ days |
|
|
| Status | ⬜ Not started |
|
|
| Flags | homelab |
|
|
| Spec | inlined below |
|
|
| Blocks | — |
|
|
|
|
## Goal
|
|
|
|
A serving path that can load the memory adapter — because the current one cannot.
|
|
|
|
## Facts (inlined — no spec read needed)
|
|
|
|
**Ollama cannot hot-swap LoRA adapters.** The controller runs on
|
|
`qwen2.5:3b-instruct` served by Ollama today, which is fine for prompted-only use
|
|
and a dead end for post-training. vLLM supports `--enable-lora` with
|
|
`--lora-modules name=path`, serving base plus adapters from one resident model.
|
|
|
|
The pattern already exists in this cluster: the `reasoning` predictor is
|
|
`vllm/vllm-openai:v0.11.0` under KServe, adopted into ArgoCD as
|
|
`k8s/apps/llm-serving/reasoning.yaml`. Copy its shape.
|
|
|
|
**VRAM is the constraint that makes an adapter the right answer.** One GPU,
|
|
`OLLAMA_MAX_LOADED_MODELS=2`, currently holding `ornith:35b` + `qwen2.5:3b`. A
|
|
separate full memory model evicts something, and eviction is a weights reload
|
|
measured in tens of seconds — `ornith`'s cold start already blew a 60s gateway
|
|
timeout once. A LoRA rides on a resident base for near-zero extra VRAM.
|
|
|
|
Two hard-won operational facts to carry over:
|
|
|
|
1. **Kong reads timeouts from the Service, not the Ingress.** `konghq.com/read-timeout`
|
|
on an Ingress is ignored; it must reach the predictor Service, and KServe
|
|
propagates InferenceService annotations there. Getting this wrong produces a
|
|
504 at exactly 60s on the first cold request.
|
|
2. **Readiness must mean "can serve", not "process is up."** vLLM's startup probe
|
|
needs a long `failureThreshold` — model load plus torch compile measured 108s
|
|
+ 55s on the 32B. Report ready too early and the first request 504s.
|
|
|
|
## Steps
|
|
|
|
1. `k8s/apps/llm-serving/memory.yaml` — InferenceService, vLLM, Qwen2.5-3B-Instruct.
|
|
2. Args: `--enable-lora`, `--max-lora-rank 32`, `--max-model-len 32768`,
|
|
`--served-model-name memory`.
|
|
3. Adapter storage: a PVC or an initContainer fetching from object storage;
|
|
`--lora-modules memory-v1=/mnt/adapters/memory-v1`.
|
|
4. Kong timeout annotations on the **InferenceService** metadata so KServe
|
|
propagates them to the Service.
|
|
5. Startup probe with a generous `failureThreshold`, gated on the OpenAI
|
|
`/health` endpoint.
|
|
6. New Kong route `/v1/memory/chat/completions` in `llm-routes.yaml`, with the
|
|
`model-key-auth` plugin — in namespace `llm-serving`, since a KongPlugin
|
|
reference resolves in the annotated object's own namespace and a dangling one
|
|
fails open.
|
|
7. Commit, push, let ArgoCD sync. No `kubectl apply`.
|
|
|
|
## Acceptance
|
|
|
|
- Base model answers through `/v1/memory/chat/completions`.
|
|
- A named adapter is selectable via the `model` field.
|
|
- Unauthenticated requests to the new route return 401.
|
|
- Cold start does not 504.
|
|
|
|
## Verify
|
|
|
|
**Harness:** `kubectl` and `curl` against the live gateway after sync.
|
|
|
|
**Integration test** — `verify/m5.4.sh` diffed against `expected/m5.4.txt`:
|
|
1. `a1_isvc_ready` — InferenceService reports Ready.
|
|
2. `a2_base_completion` — a completion naming the base model returns 200.
|
|
3. `a3_adapter_selectable` — with a dummy adapter mounted, request `model:
|
|
memory-v1`; assert 200 and that it differs from the base response.
|
|
4. `a4_auth_enforced` — no key → 401; `apikey` header → 200.
|
|
5. `a5_cold_start_no_504` — delete the pod, wait for Ready, immediately send a
|
|
request; assert 200, not 504. This is the regression test for the timeout bug.
|
|
6. `a6_timeouts_on_service` — assert `konghq.com/read-timeout` is present on the
|
|
predictor **Service**, not only the Ingress.
|
|
7. `a7_vram_headroom` — with the memory model resident alongside the others,
|
|
assert `nvidia-smi` free memory stays above a threshold.
|
|
|
|
**Command:** `bash verify/m5.4.sh | diff - expected/m5.4.txt`
|
|
|
|
**False pass:**
|
|
- Testing after the model is warm. The 504 bug only appears on the first request
|
|
after a restart, which is exactly the case assertion 5 forces.
|
|
- Checking the timeout annotation on the Ingress. It is ignored there, and its
|
|
presence is what makes the bug hard to see.
|
|
|
|
## Traps
|
|
|
|
- Adding a third resident model without checking VRAM. Something gets evicted,
|
|
and the symptom is a slow unrelated model rather than an obvious failure.
|
|
- Putting the KongPlugin in the wrong namespace. It fails **open** — the route
|
|
serves unauthenticated and looks healthy, which is how the model routes ran
|
|
with no auth for a while.
|
|
|
|
---
|
|
|
|
Background: [DESIGN.md](../DESIGN.md) — Separate weights · `k8s/apps/llm-serving/`
|