(plan) system review and break down plans

2026-08-19 09:52:07 -07:00
commit 9d28b63ff2
46 changed files with 5620 additions and 0 deletions
@@ -0,0 +1,102 @@
+# M5.4 — vLLM + `--enable-lora`
+
+| Field | Value |
+|---|---|
+| Phase | M5 — Post-training |
+| Size | L — 3+ days |
+| Status | ⬜ Not started |
+| Flags | homelab |
+| Spec | inlined below |
+| Blocks | — |
+
+## Goal
+
+A serving path that can load the memory adapter — because the current one cannot.
+
+## Facts (inlined — no spec read needed)
+
+**Ollama cannot hot-swap LoRA adapters.** The controller runs on
+`qwen2.5:3b-instruct` served by Ollama today, which is fine for prompted-only use
+and a dead end for post-training. vLLM supports `--enable-lora` with
+`--lora-modules name=path`, serving base plus adapters from one resident model.
+
+The pattern already exists in this cluster: the `reasoning` predictor is
+`vllm/vllm-openai:v0.11.0` under KServe, adopted into ArgoCD as
+`k8s/apps/llm-serving/reasoning.yaml`. Copy its shape.
+
+**VRAM is the constraint that makes an adapter the right answer.** One GPU,
+`OLLAMA_MAX_LOADED_MODELS=2`, currently holding `ornith:35b` + `qwen2.5:3b`. A
+separate full memory model evicts something, and eviction is a weights reload
+measured in tens of seconds — `ornith`'s cold start already blew a 60s gateway
+timeout once. A LoRA rides on a resident base for near-zero extra VRAM.
+
+Two hard-won operational facts to carry over:
+
+1. **Kong reads timeouts from the Service, not the Ingress.** `konghq.com/read-timeout`
+   on an Ingress is ignored; it must reach the predictor Service, and KServe
+   propagates InferenceService annotations there. Getting this wrong produces a
+   504 at exactly 60s on the first cold request.
+2. **Readiness must mean "can serve", not "process is up."** vLLM's startup probe
+   needs a long `failureThreshold` — model load plus torch compile measured 108s
+   + 55s on the 32B. Report ready too early and the first request 504s.
+
+## Steps
+
+1. `k8s/apps/llm-serving/memory.yaml` — InferenceService, vLLM, Qwen2.5-3B-Instruct.
+2. Args: `--enable-lora`, `--max-lora-rank 32`, `--max-model-len 32768`,
+   `--served-model-name memory`.
+3. Adapter storage: a PVC or an initContainer fetching from object storage;
+   `--lora-modules memory-v1=/mnt/adapters/memory-v1`.
+4. Kong timeout annotations on the **InferenceService** metadata so KServe
+   propagates them to the Service.
+5. Startup probe with a generous `failureThreshold`, gated on the OpenAI
+   `/health` endpoint.
+6. New Kong route `/v1/memory/chat/completions` in `llm-routes.yaml`, with the
+   `model-key-auth` plugin — in namespace `llm-serving`, since a KongPlugin
+   reference resolves in the annotated object's own namespace and a dangling one
+   fails open.
+7. Commit, push, let ArgoCD sync. No `kubectl apply`.
+
+## Acceptance
+
+- Base model answers through `/v1/memory/chat/completions`.
+- A named adapter is selectable via the `model` field.
+- Unauthenticated requests to the new route return 401.
+- Cold start does not 504.
+
+## Verify
+
+**Harness:** `kubectl` and `curl` against the live gateway after sync.
+
+**Integration test** — `verify/m5.4.sh` diffed against `expected/m5.4.txt`:
+1. `a1_isvc_ready` — InferenceService reports Ready.
+2. `a2_base_completion` — a completion naming the base model returns 200.
+3. `a3_adapter_selectable` — with a dummy adapter mounted, request `model:
+   memory-v1`; assert 200 and that it differs from the base response.
+4. `a4_auth_enforced` — no key → 401; `apikey` header → 200.
+5. `a5_cold_start_no_504` — delete the pod, wait for Ready, immediately send a
+   request; assert 200, not 504. This is the regression test for the timeout bug.
+6. `a6_timeouts_on_service` — assert `konghq.com/read-timeout` is present on the
+   predictor **Service**, not only the Ingress.
+7. `a7_vram_headroom` — with the memory model resident alongside the others,
+   assert `nvidia-smi` free memory stays above a threshold.
+
+**Command:** `bash verify/m5.4.sh | diff - expected/m5.4.txt`
+
+**False pass:**
+- Testing after the model is warm. The 504 bug only appears on the first request
+  after a restart, which is exactly the case assertion 5 forces.
+- Checking the timeout annotation on the Ingress. It is ignored there, and its
+  presence is what makes the bug hard to see.
+
+## Traps
+
+- Adding a third resident model without checking VRAM. Something gets evicted,
+  and the symptom is a slow unrelated model rather than an obvious failure.
+- Putting the KongPlugin in the wrong namespace. It fails **open** — the route
+  serves unauthenticated and looks healthy, which is how the model routes ran
+  with no auth for a while.
+
+---
+
+Background: [DESIGN.md](../DESIGN.md) — Separate weights · `k8s/apps/llm-serving/`