Homelab LLM Stack

API gateway routes model requests to vLLM, Ollama, TEI predictors on GPU worker

Homelab LLM Stack API gateway routes model requests to vLLM, Ollama, TEI predictors on GPU worker Client · api.riotpiao.com · Architecture component Client api.riotpiao.com Nginx · TLS · Public API Nginx TLS API Gateway · Go, routes model · Architecture component · dispatch API Gateway Go, routes model dispatch vLLM · Qwen3-32B · LLM Serving (llm-serving namespace) · reasoning vLLM Qwen3-32B reasoning Ollama · 35B + 3B · LLM Serving (llm-serving namespace) · ornith Ollama 35B + 3B ornith TEI · embed + rerank · LLM Serving (llm-serving namespace) · TEI TEI embed + rerank TEI Worker GPU · V100, all pods · LLM Serving (llm-serving namespace) Worker GPU V100, all pods llm-models PVC · Weights shared · Architecture component llm-models PVC Weights shared Authentik · JWT verify · Architecture component Authentik JWT verify HTTPS proxy reasoning ornith nomic-embed JWT mount Public API LLM Serving (llm-serving namespace) Legend Backend Database Cloud Security External

Request Routing

  • • Client POST /v1/chat/completions with model in body
  • • Nginx TLS termination → API gateway :8080
  • • Gateway dispatches to correct predictor endpoint

Inference Engines

  • • vLLM: Qwen3-32B bnb-4bit quantization (2 replicas)
  • • Ollama: ornith:35b + qwen2.5:3b-instruct (same pod, max 2 loaded)
  • • TEI: embeddings (nomic-embed-v2-moe) + reranker (BGE-base)

Deployment

  • • All predictor pods pinned to worker-1 (V100, sm70)
  • • Model weights on shared llm-models PVC (Longhorn)
  • • No sm80 support → bnb-4bit instead of Marlin INT4