LLM Production Hardening

New Issue

Networking, GPU memory, observability, infrastructure, and workflow hardening for production LLM serving. Covers all 5 layers: Networking, GPU/Engine, Observability, Infrastructure, Workflow.

No due date

0% Completed

[Workflow] Circuit breaker with model degradation

#43

opened 2026-09-11 01:27:38 +00:00 by poimen

[Workflow] Structured output enforcement (JSON grammar)

#42

opened 2026-09-11 01:27:37 +00:00 by poimen

[Infra] NVIDIA runtime declarative config for Talos

#41

opened 2026-09-11 01:27:35 +00:00 by poimen

[Infra] Local PV on NVMe for model weight cold starts

#40

opened 2026-09-11 01:27:34 +00:00 by poimen

[Observability] HPA driven by GPU KV Cache usage

#39

opened 2026-09-11 01:27:33 +00:00 by poimen

[Observability] Grafana dashboard for TTFT + ITL

#38

opened 2026-09-11 01:27:28 +00:00 by poimen

[Observability] vLLM Prometheus metrics scraping

#37

opened 2026-09-11 01:27:24 +00:00 by poimen

[GPU] Preemption swapping for concurrency spikes

#36

opened 2026-09-11 01:27:20 +00:00 by poimen

[GPU] Enable prefix caching for RAG contexts

#35

opened 2026-09-11 01:27:15 +00:00 by poimen

[GPU] KV Cache VRAM monitoring + PagedAttention

#34

opened 2026-09-11 01:27:11 +00:00 by poimen

[Network] Disable proxy buffering for SSE

#33

opened 2026-09-11 01:26:20 +00:00 by poimen

[Network] HTTP/2 multiplexing for concurrent streams

#32

opened 2026-09-11 01:26:16 +00:00 by poimen

[Network] TCP backpressure for streaming LLM responses

#31

opened 2026-09-11 01:26:14 +00:00 by poimen