feat(monitoring): LLM observability dashboard + fix dead CAG rule

New Grafana dashboard (llm-observability) with 27 panels across 5 rows:
TTFT, inter-token latency, batching/scheduling, KV cache, throughput.

Both metric sources are shown side by side because they answer different
questions and neither is complete alone:

  vLLM 0.11.0 (reasoning) — engine truth: kv_cache_usage_perc,
    num_requests_running/waiting, num_preemptions_total, prefill vs
    decode time, prefix cache, per-request histogram quantiles.
  TEI 1.8.2 (embeddings, reranker) — te_batch_next_size, te_queue_size,
    te_embed_duration, te_embed_queue_duration.
  api-gateway — the ONLY per-request latency source covering Ollama
    models (ornith:35b, qwen2.5:3b-instruct), which expose no metrics.
    It is derived from the SSE byte stream, so it is streaming-only and
    counts SSE chunks rather than true tokens. vLLM's own histograms are
    authoritative for reasoning.

All 44 panel queries were executed against live Prometheus before commit —
all return success. vLLM/TEI panels have live series; gateway panels show
no data until the netpol fix in homelab-frontend PR #62 merges.

Also fixes a dead alert. alerts/cag-kv-cache-rules.yaml derived
vllm:prefix_cache_hit_rate from vllm:cache_query_hit_total and
vllm:cache_query_miss_total. Those metric names do not exist in vLLM
0.11.0 — the real series are vllm:prefix_cache_hits_total and
vllm:prefix_cache_queries_total. The recording rule produced no data, so
VLLMPrefixCacheHitRateLow could never fire.

Not included: real GPU hardware metrics (SM utilisation, framebuffer,
power, temperature). Only nvidia-device-plugin is deployed, which exposes
none of them; DCGM exporter is required. vllm:kv_cache_usage_perc and
preemption rate are the only GPU-pressure proxies available today.

k8s/infra/monitoring/alerts/cag-kv-cache-rules.yaml

@@ -7,19 +7,24 @@ metadata:
     release: prometheus
 spec:
   groups:
-    # Recording rules: derive vLLM prefix cache hit rate from raw counters
+    # Recording rules: derive vLLM prefix cache hit rate from raw counters.
+    # Metric names verified against live vLLM 0.11.0 scrape (2026-09-29).
+    # The previous names (vllm:cache_query_hit_total / vllm:cache_query_miss_total)
+    # do not exist in this version — vLLM exposes prefix_cache_hits_total and
+    # prefix_cache_queries_total — so this rule produced no data and
+    # VLLMPrefixCacheHitRateLow could never fire.
     - name: vllm.recording
       rules:
         - record: vllm:prefix_cache_hit_rate
           expr: |
-            rate(vllm:cache_query_hit_total[5m])
+            rate(vllm:prefix_cache_hits_total[5m])
             /
             clamp_min(
-              rate(vllm:cache_query_hit_total[5m]) + rate(vllm:cache_query_miss_total[5m]),
+              rate(vllm:prefix_cache_queries_total[5m]),
               0.001
             )
           labels:
-            job: "vllm"
+            job: "reasoning-predictor"
 
     # Alert rules: CAG pipeline health
     - name: cag.alerts