rock
e6fd6f2b62
fix(prometheus): cut TSDB volume 100Gi -> 25Gi
...
Actual TSDB data is ~2Gi. 100Gi over-provisioned Longhorn storage
contributed to the cluster-wide scheduling pressure; 25Gi is ample
with room for retention growth.
2026-09-29 21:59:55 +08:00
rock
605bc706e0
feat(cag): PrometheusRule + Grafana panels for KV cache observability
...
- cag-kv-cache-rules.yaml: recording rule vllm:prefix_cache_hit_rate
(from cache_query_hit/miss counters), 3 alert rules:
KVWarmupFailureHigh, KVWarmupMissHigh, VLLMPrefixCacheHitRateLow
- api-latency-dashboard.yaml: 5 new panels (id 7-11):
prefix cache hit rate, warmup hit/miss, warmup latency p50/p95,
standing context tier hits, alert panel with fixed threshold 0.01/s
2026-09-27 18:37:03 +08:00
rock
795cef5a85
feat(prometheus): scale up storage to 100Gi and move to worker-1
...
- Increase PVC from 20Gi to 100Gi (talos-cp-1 disk full)
- Move from az-a (talos-cp-1) to worker-1 (648GB available)
- Increase retention from 15d to 30d (95GB limit)
- Fixes Prometheus pod restarts due to disk pressure
2026-09-15 08:16:01 +09:00
rock
179e12b9a9
feat: vLLM Prometheus metrics scraping ( #37 )
...
Add ServiceMonitor for llm-serving namespace. Wire prometheus.io annotations to all vLLM pods (reasoning, ornith, embeddings, reranker). Scrape /metrics@8080 every 30s with proper relabeling.
2026-09-11 10:46:15 +09:00
rock
edc5dadd82
feat: add nginx-ingress ServiceMonitor for gateway traffic metrics
2026-09-04 23:07:44 -07:00
rock
efbe530b5c
refactor: consolidate 13 dashboards into 2 (cluster-infrastructure + api-gateway)
2026-09-04 22:58:53 -07:00
rock
4c63f8b125
feat: add cluster and api-gateway alert rules with SLA targets
2026-09-04 22:37:48 -07:00
rock
2a9220b576
feat: add cluster-infrastructure and api-gateway grafana dashboards
2026-09-04 22:31:51 -07:00
rock
26714d2ef3
feat: add api-gateway blackbox probes for healthz and /v1/models
2026-09-04 22:13:00 -07:00
Story Crater Bot
7e40a4baf6
feat: cut api.riotpiao.com over to the Go gateway and retire Kong
...
Ingress api/api now backs onto api-gateway:8080; the kong Application, its
Helm values, plugins and llm-routes are removed. Gateway image v0.0.0 is in
the Forgejo registry and the pull secret is in the api namespace.
2026-08-19 22:51:53 -07:00
Story Crater Bot
afb9b35292
fix(agent-pod): remote tui session for multi-agent
2026-08-18 15:08:04 -07:00
Story Crater Bot
61b906cce1
feat(monitoring): enable Alertmanager (null receiver, longhorn PVC, az-a) + fix forgejo-rules ns forgejo->cicd — alerting delivery was disabled; forgejo PrometheusRule targeted a nonexistent namespace
2026-08-18 15:08:04 -07:00
Story Crater Bot
9e0a83f8ca
fix(prometheus): use longhorn StorageClass, drop nonexistent longhorn-wffc — Prometheus CR requested storageClass longhorn-wffc which doesn't exist (deprecated), so operator never created the StatefulSet (Reconciled=False, no metrics server)
2026-08-18 15:08:04 -07:00
Story Crater Bot
41f5b05395
refactor(k8s): consolidate to infra/+apps/ single-source tree, dedicated per-app CNPG (authentik-db/temporal-db), wire monitoring-config, forgejo→cicd ns, drop orphan/stale (data-schemas, ollama, story-crater, sqs/argocd, key-rotation)
2026-08-18 15:08:03 -07:00
rock
Story Crater Bot