CI / CI (push) Successful in 41m20s
Closes #64 Replaces the per-pod concurrency semaphore as the effective cap with admission gated on the vLLM engine's own scheduler gauges. ## Why `internal/proxy/proxy.go:119` built a semaphore of 4 slots *inside each gateway process*, and `api-gateway` runs 3 replicas, so the advertised cap was 12 while the engine's `--max-num-seqs` is 4. Nothing coordinated replicas, and the mismatch was invisible because the queue never rejected anything (`queue_full_requests_total 0`). Measured on the engine while investigating a 10-minute TTFT: - `vllm:cache_config_info{num_gpu_blocks="9527",block_size="16"}` = **152,432 tokens** of KV capacity for the whole engine. PP=2 splits layers, so both ranks hold KV for the same tokens — not 2x that. - Uncached prefill runs at **262 tok/s** contended, 525 tok/s over 7d (`vllm:request_prefill_time_seconds_sum/count`). - One request showed `llm_ttft_seconds_sum 590.981 / count 5` at the gateway and a `(480, 960]` s entry in `vllm:request_prefill_time_seconds_bucket`, with engine-side generation throughput at 0.0-0.1 tok/s throughout: a 65k-token prompt costs ~250 s of prefill and starves every other request while it runs. - `vllm:num_preemptions_total = 0`; KV peaked at `0.5399`. Reading the engine's own gauges is global by construction, needs no lock and no new datastore, and it also accounts for callers that never pass the gateway — namespace `poimen` is admitted directly on 80/8080 by `llm-serving-ingress`, and `memory/crates/mem-cli/src/kv_warmer.rs` is such a caller. A distributed lock would have been blind to that traffic. ## Changes - **new `internal/proxy/engine_state.go`** — 2s scrape of `http://<addr>/metrics` on its own client (the proxy transports carry socket options tuned for inference), hand-rolled exposition parse, `Healthy=false` after 3 consecutive failed polls with `engine_state_stale_total` counted once at the transition, and a startup self-check that raises `observer_degraded` and logs at error level instead of degrading to fail-open silently. - **admission gate** (`internal/proxy/scheduler.go`) — parks while `kv_cache_usage_perc >= 0.90` or `running + waiting >= ADMISSION_MAX_INFLIGHT`, bounded by `ADMISSION_MAX_PARK_SECONDS`, then `503 + Retry-After: 5`. Unknown or unhealthy state fails **open** on purpose: a metrics outage must not become an inference outage. - **parked streaming heartbeat** (`internal/proxy/proxy.go`) — `stream: true` requests get 200 + `text/event-stream` and a `: keep-alive` comment every 15s while parked, so a client idle timeout does not kill a request that is legitimately waiting. Non-streaming requests park silently. - **`internal/proxy/router.go`** — extract only the top-level `model` field with a `json.Decoder` token walk instead of unmarshalling the whole body into `map[string]interface{}` (that built the entire `messages`+`tools` tree per request, ~300 KB for a 65k-token prompt). `MaxBodySize` now comes from `MAX_BODY_BYTES`; the model path previously inherited the 100 MB default at `proxy.go:93` against a 512Mi container limit. - **`cmd/gateway/main.go`** — `GOMAXPROCS` from the cgroup CPU quota (cgroup v2 `cpu.max`, v1 fallback). The container is limited to `500m` while Go defaulted to the host core count and was being CFS-throttled. - **`k8s/deployment.yaml`** — `ADMISSION_MAX_PARK_SECONDS=120`, `ADMISSION_OBSERVE_MODELS=reasoning`, `VLLM_MAX_QUEUE_DEPTH=8`, `MAX_BODY_BYTES=1048576`; corrected the `llm-client` comment (the NetworkPolicy selects by namespace now, not by that label). - **deleted `k8s/configmap.yaml`** — absent from `k8s/kustomization.yaml`, and the live ConfigMap of that name carries an unrelated key (`gw-config-fixed.yaml`) plus a stale routing table. - **`internal/observability/`** — `admission_parked_total`, `admission_rejected_total{reason}`, `engine_state_stale_total`, and gauges `vllm_requests_running` / `vllm_kv_usage_perc` (the gateway's own reading, deliberately mirroring the vLLM series so a disagreement is visible) and `observer_degraded`. Names follow the existing convention, which carries no prefix. Out of scope here, in `riotpiao-poimen/homelab#51`: the engine flags (`--max-num-partial-prefills`, `--max-long-partial-prefills`, `--long-prefill-token-threshold`, `--max-num-batched-tokens`, `--max-num-seqs`, `--gpu-memory-utilization`) and the four dashboard panels in `generate-dashboards.py`. ## Verification - `go build ./...` clean, `go vet ./...` clean. - `go test ./...` — 368 tests, 0 failures (baseline on `main` is 368/0). - `structural-check.sh` on the changed files — 0 errors, 1 pre-existing warning (`:latest` image tag). - `go.mod` / `go.sum` byte-identical: no new dependency. Not yet verified: the real-engine load test (2x65k + 15x8k against `reasoning`). That run also produces the number for `ADMISSION_MAX_PARK_SECONDS`, which today is the provisional 120; retuning it is a follow-up since it is env-only and needs no rebuild. ## Accepted risks - **No tests were added** for the observer or the gate (deliberate: production is the test bed). The mitigation is the startup self-check, which is production self-diagnosis rather than a test, plus the `observer_degraded` alert. A parse failure that outlives the first poll logs and surfaces as a metric; it does not silently disable gating without a signal. - **The cap is soft, not hard.** Between polls each replica may admit more than one request, so there is no clean bound. Exposure is low (~11,500 requests over 7d ≈ 1.4/min) and the engine remains the hard limit — a race parks requests in vLLM's queue rather than over-committing KV. - **`ADMISSION_MAX_INFLIGHT` is manually coupled to `--max-num-seqs`** and can drift silently. It defaults to `VLLM_MAX_NUM_SEQS`, and startup logs the effective value. - The first scrape waits one interval, so the first ~2s of admission is ungated (fail-open). Scraping at `Start()` instead would make the gateway contact every observed backend before any request needs one, which is wasted work and a surprising side effect. ## Rollback Revert this commit and redeploy. The admission path is fail-open, so the code can also be neutralised without reverting by unsetting `ADMISSION_OBSERVE_MODELS`. Separately, the orphaned cluster object can be removed by hand: `kubectl delete cm -n api api-gateway-config` (not a code change, deliberately left to a human). --------- Reviewed-on: #65 Co-authored-by: poimen <[email protected]>
173 lines
5.3 KiB
YAML
173 lines
5.3 KiB
YAML
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: api-gateway
|
|
namespace: api
|
|
labels:
|
|
app: api-gateway
|
|
component: gateway
|
|
spec:
|
|
replicas: 3
|
|
strategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxSurge: 1
|
|
maxUnavailable: 0
|
|
selector:
|
|
matchLabels:
|
|
app: api-gateway
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: api-gateway
|
|
component: gateway
|
|
# The llm-serving-ingress NetworkPolicy in the llm-serving namespace now
|
|
# selects by namespace (kubernetes.io/metadata.name: api), not by this
|
|
# label. Kept for older NetworkPolicy revisions that still admit on
|
|
# llm-client=true.
|
|
llm-client: "true"
|
|
annotations:
|
|
reloader.stakater.com/auto: "true"
|
|
prometheus.io/scrape: "true"
|
|
prometheus.io/port: "8080"
|
|
prometheus.io/path: "/metrics"
|
|
spec:
|
|
serviceAccountName: api-gateway
|
|
# The registry at forgejo.riotpiao.com requires auth — /v2/ answers 401.
|
|
# This Secret must exist in the api namespace before the first rollout.
|
|
imagePullSecrets:
|
|
- name: forgejo-registry
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
# 65532 is distroless's nonroot user, matching USER in the Dockerfile.
|
|
runAsUser: 65532
|
|
fsGroup: 65532
|
|
containers:
|
|
- name: gateway
|
|
# Tag is pinned in kustomization.yaml so there is exactly one place to
|
|
# bump it. Never :latest — Argo cannot make a deterministic rollout
|
|
# decision from a mutable tag, and 6.1 requires SHA tags.
|
|
image: forgejo.riotpiao.com/rock/api-gateway:latest
|
|
imagePullPolicy: Always
|
|
ports:
|
|
- name: http
|
|
containerPort: 8080
|
|
protocol: TCP
|
|
env:
|
|
- name: LISTEN_ADDR
|
|
value: "0.0.0.0:8080"
|
|
- name: CONFIG_PATH
|
|
value: "/etc/gateway/config.yaml"
|
|
- name: AUTH_CLIENT_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: api-gw-client-secret
|
|
key: client-secret
|
|
optional: true
|
|
- name: SHUTDOWN_TIMEOUT
|
|
value: "5m"
|
|
- name: LOG_LEVEL
|
|
value: "info"
|
|
- name: ADMISSION_MAX_PARK_SECONDS
|
|
value: "120"
|
|
# Only backends that publish the vLLM scheduler gauges may be observed.
|
|
# `reasoning` is the only vLLM engine; ornith/qwen-cpu are Ollama and
|
|
# embeddings/rerank are TEI, so observing them would hold
|
|
# observer_degraded at 1 forever.
|
|
- name: ADMISSION_OBSERVE_MODELS
|
|
value: "reasoning"
|
|
- name: VLLM_MAX_QUEUE_DEPTH
|
|
value: "8"
|
|
- name: MAX_BODY_BYTES
|
|
value: "1048576"
|
|
# OpenTelemetry tracing configuration
|
|
- name: OTEL_EXPORTER_OTLP_ENDPOINT
|
|
value: "otel-collector.tracing.svc.cluster.local:4317"
|
|
- name: OTEL_SERVICE_NAME
|
|
value: "api-gateway"
|
|
- name: OTEL_SERVICE_VERSION
|
|
value: "1.0.0"
|
|
- name: OTEL_ENVIRONMENT
|
|
value: "production"
|
|
# Gotify integration — Forgejo webhook → push notifications
|
|
- name: GOTIFY_URL
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gotify-webhook-secret
|
|
key: gotify-url
|
|
optional: true
|
|
- name: GOTIFY_APP_TOKEN
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gotify-webhook-secret
|
|
key: gotify-app-token
|
|
optional: true
|
|
- name: FORGEJO_WEBHOOK_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gotify-webhook-secret
|
|
key: forgejo-webhook-secret
|
|
optional: true
|
|
- name: REGISTRY_PAT
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: forgejo-registry-pat
|
|
key: pat
|
|
optional: true
|
|
- name: REGISTRY_BASE
|
|
value: "http://forgejo-gitea-http.cicd.svc.cluster.local:3000"
|
|
volumeMounts:
|
|
- name: config
|
|
mountPath: /etc/gateway
|
|
readOnly: true
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /healthz
|
|
port: http
|
|
scheme: HTTP
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 10
|
|
timeoutSeconds: 2
|
|
failureThreshold: 3
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /readyz
|
|
port: http
|
|
scheme: HTTP
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 5
|
|
timeoutSeconds: 2
|
|
failureThreshold: 2
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 128Mi
|
|
limits:
|
|
cpu: 500m
|
|
memory: 512Mi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
runAsNonRoot: true
|
|
runAsUser: 65532
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
volumes:
|
|
- name: config
|
|
secret:
|
|
secretName: api-gateway-config
|
|
affinity:
|
|
podAntiAffinity:
|
|
preferredDuringSchedulingIgnoredDuringExecution:
|
|
- weight: 100
|
|
podAffinityTerm:
|
|
labelSelector:
|
|
matchExpressions:
|
|
- key: app
|
|
operator: In
|
|
values:
|
|
- api-gateway
|
|
topologyKey: kubernetes.io/hostname
|
|
terminationGracePeriodSeconds: 300
|