727b9e1cb0
CI / CI (push) Successful in 43m8s
Follow-up to #65, which is already merged. The review of that PR returned BLOCK with two high-severity defects; the fix commit was pushed **after** the merge landed, so none of it is in `main`. ## What happened - `main` @ `369d51a` is the squash of the first commit only (`c5a91cf`). - `d71c884` (review fixes) is on the old branch and was never merged — PR #65 is closed. - Verified against `main` directly: `ADMISSION_MAX_INFLIGHT` 0 hits, `func (rw *responseWriter) Flush` 0 hits, `ModifyResponse` 0 hits, `Snapshot` 0 hits. - The image in production (`forgejo.riotpiao.com/rock/api-gateway@sha256:64f2bfc1…`, 3/3 replicas) is built from that squash, so the defects below are **live right now**. ## Live state confirmed `https://api.riotpiao.com/metrics` on the running pods: ``` observer_degraded 0 vllm_requests_running 1 vllm_kv_usage_perc 0.008293092588704587 admission_parked_total 0 engine_state_stale_total 0 ``` The observer itself works: it is reading the real vLLM exposition body and the startup self-check passed (`observer_degraded 0`). No requests have parked yet, so the two defects below have not been observable in the wild — they are proven from code, not from a live failure. ## Defects fixed here **1. The parked-stream keep-alive never runs in production (high).** `internal/tracing/middleware.go` wraps the `ResponseWriter` in a struct that embeds the interface and implements only `WriteHeader`/`Write`, so `w.(http.Flusher)` in `internal/proxy/proxy.go` is false for every real request (`cmd/gateway/main.go` → `tracing.Middleware` → `router`). `prewritten` therefore never becomes true, the heartbeat never starts, and a parked streaming request writes zero bytes for up to `ADMISSION_MAX_PARK_SECONDS` (120) — the client timeout the feature exists to prevent. Fixed by adding `Flush()` and `Unwrap()` to the wrapper. **2. Upstream status masked once the SSE status is spent (high, latent behind 1).** `httputil.ReverseProxy` calls `rw.WriteHeader(res.StatusCode)` unconditionally; after the park wrote 200, net/http keeps 200 and the upstream error body is appended raw into the event stream, which a streaming client reads as an empty completion and reports as success. Fixed with a `ModifyResponse` that frames an upstream `>= 400` as an `event: error` when the status has already been written. **3. Lost wakeup in the park loop.** `Get()` and `Tick()` were taken under two separate lock acquisitions, so a poll landing between them pushed the waiter onto the next generation's channel — up to one extra poll interval (2s) parked. `Snapshot()` now returns state and channel under one lock. **4. `observer_degraded` pinned at 1 forever.** It was raised only on the *first* poll's parse failure and `SetObserverDegraded(false)` had no call site, so one truncated response pinned the gauge for the pod's life, while genuine metric-name drift appearing later was never reported at all. `markDegraded` now tracks current parsability in both directions and only touches the exporter on a change. **5. `ADMISSION_MAX_INFLIGHT` silently defaulted.** Now set explicitly to `4` in `k8s/deployment.yaml` with the coupling to the engine's `--max-num-seqs` documented, instead of quietly inheriting `VLLM_MAX_NUM_SEQS`. ## Verification `go build ./...` clean · `go vet ./...` clean · `go test ./...` 368 passed, 0 failed · `structural-check.sh` 0 errors (1 pre-existing `:latest` warning) · `go.mod` unchanged. Cherry-picked onto `main` rather than merged from the old branch, so this diff is exactly the five fixes and nothing else: 5 files, +80/−8. ## Still open The reviewer's BLOCK also rested on nine functions over CRAP>30 purely from 0% coverage — the deliberate "no new tests" decision. Worth recording that defect 1 is a one-line type assertion that a `httptest` case through the real middleware chain would have caught immediately. Raising it here so the decision is visible rather than implied. --------- Co-authored-by: Poimen <[email protected]> Reviewed-on: #66 Co-authored-by: poimen <[email protected]>
179 lines
5.6 KiB
YAML
179 lines
5.6 KiB
YAML
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: api-gateway
|
|
namespace: api
|
|
labels:
|
|
app: api-gateway
|
|
component: gateway
|
|
spec:
|
|
replicas: 3
|
|
strategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxSurge: 1
|
|
maxUnavailable: 0
|
|
selector:
|
|
matchLabels:
|
|
app: api-gateway
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: api-gateway
|
|
component: gateway
|
|
# The llm-serving-ingress NetworkPolicy in the llm-serving namespace now
|
|
# selects by namespace (kubernetes.io/metadata.name: api), not by this
|
|
# label. Kept for older NetworkPolicy revisions that still admit on
|
|
# llm-client=true.
|
|
llm-client: "true"
|
|
annotations:
|
|
reloader.stakater.com/auto: "true"
|
|
prometheus.io/scrape: "true"
|
|
prometheus.io/port: "8080"
|
|
prometheus.io/path: "/metrics"
|
|
spec:
|
|
serviceAccountName: api-gateway
|
|
# The registry at forgejo.riotpiao.com requires auth — /v2/ answers 401.
|
|
# This Secret must exist in the api namespace before the first rollout.
|
|
imagePullSecrets:
|
|
- name: forgejo-registry
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
# 65532 is distroless's nonroot user, matching USER in the Dockerfile.
|
|
runAsUser: 65532
|
|
fsGroup: 65532
|
|
containers:
|
|
- name: gateway
|
|
# Tag is pinned in kustomization.yaml so there is exactly one place to
|
|
# bump it. Never :latest — Argo cannot make a deterministic rollout
|
|
# decision from a mutable tag, and 6.1 requires SHA tags.
|
|
image: forgejo.riotpiao.com/rock/api-gateway:latest
|
|
imagePullPolicy: Always
|
|
ports:
|
|
- name: http
|
|
containerPort: 8080
|
|
protocol: TCP
|
|
env:
|
|
- name: LISTEN_ADDR
|
|
value: "0.0.0.0:8080"
|
|
- name: CONFIG_PATH
|
|
value: "/etc/gateway/config.yaml"
|
|
- name: AUTH_CLIENT_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: api-gw-client-secret
|
|
key: client-secret
|
|
optional: true
|
|
- name: SHUTDOWN_TIMEOUT
|
|
value: "5m"
|
|
- name: LOG_LEVEL
|
|
value: "info"
|
|
- name: ADMISSION_MAX_PARK_SECONDS
|
|
value: "120"
|
|
# Must track the engine's --max-num-seqs (homelab: reasoning.yaml).
|
|
# The gauge this compares against is engine-wide, so this number is a
|
|
# ceiling on concurrent requests across all gateway replicas, not a
|
|
# per-pod slot count. Drift here is silent; startup logs the value.
|
|
- name: ADMISSION_MAX_INFLIGHT
|
|
value: "4"
|
|
# Only backends that publish the vLLM scheduler gauges may be observed.
|
|
# `reasoning` is the only vLLM engine; ornith/qwen-cpu are Ollama and
|
|
# embeddings/rerank are TEI, so observing them would hold
|
|
# observer_degraded at 1 forever.
|
|
- name: ADMISSION_OBSERVE_MODELS
|
|
value: "reasoning"
|
|
- name: VLLM_MAX_QUEUE_DEPTH
|
|
value: "8"
|
|
- name: MAX_BODY_BYTES
|
|
value: "1048576"
|
|
# OpenTelemetry tracing configuration
|
|
- name: OTEL_EXPORTER_OTLP_ENDPOINT
|
|
value: "otel-collector.tracing.svc.cluster.local:4317"
|
|
- name: OTEL_SERVICE_NAME
|
|
value: "api-gateway"
|
|
- name: OTEL_SERVICE_VERSION
|
|
value: "1.0.0"
|
|
- name: OTEL_ENVIRONMENT
|
|
value: "production"
|
|
# Gotify integration — Forgejo webhook → push notifications
|
|
- name: GOTIFY_URL
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gotify-webhook-secret
|
|
key: gotify-url
|
|
optional: true
|
|
- name: GOTIFY_APP_TOKEN
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gotify-webhook-secret
|
|
key: gotify-app-token
|
|
optional: true
|
|
- name: FORGEJO_WEBHOOK_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gotify-webhook-secret
|
|
key: forgejo-webhook-secret
|
|
optional: true
|
|
- name: REGISTRY_PAT
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: forgejo-registry-pat
|
|
key: pat
|
|
optional: true
|
|
- name: REGISTRY_BASE
|
|
value: "http://forgejo-gitea-http.cicd.svc.cluster.local:3000"
|
|
volumeMounts:
|
|
- name: config
|
|
mountPath: /etc/gateway
|
|
readOnly: true
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /healthz
|
|
port: http
|
|
scheme: HTTP
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 10
|
|
timeoutSeconds: 2
|
|
failureThreshold: 3
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /readyz
|
|
port: http
|
|
scheme: HTTP
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 5
|
|
timeoutSeconds: 2
|
|
failureThreshold: 2
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 128Mi
|
|
limits:
|
|
cpu: 500m
|
|
memory: 512Mi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
runAsNonRoot: true
|
|
runAsUser: 65532
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
volumes:
|
|
- name: config
|
|
secret:
|
|
secretName: api-gateway-config
|
|
affinity:
|
|
podAntiAffinity:
|
|
preferredDuringSchedulingIgnoredDuringExecution:
|
|
- weight: 100
|
|
podAffinityTerm:
|
|
labelSelector:
|
|
matchExpressions:
|
|
- key: app
|
|
operator: In
|
|
values:
|
|
- api-gateway
|
|
topologyKey: kubernetes.io/hostname
|
|
terminationGracePeriodSeconds: 300
|