CI / CI (push) Successful in 43m8s
Follow-up to #65, which is already merged. The review of that PR returned BLOCK with two high-severity defects; the fix commit was pushed **after** the merge landed, so none of it is in `main`. ## What happened - `main` @ `369d51a` is the squash of the first commit only (`c5a91cf`). - `d71c884` (review fixes) is on the old branch and was never merged — PR #65 is closed. - Verified against `main` directly: `ADMISSION_MAX_INFLIGHT` 0 hits, `func (rw *responseWriter) Flush` 0 hits, `ModifyResponse` 0 hits, `Snapshot` 0 hits. - The image in production (`forgejo.riotpiao.com/rock/api-gateway@sha256:64f2bfc1…`, 3/3 replicas) is built from that squash, so the defects below are **live right now**. ## Live state confirmed `https://api.riotpiao.com/metrics` on the running pods: ``` observer_degraded 0 vllm_requests_running 1 vllm_kv_usage_perc 0.008293092588704587 admission_parked_total 0 engine_state_stale_total 0 ``` The observer itself works: it is reading the real vLLM exposition body and the startup self-check passed (`observer_degraded 0`). No requests have parked yet, so the two defects below have not been observable in the wild — they are proven from code, not from a live failure. ## Defects fixed here **1. The parked-stream keep-alive never runs in production (high).** `internal/tracing/middleware.go` wraps the `ResponseWriter` in a struct that embeds the interface and implements only `WriteHeader`/`Write`, so `w.(http.Flusher)` in `internal/proxy/proxy.go` is false for every real request (`cmd/gateway/main.go` → `tracing.Middleware` → `router`). `prewritten` therefore never becomes true, the heartbeat never starts, and a parked streaming request writes zero bytes for up to `ADMISSION_MAX_PARK_SECONDS` (120) — the client timeout the feature exists to prevent. Fixed by adding `Flush()` and `Unwrap()` to the wrapper. **2. Upstream status masked once the SSE status is spent (high, latent behind 1).** `httputil.ReverseProxy` calls `rw.WriteHeader(res.StatusCode)` unconditionally; after the park wrote 200, net/http keeps 200 and the upstream error body is appended raw into the event stream, which a streaming client reads as an empty completion and reports as success. Fixed with a `ModifyResponse` that frames an upstream `>= 400` as an `event: error` when the status has already been written. **3. Lost wakeup in the park loop.** `Get()` and `Tick()` were taken under two separate lock acquisitions, so a poll landing between them pushed the waiter onto the next generation's channel — up to one extra poll interval (2s) parked. `Snapshot()` now returns state and channel under one lock. **4. `observer_degraded` pinned at 1 forever.** It was raised only on the *first* poll's parse failure and `SetObserverDegraded(false)` had no call site, so one truncated response pinned the gauge for the pod's life, while genuine metric-name drift appearing later was never reported at all. `markDegraded` now tracks current parsability in both directions and only touches the exporter on a change. **5. `ADMISSION_MAX_INFLIGHT` silently defaulted.** Now set explicitly to `4` in `k8s/deployment.yaml` with the coupling to the engine's `--max-num-seqs` documented, instead of quietly inheriting `VLLM_MAX_NUM_SEQS`. ## Verification `go build ./...` clean · `go vet ./...` clean · `go test ./...` 368 passed, 0 failed · `structural-check.sh` 0 errors (1 pre-existing `:latest` warning) · `go.mod` unchanged. Cherry-picked onto `main` rather than merged from the old branch, so this diff is exactly the five fixes and nothing else: 5 files, +80/−8. ## Still open The reviewer's BLOCK also rested on nine functions over CRAP>30 purely from 0% coverage — the deliberate "no new tests" decision. Worth recording that defect 1 is a one-line type assertion that a `httptest` case through the real middleware chain would have caught immediately. Raising it here so the decision is visible rather than implied. --------- Co-authored-by: Poimen <[email protected]> Reviewed-on: #66 Co-authored-by: poimen <[email protected]>
125 lines
3.4 KiB
Go
125 lines
3.4 KiB
Go
package tracing
|
|
|
|
import (
|
|
"net/http"
|
|
"time"
|
|
|
|
"go.opentelemetry.io/otel"
|
|
"go.opentelemetry.io/otel/attribute"
|
|
"go.opentelemetry.io/otel/codes"
|
|
"go.opentelemetry.io/otel/propagation"
|
|
semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
|
|
"go.opentelemetry.io/otel/trace"
|
|
)
|
|
|
|
const tracerName = "api-gateway"
|
|
|
|
// responseWriter wraps http.ResponseWriter to capture status code.
|
|
type responseWriter struct {
|
|
http.ResponseWriter
|
|
statusCode int
|
|
written int64
|
|
}
|
|
|
|
func newResponseWriter(w http.ResponseWriter) *responseWriter {
|
|
return &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}
|
|
}
|
|
|
|
func (rw *responseWriter) WriteHeader(code int) {
|
|
rw.statusCode = code
|
|
rw.ResponseWriter.WriteHeader(code)
|
|
}
|
|
|
|
func (rw *responseWriter) Write(b []byte) (int, error) {
|
|
n, err := rw.ResponseWriter.Write(b)
|
|
rw.written += int64(n)
|
|
return n, err
|
|
}
|
|
|
|
// Flush forwards to the wrapped writer so that streaming handlers can still
|
|
// detect http.Flusher through this wrapper. Without it a bare
|
|
// `w.(http.Flusher)` assertion fails for every request that passes through
|
|
// this middleware, which silently disables SSE flushing and keep-alives.
|
|
func (rw *responseWriter) Flush() {
|
|
if f, ok := rw.ResponseWriter.(http.Flusher); ok {
|
|
f.Flush()
|
|
}
|
|
}
|
|
|
|
// Unwrap exposes the wrapped writer to http.ResponseController.
|
|
func (rw *responseWriter) Unwrap() http.ResponseWriter {
|
|
return rw.ResponseWriter
|
|
}
|
|
|
|
// Middleware returns an HTTP middleware that adds tracing to requests.
|
|
func Middleware(next http.Handler) http.Handler {
|
|
tracer := otel.Tracer(tracerName)
|
|
propagator := otel.GetTextMapPropagator()
|
|
|
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
start := time.Now()
|
|
|
|
// Extract any existing trace context from incoming request
|
|
ctx := propagator.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
|
|
|
|
// Start a new span
|
|
spanName := r.Method + " " + r.URL.Path
|
|
ctx, span := tracer.Start(ctx, spanName,
|
|
trace.WithSpanKind(trace.SpanKindServer),
|
|
trace.WithAttributes(
|
|
semconv.HTTPRequestMethodKey.String(r.Method),
|
|
semconv.URLPath(r.URL.Path),
|
|
semconv.URLScheme(scheme(r)),
|
|
semconv.ServerAddress(r.Host),
|
|
semconv.UserAgentOriginal(r.UserAgent()),
|
|
semconv.NetworkPeerAddress(r.RemoteAddr),
|
|
),
|
|
)
|
|
defer span.End()
|
|
|
|
// Add query parameters if present
|
|
if r.URL.RawQuery != "" {
|
|
span.SetAttributes(semconv.URLQuery(r.URL.RawQuery))
|
|
}
|
|
|
|
// Add model attribute for LLM requests
|
|
if model := r.Header.Get("X-Model"); model != "" {
|
|
span.SetAttributes(attribute.String("llm.model", model))
|
|
}
|
|
|
|
// Wrap response writer to capture status
|
|
rw := newResponseWriter(w)
|
|
|
|
// Inject trace context into response headers (for debugging)
|
|
propagator.Inject(ctx, propagation.HeaderCarrier(w.Header()))
|
|
|
|
// Call the next handler with traced context
|
|
next.ServeHTTP(rw, r.WithContext(ctx))
|
|
|
|
// Record response attributes
|
|
duration := time.Since(start)
|
|
span.SetAttributes(
|
|
semconv.HTTPResponseStatusCode(rw.statusCode),
|
|
attribute.Int64("http.response.body.size", rw.written),
|
|
attribute.Float64("http.request.duration_ms", float64(duration.Milliseconds())),
|
|
)
|
|
|
|
// Set span status based on HTTP status code
|
|
if rw.statusCode >= 400 {
|
|
span.SetStatus(codes.Error, http.StatusText(rw.statusCode))
|
|
} else {
|
|
span.SetStatus(codes.Ok, "")
|
|
}
|
|
})
|
|
}
|
|
|
|
func scheme(r *http.Request) string {
|
|
if r.TLS != nil {
|
|
return "https"
|
|
}
|
|
if s := r.Header.Get("X-Forwarded-Proto"); s != "" {
|
|
return s
|
|
}
|
|
return "http"
|
|
}
|