Files
homelab-frontend/internal/tracing/middleware.go
T
727b9e1cb0
CI / CI (push) Successful in 43m8s
fix(proxy): admission-control review fixes missing from the #65 merge (#66)
Follow-up to #65, which is already merged. The review of that PR returned BLOCK with two high-severity defects; the fix commit was pushed **after** the merge landed, so none of it is in `main`.

## What happened

- `main` @ `369d51a` is the squash of the first commit only (`c5a91cf`).
- `d71c884` (review fixes) is on the old branch and was never merged — PR #65 is closed.
- Verified against `main` directly: `ADMISSION_MAX_INFLIGHT` 0 hits, `func (rw *responseWriter) Flush` 0 hits, `ModifyResponse` 0 hits, `Snapshot` 0 hits.
- The image in production (`forgejo.riotpiao.com/rock/api-gateway@sha256:64f2bfc1…`, 3/3 replicas) is built from that squash, so the defects below are **live right now**.

## Live state confirmed

`https://api.riotpiao.com/metrics` on the running pods:

```
observer_degraded 0
vllm_requests_running 1
vllm_kv_usage_perc 0.008293092588704587
admission_parked_total 0
engine_state_stale_total 0
```

The observer itself works: it is reading the real vLLM exposition body and the startup self-check passed (`observer_degraded 0`). No requests have parked yet, so the two defects below have not been observable in the wild — they are proven from code, not from a live failure.

## Defects fixed here

**1. The parked-stream keep-alive never runs in production (high).** `internal/tracing/middleware.go` wraps the `ResponseWriter` in a struct that embeds the interface and implements only `WriteHeader`/`Write`, so `w.(http.Flusher)` in `internal/proxy/proxy.go` is false for every real request (`cmd/gateway/main.go` → `tracing.Middleware` → `router`). `prewritten` therefore never becomes true, the heartbeat never starts, and a parked streaming request writes zero bytes for up to `ADMISSION_MAX_PARK_SECONDS` (120) — the client timeout the feature exists to prevent. Fixed by adding `Flush()` and `Unwrap()` to the wrapper.

**2. Upstream status masked once the SSE status is spent (high, latent behind 1).** `httputil.ReverseProxy` calls `rw.WriteHeader(res.StatusCode)` unconditionally; after the park wrote 200, net/http keeps 200 and the upstream error body is appended raw into the event stream, which a streaming client reads as an empty completion and reports as success. Fixed with a `ModifyResponse` that frames an upstream `>= 400` as an `event: error` when the status has already been written.

**3. Lost wakeup in the park loop.** `Get()` and `Tick()` were taken under two separate lock acquisitions, so a poll landing between them pushed the waiter onto the next generation's channel — up to one extra poll interval (2s) parked. `Snapshot()` now returns state and channel under one lock.

**4. `observer_degraded` pinned at 1 forever.** It was raised only on the *first* poll's parse failure and `SetObserverDegraded(false)` had no call site, so one truncated response pinned the gauge for the pod's life, while genuine metric-name drift appearing later was never reported at all. `markDegraded` now tracks current parsability in both directions and only touches the exporter on a change.

**5. `ADMISSION_MAX_INFLIGHT` silently defaulted.** Now set explicitly to `4` in `k8s/deployment.yaml` with the coupling to the engine's `--max-num-seqs` documented, instead of quietly inheriting `VLLM_MAX_NUM_SEQS`.

## Verification

`go build ./...` clean · `go vet ./...` clean · `go test ./...` 368 passed, 0 failed · `structural-check.sh` 0 errors (1 pre-existing `:latest` warning) · `go.mod` unchanged.

Cherry-picked onto `main` rather than merged from the old branch, so this diff is exactly the five fixes and nothing else: 5 files, +80/−8.

## Still open

The reviewer's BLOCK also rested on nine functions over CRAP>30 purely from 0% coverage — the deliberate "no new tests" decision. Worth recording that defect 1 is a one-line type assertion that a `httptest` case through the real middleware chain would have caught immediately. Raising it here so the decision is visible rather than implied.

---------

Co-authored-by: Poimen <[email protected]>
Reviewed-on: #66
Co-authored-by: poimen <[email protected]>
2026-09-30 12:19:24 +00:00

125 lines
3.4 KiB
Go

package tracing
import (
"net/http"
"time"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/propagation"
semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
"go.opentelemetry.io/otel/trace"
)
const tracerName = "api-gateway"
// responseWriter wraps http.ResponseWriter to capture status code.
type responseWriter struct {
http.ResponseWriter
statusCode int
written int64
}
func newResponseWriter(w http.ResponseWriter) *responseWriter {
return &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}
}
func (rw *responseWriter) WriteHeader(code int) {
rw.statusCode = code
rw.ResponseWriter.WriteHeader(code)
}
func (rw *responseWriter) Write(b []byte) (int, error) {
n, err := rw.ResponseWriter.Write(b)
rw.written += int64(n)
return n, err
}
// Flush forwards to the wrapped writer so that streaming handlers can still
// detect http.Flusher through this wrapper. Without it a bare
// `w.(http.Flusher)` assertion fails for every request that passes through
// this middleware, which silently disables SSE flushing and keep-alives.
func (rw *responseWriter) Flush() {
if f, ok := rw.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
// Unwrap exposes the wrapped writer to http.ResponseController.
func (rw *responseWriter) Unwrap() http.ResponseWriter {
return rw.ResponseWriter
}
// Middleware returns an HTTP middleware that adds tracing to requests.
func Middleware(next http.Handler) http.Handler {
tracer := otel.Tracer(tracerName)
propagator := otel.GetTextMapPropagator()
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
// Extract any existing trace context from incoming request
ctx := propagator.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
// Start a new span
spanName := r.Method + " " + r.URL.Path
ctx, span := tracer.Start(ctx, spanName,
trace.WithSpanKind(trace.SpanKindServer),
trace.WithAttributes(
semconv.HTTPRequestMethodKey.String(r.Method),
semconv.URLPath(r.URL.Path),
semconv.URLScheme(scheme(r)),
semconv.ServerAddress(r.Host),
semconv.UserAgentOriginal(r.UserAgent()),
semconv.NetworkPeerAddress(r.RemoteAddr),
),
)
defer span.End()
// Add query parameters if present
if r.URL.RawQuery != "" {
span.SetAttributes(semconv.URLQuery(r.URL.RawQuery))
}
// Add model attribute for LLM requests
if model := r.Header.Get("X-Model"); model != "" {
span.SetAttributes(attribute.String("llm.model", model))
}
// Wrap response writer to capture status
rw := newResponseWriter(w)
// Inject trace context into response headers (for debugging)
propagator.Inject(ctx, propagation.HeaderCarrier(w.Header()))
// Call the next handler with traced context
next.ServeHTTP(rw, r.WithContext(ctx))
// Record response attributes
duration := time.Since(start)
span.SetAttributes(
semconv.HTTPResponseStatusCode(rw.statusCode),
attribute.Int64("http.response.body.size", rw.written),
attribute.Float64("http.request.duration_ms", float64(duration.Milliseconds())),
)
// Set span status based on HTTP status code
if rw.statusCode >= 400 {
span.SetStatus(codes.Error, http.StatusText(rw.statusCode))
} else {
span.SetStatus(codes.Ok, "")
}
})
}
func scheme(r *http.Request) string {
if r.TLS != nil {
return "https"
}
if s := r.Header.Get("X-Forwarded-Proto"); s != "" {
return s
}
return "http"
}