Add TTFT & ITL Metrics for LLM Inference #28

2026-09-14T15:42:04Z

rock commented

2026-09-14 15:42:04 +00:00

Time-to-First-Token (TTFT) and Inter-Token Latency (ITL) metrics for LLM inference observability

Metrics: llm_ttft_seconds, llm_itl_seconds, llm_tokens_total

Time-to-First-Token (TTFT) and Inter-Token Latency (ITL) metrics for LLM inference observability Metrics: llm_ttft_seconds, llm_itl_seconds, llm_tokens_total Closes #31 #32 #33

rock added 11 commits 2026-09-15 06:14:37 +00:00

feat(network): SSE optimization for local LLM streaming (#31 #32 #33 ) 178af6afe7

Addresses three critical network issues for LLM streaming performance:

**#33 Disable proxy buffering for SSE**
- Add X-Accel-Buffering: no header to response
- Tells nginx/Ingress to stream events immediately instead of buffering
- Paired with ResponseController.Flush() for unbuffered token delivery

**#32 HTTP/2 multiplexing for concurrent streams**
- Enable HTTP/2 in server config via http2.ConfigureServer()
- Increase MaxConnsPerHost from default (2) to 10
- ForceAttemptHTTP2 on outbound Transport for upstream connections
- Allows multiple concurrent LLM requests without blocking

**#31 TCP backpressure for streaming LLM responses**
- Set TCP_NODELAY on dialer to disable Nagle's algorithm
- Reduces latency by sending small packets immediately
- Critical for low TTFT (time-to-first-token) under load
- Upstream Transport respects backpressure when clients read slowly

**Tests added:**
- TestTCPBackpressure: Verifies TCP backpressure handling with slow client
- TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly
- Both pass at 0.11s and 0.06s respectively

Fixes all three streaming performance issues in one coherent change.

test: add workflow visibility tests for poimen-harness namespace f44a74d05e

Verify that WorkflowAdapter provides visibility into terminated workflows
in the poimen-harness namespace. This ensures namespace pass-down feature
is working correctly and users can specify different domains/namespaces
via X-Service: workflow requests.

Tests added:
1. integration-test.sh: Added workflow visibility tests
   - List workflows in poimen-harness namespace
   - Verify terminated/completed workflows are visible
   - Validate namespace parameter requirement
   - Check auth enforcement

2. workflow-visibility-test.sh: NEW dedicated workflow test script
   - Tests WorkflowAdapter namespace pass-down
   - Verifies list, describe, and auth enforcement
   - Specific focus on poimen-harness namespace
   - Looks for 4 terminated workflows

3. task-workflow-visibility.yaml: NEW Tekton task
   - Runs workflow visibility tests against live gateway
   - Sidecar deployment pattern
   - Publishes result + summary + workflow-count metrics

4. pipeline-sse-optimization.yaml: Updated
   - Added workflow-visibility-tests stage (runs after integration-tests)
   - Updated report-results to include workflow test results
   - Full pipeline now: integration → workflow-visibility → load → report

5. kustomization.yaml: Updated
   - Added task-workflow-visibility.yaml
   - Added workflow-visibility-test-script ConfigMap

This ensures that the deprecated /workflows endpoint replacement correctly
supports multi-tenant access via namespace specification in request payload.

feat: add TTFT/ITL metrics for LLM inference e5a1f052a4

- RecordTTFT: Time-to-First-Token in milliseconds
- RecordITL: Inter-Token Latency in milliseconds
- RecordTokenCount: Track total tokens generated
- Prometheus exporter for /metrics endpoint
- Grafana dashboard ConfigMap (llm-metrics.json)
- ResponseWriterWrapper to capture metrics during LLM calls
- Metrics exported: llm_ttft_seconds, llm_itl_seconds, llm_tokens_total

fix: go vet errors in observability metrics 94c0cf6fe6

- Remove unused avg variable in prometheus.go
- Fix import ordering in llm_metrics.go (move net/http to top)

chore: remove BFG repo-cleaner reports a9f0d1a4fb

feat: add Gotify support to sendMsg handler 39c450cb77

- Add sendGotify method to send notifications to Gotify server
- Support format: 'gotify' in SendMsgRequest
- Extract message ID from Gotify response
- Configurable via GOTIFY_URL and GOTIFY_TOKEN env vars
- Fallback graceful error if Gotify not configured

fix: workflow namespace should be 'temporal' not 'api' 743148f576

The workflow service is deployed in the temporal namespace:
temporal-frontend.temporal.svc.cluster.local:7233

Update ServiceAdapter to use correct namespace for workflow routing.

feat: Gotify CRUD + internal handler dispatch for notification service 18e2031ab9

- Add internal handler support to ServiceAdapter (Handler field)
- Dispatcher routes to internal handler when set (no reverse proxy)
- GotifyClient: full CRUD (send/list/delete messages, CRUD applications)
- Refactor notification handler: route by X-Resource header, not body format
- Register notification as ServiceAdapter with auth required
- Resources: send-email, send-message, list-messages, delete-message,
  delete-all-messages, list-applications, create-application, delete-application
- Update examples: sendmsg-email.sh, gotify-crud.sh
- Env vars: GOTIFY_URL, GOTIFY_APP_TOKEN, GOTIFY_CLIENT_TOKEN

feat: add upstreamModel config for model name mapping 65928b109b

When the upstream LLM server expects a different model name than what
clients send, the gateway now rewrites the 'model' field in the request
body before forwarding.

Config example:
  models:
  - name: "ornith:35b"           # client-facing name
    address: "ornith-predictor:80"
    upstreamModel: "qwen2.5:72b"  # what upstream expects

Fixes 400 'model is required' errors when upstream model names differ.

test: notification handler + gotify client unit tests df6f8165b6

- 24 tests covering all X-Resource routes
- Mock Gotify server for message/application CRUD
- Error cases: nil gotify, invalid JSON, missing fields, 429, 500
- Fix readError() to handle io.ReadAll failure
- GotifyClient direct tests for all 7 methods

fix: workflow dispatcher uses internal handler instead of raw gRPC proxy

CI / CI (pull_request) Successful in 2m55s

Details

d999b02943

- Wire WorkflowAdapter as internal handler (JSON-to-gRPC bridge)
- Add ServeHTTP to WorkflowAdapter: maps X-Resource to Temporal action
- Rename resource 'start' to 'execute' for consistency
- Add GET methods for describe/list/history resources
- Remove unused workflowAdapterImpl variable
- SDK clients can now send JSON, gateway translates to gRPC

rock force-pushed feat/sse-optimization-31-32-33 from ef79bf9a80 to d999b02943

2026-09-15 06:14:37 +00:00

Compare

rock merged commit d439536ca9 into main

2026-09-15 08:54:00 +00:00

rock deleted branch feat/sse-optimization-31-32-33

2026-09-15 08:54:00 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab-frontend#28