homelab-frontend/OBSERVABILITY.md

Transaction-Based Observability Plan

Overview

Implement transaction-driven observability with correlation ID propagation across all gateway services (LLM, SQS, Temporal, Memory, S3, IAM). Pattern inspired by Amazon's SQS-based transaction tracing.

Architecture

Client Request
      │
      ▼
┌─────────────────────────────────────────────────────────┐
│  Gateway                                                │
│  ┌─────────────────────────────────────────────────┐   │
│  │ 1. Generate/extract X-Request-ID                │   │
│  │ 2. Start timer                                  │   │
│  │ 3. Validate JWT                                 │   │
│  │ 4. Route to upstream                            │   │
│  │ 5. Stream response                              │   │
│  │ 6. Emit metrics (request_id, model, status)     │   │
│  │ 7. Log with correlation ID                      │   │
│  └─────────────────────────────────────────────────┘   │
│                         │                               │
│                         ▼                               │
│              ┌──────────────────┐                       │
│              │ /metrics         │ ◄── Prometheus scrape │
│              │ (RED + tokens)   │                       │
│              └──────────────────┘                       │
└─────────────────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────┐
│ LLM Upstream        │
│ (vLLM/Ollama/TEI)   │
│ X-Request-ID passed │
└─────────────────────┘

Metrics

Request Metrics (RED)

# Rate - requests per second
llm_requests_total{model="reasoning", status="success|error", error_type="none|timeout|upstream|auth"}

# Errors - error rate by type
llm_request_errors_total{model="reasoning", error_type="timeout|upstream_5xx|auth_failed|invalid_request"}

# Duration - latency histograms
llm_request_duration_seconds{model="reasoning", phase="total|ttft|generation"}

Token Metrics

# Token throughput
llm_tokens_total{model="reasoning", direction="prompt|completion"}

# Tokens per request (histogram)
llm_tokens_per_request{model="reasoning", direction="prompt|completion"}

Connection Metrics

# Active requests (gauge)
llm_active_requests{model="reasoning"}

# Upstream health
llm_upstream_health{upstream="reasoning-predictor.llm-serving:80", status="healthy|unhealthy"}

Service-Specific Metrics

SQS (Queue Operations)

# Message latency
sqs_message_latency_seconds{queue="orders", operation="send|receive|delete", status="success|error"}

# Message throughput
sqs_messages_total{queue="orders", operation="send|receive|delete"}

# DLQ errors
sqs_dlq_messages_total{queue="orders", reason="timeout|invalid_format|permission_denied"}

# Queue depth (gauge)
sqs_queue_depth{queue="orders"}

# Error rate by type
sqs_errors_total{queue="orders", error_type="timeout|network|auth|rate_limit"}

Temporal (Workflow Orchestration)

# Workflow execution time
temporal_workflow_duration_seconds{workflow_type="ProcessOrder", status="completed|failed|timeout"}

# Workflow state transitions
temporal_workflow_state_transitions_total{workflow_type="ProcessOrder", from="PENDING", to="RUNNING|COMPLETED|FAILED"}

# Activity success rate
temporal_activity_success_rate{activity="CloneRepo|AnalyzeCode|Deploy", status="success|failure|retry"}

# Workflow failures by reason
temporal_workflow_failures_total{workflow_type="ProcessOrder", reason="timeout|activity_failed|network|permission_denied"}

# Active workflows (gauge)
temporal_active_workflows{workflow_type="ProcessOrder"}

Memory / Poimen (Semantic Search)

# Query latency
memory_query_latency_seconds{level="L1|L2", status="success|error"}

# Ingestion throughput
memory_documents_ingested_total{project="homelab|portfolio", level="L1|L2"}

# Rerank scoring
memory_rerank_score{query_type="similarity|relevance", percentile="p50|p95|p99"}

# Vector DB errors
memory_errors_total{operation="query|ingest", error_type="timeout|invalid_query|capacity_exceeded"}

# Cache hits
memory_cache_hits_total{level="L1|L2"}

S3 (Object Storage)

# Upload/download bandwidth
s3_bytes_transferred{operation="put|get", bucket="data|backups", status="success|error"}

# Object operations latency
s3_operation_duration_seconds{operation="put|get|delete|list", bucket="data"}

# Multipart upload failures
s3_multipart_failures_total{bucket="data", reason="timeout|part_mismatch|abort"}

# Bucket space usage (gauge)
s3_bucket_size_bytes{bucket="data|backups"}

# Error rate
s3_errors_total{bucket="data", error_type="network|auth|timeout|quota_exceeded"}

IAM (Identity & Access Management)

# User/role mutations
iam_mutations_total{operation="create_user|update_user|delete_user|add_role|remove_role", status="success|failed"}

# Permission cache
iam_permission_cache_hits_total{resource="user|role|group"}

# Auth flow latency
iam_auth_latency_seconds{flow="client_credentials|authorization_code|refresh", status="success|failed"}

# JWT validation
iam_jwt_validations_total{result="valid|expired|invalid_signature|insufficient_permissions"}

# Error rate by type
iam_errors_total{operation="list_users|get_user|create_user", error_type="auth_failed|not_found|permission_denied|network"}

Gateway (All Services)

# Request rate by service
gateway_requests_total{service="llm|sqs|workflow|memory|s3|iam", status="2xx|4xx|5xx"}

# Latency by service
gateway_request_duration_seconds{service="llm|sqs|workflow|memory|s3|iam", phase="total|auth|upstream"}

# Correlation ID tracking
gateway_correlation_id_errors_total{reason="missing|invalid_format|timeout"}

Correlation ID Flow

Request ID Generation

func RequestIDMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        requestID := r.Header.Get("X-Request-ID")
        if requestID == "" {
            requestID = uuid.New().String()
        }
        
        // Set in context for logging
        ctx := context.WithValue(r.Context(), RequestIDKey, requestID)
        
        // Echo back to client
        w.Header().Set("X-Request-ID", requestID)
        
        // Propagate to upstream
        r.Header.Set("X-Request-ID", requestID)
        
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

Structured Logging

{
  "timestamp": "2026-09-01T12:00:00Z",
  "level": "info",
  "message": "request completed",
  "request_id": "abc-123-def",
  "model": "reasoning",
  "status": "success",
  "duration_ms": 1523,
  "tokens_prompt": 150,
  "tokens_completion": 89,
  "user_id": "rock",
  "client_ip": "192.168.1.100"
}

Error case:

{
  "timestamp": "2026-09-01T12:00:00Z",
  "level": "error",
  "message": "request failed",
  "request_id": "abc-123-def",
  "model": "reasoning",
  "status": "error",
  "error_type": "upstream_timeout",
  "error_detail": "context deadline exceeded",
  "duration_ms": 30000,
  "action": "LLM_INFERENCE",
  "exception": "UPSTREAM_TIMEOUT"
}

Drill-Down Query Flow

1. Dashboard Alert

Grafana shows spike in error rate:

rate(llm_request_errors_total{error_type="upstream_timeout"}[5m]) > 0.1

2. Error Breakdown

Click through to see error distribution:

sum by (error_type, model) (rate(llm_request_errors_total[5m]))

3. Find Affected Requests

Loki query for specific error type:

{namespace="api", app="api-gateway"} 
| json 
| error_type="upstream_timeout" 
| line_format "{{.request_id}} {{.model}} {{.duration_ms}}ms"

4. Trace Single Request

Drill into specific request_id:

{namespace=~"api|llm-serving"} |= "abc-123-def"

5. Correlate with Upstream

If vLLM/Ollama logs also include X-Request-ID:

{namespace="llm-serving"} |= "abc-123-def"

Implementation Tasks

Phase 1: Metrics Foundation

• Create internal/observability/metrics.go
  • Define Prometheus metrics (counters, histograms, gauges)
  • Register with default registry
• Add /metrics endpoint to server
• Create ServiceMonitor for gateway

Phase 2: Request ID Middleware

• Create internal/observability/requestid.go
  • Generate/extract X-Request-ID
  • Store in context
  • Propagate to upstream
• Wire into server router

Phase 3: Instrumentation

• Instrument proxy handler
  • Record request start/end times
  • Count tokens from response (parse SSE stream)
  • Emit metrics on completion
• Update structured logging
  • Add request_id to all log entries
  • Add action/exception pattern for errors

Phase 4: Dashboard

• Update llm-frontend dashboard
  • Request rate panel
  • Error rate by type panel
  • Latency percentiles (p50, p95, p99)
  • Token throughput panel
  • Active requests gauge
• Add drill-down links to Loki

Phase 5: Alerting

• Create PrometheusRule for LLM alerts
  • High error rate
  • High latency (p99 > threshold)
  • Upstream unhealthy
  • Token rate anomaly

File Structure

internal/observability/
├── metrics.go       # Prometheus metric definitions
├── requestid.go     # Request ID middleware
├── instrumentation.go # Metric recording helpers
└── logging.go       # Structured logging with correlation

k8s/
├── servicemonitor.yaml  # Prometheus scrape config
└── prometheusrule.yaml  # Alert rules

k8s/infra/monitoring/dashboards/
└── llm-frontend.yaml    # Updated dashboard

Metric Labels Convention

Label	Values	Description
model	reasoning, ornith:35b, qwen2.5:3b-instruct, etc.	LLM model name
status	success, error	Request outcome
error_type	none, timeout, upstream_5xx, auth_failed, invalid_request, rate_limited	Error category
phase	total, ttft, generation	Latency measurement phase
direction	prompt, completion	Token direction
upstream	host:port	Upstream address

Action-Exception Pattern

For structured error tracking (Amazon-style):

Action	Exception	Metric
LLM_INFERENCE	UPSTREAM_TIMEOUT	llm_action_exception_total{action="LLM_INFERENCE", exception="UPSTREAM_TIMEOUT"}
LLM_INFERENCE	UPSTREAM_5XX	llm_action_exception_total{action="LLM_INFERENCE", exception="UPSTREAM_5XX"}
JWT_VALIDATION	TOKEN_EXPIRED	llm_action_exception_total{action="JWT_VALIDATION", exception="TOKEN_EXPIRED"}
JWT_VALIDATION	INVALID_SIGNATURE	llm_action_exception_total{action="JWT_VALIDATION", exception="INVALID_SIGNATURE"}
BODY_PARSE	INVALID_JSON	llm_action_exception_total{action="BODY_PARSE", exception="INVALID_JSON"}
MODEL_DISPATCH	UNKNOWN_MODEL	llm_action_exception_total{action="MODEL_DISPATCH", exception="UNKNOWN_MODEL"}

Query pattern:

# All exceptions for an action
sum by (exception) (rate(llm_action_exception_total{action="LLM_INFERENCE"}[5m]))

# Specific exception rate
rate(llm_action_exception_total{action="LLM_INFERENCE", exception="UPSTREAM_TIMEOUT"}[5m])

Dependencies

// go.mod additions
require (
    github.com/prometheus/client_golang v1.19.0
    github.com/google/uuid v1.6.0
)

References

• Prometheus Go client
• OpenTelemetry metrics
• Grafana Loki LogQL
• RED method