# User Group & Client Identification Logging

## Problem

Standard observability only tracks requests globally. Root cause analysis fails when:
- "Chat API fails for 10% of users" → which users? which client type?
- "SQS queue slow for portfolio team" → portfolio admins or agents?
- "Memory ingestion fails in batches" → browser JS or CLI tool?

**We need:** User group + client type tagged in every log, trace, and metric for surgical debugging.

---

## Enterprise Approaches

### Approach 1: OpenTelemetry Baggage + W3C Trace Context

**Pattern:** Inject user metadata into distributed trace from gateway, propagate via HTTP headers to all upstream services.

**Implementation:**
```go
// Gateway middleware extracts JWT claims → baggage
func UserBaggageMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Parse JWT (already done in auth middleware)
        claims := r.Context().Value("claims").(jwt.MapClaims)
        
        // Extract user info
        userID := claims["sub"].(string)
        groups := claims["groups"].([]interface{})
        
        // Create baggage (OTel standard)
        ctx := baggage.ContextWithValues(r.Context(),
            baggage.NewKeyValueProperty("user_id", userID),
            baggage.NewKeyValueProperty("user_group", groups[0].(string)),
            baggage.NewKeyValueProperty("client_id", r.Header.Get("X-Client-ID")),
            baggage.NewKeyValueProperty("sdk_version", r.Header.Get("User-Agent")),
        )
        
        // Baggage auto-propagates in outbound headers (traceparent, baggage)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}
```

**Logs:**
```json
{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "baggage": {
    "user_id": "rock",
    "user_group": "homelab-admins",
    "client_id": "portfolio-web-v1.2.3",
    "sdk_version": "LLM-SDK/2.0.1"
  },
  "message": "chat request routed to llm-serving",
  "status": "success"
}
```

**Metrics tags:**
```prometheus
llm_requests_total{
  user_group="homelab-admins",
  client_type="web",
  sdk_version="2.0.1",
  status="success"
}
```

**Pros:**
- ✅ **CNCF standard** (OTel baggage spec)
- ✅ Propagates automatically via W3C headers
- ✅ Works across Kubernetes, Temporal, Loki
- ✅ SDK integration (Python, JS, Go all support it)
- ✅ No extra plumbing needed

**Cons:**
- ❌ Baggage size limited (~8KB per W3C spec)
- ❌ Requires OTel SDK in all services (SDK cost)
- ❌ Not all upstream services parse baggage (vLLM, Ollama don't natively)

---

### Approach 2: Structured Logging with User Context Injection

**Pattern:** Gateway logs include user/client metadata; downstream services receive request ID + log to same correlator (Loki).

**Implementation:**
```go
// Gateway request logger
func StructuredRequestLogger(req *http.Request) {
    claims := req.Context().Value("claims").(jwt.MapClaims)
    
    log := map[string]interface{}{
        "request_id": req.Header.Get("X-Request-ID"),
        "timestamp": time.Now().UTC(),
        "user_id": claims["sub"],
        "user_group": claims["groups"].([]interface{})[0],
        "user_email": claims["email"],
        "org_id": claims["org_id"], // custom claim from Authentik
        
        // Client info
        "client_type": parseClientType(req.Header.Get("User-Agent")),
        "client_id": req.Header.Get("X-Client-ID"),
        "sdk_version": parseSDKVersion(req.Header.Get("User-Agent")),
        "client_ip": ClientIP(req),
        "client_country": GeoIP(req),
        
        // Request context
        "service": "gateway",
        "path": req.URL.Path,
        "method": req.Method,
        "model": req.Header.Get("X-Model"),
        "api_version": "v1",
        
        // Performance
        "start_time": time.Now(),
    }
    
    // Log as JSON to stdout → Loki
    jsonLog, _ := json.Marshal(log)
    fmt.Println(string(jsonLog))
}

func parseClientType(userAgent string) string {
    if strings.Contains(userAgent, "Chrome") { return "browser-chrome" }
    if strings.Contains(userAgent, "Safari") { return "browser-safari" }
    if strings.Contains(userAgent, "LLM-SDK/") { return "sdk-python" }
    if strings.Contains(userAgent, "node-llm/") { return "sdk-node" }
    if strings.Contains(userAgent, "curl") { return "cli-curl" }
    if strings.Contains(userAgent, "PostmanRuntime") { return "postman" }
    return "unknown"
}
```

**Loki query to find impacted group:**
```logql
{namespace="portfolio"} 
| json 
| user_group="portfolio-agents" 
| client_type="sdk-python" 
| status="error"
| stats count() by error_reason
```

**Pros:**
- ✅ **Simple to implement** (just structured logging)
- ✅ Works with existing Loki + Grafana
- ✅ No SDK overhead
- ✅ Full flexibility (any custom claim)
- ✅ Easy to debug (plain JSON)

**Cons:**
- ❌ Logs only; doesn't tag metrics automatically
- ❌ Needs Loki for log aggregation (additional infra)
- ❌ Can't correlate metrics and logs without additional work
- ❌ Log volume explodes (every request logged)

---

### Approach 3: HTTP Headers Passthrough + Service-Level Tagging

**Pattern:** Gateway adds custom headers (X-User-Group, X-Client-Type, X-Org-ID); every upstream service logs them.

**Implementation:**
```go
// Gateway outbound handler
func EnrichUpstreamRequest(upstream string, req *http.Request) *http.Request {
    claims := req.Context().Value("claims").(jwt.MapClaims)
    
    // Inject user metadata as headers
    req.Header.Set("X-User-ID", claims["sub"].(string))
    req.Header.Set("X-User-Group", claims["groups"].([]interface{})[0].(string))
    req.Header.Set("X-Org-ID", claims["org_id"].(string))
    req.Header.Set("X-Client-Type", parseClientType(req.Header.Get("User-Agent")))
    req.Header.Set("X-Request-ID", uuid.New().String()) // existing
    req.Header.Set("X-Trace-ID", req.Header.Get("X-Trace-ID")) // OpenTelemetry
    
    return req
}

// Upstream service (vLLM, Temporal, S3) middleware
func LogUserContextMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        log := map[string]interface{}{
            "request_id": r.Header.Get("X-Request-ID"),
            "trace_id": r.Header.Get("X-Trace-ID"),
            "user_id": r.Header.Get("X-User-ID"),
            "user_group": r.Header.Get("X-User-Group"),
            "org_id": r.Header.Get("X-Org-ID"),
            "client_type": r.Header.Get("X-Client-Type"),
            "service": "llm-serving",
            "endpoint": r.URL.Path,
        }
        
        // Log at start and end of request
        jsonLog, _ := json.Marshal(log)
        fmt.Println(string(jsonLog))
        
        next.ServeHTTP(w, r)
    })
}
```

**Prometheus labels (emit from upstream service):**
```go
llm_requests_total{
  user_group="homelab-admins",
  org_id="portfolio",
  client_type="browser-chrome",
  status="success"
}
```

**Pros:**
- ✅ **Works everywhere** (simple HTTP headers)
- ✅ No SDK/baggage overhead
- ✅ Metrics can be tagged per service
- ✅ Compatible with all upstream services (vLLM, Ollama, Temporal, etc.)
- ✅ Easy to add/debug headers

**Cons:**
- ❌ Manual propagation needed in each service
- ❌ Header limits (~8KB total)
- ❌ Not standard (custom X-* headers)
- ❌ Hard to enforce consistency across teams

---

### Approach 4: Synthetic User Journeys + Client Fingerprinting

**Pattern:** Portfolio + Poimen + CLI clients → identify as "portfolio-web-users", "poimen-cli-users"; track their entire journey through system.

**Implementation:**
```go
// Client registration (SDK or browser)
type ClientMetadata struct {
    ClientID      string // generated once, persisted locally
    ClientType    string // "portfolio-web", "poimen-cli", "memory-api-client"
    ClientVersion string
    UserGroup     string // from JWT
    Org           string
}

// Gateway: extract or generate client fingerprint
func GetClientFingerprint(req *http.Request) string {
    clientID := req.Header.Get("X-Client-ID") // SDK provides
    if clientID == "" {
        // Browser: fingerprint from User-Agent + IP + TLS
        clientID = Hash(
            req.Header.Get("User-Agent"),
            ClientIP(req),
            req.TLS.PeerCertificates[0].SerialNumber,
        )
    }
    return clientID
}

// Tag all downstream operations with client fingerprint
func RecordClientJourney(clientID, userGroup, clientType string, operation string, status string) {
    // Prometheus
    clientJourneyDuration.WithLabelValues(clientID, userGroup, clientType, operation, status).Observe(duration)
    
    // Loki (full request chain)
    log := map[string]interface{}{
        "client_fingerprint": clientID,
        "user_group": userGroup,
        "client_type": clientType,
        "operation": operation,
        "status": status,
        "timestamp": time.Now(),
    }
    LogToLoki(log)
}
```

**Query to find impacted users:**
```promql
# Alert: "Chat API failures for portfolio-web-users"
(
  rate(llm_requests_total{client_type="portfolio-web", status="error"}[5m]) > 0.1
)
```

**Grafana dashboard:**
- Heatmap: client_type × user_group vs latency
- Time series: error rate by client_fingerprint
- Table: top 10 impacted client IPs/fingerprints

**Pros:**
- ✅ **Surgical root cause** ("portfolio-cli-v2.1 broken for homelab-admins")
- ✅ Can replay exact user journey from logs
- ✅ Cohort-based alerting ("if 5% of portfolio-web fail, alert")
- ✅ User behavior analysis (which clients call what)

**Cons:**
- ❌ Privacy concerns (fingerprinting users)
- ❌ Complex state management (client IDs, persistent storage)
- ❌ High cardinality (can explode metrics cardinality if not capped)
- ❌ Requires SDK integration

---

### Approach 5: Event Sourcing + User Action Audit Log

**Pattern:** Every operation → immutable audit log entry with user, client, action, outcome. Query for "all portfolio-web users who failed chat between 2-3pm".

**Implementation:**
```go
type AuditEvent struct {
    EventID       string    // unique per operation
    Timestamp     time.Time
    RequestID     string    // correlation ID
    
    // Who
    UserID        string
    UserGroup     string
    Email         string
    
    // What
    Action        string // "chat_completion", "sqs_send", "memory_ingest"
    Service       string // "llm-serving", "sqs", "memory"
    ResourceID    string // model name, queue name, etc.
    
    // How
    ClientType    string // "browser-chrome", "sdk-python", "cli-curl"
    ClientID      string
    ClientVersion string
    ClientIP      string
    
    // Result
    Status        string // "success", "error", "timeout"
    ErrorCode     string
    ErrorMessage  string
    Duration      int64 // milliseconds
    
    // Context
    OrgID         string
    ProjectID     string
    Metadata      map[string]interface{}
}

// Emit audit event (to Postgres/Clickhouse)
func EmitAuditEvent(ctx context.Context, event AuditEvent) {
    db.InsertAuditEvent(event)
    // Also send to Loki for real-time analysis
    LogToLoki(event)
}

// SQL query: "Find all users in portfolio-agents impacted by chat failure"
// SELECT DISTINCT user_id, email FROM audit_events 
// WHERE user_group='portfolio-agents' 
//   AND action='chat_completion' 
//   AND status='error' 
//   AND timestamp > NOW() - INTERVAL '1 hour'
//   AND client_type IN ('browser-chrome', 'sdk-python')
```

**Grafana query (using Loki or Clickhouse):**
```
SELECT 
  user_group,
  client_type,
  COUNT(*) as total_requests,
  SUM(CASE WHEN status='error' THEN 1 ELSE 0 END) as failures,
  SUM(CASE WHEN status='error' THEN 1 ELSE 0 END) * 100.0 / COUNT(*) as error_rate
FROM audit_events
WHERE timestamp > NOW() - INTERVAL '1 hour'
GROUP BY user_group, client_type
ORDER BY error_rate DESC
```

**Pros:**
- ✅ **Immutable audit trail** (compliance, forensics)
- ✅ Can answer "Who was affected and how?" precisely
- ✅ Excellent for root cause (filter by user_group + timestamp + client_type)
- ✅ Supports RBAC/compliance reporting
- ✅ Works with both logs and metrics

**Cons:**
- ❌ **Storage heavy** (every operation stored)
- ❌ Requires additional data store (Postgres/Clickhouse)
- ❌ Latency for complex queries (especially on large datasets)
- ❌ Schema evolution complexity

---

## Recommended Hybrid Approach for Your Setup

**Combine Approaches 1 + 3 + 5:**

```
┌─────────────────────────────────────────────────────────────┐
│ Gateway (homelab-frontend)                                  │
│                                                             │
│ 1. Parse JWT → extract user_id, user_group, org_id        │
│ 2. Add OTel baggage (Approach 1)                           │
│ 3. Add X-* headers (Approach 3)                            │
│ 4. Emit audit event to Loki + Postgres (Approach 5)       │
│                                                             │
│ Headers sent upstream:                                      │
│   X-Request-ID (existing)                                  │
│   X-Trace-ID (OTel)                                        │
│   traceparent, baggage (OTel)                              │
│   X-User-ID, X-User-Group, X-Org-ID                        │
│   X-Client-Type, X-Client-ID                               │
└─────────────────────────────────────────────────────────────┘
         │
         ├──→ llm-serving (vLLM)
         │    - Log headers
         │    - Emit metrics {user_group, client_type}
         │    - Propagate X-Request-ID
         │
         ├──→ sqs (Temporal)
         │    - Log headers
         │    - Emit metrics {user_group, org_id}
         │
         └──→ memory (Poimen)
              - Log headers
              - Emit metrics {user_group, client_type}
              
         └──→ Loki (structured logs)
         └──→ Prometheus (metrics with user/client tags)
         └──→ Postgres/Clickhouse (audit events)
```

**Workflow:**
1. User calls gateway with Bearer token + User-Agent
2. Gateway parses JWT → creates OTel baggage + X-* headers
3. Gateway logs audit event (user_id, user_group, client_type, action, status)
4. Headers propagated to upstream services
5. Root cause query:
   - **Metrics:** `llm_requests_total{user_group="portfolio-agents", client_type="sdk-python", status="error"}`
   - **Logs (Loki):** `{namespace="portfolio"} | json | user_group="portfolio-agents" | client_type="sdk-python" | status="error"`
   - **Audit trail (Postgres):** `SELECT * FROM audit_events WHERE user_group='portfolio-agents' AND client_type='sdk-python' AND timestamp>'2026-09-04'`

---

## Implementation Roadmap

| Phase | What | Where | Effort |
|-------|------|-------|--------|
| 1 | JWT claims extraction | `internal/auth/jwt.go` | 1 day |
| 2 | OTel baggage middleware | `internal/middleware/baggage.go` | 1 day |
| 3 | X-* header propagation | `internal/proxy/proxy.go` | 0.5 day |
| 4 | Structured logging (Loki) | `internal/logging/structured.go` | 1 day |
| 5 | Audit events (Postgres) | `internal/audit/events.go` + schema | 2 days |
| 6 | Prometheus labels per service | `internal/observability/*.go` | 1 day |
| 7 | Grafana dashboards | `k8s/grafana-dashboards/` | 1 day |

---

## Which Approach for Your Use Case?

**User Group Identification:**
- Use **OTel Baggage** (Approach 1) — auto-propagates
- Fallback to **X-* headers** (Approach 3) if OTel not available

**Client Type Detection:**
- Parse User-Agent in gateway
- Tag as: `browser-chrome`, `browser-safari`, `sdk-python`, `sdk-node`, `cli-curl`, `postman`, `unknown`

**Root Cause (Surgical Debugging):**
- Use **Audit Events** (Approach 5) with Postgres
- Query: "Who from portfolio-agents using sdk-python failed in last hour?"

**High-Cardinality Metrics:**
- Avoid tagging by individual user_id (cardinality explosion)
- Use: user_group, org_id, client_type, sdk_version (bounded)

**Real-Time Alerting:**
- Prometheus metrics with `{user_group, client_type, status}` labels
- Alert: "error_rate > 5% for user_group='portfolio-agents' for 5min"