poimen-workflows/tasks/T1.2.md

T1.2: Structured Logging + Prometheus Metrics

Submilestone: T1 (Production Hardening) Status: ✅ COMPLETE
Branch: task/T1.2

Overview

Implement structured JSON logging with zap and comprehensive Prometheus metrics export for observability.

Requirements

Structured Logging

• Replace all log.Printf / log.Fatalf with structured logging
• Use go.uber.org/zap for structured JSON logging
• Support both development (colored) and production (JSON) modes
• Easy field attachment: logging.Info("message", logging.String("key", "value"))

Prometheus Metrics

• 16 comprehensive metrics covering workflows, activities, LLM calls, git operations, judge decisions
• Counter metrics: workflow starts/completions, activity starts/completions, retries, LLM calls, git operations, judge decisions
• Histogram metrics: workflow duration, activity duration, LLM latency, git operation duration
• Gauge metrics: tasks in progress
• Error tracking: Temporal connection errors, cache hit/miss ratio
• Metrics exported on /metrics HTTP endpoint (Prometheus format)

Integration

• Health check server (port 8081) now serves both /health* and /metrics
• Graceful logging shutdown with logging.Sync()
• Both worker and starter commands use structured logging

Implementation

Internal Package: internal/logging

logger.go

• InitLogger() - Initialize global logger (dev or prod mode)
• GetLogger() - Get logger instance
• Info(), Error(), Warn(), Debug(), Fatal() - Log functions
• Field helpers: String(), Int(), Int64(), Err()
• Sync() - Flush buffered logs
• With() - Create logger with additional fields
• 8/8 unit tests passing ✅

logger_test.go

• Tests for logger initialization, field creation, logging functions
• Verifies no panics on concurrent logging

Internal Package: internal/metrics

metrics.go

• 16 pre-registered Prometheus metrics
• Helper functions for recording each metric type
• Metrics organized by concern: workflows, activities, LLM, git, judge, temporal, cache
• 13/13 unit tests passing ✅

metrics_test.go

• Tests that all metrics are registered
• Tests that recording functions don't panic
• Verifies metric registration

Integration Points

cmd/worker/main.go

• Initializes logger on startup
• Uses logging.Info(), logging.Fatal(), logging.Warn() throughout
• Health server serves /metrics endpoint
• Structured shutdown logging

cmd/starter/main.go

• Initializes logger on startup
• Logs configuration load, Temporal connection, workflow start
• Supports --health command with structured logging
• Clean shutdown with logging.Sync()

internal/health/handler.go

• Prometheus handler integrated via promhttp.Handler()
• /metrics endpoint available on all deployments

Verification Criteria

✅ All criteria met:

1. Structured logging deployed

   • All log statements use structured fields
   • JSON output in production
   • Colored output in development

2. Prometheus metrics exposed

   • 16 comprehensive metrics registered
   • /metrics endpoint returns Prometheus text format
   • Metrics include latencies, counters, and gauges

3. All metrics functional

   • WorkflowExecutionsStarted - workflow launch tracking
   • WorkflowExecutionsCompleted - workflow completion with status
   • ActivityExecutionsStarted/Completed/Duration - activity lifecycle
   • ActivityRetries - retry tracking
   • LLMAPICallsTotal / LLMAPILatency - LLM performance
   • GitOperationsTotal / GitOperationsDuration - git operation tracking
   • TasksInProgress - real-time task load
   • JudgeDecisionsTotal - decision tracking
   • TemporalConnectionErrors - error tracking
   • CacheHits / CacheMisses - cache efficiency

4. Integration complete

   • Worker uses structured logging throughout
   • Starter uses structured logging throughout
   • Both commands can use --health to check system status
   • Graceful shutdown flushes logs

5. Test coverage

   • 8/8 logging tests passing
   • 13/13 metrics tests passing
   • All unit tests pass
   • No panics on concurrent logging

Testing

# Unit tests
go test -v ./internal/logging ./internal/metrics
# Result: PASS (21/21 tests)

# Full test suite
go test -v ./...
# Result: All tests pass

# Integration test (requires running worker)
curl http://localhost:8081/metrics
# Returns: Prometheus metrics in text format

# Logging output
ENVIRONMENT=development go run ./cmd/worker
# Output: Colored JSON logs with structured fields

ENVIRONMENT=production go run ./cmd/worker
# Output: JSON logs suitable for Loki/ELK

Kubernetes Configuration

Example logging in pods:

env:
- name: ENVIRONMENT
  value: "production"

Example Prometheus scrape config:

scrape_configs:
- job_name: 'poimen-worker'
  static_configs:
  - targets: ['localhost:8081']
  metrics_path: '/metrics'

Metrics Schema

All metrics prefixed with poimen_:

Workflow Metrics

• poimen_workflow_executions_started_total{workflow_type} - Counter
• poimen_workflow_executions_completed_total{workflow_type, status} - Counter
• poimen_workflow_duration_seconds{workflow_type} - Histogram

Activity Metrics

• poimen_activity_executions_started_total{activity_type} - Counter
• poimen_activity_executions_completed_total{activity_type, status} - Counter
• poimen_activity_duration_seconds{activity_type} - Histogram
• poimen_activity_retries_total{activity_type} - Counter

LLM Metrics

• poimen_llm_api_calls_total{model_id, status} - Counter
• poimen_llm_api_latency_seconds{model_id} - Histogram

Git Metrics

• poimen_git_operations_total{operation, status} - Counter
• poimen_git_operations_duration_seconds{operation} - Histogram

Other Metrics

• poimen_tasks_in_progress{task_type} - Gauge
• poimen_judge_decisions_total{decision} - Counter
• poimen_temporal_connection_errors_total{error_type} - Counter
• poimen_cache_hits_total{cache_type} - Counter
• poimen_cache_misses_total{cache_type} - Counter

Files Changed

• ✅ internal/logging/logger.go - Structured logger (71 lines)
• ✅ internal/logging/logger_test.go - Logger tests (70 lines)
• ✅ internal/metrics/metrics.go - Prometheus metrics (222 lines)
• ✅ internal/metrics/metrics_test.go - Metrics tests (87 lines)
• ✅ internal/health/handler.go - Added /metrics endpoint
• ✅ cmd/worker/main.go - Structured logging integration
• ✅ cmd/starter/main.go - Structured logging integration
• ✅ go.mod - Added zap, prometheus/client_golang dependencies
• ✅ tasks/board-T1.md - Task board update

Dependencies Added

• go.uber.org/zap v1.28.0 - Structured logging
• github.com/prometheus/client_golang v1.24.1 - Prometheus metrics
• Plus 8 transitive dependencies for Prometheus support

Next Steps (T1.1 → T1.3 → T1.4)

1. T1.1: Workflow error recovery & deadletter handling
2. T1.3: Timeout tuning automation based on historical failures
3. T1.4: Board state validation & auto-heal from corruption

Notes

• Logger uses global singleton pattern for simplicity (can be refactored to DI if needed)
• Metrics are auto-registered via promauto (thread-safe, idempotent)
• /metrics endpoint serves standard Prometheus text format (compatible with all scraping systems)
• Logging mode controlled by ENVIRONMENT env var (default: development)
• All metric labels are strings (Prometheus requirement)
• Histograms use default buckets (10ms, 100ms, 1s, 10s, etc.)