[Event-Bus P4] Dead letter queue + Prometheus metrics + Grafana dashboard #26

New Issue

2026-09-09T23:43:13Z

poimen commented

2026-09-09 23:43:13 +00:00

What

Observability for the event pipeline: failed dispatches go to a DLQ, metrics exposed for Prometheus, Grafana dashboard.

Depends On

#21 Event router
#23 Target dispatchers

Acceptance

Dead Letter Queue

Failed dispatch after all retries → event sent to SQS queue events-dlq
DLQ message contains: original event, target name, error message, retry count, timestamp
DLQ message inspectable: curl sqs-mgmt/queue/events-dlq/messages returns failed events
Manual replay: take DLQ message, POST to /v1/events → re-routed through pipeline

Prometheus Metrics

gateway_events_received_total{source, type} — incremented on every POST /v1/events
gateway_events_dispatched_total{source, type, target, status} — incremented per target dispatch (status: success/failure)
gateway_events_dispatch_duration_seconds{target} — histogram of dispatch latency per target
gateway_events_dlq_total{source, type, target} — incremented when event sent to DLQ
curl api.riotpiao.com/metrics | grep gateway_events returns all 4 metric families

Grafana Dashboard

Dashboard deployed via k8s/infra/monitoring/dashboards/event-bus.yaml
Panel: event throughput by source (stacked bar, 5m rate)
Panel: dispatch success/failure rate by target (pie chart)
Panel: DLQ depth over time (line graph)
Panel: P50/P95/P99 dispatch latency by target (heatmap)
Dashboard accessible at grafana.riotpiao.com under Event Bus folder

Tests

go test ./internal/events/ — dispatch failure → DLQ message written with correct schema
go test ./internal/events/ — metrics incremented correctly on success and failure paths

## What Observability for the event pipeline: failed dispatches go to a DLQ, metrics exposed for Prometheus, Grafana dashboard. ## Depends On - #21 Event router - #23 Target dispatchers ## Acceptance ### Dead Letter Queue - [ ] Failed dispatch after all retries → event sent to SQS queue `events-dlq` - [ ] DLQ message contains: original event, target name, error message, retry count, timestamp - [ ] DLQ message inspectable: `curl sqs-mgmt/queue/events-dlq/messages` returns failed events - [ ] Manual replay: take DLQ message, POST to `/v1/events` → re-routed through pipeline ### Prometheus Metrics - [ ] `gateway_events_received_total{source, type}` — incremented on every POST /v1/events - [ ] `gateway_events_dispatched_total{source, type, target, status}` — incremented per target dispatch (status: success/failure) - [ ] `gateway_events_dispatch_duration_seconds{target}` — histogram of dispatch latency per target - [ ] `gateway_events_dlq_total{source, type, target}` — incremented when event sent to DLQ - [ ] `curl api.riotpiao.com/metrics | grep gateway_events` returns all 4 metric families ### Grafana Dashboard - [ ] Dashboard deployed via `k8s/infra/monitoring/dashboards/event-bus.yaml` - [ ] Panel: event throughput by source (stacked bar, 5m rate) - [ ] Panel: dispatch success/failure rate by target (pie chart) - [ ] Panel: DLQ depth over time (line graph) - [ ] Panel: P50/P95/P99 dispatch latency by target (heatmap) - [ ] Dashboard accessible at `grafana.riotpiao.com` under Event Bus folder ### Tests - [ ] `go test ./internal/events/` — dispatch failure → DLQ message written with correct schema - [ ] `go test ./internal/events/` — metrics incremented correctly on success and failure paths

poimen added this to the Event-Driven Platform milestone 2026-09-09 23:43:13 +00:00

poimen added the type/infra area/event-bus area/gateway priority/medium status/todo labels 2026-09-09 23:43:15 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#26