O12: Prometheus alerting rules for all SLOs #38

New Issue

2026-09-08T05:02:32Z

rock commented

2026-09-08 05:02:32 +00:00

Create PrometheusRule CRD with all alerting rules from observability doc.

20 SLOs to alert on:

Critical:

Ingest pgvector p99 > 1s
Query p99 > 1s
Availability < 99.9% over 1h
5xx rate > 10/min for 5min
Postgres down (dependency_up = 0)
Pod memory pressure > 85% for 5min
Pod restart
PG cache hit ratio < 95%
PG seq scan on memory_vector
PG connection ratio > 80%
Ingest queue depth > 1000 for 15min

Warning:

NDCG@10 7-day avg < 0.80
NDCG@10 drops > 0.05 in 24h
Query empty rate > 20%
Tier-1 hit rate < 60%
Review queue depth > 100 for 24h
OpenSearch write failures > 5/min
CPU throttling sustained 10min
PG dead tuple ratio > 20%
PG replication lag > 30s
Ingest rate = 0 for 30min

Deploy to: k8s/monitoring/prometheus-rules-memory.yaml

Reference: docs/OBSERVABILITY.md Section 14 (SLOs)

Create PrometheusRule CRD with all alerting rules from observability doc. 20 SLOs to alert on: Critical: - Ingest pgvector p99 > 1s - Query p99 > 1s - Availability < 99.9% over 1h - 5xx rate > 10/min for 5min - Postgres down (dependency_up = 0) - Pod memory pressure > 85% for 5min - Pod restart - PG cache hit ratio < 95% - PG seq scan on memory_vector - PG connection ratio > 80% - Ingest queue depth > 1000 for 15min Warning: - NDCG@10 7-day avg < 0.80 - NDCG@10 drops > 0.05 in 24h - Query empty rate > 20% - Tier-1 hit rate < 60% - Review queue depth > 100 for 24h - OpenSearch write failures > 5/min - CPU throttling sustained 10min - PG dead tuple ratio > 20% - PG replication lag > 30s - Ingest rate = 0 for 30min Deploy to: k8s/monitoring/prometheus-rules-memory.yaml Reference: docs/OBSERVABILITY.md Section 14 (SLOs)

rock added the memory-observability label 2026-09-08 05:02:32 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/poimen-memory#38