feat: complete observability stack (O1-O13) (#52)

## Complete Observability Stack (O1-O13)

Implements all 13 observability issues in a single PR. 119 metrics total.

### Commits (one per issue)

| Issue | Title | Metrics |
|-------|-------|---------|
| **O10** | Prometheus metrics module + /metrics endpoint | Foundation |
| **O1** | Instrument ingest handler | I1-I12 (12) |
| **O2** | Instrument query handler | Q1-Q12 (12) |
| **O3** | Instrument context endpoint | C1-C8 (8) |
| **O4** | Relevance judge | R1-R9 (9) |
| **O5** | Write volume + storage metrics | W1-W12 (12) |
| **O6** | Pod resource observability | P1-P13 |
| **O7** | Availability + dependency health | A1-A10 (10) |
| **O8** | Ingest rate pattern tracking | IR1-IR10 (10) |
| **O9** | Postgres internal observability | PG1-PG33 |
| **O11** | Grafana dashboard | 12 panels |
| **O12** | Prometheus alerting rules | 11 alerts |
| **O13** | Relevance evaluation CronJob | K8s manifest |

### Key Changes

- **metrics.rs**: Zero-dependency Prometheus metrics (Counter, Gauge, Histogram, Timer)
- **GET /metrics**: Prometheus text exposition format endpoint
- **Ingest/Query/Context handlers**: Instrumented with latency, errors, auth failures
- **Health check**: DB dependency check with latency tracking
- **Background task**: Periodic DB stats collection (entity/edge counts, pool stats)
- **Relevance judge**: Threshold-based eval with precision/recall/F1 tracking
- **Grafana dashboard**: 12 panels covering all metric groups
- **Alert rules**: 11 PrometheusRule alerts (availability, latency, errors, quality)
- **CronJob**: Periodic relevance evaluation with sample queries

### Testing

- 506 tests passing (0 failures)
- All metrics modules have unit tests
- Relevance judge: 4 tests

### Deploy

```bash
# Grafana dashboard
kubectl apply -f k8s/infra/grafana-dashboard.json

# Prometheus alerts
kubectl apply -f k8s/infra/prometheus-alerts.yaml

# Relevance eval CronJob
kubectl apply -f k8s/infra/relevance-eval-cronjob.yaml
```

Closes #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39

---------

Co-authored-by: rock <[email protected]>
Reviewed-on: #52
Co-authored-by: poimen <[email protected]>

k8s/infra/prometheus-alerts.yaml

@@ -0,0 +1,130 @@
+# Prometheus alerting rules for Poimen Memory (O12)
+# Deploy: kubectl apply -f k8s/infra/prometheus-alerts.yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: poimen-memory-alerts
+  namespace: poimen
+  labels:
+    app: poimen-memory
+    prometheus: k8s
+    role: alert-rules
+spec:
+  groups:
+    - name: poimen-memory.availability
+      rules:
+        - alert: MemoryServiceDown
+          expr: up{job="poimen-memory"} == 0
+          for: 2m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Poimen memory service is down"
+            description: "Memory service has been unreachable for > 2 minutes"
+
+        - alert: MemoryDBDown
+          expr: memory_dependency_db_up == 0
+          for: 1m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Memory service cannot reach database"
+            description: "DB dependency health check failing for > 1 minute"
+
+        - alert: MemoryEmbeddingDown
+          expr: memory_dependency_embedding_up == 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Embedding service unreachable"
+            description: "Embedding dependency health check failing for > 5 minutes"
+
+    - name: poimen-memory.latency
+      rules:
+        - alert: MemoryIngestLatencyHigh
+          expr: histogram_quantile(0.95, rate(memory_ingest_duration_seconds_bucket[5m])) > 5
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Ingest p95 latency > 5s"
+            description: "95th percentile ingest latency is {{ $value }}s"
+
+        - alert: MemoryQueryLatencyHigh
+          expr: histogram_quantile(0.95, rate(memory_query_duration_seconds_bucket[5m])) > 2
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Query p95 latency > 2s"
+            description: "95th percentile query latency is {{ $value }}s"
+
+        - alert: MemoryEmbeddingLatencyHigh
+          expr: histogram_quantile(0.95, rate(memory_query_embedding_duration_seconds_bucket[5m])) > 10
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Embedding p95 latency > 10s"
+            description: "95th percentile embedding call latency is {{ $value }}s"
+
+    - name: poimen-memory.errors
+      rules:
+        - alert: MemoryIngestErrorRateHigh
+          expr: rate(memory_ingest_errors_total[5m]) / rate(memory_ingest_requests_total[5m]) > 0.1
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Ingest error rate > 10%"
+            description: "{{ $value | humanizePercentage }} of ingest requests are failing"
+
+        - alert: MemoryQueryErrorRateHigh
+          expr: rate(memory_query_errors_total[5m]) / rate(memory_query_requests_total[5m]) > 0.1
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Query error rate > 10%"
+            description: "{{ $value | humanizePercentage }} of query requests are failing"
+
+        - alert: MemoryEmbeddingFailureRate
+          expr: rate(memory_query_embedding_failures_total[5m]) > 0.5
+          for: 3m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Embedding failures > 0.5/s"
+            description: "Embedding service failing at {{ $value }}/s — queries cannot embed"
+
+    - name: poimen-memory.storage
+      rules:
+        - alert: MemoryDBPoolExhausted
+          expr: memory_db_pool_idle == 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "DB connection pool exhausted"
+            description: "No idle DB connections for > 5 minutes"
+
+        - alert: MemoryWriteErrorsHigh
+          expr: rate(memory_write_errors_total[5m]) > 1
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Write errors > 1/s"
+            description: "Database write errors at {{ $value }}/s"
+
+    - name: poimen-memory.quality
+      rules:
+        - alert: MemoryRelevanceLow
+          expr: memory_relevance_precision < 0.3
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Retrieval relevance precision < 30%"
+            description: "Relevance precision is {{ $value | humanizePercentage }}"