feat: complete observability stack (O1-O13) (#52)
CI / CI (push) Successful in 12m36s
Deploy / Tag & Push Latest (push) Successful in 1m56s

## Complete Observability Stack (O1-O13)

Implements all 13 observability issues in a single PR. 119 metrics total.

### Commits (one per issue)

| Issue | Title | Metrics |
|-------|-------|---------|
| **O10** | Prometheus metrics module + /metrics endpoint | Foundation |
| **O1** | Instrument ingest handler | I1-I12 (12) |
| **O2** | Instrument query handler | Q1-Q12 (12) |
| **O3** | Instrument context endpoint | C1-C8 (8) |
| **O4** | Relevance judge | R1-R9 (9) |
| **O5** | Write volume + storage metrics | W1-W12 (12) |
| **O6** | Pod resource observability | P1-P13 |
| **O7** | Availability + dependency health | A1-A10 (10) |
| **O8** | Ingest rate pattern tracking | IR1-IR10 (10) |
| **O9** | Postgres internal observability | PG1-PG33 |
| **O11** | Grafana dashboard | 12 panels |
| **O12** | Prometheus alerting rules | 11 alerts |
| **O13** | Relevance evaluation CronJob | K8s manifest |

### Key Changes

- **metrics.rs**: Zero-dependency Prometheus metrics (Counter, Gauge, Histogram, Timer)
- **GET /metrics**: Prometheus text exposition format endpoint
- **Ingest/Query/Context handlers**: Instrumented with latency, errors, auth failures
- **Health check**: DB dependency check with latency tracking
- **Background task**: Periodic DB stats collection (entity/edge counts, pool stats)
- **Relevance judge**: Threshold-based eval with precision/recall/F1 tracking
- **Grafana dashboard**: 12 panels covering all metric groups
- **Alert rules**: 11 PrometheusRule alerts (availability, latency, errors, quality)
- **CronJob**: Periodic relevance evaluation with sample queries

### Testing

- 506 tests passing (0 failures)
- All metrics modules have unit tests
- Relevance judge: 4 tests

### Deploy

```bash
# Grafana dashboard
kubectl apply -f k8s/infra/grafana-dashboard.json

# Prometheus alerts
kubectl apply -f k8s/infra/prometheus-alerts.yaml

# Relevance eval CronJob
kubectl apply -f k8s/infra/relevance-eval-cronjob.yaml
```

Closes #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39

---------

Co-authored-by: rock <[email protected]>
Reviewed-on: #52
Co-authored-by: poimen <[email protected]>
This commit was merged in pull request #52.
This commit is contained in:
2026-09-13 13:53:50 +00:00
committed by rock
co-authored by rock
parent d7a36ce9e8
commit 4169effd8a
14 changed files with 1873 additions and 13 deletions
+50
View File
@@ -0,0 +1,50 @@
# Local development environment (.env file)
# Copy to .env and fill in your local/dev URLs
# .env is gitignored - never commit
# Auth mode: jwt | apikey | none
MEM_AUTH_MODE=none
# Rate limiting
MEM_RATE_LIMIT_INGEST=1000
MEM_RATE_LIMIT_QUERY=10000
MEM_IDEMPOTENCY_TTL_SECS=86400
# Embeddings
MEM_EMBEDDING_BATCH_SIZE=32
# Database (local or remote)
DATABASE_URL=postgresql://user:password@localhost:5432/memory
# Downstream services - point to your local/dev endpoints
# LLM Service (entity extraction, fact extraction)
LLM_ENDPOINT=http://localhost:11434/v1/chat/completions
LLM_API_BASE=http://localhost:11434/v1
LLM_MODEL=qwen:7b
LLM_TIMEOUT_SECS=60
ENABLE_LLM_EXTRACTION=true
# OpenSearch (vector store, BM25)
OPENSEARCH_HOST=localhost:9200
OPENSEARCH_SCHEME=http
OPENSEARCH_VERIFY_CERTS=false
# Authentik (OIDC - optional for local dev)
AUTHENTIK_ISSUER=https://authentik.riotpiao.com/application/o/poimen/
AUTHENTIK_CLIENT_ID=
AUTHENTIK_CLIENT_SECRET=
TOKEN_URL=https://authentik.riotpiao.com/application/o/token/
AUTHENTIK_VERIFY_SSL=false
# Temporal (workflow orchestration - future)
TEMPORAL_ENDPOINT=localhost:7233
TEMPORAL_NAMESPACE=poimen
# API Gateway (route optimization - future)
GATEWAY_URL=http://localhost:8080
# Server config
MEM_PORT=8080
MEM_API_KEY=test-key
MEM_HOME=/tmp