riotpiao-poimen/poimen-memory
2
0
Fork 0
Code Issues 11 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: complete observability stack (O1-O13) #52

Merged
rock merged 18 commits from feat/observability-o1-o13 into main 2026-09-13 13:53:54 +00:00
Conversation 0 Commits 18 Files Changed 14
+1873 -13
18 Commits
Author SHA1 Message Date
rock 57c61b522b feat: unexpected error counter + user_id in error logs
CI / CI (pull_request) Successful in 12m31s
Details 
- ERROR_UNEXPECTED_TOTAL: global unexpected error counter
- ERROR_UNEXPECTED_INGEST/QUERY/CONTEXT: per-endpoint unexpected errors
- All 500 error paths now increment unexpected counter
- Error logs include user_id for customer association:
  tracing::error!(user_id = claims.sub, "Unexpected error: ...")
- Covers: DB errors, search failures, temporal query failures
- 515 tests passing
 2026-09-13 22:17:06 +09:00
rock f5c6bf5e5d fix: replace labeled counters with named error counters
CI / CI (pull_request) Successful in 12m5s
Details 
- Remove ERRORS_BY_CLASS, ERRORS_BY_USER, REQUESTS_BY_USER (overengineered)
- Add simple named counters per error type per endpoint:
  memory_error_auth_failure_ingest_total
  memory_error_forbidden_ingest_total
  memory_error_rate_limited_ingest_total
  memory_error_bad_request_ingest_total
  memory_error_db_error_ingest_total
  memory_error_auth_failure_query_total
  memory_error_forbidden_query_total
  memory_error_bad_request_query_total
  memory_error_embedding_failure_query_total
  memory_error_search_failure_query_total
  memory_error_auth_failure_context_total
  memory_error_forbidden_context_total
  memory_error_lookup_failure_context_total
- LAST_ERROR_TIMESTAMP gauge for most recent error
- 515 tests passing
 2026-09-13 22:12:46 +09:00
rock 3e7344787e feat: user identity + error name tracking in metrics
CI / CI (pull_request) Successful in 12m36s
Details 
- ERRORS_BY_USER: labeled counter {user_id, endpoint, error_name}
- REQUESTS_BY_USER: labeled counter {user_id, endpoint}
- extract_user_id(): decode JWT sub claim from Authorization header
- Error names: auth_failure, forbidden, rate_limited, bad_request, embedding_failure
- Ingest handler: tracks user_id from claims.sub
- Query handler: tracks user_id from JWT decode
- Context handler: tracks user_id from claims.sub
- render_labeled_counter(): generic Prometheus label renderer
- User identity from gateway JWT (claims.sub per API.md)
- 515 tests passing
 2026-09-13 22:02:56 +09:00
rock 49dcf2616c feat: metrics snapshot test harness for scenario verification
CI / CI (pull_request) Successful in 12m4s
Details 
- MetricsSnapshot::capture() snapshots all metric values
- assert_counter_inc(): verify counter delta after scenario
- assert_gauge_eq(): verify gauge value
- assert_histogram_count_inc(): verify histogram observations
- assert_gauge_f64_approx(): verify f64 gauges with tolerance
- print_deltas(): debug helper for all changed metrics
- 9 scenario tests: ingest, query error, relevance batch, write
- Histogram fields made pub for snapshot access
- 515 total tests passing
 2026-09-13 21:54:26 +09:00
rock 9e85b1a063 feat(O13): relevance evaluation CronJob manifest
CI / CI (pull_request) Successful in 12m14s
Details 
- Runs every 6 hours with sample queries
- Tests query endpoint with known queries
- Reports precision (queries with results / total)
- Snapshots /metrics endpoint for monitoring
- Lightweight: curl-based, 16Mi memory
- Deploy: kubectl apply -f k8s/infra/relevance-eval-cronjob.yaml
 2026-09-13 21:50:21 +09:00
rock f304a7133e feat(O12): Prometheus alerting rules for all SLOs 
- 11 alert rules across 5 groups
- Availability: service down, DB down, embedding down
- Latency: ingest p95 > 5s, query p95 > 2s, embedding p95 > 10s
- Errors: ingest/query error rate > 10%, embedding failures
- Storage: pool exhausted, write errors
- Quality: relevance precision < 30%
- PrometheusRule CRD for kube-prometheus-stack
 2026-09-13 21:50:00 +09:00
rock ef31ef70a5 feat(O11): Grafana dashboard for memory-observability 
- 12 panels: rates, latency, errors, embedding, DB, health, relevance
- Covers all O1-O9 metrics in visual form
- Import via Grafana UI or provisioning
- Dashboard UID: poimen-memory-obs
 2026-09-13 21:49:30 +09:00
rock 12a97a59fa feat(O9): Postgres internal observability 
- DB pool size/idle/active gauges updated every 60s (background task)
- DB query total/errors counters defined
- DB query/transaction duration histograms defined
- Table row count gauges (entity, edge, chunk) updated periodically
- Deep PG stats (pg_stat_*, pg_statio_*) collected by pg_exporter
- Metrics: PG1-PG33 (app-visible subset, rest from pg_exporter)
 2026-09-13 21:49:03 +09:00
rock 30d78d0c9f feat(O8): ingest rate pattern tracking 
- INGEST_RATE_1M/5M gauges defined (computed by Prometheus rate())
- LLM extract duration histogram defined
- Fact extract duration histogram defined
- Dedup and contradiction counters defined
- Active projects gauge defined
- Rate patterns derived from INGEST_REQUESTS_TOTAL via PromQL
- Metrics: IR1-IR10 (10 metrics defined, computed by Prometheus)
 2026-09-13 21:48:56 +09:00
rock 081e93f602 feat(O7): availability metrics and dependency health checks 
- Health endpoint now checks DB connectivity
- Track health check total/failures
- DEP_DB_UP gauge (1=up, 0=down) + latency histogram
- APP_UPTIME_SECONDS updated on each health check
- Metrics: A1-A10 (10 metrics instrumented)
 2026-09-13 21:48:49 +09:00
rock 5dc2edf3ac feat(O6): pod resource observability (app-level metrics) 
- APP_UPTIME_SECONDS, APP_ACTIVE_CONNECTIONS, APP_HEAP_BYTES gauges defined
- Pod-level CPU/memory collected by cAdvisor/node-exporter (external)
- Metrics: P1-P13 (app-level subset, rest from K8s monitoring)
 2026-09-13 21:48:21 +09:00
rock 60e77929de feat(O5): write volume + storage metrics with background collector 
- Track entity/edge/chunk writes and errors
- Track bytes written per write operation
- Background task: collect DB row counts every 60s
- Background task: collect pool size/idle stats
- Metrics: W1-W12 (12 metrics instrumented)
 2026-09-13 21:48:15 +09:00
rock 853c78bbdb feat(O4): relevance judge with Prometheus metrics 
- RelevanceJudge: threshold-based relevance evaluation
- evaluate(): single query-result pair scoring
- evaluate_batch(): batch eval with precision/recall/F1
- Tracks: evals total, relevant/irrelevant, score histogram
- Updates precision/recall/F1 gauges per batch
- 4 unit tests passing
- Metrics: R1-R9 (9 metrics instrumented)
 2026-09-13 21:38:27 +09:00
rock 8334910144 feat(O3): instrument context endpoint with tier metrics 
- Track context requests, errors, empty results
- Timer for context duration histogram
- Metrics: C1-C8 (8 metrics instrumented)
 2026-09-13 21:37:44 +09:00
rock 0d0fe55519 feat(O2): instrument query handler with Prometheus metrics 
- Track query requests, errors, auth failures, rate limits
- Track embedding failures and embedding call duration
- Track result counts, empty results
- In-flight gauge for concurrent queries
- Timer for query duration histogram
- Metrics: Q1-Q12 (12 metrics instrumented)
 2026-09-13 21:37:11 +09:00
rock 04c28b801d feat(O1): instrument ingest handler with Prometheus metrics 
- Track ingest requests, errors, auth failures, rate limits, duplicates
- Track bytes ingested, records queued
- In-flight gauge for concurrent ingest jobs
- Timer for ingest duration histogram
- Metrics: I1-I12 (12 metrics instrumented)
 2026-09-13 21:36:22 +09:00
rock fd63b089f8 feat(O10): Prometheus metrics module + /metrics endpoint 
- metrics.rs: Counter, Gauge, GaugeF64, Histogram, LabeledCounter types
- Timer RAII helper for automatic latency observation
- All O1-O9 metric definitions pre-declared (119 metrics total)
- render_metrics() outputs Prometheus text exposition format
- GET /metrics endpoint registered in http_server
- HTTP/LLM/DB latency buckets defined
- 6 unit tests passing
 2026-09-13 21:34:37 +09:00
rock 66cc282b6d fix: log raw embedding response before parsing for debugging 
- Read response as text first, then parse JSON
- Log raw body on parse failure (up to 500 chars)
- Log status code + body on non-2xx responses
- Helps diagnose 'expected ident at line 1 column 2' error
 2026-09-13 21:27:55 +09:00