poimen-memory/docs/OBSERVABILITY.md

Memory Service Observability

Why Observe a Knowledge Base System

A memory service that retrieves wrong facts is worse than one that retrieves nothing — it causes hallucination. Traditional web services measure uptime and latency. A knowledge-base service must also measure whether the answer was correct, whether the stored fact was accurate, and whether stale or contradictory information leaked through.

Every metric in this document exists to answer one question: "Did the user get the right information, fast enough, from a source we trust?"

---

Call Flow: Left to Right

                         INGEST PATH
                         ===========

 Client ─── POST /memory/ingest ─── Auth + Rate Limit ─── Dedup Check ─── Embed (768d) ─── Dual Write ─── Done
   │              │                       │                    │               │                │
   │         [I1: req_count]         [I2: auth_ms]      [I3: dedup_hit]  [I4: embed_ms]   [I5: write_ms]
   │                                                                         │                │
   │                                                                    pgvector INSERT   OpenSearch INDEX
   │                                                                         │                │
   │                                                                    [I6: pg_ms]      [I7: os_ms]
   │                                                                                          │
   │                                                                                   [I8: os_fail_count]
   │                                                                                   (eventual consistency)
   │
   │
                         QUERY PATH
                         ==========

 Client ─── POST /memory/query ─── Auth + Rate Limit ─── Classify Intent ─── Embed Query ─── Search ─── RRF Fusion ─── Rerank ─── Respond
   │              │                       │                    │                  │              │            │              │          │
   │         [Q1: req_count]         [Q2: auth_ms]      [Q3: intent_type]   [Q4: embed_ms]     │       [Q7: rrf_ms]   [Q8: rerank_ms] │
   │                                                                                           │                                      │
   │                                                                              ┌────────────┴──────────┐                            │
   │                                                                         pgvector cosine        OpenSearch BM25                    │
   │                                                                              │                       │                            │
   │                                                                         [Q5: sem_ms]           [Q6: lex_ms]                       │
   │                                                                         [Q5a: sem_count]       [Q6a: lex_count]                   │
   │                                                                                                                              [Q9: total_ms]
   │                                                                                                                              [Q10: result_count]
   │
   │
                         CONTEXT PATH (3-tier retrieval)
                         ==============================

 Client ─── POST /memory/context ─── Tier 1: Exact Signature ─── Tier 2: Hybrid Search ─── Tier 3: Reference Fallback ─── Budget Assembly ─── Respond
   │              │                        │                           │                          │                             │                 │
   │         [C1: req_count]          [C2: t1_hit]               [C3: t2_hit]               [C4: t3_hit]                [C5: budget_used]   [C6: total_ms]
   │                                  [C2a: t1_ms]               [C3a: t2_ms]               [C4a: t3_ms]                [C5a: dropped_count]
   │
   │
                         RELEVANCE JUDGMENT (offline, periodic)
                         =====================================

 Sampled Query Log ─── Replay Query ─── Retrieve Top-K ─── Qwen-7B Judge ─── Score (0-2) ─── Compute NDCG/MRR/Precision/Recall
        │                                     │                  │                 │                        │
   [R1: sample_size]                    [R2: replay_ms]    [R3: judge_ms]   [R4: relevance_dist]     [R5: ndcg_10]
                                                           [R3a: judge_cost]                         [R6: mrr]
                                                                                                     [R7: precision_10]
                                                                                                     [R8: recall_10]

---

1. Ingest Observability

Why

Every fact written to memory becomes a retrieval candidate. A bad write — duplicate, contradictory, or malformed — pollutes all future queries. Ingest observability answers: "How many facts are entering the system, how fast, and are any of them bad?"

Metrics

ID	Metric	Type	Unit	Why It Matters
I1	ingest_requests_total	Counter	requests	Total write demand. Capacity planning baseline. Sudden spikes = upstream behavior change.
I2	ingest_auth_duration_seconds	Histogram	seconds	JWT validation overhead. Should be < 5ms. Spike = JWKS fetch or Authentik down.
I3	ingest_dedup_hits_total	Counter	requests	Idempotency saves. High ratio = client retry storm or misconfigured source. Low = healthy unique writes.
I4	ingest_embed_duration_seconds	Histogram	seconds	Embedding latency per chunk. Budget: < 50ms for single chunk. Spike = model cold start or GPU contention.
I5	ingest_dual_write_duration_seconds	Histogram	seconds	Total time to write both stores. SLO: p99 < 500ms.
I6	ingest_pgvector_duration_seconds	Histogram	seconds	Postgres INSERT latency. Includes HNSW index update. Degrades as table grows.
I7	ingest_opensearch_duration_seconds	Histogram	seconds	OpenSearch bulk index latency. Sensitive to segment merges.
I8	ingest_opensearch_failures_total	Counter	failures	OpenSearch write failures. System is eventual-consistent: pgvector is primary. But if this counter grows, lexical search degrades silently.
I9	ingest_bytes_total	Counter	bytes	Total data volume written. Growth rate = storage budget burn.
I10	ingest_chunks_total	Counter	chunks	Write throughput in logical units. 1 ingest request may produce N chunks after splitting.
I11	ingest_contradiction_detected_total	Counter	contradictions	Facts that conflict with existing knowledge. High count = noisy source or domain shift. Each one enters review queue.
I12	ingest_review_queue_depth	Gauge	items	Pending human reviews. Growing = reviewers not keeping up. Stale contradictions = latent hallucination risk.

Alerts

Condition	Severity	Action
ingest_opensearch_failures_total rate > 5/min for 10min	Warning	Check OpenSearch cluster health. Lexical search degrading.
ingest_review_queue_depth > 100 for 24h	Warning	Unreviewed contradictions. Risk of serving conflicting facts.
ingest_pgvector_duration_seconds p99 > 1s	Critical	Postgres overloaded. HNSW index rebuild or VACUUM needed.
ingest_dedup_hits_total / ingest_requests_total > 0.5	Warning	More than half of writes are duplicates. Source misconfiguration.

---

2. Query Observability

Why

Query latency is what the user feels. But latency alone is insufficient — a fast query returning wrong results is worse than a slow correct one. Query observability answers: "Did the system respond quickly, and did the search pipeline find the right documents?"

Metrics

ID	Metric	Type	Unit	Why It Matters
Q1	query_requests_total	Counter	requests	Read demand. Ratio to ingest = read/write skew. Memory systems are read-heavy (10:1+).
Q2	query_auth_duration_seconds	Histogram	seconds	Same as ingest. Shared auth path.
Q3	query_intent_classification	Counter (labeled)	requests	Labels: bug_fix, how_to, reference, faq. Distribution reveals what users ask most. If 80% is bug_fix but recall is low for that intent, prioritize that retrieval path.
Q4	query_embed_duration_seconds	Histogram	seconds	Query embedding latency. Same model as ingest. Should match I4.
Q5	query_semantic_duration_seconds	Histogram	seconds	pgvector cosine search. SLO: p99 < 200ms. Degrades with index size.
Q5a	query_semantic_candidates	Histogram	count	Number of vectors above similarity floor. Zero = total miss. Hundreds = floor too low.
Q6	query_lexical_duration_seconds	Histogram	seconds	OpenSearch BM25 latency. SLO: p99 < 150ms.
Q6a	query_lexical_candidates	Histogram	count	BM25 hit count. Zero = query terms not in corpus (vocabulary gap).
Q7	query_rrf_fusion_duration_seconds	Histogram	seconds	RRF merge time. Should be < 5ms (in-memory). If slow, too many candidates.
Q8	query_rerank_duration_seconds	Histogram	seconds	Cross-encoder reranking. Most expensive step. Budget: < 200ms for top-20.
Q9	query_total_duration_seconds	Histogram	seconds	End-to-end latency. SLO: p99 < 500ms. User-facing number.
Q10	query_results_returned	Histogram	count	How many results pass all filters. Zero = query miss. Track per-intent.
Q11	query_empty_results_total	Counter	requests	Queries that returned nothing. High rate = coverage gap in knowledge base.
Q12	query_score_distribution	Histogram	score (0-1)	Top-1 result score distribution. Bimodal = some queries match well, others poorly. Low mean = embedding quality issue.

Alerts

Condition	Severity	Action
query_total_duration_seconds p99 > 1s	Critical	Pipeline bottleneck. Check Q5, Q6, Q8 to isolate which leg is slow.
query_empty_results_total rate > 20% of Q1	Warning	1 in 5 queries finds nothing. Coverage gap. Check if ingest is running.
query_semantic_candidates p50 = 0	Critical	Embedding search broken. Model mismatch or empty index.
query_lexical_duration_seconds p99 > 500ms	Warning	OpenSearch overloaded. Check segment count, heap usage.

---

3. Context Endpoint (Three-Tier) Observability

Why

The context endpoint is the primary consumer-facing API. It orchestrates three retrieval tiers with budget constraints. Observing tier hit rates reveals whether the knowledge base has coverage at each level, and whether the budget assembly is dropping important results.

Metrics

ID	Metric	Type	Unit	Why It Matters
C1	context_requests_total	Counter	requests	Context lookup demand. Main integration point.
C2	context_tier1_hits_total	Counter	hits	Exact signature matches. High = system is learning from repeated failures. SLO: tier-1 hit rate >= 0.80.
C2a	context_tier1_duration_seconds	Histogram	seconds	Signature lookup. Should be < 50ms (indexed hash).
C3	context_tier2_hits_total	Counter	hits	Hybrid search hits. Bulk of useful results.
C3a	context_tier2_duration_seconds	Histogram	seconds	Full hybrid search. Budget: < 500ms.
C4	context_tier3_hits_total	Counter	hits	Reference fallback. High ratio = learned knowledge insufficient, falling back to docs.
C4a	context_tier3_duration_seconds	Histogram	seconds	Obsidian API + reference retrieval. Slowest tier.
C5	context_budget_used_bytes	Histogram	bytes	How much of the token budget was consumed. Full = rich context. Low = sparse knowledge.
C5a	context_dropped_results_total	Counter	results	Results dropped to fit budget. High = budget too small or results too verbose.
C6	context_total_duration_seconds	Histogram	seconds	End-to-end context assembly. SLO: p99 < 2s.
C7	context_tier_distribution	Counter (labeled)	requests	Label: tier=1|2|3. Which tier served the primary result. Shift from tier-1 to tier-3 over time = knowledge decay.
C8	context_degraded_total	Counter	requests	Requests where a leg failed (e.g., Obsidian timeout). Partial results served.

Alerts

Condition	Severity	Action
context_tier1_hits_total / context_requests_total < 0.60	Warning	Signature match rate dropping. System not learning from failures. Check ingest pipeline.
context_dropped_results_total rate > 30% of results	Warning	Budget too tight. Users missing relevant context.
context_degraded_total rate > 5%	Warning	Partial responses. Check Obsidian API, OpenSearch health.

---

4. Relevance Judgment with Qwen-7B

Why

All the metrics above measure speed and volume. None measure correctness. A system that returns 10 results in 50ms is useless if those results are wrong. Traditional IR evaluation requires human-labeled relevance judgments — expensive and slow. Instead, we use a Qwen-7B model as an automated relevance judge on sampled queries.

This is the single most important observability signal for hallucination prevention. If retrieval precision drops, the LLM downstream gets wrong context and hallucinates. Catching it here — at the retrieval layer — is 10x cheaper than catching it at the generation layer.

Why Qwen-7B

• Cost: ~0.002 USD per judgment. At 500 samples/day = $1/day. A 70B model costs 10x more for marginal gain.
• Speed: ~200ms per judgment on 1x A10. Fast enough for daily batch evaluation.
• Accuracy: 7B models achieve 85-90% agreement with human relevance labels on standard benchmarks (BEIR, MS MARCO). Sufficient for trend detection. We are not using it for absolute measurement — we are using it for drift detection.
• Self-hosted: Runs inside the cluster. No data leaves the network. Required for security-sensitive knowledge bases.

Judgment Flow

                         DAILY RELEVANCE EVALUATION (Cron, 03:00 UTC)
                         ============================================

 Query Log (24h) ─── Sample 500 queries ─── Replay each query ─── Get top-10 results ─── For each (query, result) pair:
                          │                                                                         │
                     [R1: sample_size]                                                        Qwen-7B Prompt:
                                                                                                    │
                                                                                           ┌────────┴────────┐
                                                                                           │  "Given query:  │
                                                                                           │  '{query}'      │
                                                                                           │                 │
                                                                                           │  Rate this      │
                                                                                           │  result:        │
                                                                                           │  '{result}'     │
                                                                                           │                 │
                                                                                           │  Score:         │
                                                                                           │  0 = irrelevant │
                                                                                           │  1 = partial    │
                                                                                           │  2 = perfect    │
                                                                                           └────────┬────────┘
                                                                                                    │
                                                                                              [R4: score]
                                                                                                    │
                                                                         Aggregate: NDCG@10, MRR, Precision@10, Recall@10
                                                                                                    │
                                                                                    ┌───────────────┴───────────────┐
                                                                               [R5: ndcg_10]              [R7: precision_10]
                                                                               [R6: mrr]                  [R8: recall_10]
                                                                                    │
                                                                              Store in Postgres
                                                                              (daily time-series)
                                                                                    │
                                                                              Grafana Dashboard
                                                                              (7-day rolling avg)

Prompt Template

You are a relevance judge for a knowledge base system.

Given a user query and a retrieved document, rate the relevance:
- 0: Irrelevant. The document does not help answer the query at all.
- 1: Partially relevant. The document contains some useful information but does not fully answer the query.
- 2: Highly relevant. The document directly and completely answers the query.

Query: "{query}"

Retrieved Document:
---
{document_text}
---

Relevance Score (0, 1, or 2):

Metrics

ID	Metric	Type	Unit	Why It Matters
R1	relevance_sample_size	Gauge	queries	Number of queries evaluated. 500 gives statistically stable NDCG with ±0.02 CI.
R2	relevance_replay_duration_seconds	Histogram	seconds	Time to replay and retrieve. Should match Q9.
R3	relevance_judge_duration_seconds	Histogram	seconds	Qwen-7B inference time per pair. Budget: < 300ms.
R3a	relevance_judge_cost_usd	Counter	USD	Running cost. Alert if budget exceeded.
R4	relevance_score_distribution	Histogram	score (0-2)	Distribution of judgments. Healthy: 60%+ score=2, < 15% score=0. Drift toward 0 = retrieval degradation.
R5	relevance_ndcg_10	Gauge	ratio (0-1)	Ranking quality. Primary quality metric. SLO: >= 0.85. Measures whether relevant docs appear at the top.
R6	relevance_mrr	Gauge	ratio (0-1)	Position of first relevant result. SLO: >= 0.80. If MRR drops but NDCG holds, results exist but are buried.
R7	relevance_precision_10	Gauge	ratio (0-1)	Fraction of top-10 that is relevant. Measures noise in results.
R8	relevance_recall_10	Gauge	ratio (0-1)	Fraction of all relevant docs captured in top-10. Low = knowledge exists but search can't find it.
R9	relevance_judge_agreement	Gauge	ratio (0-1)	Weekly: re-judge 50 pairs with human labels. Agreement rate validates the judge. SLO: >= 0.85. If agreement drops, Qwen model needs recalibration.

Alerts

Condition	Severity	Action
relevance_ndcg_10 7-day avg < 0.80	Critical	Retrieval quality degraded. Root cause: embedding drift, index corruption, or knowledge gap.
relevance_ndcg_10 drops > 0.05 in 24h	Critical	Sudden quality drop. Check recent ingest for poisoned data.
relevance_mrr < 0.70	Warning	Relevant docs exist but rank poorly. Check reranker, RRF weights.
relevance_score_distribution score=0 > 25%	Warning	Quarter of results are irrelevant. Coverage gap or embedding model mismatch.
relevance_judge_agreement < 0.80	Warning	Judge drifting from human labels. Re-evaluate prompt or model.

---

5. Write Volume and Storage Observability

Why

Memory services grow unboundedly. Unlike caches (eviction policy) or databases (schema constraints), a knowledge base accumulates everything. Write volume tracking answers: "How fast is the system growing, and when do we need to intervene?"

Write volume also directly impacts retrieval quality. More documents = more noise in search results. Without compaction, precision degrades as the corpus grows.

Metrics

ID	Metric	Type	Unit	Why It Matters
W1	storage_pgvector_rows_total	Gauge	rows	Total vectors stored. Growth rate = capacity planning.
W2	storage_pgvector_bytes	Gauge	bytes	Disk usage. 768-dim float32 = ~3KB/row with overhead.
W3	storage_opensearch_docs_total	Gauge	docs	OpenSearch document count. Should match W1 (eventual consistency).
W4	storage_opensearch_bytes	Gauge	bytes	OpenSearch index size. Includes inverted index overhead.
W5	storage_parity_drift	Gauge	count	abs(W1 - W3). Should be 0 in steady state. Non-zero = dual-write inconsistency.
W6	write_rate_per_hour	Gauge	chunks/hour	Sustained write throughput. Trigger compaction planning at > 1000/hour.
W7	write_rate_per_project	Gauge (labeled)	chunks/hour	Per-project write rate. Identifies hot projects dominating storage.
W8	storage_level_distribution	Gauge (labeled)	rows	Label: level=L0|L1|L2|R. Distribution across learning levels. Healthy: L1 > L0 (facts promoted). If L0 dominates, promotion pipeline stalled.
W9	compaction_runs_total	Counter	runs	How often compaction executes.
W10	compaction_dedup_removed_total	Counter	chunks	Duplicates removed per run. High = ingest dedup isn't catching everything.
W11	compaction_stale_gc_removed_total	Counter	chunks	Stale facts garbage-collected (soft-deleted, age > 30d).
W12	compaction_space_freed_bytes	Counter	bytes	Space recovered per run. Declining = less to compact (good).

Alerts

Condition	Severity	Action
storage_parity_drift > 100 for 1h	Warning	pgvector and OpenSearch out of sync. Check dual-write failures (I8).
storage_pgvector_bytes > 80% of PVC	Critical	Storage nearing capacity. Expand PVC or run compaction.
write_rate_per_hour > 5000 sustained 2h	Warning	High write load. Check if upstream is flooding. Consider rate limiting.
storage_level_distribution{level="L0"} / W1 > 0.7	Warning	70% of storage is unprocessed L0. Promotion pipeline stalled.

---

6. Pod Resource Observability

Why

The memory service runs as a Kubernetes pod. If the pod runs out of memory, it gets OOMKilled. If it saturates CPU, latency spikes across all endpoints. These are the physical constraints that gate everything else.

Unlike stateless web services, a memory service has resident state: the embedding model weights (~500MB for MiniLM-L6), connection pools, in-flight embeddings, and cached query results. Memory usage is not flat — it grows with concurrent requests. A burst of 50 parallel ingest requests each holding a 768-dim float32 vector = 50 × 3KB = 150KB just in vectors, but the surrounding allocations (HTTP buffers, serde frames, OpenSearch bulk payloads) multiply that 10-20x.

Metrics

ID	Metric	Type	Unit	Why It Matters
P1	container_memory_working_set_bytes	Gauge	bytes	Actual memory in use (excludes reclaimable cache). This is what Kubernetes uses for OOMKill decisions.
P2	container_memory_rss	Gauge	bytes	Resident Set Size. Physical memory held. If RSS diverges from working set, fragmentation is occurring.
P3	container_memory_usage_bytes	Gauge	bytes	Total memory (includes page cache). Less useful for OOM prediction but shows total footprint.
P4	container_memory_limit_bytes	Gauge	bytes	Pod memory limit from resource spec. P1 / P4 = memory pressure ratio.
P5	container_cpu_usage_seconds_total	Counter	CPU-seconds	CPU consumption rate. rate(P5[1m]) = CPU cores used. Compare to limit.
P6	container_cpu_throttled_seconds_total	Counter	seconds	Time the pod was CPU-throttled by cgroup. Any throttling = latency impact.
P7	container_cpu_cfs_throttled_periods_total	Counter	periods	Number of CFS periods where throttling occurred. P7 / total_periods = throttle ratio.
P8	kube_pod_container_resource_requests	Gauge	cores/bytes	Requested resources. Over-request wastes cluster capacity. Under-request = eviction risk.
P9	kube_pod_container_resource_limits	Gauge	cores/bytes	Resource limits. P1 / P9{resource="memory"} > 0.85 = danger zone.
P10	kube_pod_status_phase	Gauge	phase	Running/Pending/Failed/Succeeded. Pending too long = scheduling issues.
P11	kube_pod_container_status_restarts_total	Counter	restarts	OOMKills and CrashLoopBackoff. Any restart = data in flight was lost.
P12	container_network_receive_bytes_total	Counter	bytes	Network ingress. Correlate with ingest volume. Spike = large batch ingest.
P13	container_network_transmit_bytes_total	Counter	bytes	Network egress. Correlate with query response sizes.

Memory Breakdown (What Lives in the Pod)

 Pod Memory Budget (e.g., 2Gi limit)
 ├── Embedding Model weights         ~500MB (loaded once at startup)
 ├── sqlx connection pool             ~50MB  (20 connections × ~2.5MB each)
 ├── OpenSearch HTTP client pool      ~20MB  (keep-alive connections)
 ├── In-flight ingest embeddings      ~variable (concurrent_requests × ~60KB)
 ├── In-flight query results          ~variable (concurrent_queries × ~200KB)
 ├── Tokio runtime + thread stacks    ~30MB  (worker threads × 8MB stack)
 ├── Rate limiter buckets             ~5MB   (in-memory token buckets)
 ├── Idempotency store (24h TTL)      ~10-50MB (grows with ingest volume)
 └── Heap overhead + fragmentation    ~100-200MB
                                      ─────────
                                      ~800MB baseline + ~variable per-request

Alerts

Condition	Severity	Action
P1 / P4 > 0.85 for 5min	Critical	Memory pressure. OOMKill imminent. Scale up limit or reduce concurrency.
P11 increments	Critical	Pod restarted. Check if OOMKilled (kubectl describe pod). Raise memory limit.
rate(P6[5m]) > 0 for 10min	Warning	Sustained CPU throttling. Query/ingest latency affected. Raise CPU limit.
P7 / total_periods > 0.25	Warning	25%+ of CPU periods throttled. Under-provisioned.
P1 growing monotonically over 24h	Warning	Memory leak. Check idempotency store TTL, connection pool, or embedding cache.

---

7. Availability

Why

A knowledge base that is down cannot reduce hallucination. If the memory service is unavailable during an LLM generation call, the model falls back to parametric knowledge only — which is exactly where hallucinations come from. Availability is not just uptime; it is the probability that a query gets a correct answer within the latency SLO.

Metrics

ID	Metric	Type	Unit	Why It Matters
A1	http_requests_total	Counter (labeled)	requests	Label: method, endpoint, status_code. Foundation for error rate calculation.
A2	http_requests_duration_seconds	Histogram (labeled)	seconds	Label: endpoint. Per-endpoint latency distribution.
A3	http_5xx_total	Counter	requests	Server errors. Any 5xx = something broke internally.
A4	http_4xx_total	Counter (labeled)	requests	Label: status_code. 401/403 = auth issues. 429 = rate limiting. 400 = bad client.
A5	availability_ratio	Gauge	ratio (0-1)	1 - (A3 / A1) over rolling window. SLO: >= 0.999 (three nines).
A6	successful_query_ratio	Gauge	ratio (0-1)	Queries that return 200 with >= 1 result, within 500ms. Stricter than raw availability — includes quality.
A7	health_check_consecutive_failures	Gauge	count	Consecutive /health failures. Kubernetes uses this for restart decisions (liveness probe).
A8	dependency_up	Gauge (labeled)	0/1	Label: dependency=postgres|opensearch|obsidian|embedding_model. Which backends are reachable.
A9	graceful_degradation_total	Counter (labeled)	requests	Label: degraded_component. Requests served with partial results because a dependency was down. e.g., OpenSearch down = semantic-only results.
A10	circuit_breaker_state	Gauge (labeled)	0/1/2	Label: backend. 0=closed (healthy), 1=half-open (probing), 2=open (failing). Per dependency.

Availability Calculation

                    Successful Requests (2xx, within SLO latency)
 Availability = ────────────────────────────────────────────────────
                              Total Requests

 Three tiers of availability:

 1. RAW AVAILABILITY:     1 - (5xx / total)           Target: 99.9%
    "Did it respond?"

 2. LATENCY AVAILABILITY: requests_within_slo / total  Target: 99.5%
    "Did it respond fast enough?"

 3. QUALITY AVAILABILITY: queries_with_results / total  Target: 95%
    "Did it respond with useful results?"

 Monitor all three. A system can be 99.9% available (raw) but only
 80% available (quality) if 20% of queries return empty results.

Alerts

Condition	Severity	Action
availability_ratio < 0.999 over 1h	Critical	SLO breach. Page on-call. Check A8 for which dependency is down.
http_5xx_total rate > 10/min for 5min	Critical	Error spike. Check pod logs, Postgres connectivity, OpenSearch health.
dependency_up{dependency="postgres"} = 0	Critical	Primary store down. All writes and most reads fail.
dependency_up{dependency="opensearch"} = 0	Warning	Lexical search unavailable. Semantic-only fallback active. Quality degraded.
graceful_degradation_total rate > 5% of A1	Warning	Serving partial results too often. Fix the degraded dependency.
successful_query_ratio < 0.90	Warning	10%+ of queries failing or empty. Check ingest pipeline, index health.

---

8. Ingest Rate Patterns

Why

Ingest rate is not just a throughput number. The pattern of writes reveals system behavior. Bursty writes from batch jobs behave differently from steady trickle from live sessions. A sudden drop in ingest rate may mean the upstream source broke. A sudden spike may mean a replay or backfill is running, which changes storage projections.

For a knowledge base, write rate directly affects retrieval quality: every new chunk is a new candidate that can dilute search precision. Knowing when and how fast writes happen lets you plan compaction, predict storage growth, and detect anomalies.

Metrics

ID	Metric	Type	Unit	Why It Matters
IR1	ingest_rate_1m	Gauge	chunks/min	1-minute rolling write rate. Shows bursts.
IR2	ingest_rate_1h	Gauge	chunks/hour	Hourly smoothed rate. Capacity planning baseline.
IR3	ingest_rate_by_project	Gauge (labeled)	chunks/hour	Label: project. Identifies which project dominates writes.
IR4	ingest_rate_by_level	Gauge (labeled)	chunks/hour	Label: level=L0|L1|L2|R. L0 dominance = raw data flooding. L1/L2 growing = healthy knowledge promotion.
IR5	ingest_rate_by_source	Gauge (labeled)	chunks/hour	Label: source=transcript|document|api|batch. Reveals upstream behavior.
IR6	ingest_batch_size	Histogram	chunks/batch	Size of batch ingest requests. Large batches (>100) need different backpressure.
IR7	ingest_queue_depth	Gauge	messages	External queue (kmsvc) pending messages. Growing = workers can't keep up.
IR8	ingest_queue_age_seconds	Histogram	seconds	Age of oldest message in queue. > 60s = processing lag.
IR9	ingest_bytes_per_chunk	Histogram	bytes	Average chunk size. Sudden increase = source sending larger payloads.
IR10	ingest_throughput_bytes_per_second	Gauge	bytes/sec	Sustained write bandwidth. Correlate with P12 (network ingress).

Rate Patterns and What They Mean

 Pattern 1: STEADY TRICKLE (healthy)
 ────────────────────────────────────
  chunks/min
  10 │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
   5 │
   0 └──────────────────────────── time
      Constant ~8-12 chunks/min from live sessions.
      Storage growth predictable. Compaction schedule stable.

 Pattern 2: BURST (batch job or backfill)
 ────────────────────────────────────────
  chunks/min
 500 │          ██
 250 │        ██████
   0 │──────██──────██─────────── time
      Sudden spike. Check: is this a planned backfill?
      If unexpected: rate limit may trigger, queue depth spikes.
      Action: verify source, check queue lag (IR7).

 Pattern 3: DROP TO ZERO (upstream broken)
 ─────────────────────────────────────────
  chunks/min
  10 │ ─ ─ ─ ─ ┐
   5 │          │
   0 │          └──────────────── time
      Ingest stopped. Source may be down, auth token expired,
      or network partition. Silent failure — no errors, just absence.
      Alert on: IR2 = 0 for > 30min during business hours.

 Pattern 4: MONOTONIC GROWTH (runaway source)
 ─────────────────────────────────────────────
  chunks/min
 100 │                        ╱
  50 │                   ╱───
  10 │ ─ ─ ─ ─ ─ ─ ╱───
   0 └──────────────────────────── time
      Write rate increasing over days. Source producing more data.
      Storage projection changes. Compaction may not keep up.
      Action: review source, consider sampling or filtering.

Alerts

Condition	Severity	Action
ingest_rate_1h = 0 for 30min (during business hours)	Warning	Ingest stopped. Check upstream source, auth tokens, network.
ingest_rate_1m > 200 sustained 10min	Warning	Burst ingest. Check if planned. Monitor queue depth (IR7).
ingest_queue_depth > 1000 for 15min	Critical	Workers can't keep up. Scale workers or throttle source.
ingest_queue_age_seconds p99 > 300	Warning	5+ minutes processing lag. Stale data entering the system.
ingest_rate_by_level{level="L0"} / total > 0.9 sustained 24h	Warning	90% raw data, no promotion. Knowledge extraction pipeline stalled.

---

9. Postgres Internal Observability

Why

Postgres is the primary store. Every vector lives there. Every query hits it. Postgres health directly determines memory service health. But Postgres problems are silent — a bloated table doesn't throw errors, it just gets slower. A missing VACUUM doesn't alert, it just consumes 2x disk. An HNSW index with wrong parameters doesn't fail, it just returns worse results.

These metrics catch degradation before users notice it.

Connection Pool and Session Metrics

ID	Metric	Source	Unit	Why It Matters
PG1	pg_stat_activity_count	pg_stat_activity	connections	Active connections by state. active = running query. idle = waiting. idle in transaction = dangerous — holds locks.
PG2	pg_stat_activity_max_duration_seconds	pg_stat_activity	seconds	Longest running query. > 30s = likely stuck or missing index.
PG3	pg_stat_activity_waiting_count	pg_stat_activity	connections	Queries waiting for locks. > 0 sustained = lock contention.
PG4	pg_settings_max_connections	pg_settings	connections	Max allowed connections. PG1 / PG4 > 0.8 = pool exhaustion risk.

Query Performance

ID	Metric	Source	Unit	Why It Matters
PG5	pg_stat_statements_mean_exec_time	pg_stat_statements	ms	Mean execution time per query pattern. Tracks if vector search is degrading over time.
PG6	pg_stat_statements_calls	pg_stat_statements	count	Call count per query. Identifies hot queries. Top-1 query consuming 80% of DB time = optimization target.
PG7	pg_stat_statements_rows	pg_stat_statements	rows	Rows returned per query. Vector search returning 10k rows when limit is 50 = missing index or wrong query plan.
PG8	pg_stat_user_tables_seq_scan	pg_stat_user_tables	scans	Sequential scans on memory_vector. Any seq scan on a large vector table = catastrophic. HNSW index not being used.
PG9	pg_stat_user_tables_idx_scan	pg_stat_user_tables	scans	Index scans. Should be >> seq scans for vector table.

Table and Index Health

ID	Metric	Source	Unit	Why It Matters
PG10	pg_stat_user_tables_n_live_tup	pg_stat_user_tables	tuples	Live rows in memory_vector. Growth rate = storage planning.
PG11	pg_stat_user_tables_n_dead_tup	pg_stat_user_tables	tuples	Dead tuples (deleted/updated but not vacuumed). High ratio = bloat.
PG12	pg_dead_tuple_ratio	computed	ratio	PG11 / (PG10 + PG11). > 0.2 = 20% bloat. VACUUM needed.
PG13	pg_stat_user_tables_last_autovacuum	pg_stat_user_tables	timestamp	When autovacuum last ran. > 24h ago on active table = misconfigured threshold.
PG14	pg_stat_user_tables_last_autoanalyze	pg_stat_user_tables	timestamp	When autoanalyze last ran. Stale statistics = bad query plans.
PG15	pg_table_size_bytes	pg_total_relation_size()	bytes	Total table size including indexes and TOAST.
PG16	pg_index_size_bytes	pg_indexes_size()	bytes	HNSW index size. Grows with vectors. If index > table, check parameters.
PG17	pg_index_bloat_ratio	pgstattuple	ratio	Index bloat. > 0.3 = REINDEX needed. HNSW indexes don't bloat like B-tree, but monitor anyway.

HNSW Index Specific

ID	Metric	Source	Unit	Why It Matters
PG18	pg_hnsw_index_size	pg_relation_size()	bytes	Size of the HNSW index on memory_vector.embedding. Grows as O(n × m) where m=16.
PG19	pg_hnsw_build_time_seconds	manual / CREATE INDEX	seconds	Time to rebuild HNSW index. Needed after parameter changes. At 1M vectors: ~30min. At 10M: hours. Plan maintenance windows.
PG20	pg_hnsw_recall_estimate	benchmark	ratio	Estimated recall of HNSW at current parameters (m=16, ef_construction=200). Run periodic benchmark with known queries. If recall < 0.95, increase ef_search or rebuild with higher m.

WAL and Replication (CNPG)

ID	Metric	Source	Unit	Why It Matters
PG21	pg_wal_lsn_diff	pg_current_wal_lsn()	bytes	WAL generation rate. High during bulk ingest. Correlate with IR1.
PG22	pg_replication_lag_bytes	pg_stat_replication	bytes	Replica lag in bytes. CNPG manages replicas. Lag > 100MB = replica falling behind.
PG23	pg_replication_lag_seconds	pg_stat_replication	seconds	Replica lag in time. > 10s = replica can't keep up with write rate. Read queries to replica return stale results.
PG24	pg_wal_size_bytes	pg_wal directory	bytes	Total WAL on disk. Unbounded growth = archiving broken or wal_keep_size too high.

Transaction and Lock Health

ID	Metric	Source	Unit	Why It Matters
PG25	pg_stat_database_xact_commit	pg_stat_database	transactions	Committed transactions/sec. Baseline throughput.
PG26	pg_stat_database_xact_rollback	pg_stat_database	transactions	Rolled back transactions. PG26 / PG25 > 0.01 = 1% rollback rate. Check constraint violations or deadlocks.
PG27	pg_stat_database_deadlocks	pg_stat_database	deadlocks	Any deadlock = concurrent write contention. Rare in append-mostly workload. If seen, check compaction + ingest overlap.
PG28	pg_stat_database_conflicts	pg_stat_database	conflicts	Replication conflicts. Query on replica canceled due to WAL replay. Adjust max_standby_streaming_delay.
PG29	pg_locks_count	pg_locks	locks	Lock count by mode. AccessExclusiveLock blocks everything — check for DDL during traffic.

Cache Efficiency

ID	Metric	Source	Unit	Why It Matters
PG30	pg_stat_database_blks_hit	pg_stat_database	blocks	Buffer cache hits.
PG31	pg_stat_database_blks_read	pg_stat_database	blocks	Disk reads (cache misses).
PG32	pg_cache_hit_ratio	computed	ratio	PG30 / (PG30 + PG31). SLO: >= 0.99. Below 0.95 = shared_buffers too small or working set exceeds RAM.
PG33	pg_stat_user_indexes_idx_blks_hit	pg_stat_user_indexes	blocks	HNSW index cache hits. Low hit ratio = index doesn't fit in memory. Increase shared_buffers or effective_cache_size.

Key SQL Queries for Monitoring

-- Dead tuple ratio (bloat indicator)
SELECT relname,
       n_live_tup,
       n_dead_tup,
       CASE WHEN n_live_tup > 0
            THEN round(n_dead_tup::numeric / (n_live_tup + n_dead_tup) * 100, 2)
            ELSE 0 END AS dead_pct,
       last_autovacuum,
       last_autoanalyze
FROM pg_stat_user_tables
WHERE relname IN ('memory_vector', 'memory_entity', 'memory_edge')
ORDER BY n_dead_tup DESC;

-- Slowest queries (requires pg_stat_statements)
SELECT query,
       calls,
       round(mean_exec_time::numeric, 2) AS mean_ms,
       round(max_exec_time::numeric, 2) AS max_ms,
       rows
FROM pg_stat_statements
WHERE dbid = (SELECT oid FROM pg_database WHERE datname = 'memory')
ORDER BY mean_exec_time DESC
LIMIT 10;

-- Sequential vs index scans (vector table must use index)
SELECT relname,
       seq_scan,
       idx_scan,
       CASE WHEN (seq_scan + idx_scan) > 0
            THEN round(idx_scan::numeric / (seq_scan + idx_scan) * 100, 2)
            ELSE 100 END AS idx_scan_pct
FROM pg_stat_user_tables
WHERE relname = 'memory_vector';

-- Table and index sizes
SELECT relname,
       pg_size_pretty(pg_total_relation_size(relid)) AS total_size,
       pg_size_pretty(pg_relation_size(relid)) AS table_size,
       pg_size_pretty(pg_indexes_size(relid)) AS index_size
FROM pg_stat_user_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(relid) DESC;

-- Replication lag (CNPG replicas)
SELECT client_addr,
       state,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS lag_bytes,
       extract(epoch FROM now() - replay_lag) AS lag_seconds
FROM pg_stat_replication;

-- Cache hit ratio
SELECT datname,
       round(
         blks_hit::numeric / NULLIF(blks_hit + blks_read, 0) * 100, 2
       ) AS cache_hit_pct
FROM pg_stat_database
WHERE datname = 'memory';

-- Connection state breakdown
SELECT state, count(*)
FROM pg_stat_activity
WHERE datname = 'memory'
GROUP BY state;

Alerts

Condition	Severity	Action
pg_dead_tuple_ratio > 0.20 on memory_vector	Warning	20% bloat. Run VACUUM ANALYZE memory_vector; or check autovacuum config.
pg_stat_user_tables_seq_scan on memory_vector increments	Critical	Sequential scan on vector table. HNSW index not used. Check query plan with EXPLAIN ANALYZE.
pg_cache_hit_ratio < 0.95	Critical	Cache thrashing. Increase shared_buffers or scale to larger instance.
pg_replication_lag_seconds > 30	Warning	Replica 30s behind. Read queries returning stale data. Check write rate, replica resources.
pg_stat_database_deadlocks > 0	Warning	Deadlock detected. Check concurrent write patterns (ingest + compaction).
pg_stat_activity_max_duration_seconds > 60	Warning	Query running > 60s. Likely stuck. Check for missing index or lock wait.
pg_stat_activity_count{state="idle in transaction"} > 5 for 10min	Warning	Idle-in-transaction connections holding locks. Connection pool leak or application bug.
pg_wal_size_bytes > 10GB	Warning	WAL accumulation. Check archiving, replication, or wal_keep_size setting.
PG1 / PG4 > 0.8	Critical	Connection pool near max. Add PgBouncer or increase max_connections.

---

10. System Health (Infrastructure Summary)

Metrics

ID	Metric	Type	Unit	Why It Matters
H1	health_check_status	Gauge	0/1	/health endpoint. Basic liveness.
H2	pgvector_connection_pool_active	Gauge	connections	Active DB connections. Near max = pool exhaustion risk.
H3	pgvector_connection_pool_idle	Gauge	connections	Idle connections. Zero idle + high active = under-provisioned.
H4	opensearch_cluster_status	Gauge	0/1/2	0=red, 1=yellow, 2=green. Yellow = replica missing. Red = data loss risk.
H5	embedding_model_loaded	Gauge	0/1	Model health check. 0 = all ingest and query embeds will fail.
H6	rate_limit_rejections_total	Counter	requests	Rate limit hits. High = legitimate traffic being blocked, or DDoS.
H7	auth_failures_total	Counter (labeled)	requests	Label: reason=expired|invalid|missing. Pattern reveals attack or misconfiguration.

---

11. Dashboard Layout

Grafana Rows (top to bottom)

Row 1: SYSTEM HEALTH
┌──────────────┬──────────────┬──────────────┬──────────────┐
│  Health: UP  │  PG Pool:    │  OpenSearch:  │  Embed Model │
│  (H1)        │  12/20 active│  GREEN       │  LOADED      │
└──────────────┴──────────────┴──────────────┴──────────────┘

Row 2: WRITE PATH (Ingest)
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  Ingest Rate (I1)        │  Write Latency p50/p99   │  Dedup Hit Ratio         │
│  [line chart, 24h]       │  (I5) [line chart, 24h]  │  (I3/I1) [line, 24h]     │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│  OpenSearch Failures (I8)│  Contradiction Queue (I12)│  Chunks Written (I10)   │
│  [counter, 24h]          │  [gauge, current depth]  │  [counter, 24h]          │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Row 3: READ PATH (Query)
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  Query Rate (Q1)         │  Query Latency p50/p99   │  Empty Results (Q11)     │
│  [line chart, 24h]       │  (Q9) [line chart, 24h]  │  [%, 24h]               │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│  Latency Breakdown       │  Intent Distribution     │  Score Distribution      │
│  sem/lex/rrf/rerank      │  (Q3) [pie chart]        │  (Q12) [histogram]       │
│  [stacked area, 24h]     │                          │                          │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Row 4: CONTEXT (3-Tier)
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  Tier Hit Distribution   │  Context Latency p50/p99 │  Budget Usage            │
│  (C7) [stacked bar, 7d]  │  (C6) [line chart, 24h]  │  (C5) [histogram]       │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│  Tier-1 Hit Rate         │  Degraded Responses (C8) │  Dropped Results (C5a)   │
│  (C2/C1) [gauge, target  │  [counter, 24h]          │  [counter, 24h]          │
│   >= 0.80]               │                          │                          │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Row 5: RELEVANCE (Qwen-7B Judge) ← MOST IMPORTANT ROW
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  NDCG@10 (R5)            │  MRR (R6)                │  Precision/Recall        │
│  [line chart, 30d        │  [line chart, 30d        │  (R7, R8) [line, 30d     │
│   rolling avg, target    │   rolling avg]           │   rolling avg]           │
│   >= 0.85]               │                          │                          │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│  Relevance Score Dist    │  Judge Cost (R3a)        │  Judge Agreement (R9)    │
│  (R4) [bar: 0/1/2, 7d]  │  [counter, daily USD]    │  [gauge, weekly]         │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Row 6: POD RESOURCES
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  Memory Usage (P1)       │  CPU Usage (P5 rate)     │  Pod Restarts (P11)      │
│  [line, 24h, limit line] │  [line, 24h, limit line] │  [counter, 7d]           │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│  Memory Pressure (P1/P4) │  CPU Throttle (P6 rate)  │  Network I/O (P12, P13)  │
│  [gauge, target < 0.85]  │  [line, 24h]             │  [line, 24h]             │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Row 7: AVAILABILITY
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  Availability (A5)       │  Error Rate (A3/A1)      │  Dependencies (A8)       │
│  [gauge, target >= 99.9%]│  [line, 24h]             │  [status grid: pg/os/emb]│
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│  Quality Avail (A6)      │  Degraded Responses (A9) │  4xx Breakdown (A4)      │
│  [gauge, target >= 95%]  │  [counter, 24h]          │  [stacked bar, 24h]      │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Row 8: INGEST RATE PATTERNS
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  Write Rate/Min (IR1)    │  Write Rate/Hour (IR2)   │  Queue Depth (IR7)       │
│  [line, 24h, burst high] │  [line, 7d]              │  [gauge, target < 1000]  │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│  Rate by Project (IR3)   │  Rate by Level (IR4)     │  Queue Age p99 (IR8)     │
│  [stacked area, 24h]     │  [stacked area, 24h]     │  [line, 24h]             │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Row 9: POSTGRES INTERNALS
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  Cache Hit Ratio (PG32)  │  Dead Tuple Ratio (PG12) │  Connections (PG1)       │
│  [gauge, target >= 99%]  │  [gauge, target < 20%]   │  [stacked bar by state]  │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│  Seq vs Idx Scans (PG8/9)│  Replication Lag (PG23)  │  Table Sizes (PG15)      │
│  [line, 7d]              │  [line, 24h, target < 10s]│ [bar chart, current]    │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│  Slowest Queries (PG5)   │  WAL Size (PG24)         │  Deadlocks (PG27)        │
│  [table, top 5]          │  [line, 7d]              │  [counter, 30d]          │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Row 10: STORAGE
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  Total Vectors (W1)      │  Storage Bytes           │  Write Rate/Hour (W6)    │
│  [gauge, current]        │  (W2+W4) [line, 30d]     │  [line chart, 24h]       │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│  Parity Drift (W5)       │  Level Distribution (W8) │  Compaction Freed (W12)  │
│  [gauge, target = 0]     │  [stacked bar]           │  [counter per run]       │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

---

12. Implementation: Prometheus Metrics in Rust

use prometheus::{
    register_counter, register_counter_vec, register_gauge, register_gauge_vec,
    register_histogram, register_histogram_vec,
    Counter, CounterVec, Gauge, GaugeVec, Histogram, HistogramVec,
};
use lazy_static::lazy_static;

lazy_static! {
    // === INGEST ===
    pub static ref INGEST_REQUESTS: Counter =
        register_counter!("memory_ingest_requests_total", "Total ingest requests").unwrap();
    pub static ref INGEST_CHUNKS: Counter =
        register_counter!("memory_ingest_chunks_total", "Total chunks written").unwrap();
    pub static ref INGEST_BYTES: Counter =
        register_counter!("memory_ingest_bytes_total", "Total bytes ingested").unwrap();
    pub static ref INGEST_DEDUP_HITS: Counter =
        register_counter!("memory_ingest_dedup_hits_total", "Deduplicated chunks skipped").unwrap();
    pub static ref INGEST_CONTRADICTIONS: Counter =
        register_counter!("memory_ingest_contradictions_total", "Contradictions detected").unwrap();
    pub static ref INGEST_EMBED_DURATION: Histogram =
        register_histogram!("memory_ingest_embed_seconds", "Embedding latency per chunk",
            vec![0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0]).unwrap();
    pub static ref INGEST_PGVECTOR_DURATION: Histogram =
        register_histogram!("memory_ingest_pgvector_seconds", "pgvector write latency",
            vec![0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0]).unwrap();
    pub static ref INGEST_OPENSEARCH_DURATION: Histogram =
        register_histogram!("memory_ingest_opensearch_seconds", "OpenSearch index latency",
            vec![0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0]).unwrap();
    pub static ref INGEST_OPENSEARCH_FAILURES: Counter =
        register_counter!("memory_ingest_opensearch_failures_total", "OpenSearch write failures").unwrap();
    pub static ref REVIEW_QUEUE_DEPTH: Gauge =
        register_gauge!("memory_review_queue_depth", "Pending contradiction reviews").unwrap();

    // === QUERY ===
    pub static ref QUERY_REQUESTS: Counter =
        register_counter!("memory_query_requests_total", "Total query requests").unwrap();
    pub static ref QUERY_EMPTY_RESULTS: Counter =
        register_counter!("memory_query_empty_results_total", "Queries returning zero results").unwrap();
    pub static ref QUERY_TOTAL_DURATION: Histogram =
        register_histogram!("memory_query_total_seconds", "End-to-end query latency",
            vec![0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0]).unwrap();
    pub static ref QUERY_SEMANTIC_DURATION: Histogram =
        register_histogram!("memory_query_semantic_seconds", "pgvector search latency",
            vec![0.01, 0.025, 0.05, 0.1, 0.2, 0.5]).unwrap();
    pub static ref QUERY_LEXICAL_DURATION: Histogram =
        register_histogram!("memory_query_lexical_seconds", "OpenSearch BM25 latency",
            vec![0.01, 0.025, 0.05, 0.1, 0.2, 0.5]).unwrap();
    pub static ref QUERY_INTENT: CounterVec =
        register_counter_vec!("memory_query_intent_total", "Query intent classification",
            &["intent"]).unwrap();
    pub static ref QUERY_RESULTS_COUNT: Histogram =
        register_histogram!("memory_query_results_count", "Results returned per query",
            vec![0.0, 1.0, 3.0, 5.0, 10.0, 20.0, 50.0]).unwrap();
    pub static ref QUERY_TOP1_SCORE: Histogram =
        register_histogram!("memory_query_top1_score", "Top-1 result similarity score",
            vec![0.3, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 1.0]).unwrap();

    // === CONTEXT ===
    pub static ref CONTEXT_REQUESTS: Counter =
        register_counter!("memory_context_requests_total", "Total context lookups").unwrap();
    pub static ref CONTEXT_TIER_HITS: CounterVec =
        register_counter_vec!("memory_context_tier_hits_total", "Hits per tier",
            &["tier"]).unwrap();
    pub static ref CONTEXT_TOTAL_DURATION: Histogram =
        register_histogram!("memory_context_total_seconds", "End-to-end context latency",
            vec![0.1, 0.25, 0.5, 1.0, 2.0, 5.0]).unwrap();
    pub static ref CONTEXT_DROPPED: Counter =
        register_counter!("memory_context_dropped_results_total", "Results dropped for budget").unwrap();

    // === RELEVANCE (updated daily by batch job) ===
    pub static ref RELEVANCE_NDCG: Gauge =
        register_gauge!("memory_relevance_ndcg_10", "NDCG@10 from Qwen-7B judge").unwrap();
    pub static ref RELEVANCE_MRR: Gauge =
        register_gauge!("memory_relevance_mrr", "Mean Reciprocal Rank").unwrap();
    pub static ref RELEVANCE_PRECISION: Gauge =
        register_gauge!("memory_relevance_precision_10", "Precision@10").unwrap();
    pub static ref RELEVANCE_RECALL: Gauge =
        register_gauge!("memory_relevance_recall_10", "Recall@10").unwrap();

    // === STORAGE ===
    pub static ref STORAGE_VECTORS: Gauge =
        register_gauge!("memory_storage_vectors_total", "Total vectors in pgvector").unwrap();
    pub static ref STORAGE_BYTES: Gauge =
        register_gauge!("memory_storage_bytes", "Total storage bytes (pg + os)").unwrap();
    pub static ref STORAGE_PARITY_DRIFT: Gauge =
        register_gauge!("memory_storage_parity_drift", "pgvector vs OpenSearch doc count difference").unwrap();
    pub static ref WRITE_RATE: Gauge =
        register_gauge!("memory_write_rate_per_hour", "Current write rate (chunks/hour)").unwrap();
}

Instrumentation Example (Ingest Handler)

pub async fn ingest_handler(req: HttpRequest, body: web::Json<IngestRequest>, state: web::Data<AppState>) -> HttpResponse {
    INGEST_REQUESTS.inc();

    // Auth
    let auth_timer = INGEST_AUTH_DURATION.start_timer();
    let (claims, token) = match validate_auth(&req, &state).await { ... };
    auth_timer.observe_duration();

    // Dedup
    if state.idempotency_store.is_duplicate(&body.idempotency_key) {
        INGEST_DEDUP_HITS.inc();
        return HttpResponse::Ok().json(json!({"status": "duplicate"}));
    }

    // Embed
    let embed_timer = INGEST_EMBED_DURATION.start_timer();
    let embedding = state.embeddings.embed_one(&body.text).await?;
    embed_timer.observe_duration();

    // pgvector write
    let pg_timer = INGEST_PGVECTOR_DURATION.start_timer();
    state.vector_store.insert(&body.project, &body.text, &embedding).await?;
    pg_timer.observe_duration();

    // OpenSearch write
    let os_timer = INGEST_OPENSEARCH_DURATION.start_timer();
    match state.opensearch_client.index_document(...).await {
        Ok(_) => {},
        Err(e) => {
            INGEST_OPENSEARCH_FAILURES.inc();
            tracing::warn!("OpenSearch write failed (non-blocking): {}", e);
        }
    }
    os_timer.observe_duration();

    INGEST_CHUNKS.inc();
    INGEST_BYTES.inc_by(body.text.len() as f64);

    HttpResponse::Created().json(...)
}

---

13. Relevance Evaluation CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: memory-relevance-eval
  namespace: poimen
spec:
  schedule: "0 3 * * *"  # Daily at 03:00 UTC
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: relevance-eval
              image: forgejo.riotpiao.com/rock/poimen-memory:latest
              command: ["mem", "evaluate-relevance"]
              env:
                - name: EVAL_SAMPLE_SIZE
                  value: "500"
                - name: EVAL_JUDGE_MODEL
                  value: "qwen2.5-7b"
                - name: EVAL_JUDGE_ENDPOINT
                  value: "http://ollama.poimen.svc:11434/api/generate"
                - name: EVAL_QUERY_LOG_HOURS
                  value: "24"
                - name: DATABASE_URL
                  valueFrom:
                    secretKeyRef:
                      name: memory-db-credentials
                      key: url
              resources:
                requests:
                  cpu: "500m"
                  memory: "512Mi"
                limits:
                  cpu: "1"
                  memory: "1Gi"
          restartPolicy: OnFailure

---

14. SLOs Summary

Signal	Target	Window	Consequence of Miss
Ingest p99 latency	< 500ms	24h rolling	Backpressure on upstream systems
Query p99 latency	< 500ms	24h rolling	User-perceived slowness
Context p99 latency	< 2s	24h rolling	Agent timeout, degraded assistance
Query empty rate	< 20%	24h rolling	Users get no answer, lose trust
Tier-1 hit rate	>= 80%	7d rolling	System not learning from failures
NDCG@10	>= 0.85	7d rolling	Retrieval quality degraded, hallucination risk
MRR	>= 0.80	7d rolling	Relevant results buried in ranking
Storage parity drift	= 0	1h	Dual-write inconsistency, partial search
Review queue depth	< 100	24h	Unreviewed contradictions leaking through
Relevance judge agreement	>= 85%	Weekly	Automated evaluation unreliable
Raw availability	>= 99.9%	24h rolling	Service down, LLM falls back to parametric knowledge
Quality availability	>= 95%	24h rolling	Queries succeeding but returning nothing useful
Pod memory pressure	< 85% of limit	5min	OOMKill imminent, in-flight requests lost
CPU throttle ratio	< 25%	5min	Latency degradation across all endpoints
PG cache hit ratio	>= 99%	1h	Disk thrashing, query latency spikes
PG dead tuple ratio	< 20%	24h	Table bloat, slower scans, wasted disk
PG replication lag	< 10s	5min	Stale reads from replica
Ingest rate (zero)	> 0 during business hours	30min	Silent upstream failure, knowledge going stale
Ingest queue depth	< 1000	15min	Workers can't keep up, processing lag