riotpiao-poimen/poimen-memory
2
0
Fork 0
Code Issues 13 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat: setup phase 3 agent infrastructure + enable docker ci on prs
CI / CI (push) Successful in 15m5s
Details

Browse Source 
- Enable docker build, sha extraction on PRs (validate Dockerfile)
   - Add SOPS encrypted memory-agent credentials
   - Plan 15 tasks: 5 memory service + 10 temporal workflow
   - Milestone: monitoring-agent (due 2025-03-15)
   - Ready: Forgejo API token needed for PR automation
 ```

Co-authored-by: rock <[email protected]>
This commit was merged in pull request #45.
This commit is contained in:
rock
2026-09-08 23:16:31 +00:00
committed by rock
parent 83a50844c5
commit 1e5c3d1433
20 changed files with 2538 additions and 218 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/AUTHENTIK_SOPS_SETUP.md
+321
View File
@@ -0,0 +1,321 @@
# Poimen Memory: Authentik JWT + SOPS Encryption Setup
## Overview
The Poimen Memory service uses:
1. **Authentik service account** for OAuth2 client credentials flow
2. **SOPS + Age encryption** to encrypt secrets in git
3. **JWT tokens** for authentication to LLM gateway, S3, and other services
## Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ Kubernetes (poimen) │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌─────────────────┐ │
│ │ ConfigMap │ │ Secret (SOPS) │ │
│ │ (unencrypted)│ │ (age-encrypted)│ │
│ └──────┬───────┘ └────────┬────────┘ │
│ │ │ │
│ ├─────────┬───────────┤ │
│ │ │ │ │
│ ┌────▼─────────▼───────────▼────┐ │
│ │ poimen-memory Pod │ │
│ │ Environment Variables: │ │
│ │ - LLM_ENDPOINT │ │
│ │ - AUTHENTIK_ISSUER │ │
│ │ - AUTHENTIK_CLIENT_ID │ │
│ │ - AUTHENTIK_CLIENT_SECRET │ │
│ │ - S3_ACCESS_KEY │ │
│ │ - S3_SECRET_KEY │ │
│ └────┬────────────────┬──────────┘ │
│ │ │ │
│ ┌──────▼──┐ ┌──────────▼──────┐ │
│ │ Authentik│ │ LLM Endpoint │ │
│ │ (JWT) │ │ (api.riotpiao) │ │
│ └──────────┘ └─────────────────┘ │
│ │
│ ┌─────────────────────────────────────┐ │
│ │ Entity Extraction Pipeline │ │
│ │ ┌────────────────────────────┐ │ │
│ │ │ 1. WikiLink fallback │ │ │
│ │ │ 2. LLM extraction (JWT auth)│ │ │
│ │ │ 3. Reflection verification │ │ │
│ │ │ 4. Contradiction detection │ │ │
│ │ └────────────────────────────┘ │ │
│ └──────────────┬──────────────────────┘ │
│ │ │
│ ┌───────▼────────┐ │
│ │ PostgreSQL │ │
│ │ (entities DB) │ │
│ └────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
```
## Step 1: Create Authentik Service Account
### In Authentik Admin Panel:
1. Navigate: **Settings** → **Applications** → **Create Application**
2. Name: `poimen-memory`
3. Slug: `poimen-memory`
4. Provider: Create a new OAuth2 Provider
- Name: `poimen-memory`
- Client type: `confidential`
- Client ID: `<auto-generated>`
- Client secret: `<auto-generated>`
5. Save and note the **Client ID** and **Client Secret**
### Verify OAuth2 Token Endpoint:
```bash
curl -X POST https://authentik.riotpiao.com/application/o/token/ \
-d "grant_type=client_credentials" \
-d "client_id=<CLIENT_ID>" \
-d "client_secret=<CLIENT_SECRET>"
# Response:
# {
# "access_token": "eyJ0eXAi...",
# "token_type": "Bearer",
# "expires_in": 3600
# }
```
## Step 2: Create Encrypted Secrets File
### 2.1 Ensure SOPS is configured:
```bash
# Load SOPS_AGE_KEY_FILE
export SOPS_AGE_KEY_FILE=~/.sops/key.txt
# Verify key exists
ls -la ~/.sops/key.txt
```
### 2.2 Create unencrypted secrets template:
```yaml
# k8s/app/poimen-memory-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: poimen-memory-secrets
namespace: poimen
type: Opaque
stringData:
# Authentik OAuth2 Credentials
AUTHENTIK_ISSUER: "https://authentik.riotpiao.com/application/o/memory"
AUTHENTIK_AUDIENCE: "poimen-memory"
AUTHENTIK_CLIENT_ID: "<from-authentik-app>"
AUTHENTIK_CLIENT_SECRET: "<from-authentik-app>"
# LLM Gateway API Key (optional fallback)
LLM_API_KEY: "<jwt-will-be-auto-generated>"
# S3/Minio Credentials
S3_ACCESS_KEY: "<minio-access-key>"
S3_SECRET_KEY: "<minio-secret-key>"
```
### 2.3 Encrypt with SOPS:
```bash
export SOPS_AGE_KEY_FILE=~/.sops/key.txt
cd ~/workplace/Poimen/memory
sops -e k8s/app/poimen-memory-secrets.yaml > k8s/app/poimen-memory-secrets.enc.yaml
# Verify encryption worked
sops -d k8s/app/poimen-memory-secrets.enc.yaml | head -20
```
### 2.4 Commit encrypted file only:
```bash
git add k8s/app/poimen-memory-secrets.enc.yaml
git add .sops.yaml
git rm k8s/app/poimen-memory-secrets.yaml # Remove plaintext
git commit -m "feat: add SOPS-encrypted Authentik secrets"
```
## Step 3: Deploy to Kubernetes
### 3.1 Install KSOPS plugin (if using ArgoCD):
```bash
# ArgoCD Helm values
kustomization:
plugins:
- name: Kustomize
image: ghcr.io/viaduct-ai/kustomize-sops:v4.1.1
```
### 3.2 Apply secrets manifest:
```bash
# With KSOPS: ArgoCD auto-decrypts and applies
# Without KSOPS: Manual decryption before apply
export SOPS_AGE_KEY_FILE=~/.sops/key.txt
sops -d k8s/app/poimen-memory-secrets.enc.yaml | kubectl apply -f -
# Verify secret created
kubectl -n poimen get secret poimen-memory-secrets
kubectl -n poimen describe secret poimen-memory-secrets
```
### 3.3 Update deployment envFrom:
```yaml
# k8s/app/deployment.yaml
spec:
template:
spec:
containers:
- name: poimen-memory
envFrom:
- configMapRef:
name: poimen-memory-config
- secretRef:
name: poimen-memory-secrets # <-- Add this
```
## Step 4: Entity Extractor JWT Flow
### Code: `crates/mem-ingest/src/entity_extractor.rs`
```rust
// Initialization
pub struct LlmEntityExtractor {
jwt_issuer: Option<Arc<Mutex<AuthentikJwtIssuer>>>,
}
impl LlmEntityExtractor {
pub fn new(model_name: &str) -> Self {
let jwt_issuer = AuthentikJwtIssuer::from_env().ok();
Self {
jwt_issuer: jwt_issuer.map(|iss| Arc::new(Mutex::new(iss))),
}
}
}
// LLM call with JWT
async fn call_llm_endpoint(&self, prompt: &str) -> Result<String> {
// Get JWT token from Authentik (cached, auto-refreshed)
let auth_header = if let Some(jwt_issuer) = &self.jwt_issuer {
let issuer = jwt_issuer.lock().await;
let token = issuer.get_access_token().await?;
format!("Bearer {}", token)
} else {
format!("Bearer {}", fallback_api_key)
};
// POST to LLM endpoint with JWT
client
.post(&endpoint)
.header("Authorization", auth_header)
.json(&payload)
.send()
.await?
}
```
## Step 5: Runtime Verification
### 5.1 Check JWT token exchange in logs:
```bash
kubectl -n poimen logs deployment/poimen-memory | grep -i "authentik\|jwt"
# Expected output:
# [2026-01-09T20:30:15Z] Obtained Authentik JWT token (expires in 3600 seconds)
# [2026-01-09T20:30:15Z] LLM response (via Authentik JWT): {...}
```
### 5.2 Test entity extraction end-to-end:
```bash
# Port-forward to service
kubectl -n poimen port-forward svc/poimen-memory 8080:8080 &
# Ingest a record
curl -X POST http://localhost:8080/memory/ingest \
-H "Content-Type: application/json" \
-d '{
"project": "homelab",
"source": "test://jwt",
"ingest_id": "jwt-test-001",
"records": [{
"role": "architect",
"text": "[[Kubernetes]] uses [[Docker]]. [[ArgoCD]] manages deployments.",
"timestamp": "2026-01-09T20:30:00Z",
"source_position": 0
}]
}'
# Check logs for JWT usage
kubectl -n poimen logs deployment/poimen-memory | tail -20
```
## Step 6: Monitoring & Maintenance
### Token Expiry Handling:
- JWT tokens are cached with auto-refresh
- If token expires during use, new token is fetched automatically
- No manual token rotation required
### Credential Rotation:
- Rotate Authentik client secret periodically
- Update SOPS secret file and re-encrypt
- Redeploy pod to pick up new secret
### SOPS Key Rotation (Yearly):
```bash
# Generate new age key
age-keygen -o ~/.sops/key.txt.new
# Re-encrypt all secrets with new key
for file in k8s/**/*.enc.yaml; do
sops -r $file
done
# Update ArgoCD to use new key
# Commit changes
git add k8s/**/*.enc.yaml
git commit -m "chore: rotate SOPS encryption keys"
```
## Troubleshooting
### Issue: "AUTHENTIK_ISSUER not set"
**Cause**: Secret not mounted properly
**Solution**: `kubectl -n poimen get secret poimen-memory-secrets`
### Issue: "JWT token request failed: 401"
**Cause**: Invalid client credentials
**Solution**: Verify Client ID/Secret in Authentik, check SOPS decryption
### Issue: "error loading config: no matching creation rules found"
**Cause**: SOPS .sops.yaml not configured correctly
**Solution**: Use `.sops.yaml` with explicit age key instead of config-based rules
### Issue: "LLM API error: 403 Forbidden"
**Cause**: JWT token doesn't have permission to LLM gateway
**Solution**: Add RBAC role "LLM User" to service account in Authentik
---
## Files Modified
- ✅ `crates/mem-ingest/src/authentik_jwt.rs` — JWT token exchange module
- ✅ `crates/mem-ingest/src/entity_extractor.rs` — LLM calls with JWT
- ✅ `crates/mem-ingest/src/lib.rs` — Module export
- ✅ `k8s/app/poimen-memory-secrets.yaml` — Secret template (plaintext, not committed)
- ✅ `k8s/app/poimen-memory-secrets.enc.yaml` — Secret encrypted with SOPS
- ✅ `k8s/app/deployment.yaml` — Updated envFrom for secrets
- ✅ `k8s/app/config.yaml` — LLM endpoint configuration
- ✅ `k8s/.sops.yaml` — SOPS encryption rules
docs/OBSERVABILITY.md
+982
View File
@@ -0,0 +1,982 @@
# Memory Service Observability
## Why Observe a Knowledge Base System
A memory service that retrieves wrong facts is worse than one that retrieves nothing — it causes hallucination. Traditional web services measure uptime and latency. A knowledge-base service must also measure **whether the answer was correct**, **whether the stored fact was accurate**, and **whether stale or contradictory information leaked through**.
Every metric in this document exists to answer one question: **"Did the user get the right information, fast enough, from a source we trust?"**
---
## Call Flow: Left to Right
```
INGEST PATH
===========
Client ─── POST /memory/ingest ─── Auth + Rate Limit ─── Dedup Check ─── Embed (768d) ─── Dual Write ─── Done
│ │ │ │ │ │
│ [I1: req_count] [I2: auth_ms] [I3: dedup_hit] [I4: embed_ms] [I5: write_ms]
│ │ │
│ pgvector INSERT OpenSearch INDEX
│ │ │
│ [I6: pg_ms] [I7: os_ms]
│ │
│ [I8: os_fail_count]
│ (eventual consistency)
│
│
QUERY PATH
==========
Client ─── POST /memory/query ─── Auth + Rate Limit ─── Classify Intent ─── Embed Query ─── Search ─── RRF Fusion ─── Rerank ─── Respond
│ │ │ │ │ │ │ │ │
│ [Q1: req_count] [Q2: auth_ms] [Q3: intent_type] [Q4: embed_ms] │ [Q7: rrf_ms] [Q8: rerank_ms] │
│ │ │
│ ┌────────────┴──────────┐ │
│ pgvector cosine OpenSearch BM25 │
│ │ │ │
│ [Q5: sem_ms] [Q6: lex_ms] │
│ [Q5a: sem_count] [Q6a: lex_count] │
│ [Q9: total_ms]
│ [Q10: result_count]
│
│
CONTEXT PATH (3-tier retrieval)
==============================
Client ─── POST /memory/context ─── Tier 1: Exact Signature ─── Tier 2: Hybrid Search ─── Tier 3: Reference Fallback ─── Budget Assembly ─── Respond
│ │ │ │ │ │ │
│ [C1: req_count] [C2: t1_hit] [C3: t2_hit] [C4: t3_hit] [C5: budget_used] [C6: total_ms]
│ [C2a: t1_ms] [C3a: t2_ms] [C4a: t3_ms] [C5a: dropped_count]
│
│
RELEVANCE JUDGMENT (offline, periodic)
=====================================
Sampled Query Log ─── Replay Query ─── Retrieve Top-K ─── Qwen-7B Judge ─── Score (0-2) ─── Compute NDCG/MRR/Precision/Recall
│ │ │ │ │
[R1: sample_size] [R2: replay_ms] [R3: judge_ms] [R4: relevance_dist] [R5: ndcg_10]
[R3a: judge_cost] [R6: mrr]
[R7: precision_10]
[R8: recall_10]
```
---
## 1. Ingest Observability
### Why
Every fact written to memory becomes a retrieval candidate. A bad write — duplicate, contradictory, or malformed — pollutes all future queries. Ingest observability answers: **"How many facts are entering the system, how fast, and are any of them bad?"**
### Metrics
| ID | Metric | Type | Unit | Why It Matters |
|----|--------|------|------|----------------|
| **I1** | `ingest_requests_total` | Counter | requests | Total write demand. Capacity planning baseline. Sudden spikes = upstream behavior change. |
| **I2** | `ingest_auth_duration_seconds` | Histogram | seconds | JWT validation overhead. Should be < 5ms. Spike = JWKS fetch or Authentik down. |
| **I3** | `ingest_dedup_hits_total` | Counter | requests | Idempotency saves. High ratio = client retry storm or misconfigured source. Low = healthy unique writes. |
| **I4** | `ingest_embed_duration_seconds` | Histogram | seconds | Embedding latency per chunk. Budget: < 50ms for single chunk. Spike = model cold start or GPU contention. |
| **I5** | `ingest_dual_write_duration_seconds` | Histogram | seconds | Total time to write both stores. SLO: p99 < 500ms. |
| **I6** | `ingest_pgvector_duration_seconds` | Histogram | seconds | Postgres INSERT latency. Includes HNSW index update. Degrades as table grows. |
| **I7** | `ingest_opensearch_duration_seconds` | Histogram | seconds | OpenSearch bulk index latency. Sensitive to segment merges. |
| **I8** | `ingest_opensearch_failures_total` | Counter | failures | OpenSearch write failures. System is eventual-consistent: pgvector is primary. But if this counter grows, lexical search degrades silently. |
| **I9** | `ingest_bytes_total` | Counter | bytes | Total data volume written. Growth rate = storage budget burn. |
| **I10** | `ingest_chunks_total` | Counter | chunks | Write throughput in logical units. 1 ingest request may produce N chunks after splitting. |
| **I11** | `ingest_contradiction_detected_total` | Counter | contradictions | Facts that conflict with existing knowledge. High count = noisy source or domain shift. Each one enters review queue. |
| **I12** | `ingest_review_queue_depth` | Gauge | items | Pending human reviews. Growing = reviewers not keeping up. Stale contradictions = latent hallucination risk. |
### Alerts
| Condition | Severity | Action |
|-----------|----------|--------|
| `ingest_opensearch_failures_total` rate > 5/min for 10min | **Warning** | Check OpenSearch cluster health. Lexical search degrading. |
| `ingest_review_queue_depth` > 100 for 24h | **Warning** | Unreviewed contradictions. Risk of serving conflicting facts. |
| `ingest_pgvector_duration_seconds` p99 > 1s | **Critical** | Postgres overloaded. HNSW index rebuild or VACUUM needed. |
| `ingest_dedup_hits_total` / `ingest_requests_total` > 0.5 | **Warning** | More than half of writes are duplicates. Source misconfiguration. |
---
## 2. Query Observability
### Why
Query latency is what the user feels. But latency alone is insufficient — a fast query returning wrong results is worse than a slow correct one. Query observability answers: **"Did the system respond quickly, and did the search pipeline find the right documents?"**
### Metrics
| ID | Metric | Type | Unit | Why It Matters |
|----|--------|------|------|----------------|
| **Q1** | `query_requests_total` | Counter | requests | Read demand. Ratio to ingest = read/write skew. Memory systems are read-heavy (10:1+). |
| **Q2** | `query_auth_duration_seconds` | Histogram | seconds | Same as ingest. Shared auth path. |
| **Q3** | `query_intent_classification` | Counter (labeled) | requests | Labels: `bug_fix`, `how_to`, `reference`, `faq`. Distribution reveals what users ask most. If 80% is `bug_fix` but recall is low for that intent, prioritize that retrieval path. |
| **Q4** | `query_embed_duration_seconds` | Histogram | seconds | Query embedding latency. Same model as ingest. Should match I4. |
| **Q5** | `query_semantic_duration_seconds` | Histogram | seconds | pgvector cosine search. SLO: p99 < 200ms. Degrades with index size. |
| **Q5a** | `query_semantic_candidates` | Histogram | count | Number of vectors above similarity floor. Zero = total miss. Hundreds = floor too low. |
| **Q6** | `query_lexical_duration_seconds` | Histogram | seconds | OpenSearch BM25 latency. SLO: p99 < 150ms. |
| **Q6a** | `query_lexical_candidates` | Histogram | count | BM25 hit count. Zero = query terms not in corpus (vocabulary gap). |
| **Q7** | `query_rrf_fusion_duration_seconds` | Histogram | seconds | RRF merge time. Should be < 5ms (in-memory). If slow, too many candidates. |
| **Q8** | `query_rerank_duration_seconds` | Histogram | seconds | Cross-encoder reranking. Most expensive step. Budget: < 200ms for top-20. |
| **Q9** | `query_total_duration_seconds` | Histogram | seconds | End-to-end latency. SLO: p99 < 500ms. User-facing number. |
| **Q10** | `query_results_returned` | Histogram | count | How many results pass all filters. Zero = query miss. Track per-intent. |
| **Q11** | `query_empty_results_total` | Counter | requests | Queries that returned nothing. High rate = coverage gap in knowledge base. |
| **Q12** | `query_score_distribution` | Histogram | score (0-1) | Top-1 result score distribution. Bimodal = some queries match well, others poorly. Low mean = embedding quality issue. |
### Alerts
| Condition | Severity | Action |
|-----------|----------|--------|
| `query_total_duration_seconds` p99 > 1s | **Critical** | Pipeline bottleneck. Check Q5, Q6, Q8 to isolate which leg is slow. |
| `query_empty_results_total` rate > 20% of Q1 | **Warning** | 1 in 5 queries finds nothing. Coverage gap. Check if ingest is running. |
| `query_semantic_candidates` p50 = 0 | **Critical** | Embedding search broken. Model mismatch or empty index. |
| `query_lexical_duration_seconds` p99 > 500ms | **Warning** | OpenSearch overloaded. Check segment count, heap usage. |
---
## 3. Context Endpoint (Three-Tier) Observability
### Why
The context endpoint is the primary consumer-facing API. It orchestrates three retrieval tiers with budget constraints. Observing tier hit rates reveals whether the knowledge base has coverage at each level, and whether the budget assembly is dropping important results.
### Metrics
| ID | Metric | Type | Unit | Why It Matters |
|----|--------|------|------|----------------|
| **C1** | `context_requests_total` | Counter | requests | Context lookup demand. Main integration point. |
| **C2** | `context_tier1_hits_total` | Counter | hits | Exact signature matches. High = system is learning from repeated failures. SLO: tier-1 hit rate >= 0.80. |
| **C2a** | `context_tier1_duration_seconds` | Histogram | seconds | Signature lookup. Should be < 50ms (indexed hash). |
| **C3** | `context_tier2_hits_total` | Counter | hits | Hybrid search hits. Bulk of useful results. |
| **C3a** | `context_tier2_duration_seconds` | Histogram | seconds | Full hybrid search. Budget: < 500ms. |
| **C4** | `context_tier3_hits_total` | Counter | hits | Reference fallback. High ratio = learned knowledge insufficient, falling back to docs. |
| **C4a** | `context_tier3_duration_seconds` | Histogram | seconds | Obsidian API + reference retrieval. Slowest tier. |
| **C5** | `context_budget_used_bytes` | Histogram | bytes | How much of the token budget was consumed. Full = rich context. Low = sparse knowledge. |
| **C5a** | `context_dropped_results_total` | Counter | results | Results dropped to fit budget. High = budget too small or results too verbose. |
| **C6** | `context_total_duration_seconds` | Histogram | seconds | End-to-end context assembly. SLO: p99 < 2s. |
| **C7** | `context_tier_distribution` | Counter (labeled) | requests | Label: `tier=1\|2\|3`. Which tier served the primary result. Shift from tier-1 to tier-3 over time = knowledge decay. |
| **C8** | `context_degraded_total` | Counter | requests | Requests where a leg failed (e.g., Obsidian timeout). Partial results served. |
### Alerts
| Condition | Severity | Action |
|-----------|----------|--------|
| `context_tier1_hits_total` / `context_requests_total` < 0.60 | **Warning** | Signature match rate dropping. System not learning from failures. Check ingest pipeline. |
| `context_dropped_results_total` rate > 30% of results | **Warning** | Budget too tight. Users missing relevant context. |
| `context_degraded_total` rate > 5% | **Warning** | Partial responses. Check Obsidian API, OpenSearch health. |
---
## 4. Relevance Judgment with Qwen-7B
### Why
All the metrics above measure speed and volume. None measure **correctness**. A system that returns 10 results in 50ms is useless if those results are wrong. Traditional IR evaluation requires human-labeled relevance judgments — expensive and slow. Instead, we use a **Qwen-7B model as an automated relevance judge** on sampled queries.
This is the single most important observability signal for hallucination prevention. If retrieval precision drops, the LLM downstream gets wrong context and hallucinates. Catching it here — at the retrieval layer — is 10x cheaper than catching it at the generation layer.
### Why Qwen-7B
- **Cost**: ~0.002 USD per judgment. At 500 samples/day = $1/day. A 70B model costs 10x more for marginal gain.
- **Speed**: ~200ms per judgment on 1x A10. Fast enough for daily batch evaluation.
- **Accuracy**: 7B models achieve 85-90% agreement with human relevance labels on standard benchmarks (BEIR, MS MARCO). Sufficient for trend detection. We are not using it for absolute measurement — we are using it for **drift detection**.
- **Self-hosted**: Runs inside the cluster. No data leaves the network. Required for security-sensitive knowledge bases.
### Judgment Flow
```
DAILY RELEVANCE EVALUATION (Cron, 03:00 UTC)
============================================
Query Log (24h) ─── Sample 500 queries ─── Replay each query ─── Get top-10 results ─── For each (query, result) pair:
│ │
[R1: sample_size] Qwen-7B Prompt:
│
┌────────┴────────┐
│ "Given query: │
│ '{query}' │
│ │
│ Rate this │
│ result: │
│ '{result}' │
│ │
│ Score: │
│ 0 = irrelevant │
│ 1 = partial │
│ 2 = perfect │
└────────┬────────┘
│
[R4: score]
│
Aggregate: NDCG@10, MRR, Precision@10, Recall@10
│
┌───────────────┴───────────────┐
[R5: ndcg_10] [R7: precision_10]
[R6: mrr] [R8: recall_10]
│
Store in Postgres
(daily time-series)
│
Grafana Dashboard
(7-day rolling avg)
```
### Prompt Template
```
You are a relevance judge for a knowledge base system.
Given a user query and a retrieved document, rate the relevance:
- 0: Irrelevant. The document does not help answer the query at all.
- 1: Partially relevant. The document contains some useful information but does not fully answer the query.
- 2: Highly relevant. The document directly and completely answers the query.
Query: "{query}"
Retrieved Document:
---
{document_text}
---
Relevance Score (0, 1, or 2):
```
### Metrics
| ID | Metric | Type | Unit | Why It Matters |
|----|--------|------|------|----------------|
| **R1** | `relevance_sample_size` | Gauge | queries | Number of queries evaluated. 500 gives statistically stable NDCG with ±0.02 CI. |
| **R2** | `relevance_replay_duration_seconds` | Histogram | seconds | Time to replay and retrieve. Should match Q9. |
| **R3** | `relevance_judge_duration_seconds` | Histogram | seconds | Qwen-7B inference time per pair. Budget: < 300ms. |
| **R3a** | `relevance_judge_cost_usd` | Counter | USD | Running cost. Alert if budget exceeded. |
| **R4** | `relevance_score_distribution` | Histogram | score (0-2) | Distribution of judgments. Healthy: 60%+ score=2, < 15% score=0. Drift toward 0 = retrieval degradation. |
| **R5** | `relevance_ndcg_10` | Gauge | ratio (0-1) | Ranking quality. **Primary quality metric.** SLO: >= 0.85. Measures whether relevant docs appear at the top. |
| **R6** | `relevance_mrr` | Gauge | ratio (0-1) | Position of first relevant result. SLO: >= 0.80. If MRR drops but NDCG holds, results exist but are buried. |
| **R7** | `relevance_precision_10` | Gauge | ratio (0-1) | Fraction of top-10 that is relevant. Measures noise in results. |
| **R8** | `relevance_recall_10` | Gauge | ratio (0-1) | Fraction of all relevant docs captured in top-10. Low = knowledge exists but search can't find it. |
| **R9** | `relevance_judge_agreement` | Gauge | ratio (0-1) | Weekly: re-judge 50 pairs with human labels. Agreement rate validates the judge. SLO: >= 0.85. If agreement drops, Qwen model needs recalibration. |
### Alerts
| Condition | Severity | Action |
|-----------|----------|--------|
| `relevance_ndcg_10` 7-day avg < 0.80 | **Critical** | Retrieval quality degraded. Root cause: embedding drift, index corruption, or knowledge gap. |
| `relevance_ndcg_10` drops > 0.05 in 24h | **Critical** | Sudden quality drop. Check recent ingest for poisoned data. |
| `relevance_mrr` < 0.70 | **Warning** | Relevant docs exist but rank poorly. Check reranker, RRF weights. |
| `relevance_score_distribution` score=0 > 25% | **Warning** | Quarter of results are irrelevant. Coverage gap or embedding model mismatch. |
| `relevance_judge_agreement` < 0.80 | **Warning** | Judge drifting from human labels. Re-evaluate prompt or model. |
---
## 5. Write Volume and Storage Observability
### Why
Memory services grow unboundedly. Unlike caches (eviction policy) or databases (schema constraints), a knowledge base accumulates everything. Write volume tracking answers: **"How fast is the system growing, and when do we need to intervene?"**
Write volume also directly impacts retrieval quality. More documents = more noise in search results. Without compaction, precision degrades as the corpus grows.
### Metrics
| ID | Metric | Type | Unit | Why It Matters |
|----|--------|------|------|----------------|
| **W1** | `storage_pgvector_rows_total` | Gauge | rows | Total vectors stored. Growth rate = capacity planning. |
| **W2** | `storage_pgvector_bytes` | Gauge | bytes | Disk usage. 768-dim float32 = ~3KB/row with overhead. |
| **W3** | `storage_opensearch_docs_total` | Gauge | docs | OpenSearch document count. Should match W1 (eventual consistency). |
| **W4** | `storage_opensearch_bytes` | Gauge | bytes | OpenSearch index size. Includes inverted index overhead. |
| **W5** | `storage_parity_drift` | Gauge | count | abs(W1 - W3). Should be 0 in steady state. Non-zero = dual-write inconsistency. |
| **W6** | `write_rate_per_hour` | Gauge | chunks/hour | Sustained write throughput. Trigger compaction planning at > 1000/hour. |
| **W7** | `write_rate_per_project` | Gauge (labeled) | chunks/hour | Per-project write rate. Identifies hot projects dominating storage. |
| **W8** | `storage_level_distribution` | Gauge (labeled) | rows | Label: `level=L0\|L1\|L2\|R`. Distribution across learning levels. Healthy: L1 > L0 (facts promoted). If L0 dominates, promotion pipeline stalled. |
| **W9** | `compaction_runs_total` | Counter | runs | How often compaction executes. |
| **W10** | `compaction_dedup_removed_total` | Counter | chunks | Duplicates removed per run. High = ingest dedup isn't catching everything. |
| **W11** | `compaction_stale_gc_removed_total` | Counter | chunks | Stale facts garbage-collected (soft-deleted, age > 30d). |
| **W12** | `compaction_space_freed_bytes` | Counter | bytes | Space recovered per run. Declining = less to compact (good). |
### Alerts
| Condition | Severity | Action |
|-----------|----------|--------|
| `storage_parity_drift` > 100 for 1h | **Warning** | pgvector and OpenSearch out of sync. Check dual-write failures (I8). |
| `storage_pgvector_bytes` > 80% of PVC | **Critical** | Storage nearing capacity. Expand PVC or run compaction. |
| `write_rate_per_hour` > 5000 sustained 2h | **Warning** | High write load. Check if upstream is flooding. Consider rate limiting. |
| `storage_level_distribution{level="L0"}` / W1 > 0.7 | **Warning** | 70% of storage is unprocessed L0. Promotion pipeline stalled. |
---
## 6. Pod Resource Observability
### Why
The memory service runs as a Kubernetes pod. If the pod runs out of memory, it gets OOMKilled. If it saturates CPU, latency spikes across all endpoints. These are the physical constraints that gate everything else.
Unlike stateless web services, a memory service has **resident state**: the embedding model weights (~500MB for MiniLM-L6), connection pools, in-flight embeddings, and cached query results. Memory usage is not flat — it grows with concurrent requests. A burst of 50 parallel ingest requests each holding a 768-dim float32 vector = 50 × 3KB = 150KB just in vectors, but the surrounding allocations (HTTP buffers, serde frames, OpenSearch bulk payloads) multiply that 10-20x.
### Metrics
| ID | Metric | Type | Unit | Why It Matters |
|----|--------|------|------|----------------|
| **P1** | `container_memory_working_set_bytes` | Gauge | bytes | Actual memory in use (excludes reclaimable cache). This is what Kubernetes uses for OOMKill decisions. |
| **P2** | `container_memory_rss` | Gauge | bytes | Resident Set Size. Physical memory held. If RSS diverges from working set, fragmentation is occurring. |
| **P3** | `container_memory_usage_bytes` | Gauge | bytes | Total memory (includes page cache). Less useful for OOM prediction but shows total footprint. |
| **P4** | `container_memory_limit_bytes` | Gauge | bytes | Pod memory limit from resource spec. `P1 / P4` = memory pressure ratio. |
| **P5** | `container_cpu_usage_seconds_total` | Counter | CPU-seconds | CPU consumption rate. `rate(P5[1m])` = CPU cores used. Compare to limit. |
| **P6** | `container_cpu_throttled_seconds_total` | Counter | seconds | Time the pod was CPU-throttled by cgroup. Any throttling = latency impact. |
| **P7** | `container_cpu_cfs_throttled_periods_total` | Counter | periods | Number of CFS periods where throttling occurred. `P7 / total_periods` = throttle ratio. |
| **P8** | `kube_pod_container_resource_requests` | Gauge | cores/bytes | Requested resources. Over-request wastes cluster capacity. Under-request = eviction risk. |
| **P9** | `kube_pod_container_resource_limits` | Gauge | cores/bytes | Resource limits. `P1 / P9{resource="memory"}` > 0.85 = danger zone. |
| **P10** | `kube_pod_status_phase` | Gauge | phase | Running/Pending/Failed/Succeeded. Pending too long = scheduling issues. |
| **P11** | `kube_pod_container_status_restarts_total` | Counter | restarts | OOMKills and CrashLoopBackoff. Any restart = data in flight was lost. |
| **P12** | `container_network_receive_bytes_total` | Counter | bytes | Network ingress. Correlate with ingest volume. Spike = large batch ingest. |
| **P13** | `container_network_transmit_bytes_total` | Counter | bytes | Network egress. Correlate with query response sizes. |
### Memory Breakdown (What Lives in the Pod)
```
Pod Memory Budget (e.g., 2Gi limit)
├── Embedding Model weights ~500MB (loaded once at startup)
├── sqlx connection pool ~50MB (20 connections × ~2.5MB each)
├── OpenSearch HTTP client pool ~20MB (keep-alive connections)
├── In-flight ingest embeddings ~variable (concurrent_requests × ~60KB)
├── In-flight query results ~variable (concurrent_queries × ~200KB)
├── Tokio runtime + thread stacks ~30MB (worker threads × 8MB stack)
├── Rate limiter buckets ~5MB (in-memory token buckets)
├── Idempotency store (24h TTL) ~10-50MB (grows with ingest volume)
└── Heap overhead + fragmentation ~100-200MB
─────────
~800MB baseline + ~variable per-request
```
### Alerts
| Condition | Severity | Action |
|-----------|----------|--------|
| `P1 / P4` > 0.85 for 5min | **Critical** | Memory pressure. OOMKill imminent. Scale up limit or reduce concurrency. |
| `P11` increments | **Critical** | Pod restarted. Check if OOMKilled (`kubectl describe pod`). Raise memory limit. |
| `rate(P6[5m])` > 0 for 10min | **Warning** | Sustained CPU throttling. Query/ingest latency affected. Raise CPU limit. |
| `P7 / total_periods` > 0.25 | **Warning** | 25%+ of CPU periods throttled. Under-provisioned. |
| `P1` growing monotonically over 24h | **Warning** | Memory leak. Check idempotency store TTL, connection pool, or embedding cache. |
---
## 7. Availability
### Why
A knowledge base that is down cannot reduce hallucination. If the memory service is unavailable during an LLM generation call, the model falls back to parametric knowledge only — which is exactly where hallucinations come from. Availability is not just uptime; it is **the probability that a query gets a correct answer within the latency SLO**.
### Metrics
| ID | Metric | Type | Unit | Why It Matters |
|----|--------|------|------|----------------|
| **A1** | `http_requests_total` | Counter (labeled) | requests | Label: `method`, `endpoint`, `status_code`. Foundation for error rate calculation. |
| **A2** | `http_requests_duration_seconds` | Histogram (labeled) | seconds | Label: `endpoint`. Per-endpoint latency distribution. |
| **A3** | `http_5xx_total` | Counter | requests | Server errors. Any 5xx = something broke internally. |
| **A4** | `http_4xx_total` | Counter (labeled) | requests | Label: `status_code`. 401/403 = auth issues. 429 = rate limiting. 400 = bad client. |
| **A5** | `availability_ratio` | Gauge | ratio (0-1) | `1 - (A3 / A1)` over rolling window. SLO: >= 0.999 (three nines). |
| **A6** | `successful_query_ratio` | Gauge | ratio (0-1) | Queries that return 200 with >= 1 result, within 500ms. Stricter than raw availability — includes quality. |
| **A7** | `health_check_consecutive_failures` | Gauge | count | Consecutive `/health` failures. Kubernetes uses this for restart decisions (liveness probe). |
| **A8** | `dependency_up` | Gauge (labeled) | 0/1 | Label: `dependency=postgres\|opensearch\|obsidian\|embedding_model`. Which backends are reachable. |
| **A9** | `graceful_degradation_total` | Counter (labeled) | requests | Label: `degraded_component`. Requests served with partial results because a dependency was down. e.g., OpenSearch down = semantic-only results. |
| **A10** | `circuit_breaker_state` | Gauge (labeled) | 0/1/2 | Label: `backend`. 0=closed (healthy), 1=half-open (probing), 2=open (failing). Per dependency. |
### Availability Calculation
```
Successful Requests (2xx, within SLO latency)
Availability = ────────────────────────────────────────────────────
Total Requests
Three tiers of availability:
1. RAW AVAILABILITY: 1 - (5xx / total) Target: 99.9%
"Did it respond?"
2. LATENCY AVAILABILITY: requests_within_slo / total Target: 99.5%
"Did it respond fast enough?"
3. QUALITY AVAILABILITY: queries_with_results / total Target: 95%
"Did it respond with useful results?"
Monitor all three. A system can be 99.9% available (raw) but only
80% available (quality) if 20% of queries return empty results.
```
### Alerts
| Condition | Severity | Action |
|-----------|----------|--------|
| `availability_ratio` < 0.999 over 1h | **Critical** | SLO breach. Page on-call. Check A8 for which dependency is down. |
| `http_5xx_total` rate > 10/min for 5min | **Critical** | Error spike. Check pod logs, Postgres connectivity, OpenSearch health. |
| `dependency_up{dependency="postgres"}` = 0 | **Critical** | Primary store down. All writes and most reads fail. |
| `dependency_up{dependency="opensearch"}` = 0 | **Warning** | Lexical search unavailable. Semantic-only fallback active. Quality degraded. |
| `graceful_degradation_total` rate > 5% of A1 | **Warning** | Serving partial results too often. Fix the degraded dependency. |
| `successful_query_ratio` < 0.90 | **Warning** | 10%+ of queries failing or empty. Check ingest pipeline, index health. |
---
## 8. Ingest Rate Patterns
### Why
Ingest rate is not just a throughput number. The **pattern** of writes reveals system behavior. Bursty writes from batch jobs behave differently from steady trickle from live sessions. A sudden drop in ingest rate may mean the upstream source broke. A sudden spike may mean a replay or backfill is running, which changes storage projections.
For a knowledge base, write rate directly affects retrieval quality: every new chunk is a new candidate that can dilute search precision. Knowing when and how fast writes happen lets you plan compaction, predict storage growth, and detect anomalies.
### Metrics
| ID | Metric | Type | Unit | Why It Matters |
|----|--------|------|------|----------------|
| **IR1** | `ingest_rate_1m` | Gauge | chunks/min | 1-minute rolling write rate. Shows bursts. |
| **IR2** | `ingest_rate_1h` | Gauge | chunks/hour | Hourly smoothed rate. Capacity planning baseline. |
| **IR3** | `ingest_rate_by_project` | Gauge (labeled) | chunks/hour | Label: `project`. Identifies which project dominates writes. |
| **IR4** | `ingest_rate_by_level` | Gauge (labeled) | chunks/hour | Label: `level=L0\|L1\|L2\|R`. L0 dominance = raw data flooding. L1/L2 growing = healthy knowledge promotion. |
| **IR5** | `ingest_rate_by_source` | Gauge (labeled) | chunks/hour | Label: `source=transcript\|document\|api\|batch`. Reveals upstream behavior. |
| **IR6** | `ingest_batch_size` | Histogram | chunks/batch | Size of batch ingest requests. Large batches (>100) need different backpressure. |
| **IR7** | `ingest_queue_depth` | Gauge | messages | External queue (kmsvc) pending messages. Growing = workers can't keep up. |
| **IR8** | `ingest_queue_age_seconds` | Histogram | seconds | Age of oldest message in queue. > 60s = processing lag. |
| **IR9** | `ingest_bytes_per_chunk` | Histogram | bytes | Average chunk size. Sudden increase = source sending larger payloads. |
| **IR10** | `ingest_throughput_bytes_per_second` | Gauge | bytes/sec | Sustained write bandwidth. Correlate with P12 (network ingress). |
### Rate Patterns and What They Mean
```
Pattern 1: STEADY TRICKLE (healthy)
────────────────────────────────────
chunks/min
10 │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
5 │
0 └──────────────────────────── time
Constant ~8-12 chunks/min from live sessions.
Storage growth predictable. Compaction schedule stable.
Pattern 2: BURST (batch job or backfill)
────────────────────────────────────────
chunks/min
500 │ ██
250 │ ██████
0 │──────██──────██─────────── time
Sudden spike. Check: is this a planned backfill?
If unexpected: rate limit may trigger, queue depth spikes.
Action: verify source, check queue lag (IR7).
Pattern 3: DROP TO ZERO (upstream broken)
─────────────────────────────────────────
chunks/min
10 │ ─ ─ ─ ─ ┐
5 │ │
0 │ └──────────────── time
Ingest stopped. Source may be down, auth token expired,
or network partition. Silent failure — no errors, just absence.
Alert on: IR2 = 0 for > 30min during business hours.
Pattern 4: MONOTONIC GROWTH (runaway source)
─────────────────────────────────────────────
chunks/min
100 │ ╱
50 │ ╱───
10 │ ─ ─ ─ ─ ─ ─ ╱───
0 └──────────────────────────── time
Write rate increasing over days. Source producing more data.
Storage projection changes. Compaction may not keep up.
Action: review source, consider sampling or filtering.
```
### Alerts
| Condition | Severity | Action |
|-----------|----------|--------|
| `ingest_rate_1h` = 0 for 30min (during business hours) | **Warning** | Ingest stopped. Check upstream source, auth tokens, network. |
| `ingest_rate_1m` > 200 sustained 10min | **Warning** | Burst ingest. Check if planned. Monitor queue depth (IR7). |
| `ingest_queue_depth` > 1000 for 15min | **Critical** | Workers can't keep up. Scale workers or throttle source. |
| `ingest_queue_age_seconds` p99 > 300 | **Warning** | 5+ minutes processing lag. Stale data entering the system. |
| `ingest_rate_by_level{level="L0"}` / total > 0.9 sustained 24h | **Warning** | 90% raw data, no promotion. Knowledge extraction pipeline stalled. |
---
## 9. Postgres Internal Observability
### Why
Postgres is the primary store. Every vector lives there. Every query hits it. Postgres health directly determines memory service health. But Postgres problems are **silent** — a bloated table doesn't throw errors, it just gets slower. A missing VACUUM doesn't alert, it just consumes 2x disk. An HNSW index with wrong parameters doesn't fail, it just returns worse results.
These metrics catch degradation before users notice it.
### Connection Pool and Session Metrics
| ID | Metric | Source | Unit | Why It Matters |
|----|--------|--------|------|----------------|
| **PG1** | `pg_stat_activity_count` | `pg_stat_activity` | connections | Active connections by state. `active` = running query. `idle` = waiting. `idle in transaction` = **dangerous** — holds locks. |
| **PG2** | `pg_stat_activity_max_duration_seconds` | `pg_stat_activity` | seconds | Longest running query. > 30s = likely stuck or missing index. |
| **PG3** | `pg_stat_activity_waiting_count` | `pg_stat_activity` | connections | Queries waiting for locks. > 0 sustained = lock contention. |
| **PG4** | `pg_settings_max_connections` | `pg_settings` | connections | Max allowed connections. `PG1 / PG4` > 0.8 = pool exhaustion risk. |
### Query Performance
| ID | Metric | Source | Unit | Why It Matters |
|----|--------|--------|------|----------------|
| **PG5** | `pg_stat_statements_mean_exec_time` | `pg_stat_statements` | ms | Mean execution time per query pattern. Tracks if vector search is degrading over time. |
| **PG6** | `pg_stat_statements_calls` | `pg_stat_statements` | count | Call count per query. Identifies hot queries. Top-1 query consuming 80% of DB time = optimization target. |
| **PG7** | `pg_stat_statements_rows` | `pg_stat_statements` | rows | Rows returned per query. Vector search returning 10k rows when limit is 50 = missing index or wrong query plan. |
| **PG8** | `pg_stat_user_tables_seq_scan` | `pg_stat_user_tables` | scans | Sequential scans on `memory_vector`. Any seq scan on a large vector table = catastrophic. HNSW index not being used. |
| **PG9** | `pg_stat_user_tables_idx_scan` | `pg_stat_user_tables` | scans | Index scans. Should be >> seq scans for vector table. |
### Table and Index Health
| ID | Metric | Source | Unit | Why It Matters |
|----|--------|--------|------|----------------|
| **PG10** | `pg_stat_user_tables_n_live_tup` | `pg_stat_user_tables` | tuples | Live rows in `memory_vector`. Growth rate = storage planning. |
| **PG11** | `pg_stat_user_tables_n_dead_tup` | `pg_stat_user_tables` | tuples | Dead tuples (deleted/updated but not vacuumed). High ratio = bloat. |
| **PG12** | `pg_dead_tuple_ratio` | computed | ratio | `PG11 / (PG10 + PG11)`. > 0.2 = 20% bloat. VACUUM needed. |
| **PG13** | `pg_stat_user_tables_last_autovacuum` | `pg_stat_user_tables` | timestamp | When autovacuum last ran. > 24h ago on active table = misconfigured threshold. |
| **PG14** | `pg_stat_user_tables_last_autoanalyze` | `pg_stat_user_tables` | timestamp | When autoanalyze last ran. Stale statistics = bad query plans. |
| **PG15** | `pg_table_size_bytes` | `pg_total_relation_size()` | bytes | Total table size including indexes and TOAST. |
| **PG16** | `pg_index_size_bytes` | `pg_indexes_size()` | bytes | HNSW index size. Grows with vectors. If index > table, check parameters. |
| **PG17** | `pg_index_bloat_ratio` | `pgstattuple` | ratio | Index bloat. > 0.3 = REINDEX needed. HNSW indexes don't bloat like B-tree, but monitor anyway. |
### HNSW Index Specific
| ID | Metric | Source | Unit | Why It Matters |
|----|--------|--------|------|----------------|
| **PG18** | `pg_hnsw_index_size` | `pg_relation_size()` | bytes | Size of the HNSW index on `memory_vector.embedding`. Grows as O(n × m) where m=16. |
| **PG19** | `pg_hnsw_build_time_seconds` | manual / `CREATE INDEX` | seconds | Time to rebuild HNSW index. Needed after parameter changes. At 1M vectors: ~30min. At 10M: hours. Plan maintenance windows. |
| **PG20** | `pg_hnsw_recall_estimate` | benchmark | ratio | Estimated recall of HNSW at current parameters (m=16, ef_construction=200). Run periodic benchmark with known queries. If recall < 0.95, increase ef_search or rebuild with higher m. |
### WAL and Replication (CNPG)
| ID | Metric | Source | Unit | Why It Matters |
|----|--------|--------|------|----------------|
| **PG21** | `pg_wal_lsn_diff` | `pg_current_wal_lsn()` | bytes | WAL generation rate. High during bulk ingest. Correlate with IR1. |
| **PG22** | `pg_replication_lag_bytes` | `pg_stat_replication` | bytes | Replica lag in bytes. CNPG manages replicas. Lag > 100MB = replica falling behind. |
| **PG23** | `pg_replication_lag_seconds` | `pg_stat_replication` | seconds | Replica lag in time. > 10s = replica can't keep up with write rate. Read queries to replica return stale results. |
| **PG24** | `pg_wal_size_bytes` | `pg_wal` directory | bytes | Total WAL on disk. Unbounded growth = archiving broken or wal_keep_size too high. |
### Transaction and Lock Health
| ID | Metric | Source | Unit | Why It Matters |
|----|--------|--------|------|----------------|
| **PG25** | `pg_stat_database_xact_commit` | `pg_stat_database` | transactions | Committed transactions/sec. Baseline throughput. |
| **PG26** | `pg_stat_database_xact_rollback` | `pg_stat_database` | transactions | Rolled back transactions. `PG26 / PG25` > 0.01 = 1% rollback rate. Check constraint violations or deadlocks. |
| **PG27** | `pg_stat_database_deadlocks` | `pg_stat_database` | deadlocks | Any deadlock = concurrent write contention. Rare in append-mostly workload. If seen, check compaction + ingest overlap. |
| **PG28** | `pg_stat_database_conflicts` | `pg_stat_database` | conflicts | Replication conflicts. Query on replica canceled due to WAL replay. Adjust `max_standby_streaming_delay`. |
| **PG29** | `pg_locks_count` | `pg_locks` | locks | Lock count by mode. `AccessExclusiveLock` blocks everything — check for DDL during traffic. |
### Cache Efficiency
| ID | Metric | Source | Unit | Why It Matters |
|----|--------|--------|------|----------------|
| **PG30** | `pg_stat_database_blks_hit` | `pg_stat_database` | blocks | Buffer cache hits. |
| **PG31** | `pg_stat_database_blks_read` | `pg_stat_database` | blocks | Disk reads (cache misses). |
| **PG32** | `pg_cache_hit_ratio` | computed | ratio | `PG30 / (PG30 + PG31)`. SLO: >= 0.99. Below 0.95 = shared_buffers too small or working set exceeds RAM. |
| **PG33** | `pg_stat_user_indexes_idx_blks_hit` | `pg_stat_user_indexes` | blocks | HNSW index cache hits. Low hit ratio = index doesn't fit in memory. Increase shared_buffers or effective_cache_size. |
### Key SQL Queries for Monitoring
```sql
-- Dead tuple ratio (bloat indicator)
SELECT relname,
n_live_tup,
n_dead_tup,
CASE WHEN n_live_tup > 0
THEN round(n_dead_tup::numeric / (n_live_tup + n_dead_tup) * 100, 2)
ELSE 0 END AS dead_pct,
last_autovacuum,
last_autoanalyze
FROM pg_stat_user_tables
WHERE relname IN ('memory_vector', 'memory_entity', 'memory_edge')
ORDER BY n_dead_tup DESC;
-- Slowest queries (requires pg_stat_statements)
SELECT query,
calls,
round(mean_exec_time::numeric, 2) AS mean_ms,
round(max_exec_time::numeric, 2) AS max_ms,
rows
FROM pg_stat_statements
WHERE dbid = (SELECT oid FROM pg_database WHERE datname = 'memory')
ORDER BY mean_exec_time DESC
LIMIT 10;
-- Sequential vs index scans (vector table must use index)
SELECT relname,
seq_scan,
idx_scan,
CASE WHEN (seq_scan + idx_scan) > 0
THEN round(idx_scan::numeric / (seq_scan + idx_scan) * 100, 2)
ELSE 100 END AS idx_scan_pct
FROM pg_stat_user_tables
WHERE relname = 'memory_vector';
-- Table and index sizes
SELECT relname,
pg_size_pretty(pg_total_relation_size(relid)) AS total_size,
pg_size_pretty(pg_relation_size(relid)) AS table_size,
pg_size_pretty(pg_indexes_size(relid)) AS index_size
FROM pg_stat_user_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(relid) DESC;
-- Replication lag (CNPG replicas)
SELECT client_addr,
state,
pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS lag_bytes,
extract(epoch FROM now() - replay_lag) AS lag_seconds
FROM pg_stat_replication;
-- Cache hit ratio
SELECT datname,
round(
blks_hit::numeric / NULLIF(blks_hit + blks_read, 0) * 100, 2
) AS cache_hit_pct
FROM pg_stat_database
WHERE datname = 'memory';
-- Connection state breakdown
SELECT state, count(*)
FROM pg_stat_activity
WHERE datname = 'memory'
GROUP BY state;
```
### Alerts
| Condition | Severity | Action |
|-----------|----------|--------|
| `pg_dead_tuple_ratio` > 0.20 on `memory_vector` | **Warning** | 20% bloat. Run `VACUUM ANALYZE memory_vector;` or check autovacuum config. |
| `pg_stat_user_tables_seq_scan` on `memory_vector` increments | **Critical** | Sequential scan on vector table. HNSW index not used. Check query plan with `EXPLAIN ANALYZE`. |
| `pg_cache_hit_ratio` < 0.95 | **Critical** | Cache thrashing. Increase `shared_buffers` or scale to larger instance. |
| `pg_replication_lag_seconds` > 30 | **Warning** | Replica 30s behind. Read queries returning stale data. Check write rate, replica resources. |
| `pg_stat_database_deadlocks` > 0 | **Warning** | Deadlock detected. Check concurrent write patterns (ingest + compaction). |
| `pg_stat_activity_max_duration_seconds` > 60 | **Warning** | Query running > 60s. Likely stuck. Check for missing index or lock wait. |
| `pg_stat_activity_count{state="idle in transaction"}` > 5 for 10min | **Warning** | Idle-in-transaction connections holding locks. Connection pool leak or application bug. |
| `pg_wal_size_bytes` > 10GB | **Warning** | WAL accumulation. Check archiving, replication, or `wal_keep_size` setting. |
| `PG1 / PG4` > 0.8 | **Critical** | Connection pool near max. Add PgBouncer or increase `max_connections`. |
---
## 10. System Health (Infrastructure Summary)
### Metrics
| ID | Metric | Type | Unit | Why It Matters |
|----|--------|------|------|----------------|
| **H1** | `health_check_status` | Gauge | 0/1 | `/health` endpoint. Basic liveness. |
| **H2** | `pgvector_connection_pool_active` | Gauge | connections | Active DB connections. Near max = pool exhaustion risk. |
| **H3** | `pgvector_connection_pool_idle` | Gauge | connections | Idle connections. Zero idle + high active = under-provisioned. |
| **H4** | `opensearch_cluster_status` | Gauge | 0/1/2 | 0=red, 1=yellow, 2=green. Yellow = replica missing. Red = data loss risk. |
| **H5** | `embedding_model_loaded` | Gauge | 0/1 | Model health check. 0 = all ingest and query embeds will fail. |
| **H6** | `rate_limit_rejections_total` | Counter | requests | Rate limit hits. High = legitimate traffic being blocked, or DDoS. |
| **H7** | `auth_failures_total` | Counter (labeled) | requests | Label: `reason=expired\|invalid\|missing`. Pattern reveals attack or misconfiguration. |
---
## 11. Dashboard Layout
### Grafana Rows (top to bottom)
```
Row 1: SYSTEM HEALTH
┌──────────────┬──────────────┬──────────────┬──────────────┐
│ Health: UP │ PG Pool: │ OpenSearch: │ Embed Model │
│ (H1) │ 12/20 active│ GREEN │ LOADED │
└──────────────┴──────────────┴──────────────┴──────────────┘
Row 2: WRITE PATH (Ingest)
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ Ingest Rate (I1) │ Write Latency p50/p99 │ Dedup Hit Ratio │
│ [line chart, 24h] │ (I5) [line chart, 24h] │ (I3/I1) [line, 24h] │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ OpenSearch Failures (I8)│ Contradiction Queue (I12)│ Chunks Written (I10) │
│ [counter, 24h] │ [gauge, current depth] │ [counter, 24h] │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
Row 3: READ PATH (Query)
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ Query Rate (Q1) │ Query Latency p50/p99 │ Empty Results (Q11) │
│ [line chart, 24h] │ (Q9) [line chart, 24h] │ [%, 24h] │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Latency Breakdown │ Intent Distribution │ Score Distribution │
│ sem/lex/rrf/rerank │ (Q3) [pie chart] │ (Q12) [histogram] │
│ [stacked area, 24h] │ │ │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
Row 4: CONTEXT (3-Tier)
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ Tier Hit Distribution │ Context Latency p50/p99 │ Budget Usage │
│ (C7) [stacked bar, 7d] │ (C6) [line chart, 24h] │ (C5) [histogram] │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Tier-1 Hit Rate │ Degraded Responses (C8) │ Dropped Results (C5a) │
│ (C2/C1) [gauge, target │ [counter, 24h] │ [counter, 24h] │
│ >= 0.80] │ │ │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
Row 5: RELEVANCE (Qwen-7B Judge) ← MOST IMPORTANT ROW
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ NDCG@10 (R5) │ MRR (R6) │ Precision/Recall │
│ [line chart, 30d │ [line chart, 30d │ (R7, R8) [line, 30d │
│ rolling avg, target │ rolling avg] │ rolling avg] │
│ >= 0.85] │ │ │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Relevance Score Dist │ Judge Cost (R3a) │ Judge Agreement (R9) │
│ (R4) [bar: 0/1/2, 7d] │ [counter, daily USD] │ [gauge, weekly] │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
Row 6: POD RESOURCES
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ Memory Usage (P1) │ CPU Usage (P5 rate) │ Pod Restarts (P11) │
│ [line, 24h, limit line] │ [line, 24h, limit line] │ [counter, 7d] │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Memory Pressure (P1/P4) │ CPU Throttle (P6 rate) │ Network I/O (P12, P13) │
│ [gauge, target < 0.85] │ [line, 24h] │ [line, 24h] │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
Row 7: AVAILABILITY
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ Availability (A5) │ Error Rate (A3/A1) │ Dependencies (A8) │
│ [gauge, target >= 99.9%]│ [line, 24h] │ [status grid: pg/os/emb]│
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Quality Avail (A6) │ Degraded Responses (A9) │ 4xx Breakdown (A4) │
│ [gauge, target >= 95%] │ [counter, 24h] │ [stacked bar, 24h] │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
Row 8: INGEST RATE PATTERNS
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ Write Rate/Min (IR1) │ Write Rate/Hour (IR2) │ Queue Depth (IR7) │
│ [line, 24h, burst high] │ [line, 7d] │ [gauge, target < 1000] │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Rate by Project (IR3) │ Rate by Level (IR4) │ Queue Age p99 (IR8) │
│ [stacked area, 24h] │ [stacked area, 24h] │ [line, 24h] │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
Row 9: POSTGRES INTERNALS
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ Cache Hit Ratio (PG32) │ Dead Tuple Ratio (PG12) │ Connections (PG1) │
│ [gauge, target >= 99%] │ [gauge, target < 20%] │ [stacked bar by state] │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Seq vs Idx Scans (PG8/9)│ Replication Lag (PG23) │ Table Sizes (PG15) │
│ [line, 7d] │ [line, 24h, target < 10s]│ [bar chart, current] │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Slowest Queries (PG5) │ WAL Size (PG24) │ Deadlocks (PG27) │
│ [table, top 5] │ [line, 7d] │ [counter, 30d] │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
Row 10: STORAGE
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ Total Vectors (W1) │ Storage Bytes │ Write Rate/Hour (W6) │
│ [gauge, current] │ (W2+W4) [line, 30d] │ [line chart, 24h] │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Parity Drift (W5) │ Level Distribution (W8) │ Compaction Freed (W12) │
│ [gauge, target = 0] │ [stacked bar] │ [counter per run] │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
```
---
## 12. Implementation: Prometheus Metrics in Rust
```rust
use prometheus::{
register_counter, register_counter_vec, register_gauge, register_gauge_vec,
register_histogram, register_histogram_vec,
Counter, CounterVec, Gauge, GaugeVec, Histogram, HistogramVec,
};
use lazy_static::lazy_static;
lazy_static! {
// === INGEST ===
pub static ref INGEST_REQUESTS: Counter =
register_counter!("memory_ingest_requests_total", "Total ingest requests").unwrap();
pub static ref INGEST_CHUNKS: Counter =
register_counter!("memory_ingest_chunks_total", "Total chunks written").unwrap();
pub static ref INGEST_BYTES: Counter =
register_counter!("memory_ingest_bytes_total", "Total bytes ingested").unwrap();
pub static ref INGEST_DEDUP_HITS: Counter =
register_counter!("memory_ingest_dedup_hits_total", "Deduplicated chunks skipped").unwrap();
pub static ref INGEST_CONTRADICTIONS: Counter =
register_counter!("memory_ingest_contradictions_total", "Contradictions detected").unwrap();
pub static ref INGEST_EMBED_DURATION: Histogram =
register_histogram!("memory_ingest_embed_seconds", "Embedding latency per chunk",
vec![0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0]).unwrap();
pub static ref INGEST_PGVECTOR_DURATION: Histogram =
register_histogram!("memory_ingest_pgvector_seconds", "pgvector write latency",
vec![0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0]).unwrap();
pub static ref INGEST_OPENSEARCH_DURATION: Histogram =
register_histogram!("memory_ingest_opensearch_seconds", "OpenSearch index latency",
vec![0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0]).unwrap();
pub static ref INGEST_OPENSEARCH_FAILURES: Counter =
register_counter!("memory_ingest_opensearch_failures_total", "OpenSearch write failures").unwrap();
pub static ref REVIEW_QUEUE_DEPTH: Gauge =
register_gauge!("memory_review_queue_depth", "Pending contradiction reviews").unwrap();
// === QUERY ===
pub static ref QUERY_REQUESTS: Counter =
register_counter!("memory_query_requests_total", "Total query requests").unwrap();
pub static ref QUERY_EMPTY_RESULTS: Counter =
register_counter!("memory_query_empty_results_total", "Queries returning zero results").unwrap();
pub static ref QUERY_TOTAL_DURATION: Histogram =
register_histogram!("memory_query_total_seconds", "End-to-end query latency",
vec![0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0]).unwrap();
pub static ref QUERY_SEMANTIC_DURATION: Histogram =
register_histogram!("memory_query_semantic_seconds", "pgvector search latency",
vec![0.01, 0.025, 0.05, 0.1, 0.2, 0.5]).unwrap();
pub static ref QUERY_LEXICAL_DURATION: Histogram =
register_histogram!("memory_query_lexical_seconds", "OpenSearch BM25 latency",
vec![0.01, 0.025, 0.05, 0.1, 0.2, 0.5]).unwrap();
pub static ref QUERY_INTENT: CounterVec =
register_counter_vec!("memory_query_intent_total", "Query intent classification",
&["intent"]).unwrap();
pub static ref QUERY_RESULTS_COUNT: Histogram =
register_histogram!("memory_query_results_count", "Results returned per query",
vec![0.0, 1.0, 3.0, 5.0, 10.0, 20.0, 50.0]).unwrap();
pub static ref QUERY_TOP1_SCORE: Histogram =
register_histogram!("memory_query_top1_score", "Top-1 result similarity score",
vec![0.3, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 1.0]).unwrap();
// === CONTEXT ===
pub static ref CONTEXT_REQUESTS: Counter =
register_counter!("memory_context_requests_total", "Total context lookups").unwrap();
pub static ref CONTEXT_TIER_HITS: CounterVec =
register_counter_vec!("memory_context_tier_hits_total", "Hits per tier",
&["tier"]).unwrap();
pub static ref CONTEXT_TOTAL_DURATION: Histogram =
register_histogram!("memory_context_total_seconds", "End-to-end context latency",
vec![0.1, 0.25, 0.5, 1.0, 2.0, 5.0]).unwrap();
pub static ref CONTEXT_DROPPED: Counter =
register_counter!("memory_context_dropped_results_total", "Results dropped for budget").unwrap();
// === RELEVANCE (updated daily by batch job) ===
pub static ref RELEVANCE_NDCG: Gauge =
register_gauge!("memory_relevance_ndcg_10", "NDCG@10 from Qwen-7B judge").unwrap();
pub static ref RELEVANCE_MRR: Gauge =
register_gauge!("memory_relevance_mrr", "Mean Reciprocal Rank").unwrap();
pub static ref RELEVANCE_PRECISION: Gauge =
register_gauge!("memory_relevance_precision_10", "Precision@10").unwrap();
pub static ref RELEVANCE_RECALL: Gauge =
register_gauge!("memory_relevance_recall_10", "Recall@10").unwrap();
// === STORAGE ===
pub static ref STORAGE_VECTORS: Gauge =
register_gauge!("memory_storage_vectors_total", "Total vectors in pgvector").unwrap();
pub static ref STORAGE_BYTES: Gauge =
register_gauge!("memory_storage_bytes", "Total storage bytes (pg + os)").unwrap();
pub static ref STORAGE_PARITY_DRIFT: Gauge =
register_gauge!("memory_storage_parity_drift", "pgvector vs OpenSearch doc count difference").unwrap();
pub static ref WRITE_RATE: Gauge =
register_gauge!("memory_write_rate_per_hour", "Current write rate (chunks/hour)").unwrap();
}
```
### Instrumentation Example (Ingest Handler)
```rust
pub async fn ingest_handler(req: HttpRequest, body: web::Json<IngestRequest>, state: web::Data<AppState>) -> HttpResponse {
INGEST_REQUESTS.inc();
// Auth
let auth_timer = INGEST_AUTH_DURATION.start_timer();
let (claims, token) = match validate_auth(&req, &state).await { ... };
auth_timer.observe_duration();
// Dedup
if state.idempotency_store.is_duplicate(&body.idempotency_key) {
INGEST_DEDUP_HITS.inc();
return HttpResponse::Ok().json(json!({"status": "duplicate"}));
}
// Embed
let embed_timer = INGEST_EMBED_DURATION.start_timer();
let embedding = state.embeddings.embed_one(&body.text).await?;
embed_timer.observe_duration();
// pgvector write
let pg_timer = INGEST_PGVECTOR_DURATION.start_timer();
state.vector_store.insert(&body.project, &body.text, &embedding).await?;
pg_timer.observe_duration();
// OpenSearch write
let os_timer = INGEST_OPENSEARCH_DURATION.start_timer();
match state.opensearch_client.index_document(...).await {
Ok(_) => {},
Err(e) => {
INGEST_OPENSEARCH_FAILURES.inc();
tracing::warn!("OpenSearch write failed (non-blocking): {}", e);
}
}
os_timer.observe_duration();
INGEST_CHUNKS.inc();
INGEST_BYTES.inc_by(body.text.len() as f64);
HttpResponse::Created().json(...)
}
```
---
## 13. Relevance Evaluation CronJob
```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: memory-relevance-eval
namespace: poimen
spec:
schedule: "0 3 * * *" # Daily at 03:00 UTC
jobTemplate:
spec:
template:
spec:
containers:
- name: relevance-eval
image: forgejo.riotpiao.com/rock/poimen-memory:latest
command: ["mem", "evaluate-relevance"]
env:
- name: EVAL_SAMPLE_SIZE
value: "500"
- name: EVAL_JUDGE_MODEL
value: "qwen2.5-7b"
- name: EVAL_JUDGE_ENDPOINT
value: "http://ollama.poimen.svc:11434/api/generate"
- name: EVAL_QUERY_LOG_HOURS
value: "24"
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: memory-db-credentials
key: url
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "1"
memory: "1Gi"
restartPolicy: OnFailure
```
---
## 14. SLOs Summary
| Signal | Target | Window | Consequence of Miss |
|--------|--------|--------|---------------------|
| Ingest p99 latency | < 500ms | 24h rolling | Backpressure on upstream systems |
| Query p99 latency | < 500ms | 24h rolling | User-perceived slowness |
| Context p99 latency | < 2s | 24h rolling | Agent timeout, degraded assistance |
| Query empty rate | < 20% | 24h rolling | Users get no answer, lose trust |
| Tier-1 hit rate | >= 80% | 7d rolling | System not learning from failures |
| NDCG@10 | >= 0.85 | 7d rolling | Retrieval quality degraded, hallucination risk |
| MRR | >= 0.80 | 7d rolling | Relevant results buried in ranking |
| Storage parity drift | = 0 | 1h | Dual-write inconsistency, partial search |
| Review queue depth | < 100 | 24h | Unreviewed contradictions leaking through |
| Relevance judge agreement | >= 85% | Weekly | Automated evaluation unreliable |
| Raw availability | >= 99.9% | 24h rolling | Service down, LLM falls back to parametric knowledge |
| Quality availability | >= 95% | 24h rolling | Queries succeeding but returning nothing useful |
| Pod memory pressure | < 85% of limit | 5min | OOMKill imminent, in-flight requests lost |
| CPU throttle ratio | < 25% | 5min | Latency degradation across all endpoints |
| PG cache hit ratio | >= 99% | 1h | Disk thrashing, query latency spikes |
| PG dead tuple ratio | < 20% | 24h | Table bloat, slower scans, wasted disk |
| PG replication lag | < 10s | 5min | Stale reads from replica |
| Ingest rate (zero) | > 0 during business hours | 30min | Silent upstream failure, knowledge going stale |
| Ingest queue depth | < 1000 | 15min | Workers can't keep up, processing lag |