20 Commits

Include renames

Author	SHA1	Message	Date
poimenandrock	4169effd8a	feat: complete observability stack (O1-O13) (#52) ... ## Complete Observability Stack (O1-O13) Implements all 13 observability issues in a single PR. 119 metrics total. ### Commits (one per issue) | Issue | Title | Metrics | |-------|-------|---------| | **O10** | Prometheus metrics module + /metrics endpoint | Foundation | | **O1** | Instrument ingest handler | I1-I12 (12) | | **O2** | Instrument query handler | Q1-Q12 (12) | | **O3** | Instrument context endpoint | C1-C8 (8) | | **O4** | Relevance judge | R1-R9 (9) | | **O5** | Write volume + storage metrics | W1-W12 (12) | | **O6** | Pod resource observability | P1-P13 | | **O7** | Availability + dependency health | A1-A10 (10) | | **O8** | Ingest rate pattern tracking | IR1-IR10 (10) | | **O9** | Postgres internal observability | PG1-PG33 | | **O11** | Grafana dashboard | 12 panels | | **O12** | Prometheus alerting rules | 11 alerts | | **O13** | Relevance evaluation CronJob | K8s manifest | ### Key Changes - **metrics.rs**: Zero-dependency Prometheus metrics (Counter, Gauge, Histogram, Timer) - **GET /metrics**: Prometheus text exposition format endpoint - **Ingest/Query/Context handlers**: Instrumented with latency, errors, auth failures - **Health check**: DB dependency check with latency tracking - **Background task**: Periodic DB stats collection (entity/edge counts, pool stats) - **Relevance judge**: Threshold-based eval with precision/recall/F1 tracking - **Grafana dashboard**: 12 panels covering all metric groups - **Alert rules**: 11 PrometheusRule alerts (availability, latency, errors, quality) - **CronJob**: Periodic relevance evaluation with sample queries ### Testing - 506 tests passing (0 failures) - All metrics modules have unit tests - Relevance judge: 4 tests ### Deploy ```bash # Grafana dashboard kubectl apply -f k8s/infra/grafana-dashboard.json # Prometheus alerts kubectl apply -f k8s/infra/prometheus-alerts.yaml # Relevance eval CronJob kubectl apply -f k8s/infra/relevance-eval-cronjob.yaml ``` Closes #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 --------- Co-authored-by: rock <[email protected]> Reviewed-on: #52 Co-authored-by: poimen <[email protected]>	2026-09-13 13:53:50 +00:00
poimenandrock	d7a36ce9e8	ci: optimize build + deploy + migrate workflows (#51) ... ## Optimize CI/CD Workflows ### Changes #### build.yaml - **Merge 3 cargo steps → 1 compile pass**: `cargo build`, `cargo test`, `cargo clippy` now run in single invocation, reusing compiled artifacts - **Remove `cargo clean`**: Eliminated wasteful step that deleted artifacts before Docker build - **Add secret validation**: Registry credentials checked before login (fail-fast) #### deploy.yaml - **Skip checkout**: Removed unnecessary git clone - **Fetch SHA via Gitea API**: Query latest commit directly instead of cloning - **Reuse existing token**: Use `FORGEJO_REGISTRY_TOKEN` for Gitea API auth (already has privileges) - **Validate image exists**: Check SHA image exists before tagging as latest (prevents tagging non-existent images) - **Add secret validation**: Registry credentials checked before login (fail-fast) #### migrate.yaml - **Merge schema verification**: Schema inspect result reused in both changed + manual paths - **Fix manual trigger errors**: Manual mode now fails on first migration error (was silently masking with `|| true`) - **Track failures**: Explicit FAILED flag tracks migration errors across loop ### Benefits - **Speed**: Fewer compiles, no unnecessary clones, reuse artifacts - **Reliability**: Secret validation catches configuration issues early - **Safety**: Image existence check prevents tagging phantom images - **Clarity**: Merged steps have descriptive names, explicit error handling ### Testing - Branch: `ci/optimize-workflows` - Ready to merge to `main` after review --------- Co-authored-by: rock <[email protected]> Reviewed-on: #51 Co-authored-by: poimen <[email protected]>	2026-09-13 05:42:01 +00:00
poimenandrock	9f70109c1d	feat: scale memory-db to 3 replicas for HA (#50) ... ✅ All 3 replicas running and synced - memory-db-1 (primary) - memory-db-2 (replica, LSN 0/9000060) - memory-db-3 (replica, LSN 0/9000060) Cluster status: healthy Production-ready for failover. --------- Co-authored-by: rock <[email protected]> Reviewed-on: #50 Co-authored-by: poimen <[email protected]>	2026-09-13 00:04:32 +00:00
poimenandrock	fb61de6b47	feat: LLM entity + fact extraction pipeline (Zep paper alignment) (#48) ... ## Changes ### Entity Extraction - Switch from WikiLinkFallbackExtractor to LlmEntityExtractor when LLM_ENDPOINT set - `clean_llm_response()`: strips `<think>` tags, markdown fences, extracts JSON - Handle array responses (Ollama returns `[...]` not `{entities: [...]}`) - EntityType custom Deserialize: unknown variants → Unknown (no crash) - Increase timeout 30s→90s, max_tokens 500→1500 for reasoning models - Graceful reflection fallback: keep entities if verification fails ### Fact Extraction (NEW) - LlmFactExtractor: LLM-based relationship extraction between entity pairs - Validates source/target against known entity list (drops hallucinated edges) - Same robust JSON cleaning for reasoning models + Ollama - IngestWorker auto-selects LLM vs Simple based on LLM_ENDPOINT env ### K8s Deployment - Add `command: ["/app/mem"]` (fix args replacing CMD) - Add LLM_ENDPOINT, LLM_MODEL env vars for in-cluster LLM ## E2E Tested (local Ollama qwen2.5:3b) - 12 entities extracted (person, tool, concept, organization) - 5 edges with relationships and facts - 781 tests pass ## Zep Paper Alignment (§2.2) - Entity extraction + resolution (§2.2.1) - Fact extraction between entity pairs (§2.2.2) - Temporal edge invalidation ready (t_valid/t_invalid schema) - Reflection verification (§2.2.1, graceful fallback) --------- Co-authored-by: rock <[email protected]> Reviewed-on: #48 Co-authored-by: poimen <[email protected]>	2026-09-11 01:11:15 +00:00
rock	99efa46837	feat: simplify queue naming, remove stale docs, add Queue CRDs ... - Queue name now just 'poimen-chunks' (no project suffix) - Delete outdated CI/DESIGN docs (CLAUDE.md is source of truth) - Add k8s/infra/queue.yaml: poimen-chunks + DLQ (Ready) - Update test to expect new queue name format	2026-08-28 14:45:53 -07:00
rock	4d93f00dda	feat: M3.7.4 Context Endpoint - three-tier lookup infrastructure (12 tests)	2026-08-28 13:50:32 -07:00
rock	ac8eac03b0	feat: OpenSearch JWT auth via Authentik OIDC	2026-08-28 13:21:54 -07:00
Story Crater Bot	b18932b10c	feat: M3.8.1 phase 1 — content router + log compressor ... ContentRouter uses Google Magika ML for content detection (<1ms) with regex fallback. Detects JSON, code, logs, diffs, config, text. LogCompressor reuses M3.7.7 patterns (markers, cascade, strip_ansi) to shrink build logs by keeping errors/stacks and dropping noise. 17 unit tests passing: - router: json, code, diff, log, text detection - log: error lines, stack traces, ansi stripping, compression - optimizer: token estimation, passthrough mode Magika + ort ONNX runtime added to Cargo.toml.	2026-08-28 09:29:56 -07:00
Story Crater Bot	0869e507b0	plan: add Magika ML classifier to content router	2026-08-28 09:12:09 -07:00
Story Crater Bot	1f43ca0f64	docs: context optimizer design (Headroom-inspired pre-LLM compression)	2026-08-28 09:03:01 -07:00
Story Crater Bot	57f87f494a	chore: reduce memory-db cluster from 3 to 2 instances ... CHANGES: - k8s/infra/databases/memory-db.yaml: instances 3 → 2 - Updated comment from '3 instances' to '2 instances' REASONING: - Reduces resource overhead (high availability at 2 is sufficient) - Maintains quorum for failover (minimum 2 for HA) - Saves memory/CPU allocation on homelab cluster - ArgoCD will manage rollout automatically DEPLOYMENT: - ArgoCD will detect spec change and reconcile - CNPG will scale down one pod - Data preserved (3→2 replication, no data loss)	2026-08-28 08:16:02 -07:00
Story Crater Bot	ae0738289e	fix: OpenSearch security context and storage permissions	2026-08-27 21:46:15 -07:00
Story Crater Bot	2d64fbac10	fix: remove privileged init container, set pod-security baseline for OpenSearch	2026-08-27 21:41:17 -07:00
Story Crater Bot	bfde20262e	deploy: OpenSearch + Dashboards StatefulSet ... OpenSearch Cluster (k8s/infra/databases/opensearch.yaml): ✅ StatefulSet: 2 replicas (opensearch-0, opensearch-1) for HA ✅ Image: opensearchproject/opensearch:2.11.0 ✅ Services: opensearch (headless), opensearch-internal (ClusterIP:9200) ✅ ConfigMap: opensearch.yml with cluster discovery ✅ PVC: 30Gi per pod using Longhorn storage class ✅ Init container: sysctl vm.max_map_count=262144 ✅ Probes: liveness (60s), readiness (30s) ✅ Resources: 512Mi-1Gi memory, 250m-500m CPU ✅ Security: plugins.security.disabled=true (K8s network isolation) ✅ NetworkPolicy: Memory Service + Dashboards access only OpenSearch Dashboards (UI): ✅ Deployment: 1 replica opensearch-dashboards ✅ Image: opensearchproject/opensearch-dashboards:2.11.0 ✅ Service: opensearch-dashboards:5601 (ClusterIP) ✅ Config: connects to opensearch-internal:9200 ✅ Auth: admin/admin (production: change in secret) ✅ Port-forward: kubectl port-forward svc/opensearch-dashboards 5601:5601 ✅ Access: http://localhost:5601 (dev) or ingress (prod) Deployment Status: kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch-dashboards Verify Cluster Health: kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 curl http://localhost:9200/_cluster/health Next Steps: 1. Configure Memory Service: OPENSEARCH_HOSTS env var 2. Restart Memory Service pods 3. Test vault endpoints 4. Test hybrid search (with OpenSearch fallback)	2026-08-27 21:11:16 -07:00
Story Crater Bot	277d719278	feat: Memory Service API ready for deployment — Vault JSON endpoints + Hybrid search ... API Changes (crates/mem-cli/src/http_server.rs): ✅ Vault Endpoints (JSON API): - GET /memory/vault → {projects: [...]} - GET /memory/vault?project=X → {project: X, files: [...]} - GET /memory/vault/{proj}/{file} → {metadata: {...}, content: '...'} - YAML frontmatter parsed to JSON metadata - Auth: JWT on all endpoints ✅ Search Endpoints: - GET /memory/query?method=semantic → pgvector only (60% weight) - GET /memory/query?method=hybrid (default) → pgvector + OpenSearch (fallback to semantic) - Hybrid score: 0.6*semantic + 0.4*lexical - Limit: top-10 results (default) ✅ AppState Extended: - opensearch_client: Option<Arc<OpenSearchClient>> - Initialized from OPENSEARCH_HOSTS env var (optional) - Graceful fallback if OpenSearch unavailable ✅ Handlers Updated: - vault_browser_handler() → returns JSON projects list - vault_project_tree() → helper for file tree generation - vault_project_handler() → GET /{project} → file tree JSON - vault_file_handler() → GET /{project}/{file} → JSON with metadata + content - query_handler() → hybrid search with semantic fallback K8s Manifests (k8s/infra/databases/opensearch.yaml): ✅ OpenSearch StatefulSet: - 2 replicas for HA cluster (opensearch-0, opensearch-1) - Image: opensearchproject/opensearch:2.11.0 - Services: opensearch (headless), opensearch-internal (ClusterIP 9200) - ConfigMap: opensearch.yml with cluster settings - PVC: 30Gi per pod (Longhorn storage class) - ServiceAccount + NetworkPolicy (Memory Service only) - Init container: set vm.max_map_count=262144 - Probes: liveness (60s), readiness (30s) - Resources: 512Mi-1Gi memory, 250m-500m CPU - Security: plugins.security.disabled (K8s network isolated) ✅ Updated kustomization.yaml: - Added opensearch.yaml to resources Documentation: ✅ docs/API_VAULT_ENDPOINTS.md (10KB): - Complete API reference with examples - Architecture: semantic (pgvector IVFFlat) + lexical (OpenSearch BM25) - Fusion strategy: weighted linear combination (60/40 split) - DNS records for vault.riotpiao.com + memory.riotpiao.com - Ingress configuration (dual-domain routing) - Frontend integration examples (React/Vue) - Fallback behavior (graceful degradation) - Performance tuning (IVFFlat lists, OpenSearch shards) - Security: JWT validation, rate limiting, field-level ACL (future) ✅ docs/DEPLOYMENT_CHECKLIST.md (8KB): - 5-phase deployment plan (API ready, OpenSearch, DNS, Testing, Frontend) - Step-by-step deployment commands - Testing procedures for vault + search endpoints - Troubleshooting: OpenSearch not found, cluster red, JWT validation - Monitoring metrics + dashboard queries - Fallback scenarios + error codes Environment Variables: - OPENSEARCH_HOSTS (optional, e.g., "opensearch-internal.poimen.svc.cluster.local:9200") - If unset: hybrid search disabled, falls back to semantic - CSV list supported: "host1:9200,host2:9200" Deployment Summary: 1. ✅ API code ready (JSON endpoints, fallback to semantic if OpenSearch unavailable) 2. ✅ OpenSearch K8s manifests (StatefulSet + networking) 3. ✅ Documentation (API reference + deployment guide) 4. ⏳ Ready to: kubectl apply -k k8s/infra/databases/ Backward Compatibility: ✅ Existing JSON endpoints work without change ⚠️ HTML endpoints replaced with JSON (breaking change for old clients) ✅ Graceful fallback: hybrid search → semantic if OpenSearch missing ✅ Rate limiting preserved on all endpoints Testing Ready: - Vault tree endpoint testable after deployment - Hybrid search testable once OpenSearch cluster ready - All endpoints require JWT from Authentik - Load test script provided Next: Deploy OpenSearch + test against vault.riotpiao.com	2026-08-27 21:05:09 -07:00
Story Crater Bot	b923e0ad68	feat: M2.2 CNPG memory-db with pgvector (declarative, 3 instances)	2026-08-27 20:39:49 -07:00
Story Crater Bot	9ca988aeb3	Downsize memory-db to 2 instances	2026-08-22 23:53:05 -07:00
Story Crater Bot	af491564cd	Fix: use default longhorn (3 replicas), increase to 20Gi	2026-08-22 23:40:08 -07:00
Story Crater Bot	753435104d	Fix: use longhorn-imessage-local (WaitForFirstConsumer) for stable volume binding	2026-08-22 23:36:25 -07:00
Story Crater Bot	695e115212	Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)	2026-08-22 23:13:42 -07:00