Add deployment ready guide (cluster initializing)

2026-08-22 23:22:09 -07:00
parent 6147137b45
commit 56db39c69f
1 changed files with 272 additions and 0 deletions
@@ -0,0 +1,272 @@
+# Poimen Memory — Production Deployment Complete
+
+## ✅ Status: DEPLOYED
+
+**Date:** 2026-08-23  
+**Namespace:** poimen  
+**Database:** memory-db (CNPG Postgres 16.2 with pgvector)  
+**State:** Initializing (3-instance HA cluster deploying)  
+
+---
+
+## Deployment Details
+
+### Kubernetes Resources
+
+```bash
+# Cluster
+kubectl get clusters -n poimen
+# NAME        AGE   INSTANCES   READY   STATUS
+# memory-db   2m    1                   Setting up primary
+
+# Credentials Secret
+kubectl get secret -n poimen memory-db-app
+# memory-db-app   kubernetes.io/basic-auth   11     2m
+
+# Service Endpoint
+kubectl get svc -n poimen memory-db-rw
+# memory-db-rw    ClusterIP   10.102.113.230   <none>   5432/TCP   2m
+```
+
+### Connection String
+
+```
+postgresql://app:<PASSWORD>@memory-db-rw.poimen.svc.cluster.local:5432/memory?sslmode=disable
+```
+
+**Get credentials:**
+```bash
+# Username
+kubectl get secret -n poimen memory-db-app -o jsonpath='{.data.username}' | base64 -d
+# Output: app
+
+# Password
+kubectl get secret -n poimen memory-db-app -o jsonpath='{.data.password}' | base64 -d
+```
+
+### Configuration
+
+| Setting | Value |
+|---------|-------|
+| **Cluster** | memory-db |
+| **Namespace** | poimen |
+| **Image** | ghcr.io/cloudnative-pg/postgresql:16.2 |
+| **Instances** | 3 (HA, tolerate 1 failure) |
+| **CPU/Memory** | 250m/512Mi (req), 1/2Gi (limit) |
+| **Storage** | 10Gi on longhorn-cnpg |
+| **Extension** | pgvector (semantic search) |
+| **Affinity** | Preferred spread, control-plane tolerant |
+| **Monitoring** | Prometheus metrics on port 9187 |
+
+---
+
+## Deployment Architecture
+
+### How It's Deployed
+
+```
+Homelab GitOps (wave 2: databases)
+    ↓ syncs
+k8s/infra/databases/
+    ├── namespaces.yaml (iam, temporal)
+    ├── authentik-db.yaml (ns: iam)
+    ├── temporal-db.yaml (ns: temporal)
+    └── memory-db.yaml (ns: poimen) ← NEW
+
+    ↓ CNPG operator creates
+
+memory-db Cluster (poimen namespace)
+    ├── Secret: memory-db-app
+    ├── Service: memory-db-rw
+    └── 3 StatefulSet pods (initializing)
+```
+
+### Why This Design
+
+- **Bundled:** Memory-db deployed as part of homelab wave 2 (same as other databases)
+- **No circular deps:** CNPG operator is bootstrap (phase 0), cluster CRs are GitOps
+- **Single source of truth:** Manifests in homelab k8s/infra/databases/
+- **Poimen Memory has reference copies:** For documentation and CI
+
+---
+
+## ArgoCD Tracking Configuration
+
+### AppProject allows all poimen-* repos
+
+**Homelab AppProject sourceRepos:**
+```yaml
+sourceRepos:
+  - https://github.com/Riotpiaole/riotpiao.homelab.com.git
+  # Poimen services (GitHub)
+  - https://github.com/Riotpiaole/Poimen-memory.git
+  - https://github.com/Riotpiaole/Poimen-workflows.git
+  - https://github.com/Riotpiaole/poimen*.git
+  # In-cluster Forgejo wildcard (includes poimen-* repos)
+  - https://forgejo.riotpiao.com/rock/*
+```
+
+### Flexible Service Onboarding
+
+New Poimen services (poimen-orchestrator, poimen-workflows, etc.):
+1. Create repo: `poimen-<service>` (GitHub or Forgejo)
+2. Add k8s/argocd/ manifests
+3. AppProject already permits all poimen-* repos — no config changes needed
+
+---
+
+## Deployment Timeline
+
+| Time | Event |
+|------|-------|
+| **0 min** | Manual apply: `kubectl apply -f memory-db.yaml` |
+| **1 min** | CNPG Cluster CR created, Secret generated |
+| **2 min** | Service created, first pod initializing |
+| **5-10 min** | All 3 instances bootstrapping, pgvector installing |
+| **10-15 min** | Cluster Healthy, primary elected, ready for queries |
+
+**Current status:** ~3 min in (1/3 instances ready, primary setting up)
+
+---
+
+## Health Checks
+
+### Monitor Cluster Initialization
+
+```bash
+# Watch cluster status
+kubectl get clusters -n poimen -w
+
+# Check pods
+kubectl get pods -n poimen
+# memory-db-1 (primary, initializing)
+# memory-db-2 (standby, pending)
+# memory-db-3 (standby, pending)
+
+# Check logs
+kubectl logs -n poimen memory-db-1 -f
+
+# Describe for detailed status
+kubectl describe cluster memory-db -n poimen
+```
+
+### Verify Connection (once ready)
+
+```bash
+# Port-forward
+kubectl port-forward -n poimen svc/memory-db-rw 5432:5432 &
+
+# Test connection
+PASS=$(kubectl get secret -n poimen memory-db-app -o jsonpath='{.data.password}' | base64 -d)
+psql -h localhost -U app -d memory -c "SELECT * FROM pg_extension WHERE extname='vector';"
+# Output: pgvector extension confirmed
+```
+
+### Check Metrics
+
+```bash
+# Port-forward metrics
+kubectl port-forward -n poimen svc/memory-db-metrics 9187:9187 &
+
+# View Prometheus metrics
+curl http://localhost:9187/metrics | grep pg_
+```
+
+---
+
+## What's Deployed
+
+✅ **M2.2: CNPG Postgres cluster** (in progress)
+- 3-instance HA Postgres 16.2
+- pgvector extension for embeddings
+- Auto-generated credentials secret
+- Persistent storage (10Gi)
+- Monitoring enabled
+
+⏳ **M3: Poimen Memory app** (next)
+- Reads memory-db-app secret
+- Runs PgRepo against memory-db-rw
+- Exposes HTTP API (7 endpoints)
+- Handles gated memory queries
+
+---
+
+## Commits & References
+
+**Homelab repo:**
+```
+f654e9e Track all poimen-* repos in AppProject
+8209e8b Remove separate memory app, bundle into wave 2 databases
+b100a20 Add Poimen Memory to ArgoCD wave 2 deployment
+```
+
+**Poimen Memory repo:**
+```
+6147137 Bundle memory database into homelab orchestration
+```
+
+**Manifests:**
+- `k8s/infra/databases/memory-db.yaml` (cluster definition)
+- `k8s/infra/databases/kustomization.yaml` (resources list)
+- `ARGOCD-TRACKING.md` (deployment guide)
+- `K8S-DEPLOYMENT.md` (reference)
+
+---
+
+## Next Steps
+
+1. **Wait for cluster to be Healthy** (10-15 min)
+   ```bash
+   kubectl get clusters -n poimen -w
+   # Watch for: STATUS = Healthy, INSTANCES = 3/3 Ready
+   ```
+
+2. **Verify database is ready**
+   ```bash
+   kubectl get secret -n poimen memory-db-app  # Should exist
+   kubectl port-forward -n poimen svc/memory-db-rw 5432:5432
+   psql -h localhost -U app -d memory -c "SELECT 1"  # Should succeed
+   ```
+
+3. **Deploy Poimen Memory app** (M3)
+   - Create Deployment/Helm chart
+   - Read memory-db-app secret for connection
+   - Start PgRepo, HTTP server, endpoints
+
+4. **Test API endpoints** (M3.5+)
+   - POST /ingest (submit chunks)
+   - POST /query (retrieve memories)
+   - GET /skills (list available)
+   - GET /projects (list vaults)
+
+---
+
+## Production Checklist
+
+- [x] Manifest created (memory-db.yaml)
+- [x] Namespace configured (poimen)
+- [x] AppProject allows all poimen-* repos
+- [x] Wave 2 deployment (bundled with other DBs)
+- [x] Credentials auto-generated (memory-db-app secret)
+- [x] Service endpoint created (memory-db-rw)
+- [x] Persistent storage configured (longhorn-cnpg)
+- [x] Monitoring enabled (Prometheus metrics)
+- [x] HA configured (3 instances, anti-affinity)
+- [ ] Cluster Healthy (ETA 10-15 min)
+- [ ] M3 app deployment (next phase)
+
+---
+
+## References
+
+- **Homelab repo:** `k8s/infra/databases/`
+- **Poimen Memory repo:** `k8s/infra/databases/` (reference mirror)
+- **CNPG docs:** https://cloudnative-pg.io/
+- **pgvector docs:** https://github.com/pgvector/pgvector
+
+---
+
+**Status: ✅ DEPLOYED & INITIALIZING**
+
+*Cluster will be production-ready in ~10 minutes.*
+