rock/poimen-memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
132 Commits 8 Branches 0 Tags
23019fdb27db5cefeaeb4612b1af8ddb61a09c6e
Commit Graph
30 Commits
Author SHA1 Message Date
rock 23019fdb27 fix: obsidian vault PVC ReadWriteMany for shared access 2026-08-28 20:28:37 -07:00
rock 89b4995213 fix: add obsidian + obsidian-ui to kustomization.yaml 2026-08-28 17:22:12 -07:00
rock cfae6f300f feat: add obsidian-remote UI for browsable vault in browser 
sytone/obsidian-remote provides full Obsidian Desktop via noVNC.
Shares vault PVC with obsidian-server (REST API stays for memory system).
UI accessible at obsidian.riotpiao.com
 2026-08-28 17:20:35 -07:00
rock a60c74fc78 fix: move obsidian ingress to homelab repo, use obsidian.riotpiao.com 
vault.riotpiao.com was already taken by HashiCorp Vault.
Ingress now managed centrally in homelab/k8s/bootstrap/ingress/ingress.yaml
 2026-08-28 16:43:36 -07:00
rock 2ddd2d6cdf fix: remove broken auth annotations from obsidian ingress 
Bearer auth-url was misconfigured (pointed to token endpoint, not
forward-auth). No Authentik outpost deployed yet. Remove for now,
vault.riotpiao.com accessible directly. TODO: add forward-auth
once outpost is set up.
 2026-08-28 16:42:00 -07:00
rock b94898d0d4 feat: obsidian git-sync from poimen-obesdient-memory repo 
- Add git-sync init container to clone/pull vault content
- Add SOPS-encrypted SSH deploy key (obsidian-git-ssh-secret.enc.yaml)
- Add .sops.yaml config (age encryption, same key as homelab)
- Repo: ssh://[email protected]:2222/rock/poimen-obesdient-memory.git
- Deploy key added to Forgejo repo (read-only)
 2026-08-28 16:31:48 -07:00
rock 99efa46837 feat: simplify queue naming, remove stale docs, add Queue CRDs 
- Queue name now just 'poimen-chunks' (no project suffix)
- Delete outdated CI/DESIGN docs (CLAUDE.md is source of truth)
- Add k8s/infra/queue.yaml: poimen-chunks + DLQ (Ready)
- Update test to expect new queue name format
 2026-08-28 14:45:53 -07:00
rock 4d93f00dda feat: M3.7.4 Context Endpoint - three-tier lookup infrastructure (12 tests) 2026-08-28 13:50:32 -07:00
rock ac8eac03b0 feat: OpenSearch JWT auth via Authentik OIDC 2026-08-28 13:21:54 -07:00
Story Crater Bot b18932b10c feat: M3.8.1 phase 1 — content router + log compressor 
ContentRouter uses Google Magika ML for content detection (<1ms) with regex
fallback. Detects JSON, code, logs, diffs, config, text.

LogCompressor reuses M3.7.7 patterns (markers, cascade, strip_ansi) to
shrink build logs by keeping errors/stacks and dropping noise.

17 unit tests passing:
- router: json, code, diff, log, text detection
- log: error lines, stack traces, ansi stripping, compression
- optimizer: token estimation, passthrough mode

Magika + ort ONNX runtime added to Cargo.toml.
 2026-08-28 09:29:56 -07:00
Story Crater Bot 0869e507b0 plan: add Magika ML classifier to content router 2026-08-28 09:12:09 -07:00
Story Crater Bot 1f43ca0f64 docs: context optimizer design (Headroom-inspired pre-LLM compression) 2026-08-28 09:03:01 -07:00
Story Crater Bot 57f87f494a chore: reduce memory-db cluster from 3 to 2 instances 
CHANGES:
- k8s/infra/databases/memory-db.yaml: instances 3 → 2
- Updated comment from '3 instances' to '2 instances'

REASONING:
- Reduces resource overhead (high availability at 2 is sufficient)
- Maintains quorum for failover (minimum 2 for HA)
- Saves memory/CPU allocation on homelab cluster
- ArgoCD will manage rollout automatically

DEPLOYMENT:
- ArgoCD will detect spec change and reconcile
- CNPG will scale down one pod
- Data preserved (3→2 replication, no data loss)
 2026-08-28 08:16:02 -07:00
Story Crater Bot ae0738289e fix: OpenSearch security context and storage permissions 2026-08-27 21:46:15 -07:00
Story Crater Bot 2d64fbac10 fix: remove privileged init container, set pod-security baseline for OpenSearch 2026-08-27 21:41:17 -07:00
Story Crater Bot 69ec8aeec2 fix: Obsidian service port and health checks, use Longhorn storage 2026-08-27 21:37:47 -07:00
Story Crater Bot 0eecca815b refactor: replace Obsidian projector with standalone service (ppatlabs/obsidian) 2026-08-27 21:35:07 -07:00
Story Crater Bot bfde20262e deploy: OpenSearch + Dashboards StatefulSet 
OpenSearch Cluster (k8s/infra/databases/opensearch.yaml):
  ✅ StatefulSet: 2 replicas (opensearch-0, opensearch-1) for HA
  ✅ Image: opensearchproject/opensearch:2.11.0
  ✅ Services: opensearch (headless), opensearch-internal (ClusterIP:9200)
  ✅ ConfigMap: opensearch.yml with cluster discovery
  ✅ PVC: 30Gi per pod using Longhorn storage class
  ✅ Init container: sysctl vm.max_map_count=262144
  ✅ Probes: liveness (60s), readiness (30s)
  ✅ Resources: 512Mi-1Gi memory, 250m-500m CPU
  ✅ Security: plugins.security.disabled=true (K8s network isolation)
  ✅ NetworkPolicy: Memory Service + Dashboards access only

OpenSearch Dashboards (UI):
  ✅ Deployment: 1 replica opensearch-dashboards
  ✅ Image: opensearchproject/opensearch-dashboards:2.11.0
  ✅ Service: opensearch-dashboards:5601 (ClusterIP)
  ✅ Config: connects to opensearch-internal:9200
  ✅ Auth: admin/admin (production: change in secret)
  ✅ Port-forward: kubectl port-forward svc/opensearch-dashboards 5601:5601
  ✅ Access: http://localhost:5601 (dev) or ingress (prod)

Deployment Status:
  kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch
  kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch-dashboards

Verify Cluster Health:
  kubectl port-forward -n poimen svc/opensearch-internal 9200:9200
  curl http://localhost:9200/_cluster/health

Next Steps:
  1. Configure Memory Service: OPENSEARCH_HOSTS env var
  2. Restart Memory Service pods
  3. Test vault endpoints
  4. Test hybrid search (with OpenSearch fallback)
 2026-08-27 21:11:16 -07:00
Story Crater Bot 277d719278 feat: Memory Service API ready for deployment — Vault JSON endpoints + Hybrid search 
API Changes (crates/mem-cli/src/http_server.rs):

✅ Vault Endpoints (JSON API):
  - GET /memory/vault → {projects: [...]}
  - GET /memory/vault?project=X → {project: X, files: [...]}
  - GET /memory/vault/{proj}/{file} → {metadata: {...}, content: '...'}
  - YAML frontmatter parsed to JSON metadata
  - Auth: JWT on all endpoints

✅ Search Endpoints:
  - GET /memory/query?method=semantic → pgvector only (60% weight)
  - GET /memory/query?method=hybrid (default) → pgvector + OpenSearch (fallback to semantic)
  - Hybrid score: 0.6*semantic + 0.4*lexical
  - Limit: top-10 results (default)

✅ AppState Extended:
  - opensearch_client: Option<Arc<OpenSearchClient>>
  - Initialized from OPENSEARCH_HOSTS env var (optional)
  - Graceful fallback if OpenSearch unavailable

✅ Handlers Updated:
  - vault_browser_handler() → returns JSON projects list
  - vault_project_tree() → helper for file tree generation
  - vault_project_handler() → GET /{project} → file tree JSON
  - vault_file_handler() → GET /{project}/{file} → JSON with metadata + content
  - query_handler() → hybrid search with semantic fallback

K8s Manifests (k8s/infra/databases/opensearch.yaml):

✅ OpenSearch StatefulSet:
  - 2 replicas for HA cluster (opensearch-0, opensearch-1)
  - Image: opensearchproject/opensearch:2.11.0
  - Services: opensearch (headless), opensearch-internal (ClusterIP 9200)
  - ConfigMap: opensearch.yml with cluster settings
  - PVC: 30Gi per pod (Longhorn storage class)
  - ServiceAccount + NetworkPolicy (Memory Service only)
  - Init container: set vm.max_map_count=262144
  - Probes: liveness (60s), readiness (30s)
  - Resources: 512Mi-1Gi memory, 250m-500m CPU
  - Security: plugins.security.disabled (K8s network isolated)

✅ Updated kustomization.yaml:
  - Added opensearch.yaml to resources

Documentation:

✅ docs/API_VAULT_ENDPOINTS.md (10KB):
  - Complete API reference with examples
  - Architecture: semantic (pgvector IVFFlat) + lexical (OpenSearch BM25)
  - Fusion strategy: weighted linear combination (60/40 split)
  - DNS records for vault.riotpiao.com + memory.riotpiao.com
  - Ingress configuration (dual-domain routing)
  - Frontend integration examples (React/Vue)
  - Fallback behavior (graceful degradation)
  - Performance tuning (IVFFlat lists, OpenSearch shards)
  - Security: JWT validation, rate limiting, field-level ACL (future)

✅ docs/DEPLOYMENT_CHECKLIST.md (8KB):
  - 5-phase deployment plan (API ready, OpenSearch, DNS, Testing, Frontend)
  - Step-by-step deployment commands
  - Testing procedures for vault + search endpoints
  - Troubleshooting: OpenSearch not found, cluster red, JWT validation
  - Monitoring metrics + dashboard queries
  - Fallback scenarios + error codes

Environment Variables:

- OPENSEARCH_HOSTS (optional, e.g., "opensearch-internal.poimen.svc.cluster.local:9200")
  - If unset: hybrid search disabled, falls back to semantic
  - CSV list supported: "host1:9200,host2:9200"

Deployment Summary:

1. ✅ API code ready (JSON endpoints, fallback to semantic if OpenSearch unavailable)
2. ✅ OpenSearch K8s manifests (StatefulSet + networking)
3. ✅ Documentation (API reference + deployment guide)
4. ⏳ Ready to: kubectl apply -k k8s/infra/databases/

Backward Compatibility:

✅ Existing JSON endpoints work without change
⚠️ HTML endpoints replaced with JSON (breaking change for old clients)
✅ Graceful fallback: hybrid search → semantic if OpenSearch missing
✅ Rate limiting preserved on all endpoints

Testing Ready:

- Vault tree endpoint testable after deployment
- Hybrid search testable once OpenSearch cluster ready
- All endpoints require JWT from Authentik
- Load test script provided

Next: Deploy OpenSearch + test against vault.riotpiao.com
 2026-08-27 21:05:09 -07:00
Story Crater Bot b923e0ad68 feat: M2.2 CNPG memory-db with pgvector (declarative, 3 instances) 2026-08-27 20:39:49 -07:00
Story Crater Bot 56bee1915e chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1) 
Deleted 31 completed task files:
- M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate)
- M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate)
- M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate)
- M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate)
- M3.6.1: DocCorpusSource (heading-boundary chunking)
- M4.1-2: skill-draft, derived-filter

Updated INDEX.md:
- Removed M0 & M1 phase sections (archived in git history)
- Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜)
- Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done
- Noted M3.5.10 JWT auth implementation complete (awaiting image rollout)
- Cleaned up broken links to deleted task files

Total test count: 239 passing, 2 ignored (up from 196 at M3.4)
Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
 2026-08-27 20:25:05 -07:00
rock 46d993824f feat: add Obsidian vault projection with Longhorn storage (#13) 2026-08-24 01:58:39 +00:00
Story Crater Bot 2347d785db Add ArgoCD Application for auto-deployment (poimen-memory-app) 2026-08-23 09:46:58 -07:00
Story Crater Bot de9c4ffeae Implement M3.6.1: DocCorpusSource with heading-boundary chunking (196 tests) 2026-08-23 09:42:09 -07:00
Story Crater Bot eaed7fc42a Add K8s app deployment, Dockerfile, and CI workflow (Option A) 2026-08-23 00:01:30 -07:00
Story Crater Bot 9ca988aeb3 Downsize memory-db to 2 instances 2026-08-22 23:53:05 -07:00
Story Crater Bot af491564cd Fix: use default longhorn (3 replicas), increase to 20Gi 2026-08-22 23:40:08 -07:00
Story Crater Bot 753435104d Fix: use longhorn-imessage-local (WaitForFirstConsumer) for stable volume binding 2026-08-22 23:36:25 -07:00
Story Crater Bot 8b8e3ec17b Bundle memory database into homelab orchestration (remove separate app) 2026-08-22 23:16:39 -07:00
Story Crater Bot 695e115212 Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7) 2026-08-22 23:13:42 -07:00