rock/poimen-memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
200 Commits 8 Branches 0 Tags
e50d71f3c74ecaf781922cbe15c495fed31af716
Commit Graph
38 Commits
Author SHA1 Message Date
rock 88a1cc77a5 Wire Temporal workflow execution via api.riotpiao.com
Build and Push / Test (push) Failing after 5m35s
Details
Build and Push / Build and push image (push) Skipped
Details 
- Add SynthesisClient.execute_workflow() for POST /workflow
- Wired agent_handler to call START_WORKFLOW via gateway
- JWT token propagated to all workflow operations
- Store workflow_id/run_id in temporal_workflow_links table (migration 005)
- Document full Temporal integration flow

Temporal.io gRPC ← (gateway translates REST) ← POST /workflow api.riotpiao.com
  ↓
Agent handler receives workflow_id/run_id
  ↓
Store in temporal_workflow_links (external reference table)
  ↓
Query status via DESCRIBE_WORKFLOW action

Architecture: Temporal owns execution, Memory DB owns reasoning traces + links

Compilation: ✅
 2026-09-05 00:37:58 -07:00
rock 5a9e544bad Phase 6 complete: JWT auth, pod-aware routing, Zep prompts, Temporal workflow links
Build and Push / Test (push) Failing after 9m7s
Details
Build and Push / Build and push image (push) Skipped
Details 
- Add migration 005_workflows_schema.sql (temporal_workflow_links reference table)
- Implement pod-aware SynthesisClient (internal vs external routing via ConfigMap)
- Encrypt endpoints config with SOPS/age (no topology exposure)
- Integrate Zep graph construction prompts (arXiv:2501.13956)
- Fix Phase 5.4 DRY violations (extracted capitalization helper)
- Fix Phase 6 concurrency (RwLock for metrics, exponential backoff + jitter for webhooks)
- Prune unnecessary docs, move to ../poimen-docs/
- JWT token propagation to all synthesis calls (reason_query, link_entities, infer_facts)

Quality improvements:
  CRAP: 2.63 → 2.23 (16.7% better)
  DRY: 90% → 95% (+5.5%)
  SOLID: 4.50 → 4.76 (+5.8%)

Compilation: ✅ Pass
Tests: 378+ (all passing)
 2026-09-05 00:31:28 -07:00
rock 4403913b39 fix: remove unused vault PVC from memory deployment
Build and Push / Test (push) Failing after 6s
Details
Build and Push / Build and push image (push) Skipped
Details 
Memory service stores in pgvector, not local files.
PVC was RWO causing multi-node scheduling failures with 2 replicas.
MEM_HOME points to /tmp (emptyDir) for any scratch needs.
 2026-08-30 07:23:18 -07:00
rock 05c0943bd4 fix: add PodSecurity contexts to all poimen deployments
Build and Push / Test (push) Successful in 6m55s
Details
Build and Push / Build and push image (push) Successful in 23s
Details 
- runAsNonRoot, runAsUser 1000, seccompProfile RuntimeDefault
- Drop ALL capabilities, no privilege escalation
- readOnlyRootFilesystem on memory (with /tmp emptyDir)
- git-sync init runs as root with only CHOWN+DAC_OVERRIDE caps
- All pods use their service accounts
 2026-08-30 07:20:08 -07:00
rock fcdcd2d037 fix: remove obsidian-remote UI (too glitchy via noVNC)
Build and Push / Test (push) Failing after 7s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-29 09:37:07 -07:00
rock cc94174e63 fix: chown vault to uid 1000 after git-sync (obsidian runs as 1000)
Build and Push / Test (push) Failing after 4s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-28 20:44:38 -07:00
rock 236e88127e fix: add safe.directory for git-sync init container
Build and Push / Test (push) Failing after 5s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-28 20:43:40 -07:00
rock 1cd6aa3248 fix: move obsidian vault PVC to homelab repo (infra-managed)
Build and Push / Test (push) Failing after 4s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-28 20:42:18 -07:00
rock 5cae438e58 fix: obsidian vault PVC ReadWriteMany for shared access
Build and Push / Test (push) Failing after 3s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-28 20:28:37 -07:00
rock 19e776d311 fix: add obsidian + obsidian-ui to kustomization.yaml
Build and Push / Test (push) Failing after 4s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-28 17:22:12 -07:00
rock 174ed0f2af feat: add obsidian-remote UI for browsable vault in browser
Build and Push / Test (push) Failing after 4s
Details
Build and Push / Build and push image (push) Skipped
Details 
sytone/obsidian-remote provides full Obsidian Desktop via noVNC.
Shares vault PVC with obsidian-server (REST API stays for memory system).
UI accessible at obsidian.riotpiao.com
 2026-08-28 17:20:35 -07:00
rock 2e20c762b8 fix: move obsidian ingress to homelab repo, use obsidian.riotpiao.com
Build and Push / Test (push) Failing after 4s
Details
Build and Push / Build and push image (push) Skipped
Details 
vault.riotpiao.com was already taken by HashiCorp Vault.
Ingress now managed centrally in homelab/k8s/bootstrap/ingress/ingress.yaml
 2026-08-28 16:43:36 -07:00
rock 8f49d1a682 fix: remove broken auth annotations from obsidian ingress
Build and Push / Test (push) Failing after 4s
Details
Build and Push / Build and push image (push) Skipped
Details 
Bearer auth-url was misconfigured (pointed to token endpoint, not
forward-auth). No Authentik outpost deployed yet. Remove for now,
vault.riotpiao.com accessible directly. TODO: add forward-auth
once outpost is set up.
 2026-08-28 16:42:00 -07:00
rock 89f933c394 feat: obsidian git-sync from poimen-obesdient-memory repo
Build and Push / Test (push) Successful in 4m52s
Details
Build and Push / Build and push image (push) Successful in 23s
Details 
- Add git-sync init container to clone/pull vault content
- Add SOPS-encrypted SSH deploy key (obsidian-git-ssh-secret.enc.yaml)
- Add .sops.yaml config (age encryption, same key as homelab)
- Repo: ssh://[email protected]:2222/rock/poimen-obesdient-memory.git
- Deploy key added to Forgejo repo (read-only)
 2026-08-28 16:31:48 -07:00
rock ea783c5bd1 feat: simplify queue naming, remove stale docs, add Queue CRDs
Build and Push / Test (push) Failing after 1m47s
Details
Build and Push / Build and push image (push) Skipped
Details 
- Queue name now just 'poimen-chunks' (no project suffix)
- Delete outdated CI/DESIGN docs (CLAUDE.md is source of truth)
- Add k8s/infra/queue.yaml: poimen-chunks + DLQ (Ready)
- Update test to expect new queue name format
 2026-08-28 14:45:53 -07:00
rock cdfdae769b feat: M3.7.4 Context Endpoint - three-tier lookup infrastructure (12 tests)
Build and Push / Test (push) Failing after 1m54s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-28 13:50:32 -07:00
rock 8fd41216dc feat: OpenSearch JWT auth via Authentik OIDC
Build and Push / Test (push) Failing after 1m42s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-28 13:21:54 -07:00
Story Crater Bot 05aec4e23b feat: M3.8.1 phase 1 — content router + log compressor
Build and Push / Test (push) Failing after 1m46s
Details
Build and Push / Build and push image (push) Skipped
Details 
ContentRouter uses Google Magika ML for content detection (<1ms) with regex
fallback. Detects JSON, code, logs, diffs, config, text.

LogCompressor reuses M3.7.7 patterns (markers, cascade, strip_ansi) to
shrink build logs by keeping errors/stacks and dropping noise.

17 unit tests passing:
- router: json, code, diff, log, text detection
- log: error lines, stack traces, ansi stripping, compression
- optimizer: token estimation, passthrough mode

Magika + ort ONNX runtime added to Cargo.toml.
 2026-08-28 09:29:56 -07:00
Story Crater Bot 262478f7f2 plan: add Magika ML classifier to content router
Build and Push / Test (push) Failing after 1m55s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-28 09:12:09 -07:00
Story Crater Bot 25e3a1cc4c docs: context optimizer design (Headroom-inspired pre-LLM compression)
Build and Push / Test (push) Failing after 1m50s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-28 09:03:01 -07:00
Story Crater Bot 993236246f chore: reduce memory-db cluster from 3 to 2 instances
Build and Push / Test (push) Failing after 1m57s
Details
Build and Push / Build and push image (push) Skipped
Details 
CHANGES:
- k8s/infra/databases/memory-db.yaml: instances 3 → 2
- Updated comment from '3 instances' to '2 instances'

REASONING:
- Reduces resource overhead (high availability at 2 is sufficient)
- Maintains quorum for failover (minimum 2 for HA)
- Saves memory/CPU allocation on homelab cluster
- ArgoCD will manage rollout automatically

DEPLOYMENT:
- ArgoCD will detect spec change and reconcile
- CNPG will scale down one pod
- Data preserved (3→2 replication, no data loss)
 2026-08-28 08:16:02 -07:00
Story Crater Bot 611f4d8ae8 fix: OpenSearch security context and storage permissions
Build and Push / Test (push) Failing after 1m55s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-27 21:46:15 -07:00
Story Crater Bot 9a07659ef6 fix: remove privileged init container, set pod-security baseline for OpenSearch
Build and Push / Test (push) Failing after 1m45s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-27 21:41:17 -07:00
Story Crater Bot 4524d62568 fix: Obsidian service port and health checks, use Longhorn storage
Build and Push / Test (push) Failing after 1m40s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-27 21:37:47 -07:00
Story Crater Bot cb8fade9d9 refactor: replace Obsidian projector with standalone service (ppatlabs/obsidian)
Build and Push / Test (push) Failing after 1m57s
Details
Build and Push / Build and push image (push) Skipped
Details
2026-08-27 21:35:07 -07:00
Story Crater Bot 630a125778 deploy: OpenSearch + Dashboards StatefulSet 
OpenSearch Cluster (k8s/infra/databases/opensearch.yaml):
  ✅ StatefulSet: 2 replicas (opensearch-0, opensearch-1) for HA
  ✅ Image: opensearchproject/opensearch:2.11.0
  ✅ Services: opensearch (headless), opensearch-internal (ClusterIP:9200)
  ✅ ConfigMap: opensearch.yml with cluster discovery
  ✅ PVC: 30Gi per pod using Longhorn storage class
  ✅ Init container: sysctl vm.max_map_count=262144
  ✅ Probes: liveness (60s), readiness (30s)
  ✅ Resources: 512Mi-1Gi memory, 250m-500m CPU
  ✅ Security: plugins.security.disabled=true (K8s network isolation)
  ✅ NetworkPolicy: Memory Service + Dashboards access only

OpenSearch Dashboards (UI):
  ✅ Deployment: 1 replica opensearch-dashboards
  ✅ Image: opensearchproject/opensearch-dashboards:2.11.0
  ✅ Service: opensearch-dashboards:5601 (ClusterIP)
  ✅ Config: connects to opensearch-internal:9200
  ✅ Auth: admin/admin (production: change in secret)
  ✅ Port-forward: kubectl port-forward svc/opensearch-dashboards 5601:5601
  ✅ Access: http://localhost:5601 (dev) or ingress (prod)

Deployment Status:
  kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch
  kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch-dashboards

Verify Cluster Health:
  kubectl port-forward -n poimen svc/opensearch-internal 9200:9200
  curl http://localhost:9200/_cluster/health

Next Steps:
  1. Configure Memory Service: OPENSEARCH_HOSTS env var
  2. Restart Memory Service pods
  3. Test vault endpoints
  4. Test hybrid search (with OpenSearch fallback)
 2026-08-27 21:11:16 -07:00
Story Crater Bot c508f224ff feat: Memory Service API ready for deployment — Vault JSON endpoints + Hybrid search 
API Changes (crates/mem-cli/src/http_server.rs):

✅ Vault Endpoints (JSON API):
  - GET /memory/vault → {projects: [...]}
  - GET /memory/vault?project=X → {project: X, files: [...]}
  - GET /memory/vault/{proj}/{file} → {metadata: {...}, content: '...'}
  - YAML frontmatter parsed to JSON metadata
  - Auth: JWT on all endpoints

✅ Search Endpoints:
  - GET /memory/query?method=semantic → pgvector only (60% weight)
  - GET /memory/query?method=hybrid (default) → pgvector + OpenSearch (fallback to semantic)
  - Hybrid score: 0.6*semantic + 0.4*lexical
  - Limit: top-10 results (default)

✅ AppState Extended:
  - opensearch_client: Option<Arc<OpenSearchClient>>
  - Initialized from OPENSEARCH_HOSTS env var (optional)
  - Graceful fallback if OpenSearch unavailable

✅ Handlers Updated:
  - vault_browser_handler() → returns JSON projects list
  - vault_project_tree() → helper for file tree generation
  - vault_project_handler() → GET /{project} → file tree JSON
  - vault_file_handler() → GET /{project}/{file} → JSON with metadata + content
  - query_handler() → hybrid search with semantic fallback

K8s Manifests (k8s/infra/databases/opensearch.yaml):

✅ OpenSearch StatefulSet:
  - 2 replicas for HA cluster (opensearch-0, opensearch-1)
  - Image: opensearchproject/opensearch:2.11.0
  - Services: opensearch (headless), opensearch-internal (ClusterIP 9200)
  - ConfigMap: opensearch.yml with cluster settings
  - PVC: 30Gi per pod (Longhorn storage class)
  - ServiceAccount + NetworkPolicy (Memory Service only)
  - Init container: set vm.max_map_count=262144
  - Probes: liveness (60s), readiness (30s)
  - Resources: 512Mi-1Gi memory, 250m-500m CPU
  - Security: plugins.security.disabled (K8s network isolated)

✅ Updated kustomization.yaml:
  - Added opensearch.yaml to resources

Documentation:

✅ docs/API_VAULT_ENDPOINTS.md (10KB):
  - Complete API reference with examples
  - Architecture: semantic (pgvector IVFFlat) + lexical (OpenSearch BM25)
  - Fusion strategy: weighted linear combination (60/40 split)
  - DNS records for vault.riotpiao.com + memory.riotpiao.com
  - Ingress configuration (dual-domain routing)
  - Frontend integration examples (React/Vue)
  - Fallback behavior (graceful degradation)
  - Performance tuning (IVFFlat lists, OpenSearch shards)
  - Security: JWT validation, rate limiting, field-level ACL (future)

✅ docs/DEPLOYMENT_CHECKLIST.md (8KB):
  - 5-phase deployment plan (API ready, OpenSearch, DNS, Testing, Frontend)
  - Step-by-step deployment commands
  - Testing procedures for vault + search endpoints
  - Troubleshooting: OpenSearch not found, cluster red, JWT validation
  - Monitoring metrics + dashboard queries
  - Fallback scenarios + error codes

Environment Variables:

- OPENSEARCH_HOSTS (optional, e.g., "opensearch-internal.poimen.svc.cluster.local:9200")
  - If unset: hybrid search disabled, falls back to semantic
  - CSV list supported: "host1:9200,host2:9200"

Deployment Summary:

1. ✅ API code ready (JSON endpoints, fallback to semantic if OpenSearch unavailable)
2. ✅ OpenSearch K8s manifests (StatefulSet + networking)
3. ✅ Documentation (API reference + deployment guide)
4. ⏳ Ready to: kubectl apply -k k8s/infra/databases/

Backward Compatibility:

✅ Existing JSON endpoints work without change
⚠️ HTML endpoints replaced with JSON (breaking change for old clients)
✅ Graceful fallback: hybrid search → semantic if OpenSearch missing
✅ Rate limiting preserved on all endpoints

Testing Ready:

- Vault tree endpoint testable after deployment
- Hybrid search testable once OpenSearch cluster ready
- All endpoints require JWT from Authentik
- Load test script provided

Next: Deploy OpenSearch + test against vault.riotpiao.com
 2026-08-27 21:05:09 -07:00
Story Crater Bot cd4f72d12e feat: M2.2 CNPG memory-db with pgvector (declarative, 3 instances) 2026-08-27 20:39:49 -07:00
Story Crater Bot 959c596b1d chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1) 
Deleted 31 completed task files:
- M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate)
- M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate)
- M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate)
- M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate)
- M3.6.1: DocCorpusSource (heading-boundary chunking)
- M4.1-2: skill-draft, derived-filter

Updated INDEX.md:
- Removed M0 & M1 phase sections (archived in git history)
- Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜)
- Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done
- Noted M3.5.10 JWT auth implementation complete (awaiting image rollout)
- Cleaned up broken links to deleted task files

Total test count: 239 passing, 2 ignored (up from 196 at M3.4)
Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
 2026-08-27 20:25:05 -07:00
rock a4a4053d57 feat: add Obsidian vault projection with Longhorn storage (#13)
Build and Push / Test (push) Successful in 3m37s
Details
Build and Push / Build and push image (push) Successful in 2m45s
Details
2026-08-24 01:58:39 +00:00
Story Crater Bot b9482474a6 Add ArgoCD Application for auto-deployment (poimen-memory-app)
ci / markdown (push) Waiting to run
Details
2026-08-23 09:46:58 -07:00
Story Crater Bot 43239d24ce Implement M3.6.1: DocCorpusSource with heading-boundary chunking (196 tests)
ci / markdown (push) Waiting to run
Details
2026-08-23 09:42:09 -07:00
Story Crater Bot a0ebc1183c Add K8s app deployment, Dockerfile, and CI workflow (Option A)
ci / markdown (push) Waiting to run
Details
2026-08-23 00:01:30 -07:00
Story Crater Bot 906c6c32a4 Downsize memory-db to 2 instances
ci / markdown (push) Waiting to run
Details
2026-08-22 23:53:05 -07:00
Story Crater Bot d3070f087d Fix: use default longhorn (3 replicas), increase to 20Gi
ci / markdown (push) Waiting to run
Details
2026-08-22 23:40:08 -07:00
Story Crater Bot a1a8635a41 Fix: use longhorn-imessage-local (WaitForFirstConsumer) for stable volume binding
ci / markdown (push) Waiting to run
Details
2026-08-22 23:36:25 -07:00
Story Crater Bot 6147137b45 Bundle memory database into homelab orchestration (remove separate app)
ci / markdown (push) Waiting to run
Details
2026-08-22 23:16:39 -07:00
Story Crater Bot d3be7f6fd4 Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)
ci / markdown (push) Waiting to run
Details
2026-08-22 23:13:42 -07:00