 Story Crater Bot
|
277d719278
|
feat: Memory Service API ready for deployment — Vault JSON endpoints + Hybrid search
API Changes (crates/mem-cli/src/http_server.rs):
✅ Vault Endpoints (JSON API):
- GET /memory/vault → {projects: [...]}
- GET /memory/vault?project=X → {project: X, files: [...]}
- GET /memory/vault/{proj}/{file} → {metadata: {...}, content: '...'}
- YAML frontmatter parsed to JSON metadata
- Auth: JWT on all endpoints
✅ Search Endpoints:
- GET /memory/query?method=semantic → pgvector only (60% weight)
- GET /memory/query?method=hybrid (default) → pgvector + OpenSearch (fallback to semantic)
- Hybrid score: 0.6*semantic + 0.4*lexical
- Limit: top-10 results (default)
✅ AppState Extended:
- opensearch_client: Option<Arc<OpenSearchClient>>
- Initialized from OPENSEARCH_HOSTS env var (optional)
- Graceful fallback if OpenSearch unavailable
✅ Handlers Updated:
- vault_browser_handler() → returns JSON projects list
- vault_project_tree() → helper for file tree generation
- vault_project_handler() → GET /{project} → file tree JSON
- vault_file_handler() → GET /{project}/{file} → JSON with metadata + content
- query_handler() → hybrid search with semantic fallback
K8s Manifests (k8s/infra/databases/opensearch.yaml):
✅ OpenSearch StatefulSet:
- 2 replicas for HA cluster (opensearch-0, opensearch-1)
- Image: opensearchproject/opensearch:2.11.0
- Services: opensearch (headless), opensearch-internal (ClusterIP 9200)
- ConfigMap: opensearch.yml with cluster settings
- PVC: 30Gi per pod (Longhorn storage class)
- ServiceAccount + NetworkPolicy (Memory Service only)
- Init container: set vm.max_map_count=262144
- Probes: liveness (60s), readiness (30s)
- Resources: 512Mi-1Gi memory, 250m-500m CPU
- Security: plugins.security.disabled (K8s network isolated)
✅ Updated kustomization.yaml:
- Added opensearch.yaml to resources
Documentation:
✅ docs/API_VAULT_ENDPOINTS.md (10KB):
- Complete API reference with examples
- Architecture: semantic (pgvector IVFFlat) + lexical (OpenSearch BM25)
- Fusion strategy: weighted linear combination (60/40 split)
- DNS records for vault.riotpiao.com + memory.riotpiao.com
- Ingress configuration (dual-domain routing)
- Frontend integration examples (React/Vue)
- Fallback behavior (graceful degradation)
- Performance tuning (IVFFlat lists, OpenSearch shards)
- Security: JWT validation, rate limiting, field-level ACL (future)
✅ docs/DEPLOYMENT_CHECKLIST.md (8KB):
- 5-phase deployment plan (API ready, OpenSearch, DNS, Testing, Frontend)
- Step-by-step deployment commands
- Testing procedures for vault + search endpoints
- Troubleshooting: OpenSearch not found, cluster red, JWT validation
- Monitoring metrics + dashboard queries
- Fallback scenarios + error codes
Environment Variables:
- OPENSEARCH_HOSTS (optional, e.g., "opensearch-internal.poimen.svc.cluster.local:9200")
- If unset: hybrid search disabled, falls back to semantic
- CSV list supported: "host1:9200,host2:9200"
Deployment Summary:
1. ✅ API code ready (JSON endpoints, fallback to semantic if OpenSearch unavailable)
2. ✅ OpenSearch K8s manifests (StatefulSet + networking)
3. ✅ Documentation (API reference + deployment guide)
4. ⏳ Ready to: kubectl apply -k k8s/infra/databases/
Backward Compatibility:
✅ Existing JSON endpoints work without change
⚠️ HTML endpoints replaced with JSON (breaking change for old clients)
✅ Graceful fallback: hybrid search → semantic if OpenSearch missing
✅ Rate limiting preserved on all endpoints
Testing Ready:
- Vault tree endpoint testable after deployment
- Hybrid search testable once OpenSearch cluster ready
- All endpoints require JWT from Authentik
- Load test script provided
Next: Deploy OpenSearch + test against vault.riotpiao.com
|
2026-08-27 21:05:09 -07:00 |
|
|
Story Crater Bot
|
56bee1915e
|
chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1)
Deleted 31 completed task files:
- M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate)
- M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate)
- M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate)
- M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate)
- M3.6.1: DocCorpusSource (heading-boundary chunking)
- M4.1-2: skill-draft, derived-filter
Updated INDEX.md:
- Removed M0 & M1 phase sections (archived in git history)
- Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜)
- Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done
- Noted M3.5.10 JWT auth implementation complete (awaiting image rollout)
- Cleaned up broken links to deleted task files
Total test count: 239 passing, 2 ignored (up from 196 at M3.4)
Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
|
2026-08-27 20:25:05 -07:00 |
|
rock