poimen-memory/docs/API_VAULT_ENDPOINTS.md

Memory Service API — Vault Endpoints & Hybrid Search

Overview

Memory Service now exposes JSON API endpoints for vault browsing and hybrid search (semantic + lexical).

Deployment: vault.riotpiao.com for vault endpoints, memory.riotpiao.com for full API

Vault Endpoints

All vault endpoints return JSON (not HTML). Authentication via JWT (Authentik).

1. List All Projects

GET /memory/vault

Returns all projects with memories.

curl -H "Authorization: Bearer $JWT" \
  http://vault.riotpiao.com/memory/vault

# Response:
{
  "projects": [
    {"name": "poimen"},
    {"name": "refcorpus"},
    {"name": "devops"}
  ]
}

2. List Files in Project

GET /memory/vault?project=

Returns file tree for a specific project.

curl -H "Authorization: Bearer $JWT" \
  'http://vault.riotpiao.com/memory/vault?project=poimen'

# Response:
{
  "project": "poimen",
  "files": [
    {
      "path": "poimen/index.md",
      "name": "index.md",
      "title": "index",
      "updated_at": "2025-01-27T15:30:45Z"
    },
    {
      "path": "poimen/query-123.md",
      "name": "query-123.md",
      "title": "query 123",
      "updated_at": "2025-01-27T15:25:00Z"
    }
  ]
}

3. Get File Content

GET /memory/vault/{project}/{file}

Returns markdown file with frontmatter parsed to JSON.

curl -H "Authorization: Bearer $JWT" \
  http://vault.riotpiao.com/memory/vault/poimen/query-123

# Response:
{
  "project": "poimen",
  "file": "query-123.md",
  "path": "poimen/query-123.md",
  "title": "query 123",
  "metadata": {
    "level": "L1",
    "query_id": "query-123",
    "updated": "2025-01-27T12:00:00Z",
    "chunks_seen": "100",
    "chunks_used": "50"
  },
  "content": "This is the memory text...\n\n## Provenance\n\n- [[pi-1]] — chunk 1\n- [[claude-2]] — chunk 2"
}

---

Search Endpoints

Hybrid search combines semantic (pgvector) + lexical (OpenSearch) retrieval.

Semantic Search Only

GET /memory/query?project=&query=&method=semantic

Uses pgvector embeddings only. Fast, but misses exact-match terms.

curl -H "Authorization: Bearer $JWT" \
  'http://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes+port+conflict&method=semantic'

# Response:
{
  "query": "kubernetes port conflict",
  "project": "poimen",
  "method": "semantic",
  "results": [
    {
      "level": "L1",
      "score": 0.92,
      "text": "To fix port conflicts in Kubernetes...",
      "source": "claude",
      "provenance": ["pi-1", "claude-2"]
    }
  ]
}

Lexical Search Only (OpenSearch not required)

GET /memory/query?project=&query=&method=lexical

Uses BM25 exact-match terms. Better for structured queries.

curl -H "Authorization: Bearer $JWT" \
  'http://memory.riotpiao.com/memory/query?project=poimen&query=fix+port&method=lexical'

# Falls back to semantic if OpenSearch not available

Hybrid Search (Recommended)

GET /memory/query?project=&query=&method=hybrid (default)

Combines semantic (60%) + lexical (40%) scores. Best accuracy.

Requires: OpenSearch deployment

curl -H "Authorization: Bearer $JWT" \
  'http://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes+port+conflict'

# Response (with fallback):
{
  "query": "kubernetes port conflict",
  "project": "poimen",
  "method": "hybrid",  # or "semantic_fallback" if OpenSearch unavailable
  "results": [
    {
      "level": "L1",
      "score": 0.992,
      "text": "...",
      "source": "claude",
      "provenance": ["pi-1", "claude-2"]
    }
  ]
}

Score Calculation (Hybrid):

final_score = 0.6 * semantic_score + 0.4 * lexical_score

---

Architecture

Semantic Path (pgvector)

Query → LLM Embed (768-dim) → pgvector IVFFlat search
        ↓
      Top-50 results (cosine distance)

Index: memory_vector (kind='Text')
Partial Index: ON (kind = 'Text') WHERE level IN ('L0', 'L1')

Lexical Path (OpenSearch BM25)

Query → Tokenize → OpenSearch BM25 search (with JWT auth)
        ↓
      Top-50 results (TF-IDF score)

Index: vault-* indices with multi_match on content^2, breadcrumb
Security: JWT realm validates Authentik tokens

Fusion (Hybrid Only)

semantic_norm[0..1] + lexical_norm[0..1]
     ↓
merge results by ID
     ↓
final_score = 0.6*sem + 0.4*lex
     ↓
sort descending → top-10

---

Deployment Checklist

Prerequisites

• Memory Service pod running (with JWT validator configured)
• pgvector database running (memory-db-0/1)
• Authentik OIDC issuer configured

Deploy OpenSearch (Optional for Hybrid)

# 1. Apply manifests
kubectl apply -k k8s/infra/databases/

# 2. Wait for OpenSearch cluster to be ready
kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch -w

# 3. Verify health
kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &
curl http://localhost:9200/_cluster/health
# Should see: "status":"green"

Configure Memory Service

Set environment variables in deployment:

env:
  - name: OPENSEARCH_HOSTS
    value: "opensearch-internal.poimen.svc.cluster.local:9200"
  - name: MEM_AUTH_MODE
    value: "jwt"

Restart pods:

kubectl rollout restart deployment poimen-memory -n poimen

Test API

# Get JWT from Authentik
TOKEN=$(curl -X POST http://authentik:9000/application/o/token/ \
  -d "client_id=..." \
  -d "grant_type=client_credentials" | jq -r .access_token)

# Test vault endpoint
curl -H "Authorization: Bearer $TOKEN" \
  http://vault.riotpiao.com/memory/vault

# Test hybrid search
curl -H "Authorization: Bearer $TOKEN" \
  'http://memory.riotpiao.com/memory/query?project=poimen&query=fix+port'

---

Migration Guide: HTML → JSON

Before (Old)

GET /memory/vault
# Returns: <html><body>..Project list...</body></html>

GET /memory/vault/poimen
# Returns: <html>File listing</html>

GET /memory/vault/poimen/query-123
# Returns: <html>Rendered markdown</html>

After (New)

GET /memory/vault
# Returns: {"projects": [...]}

GET /memory/vault?project=poimen
# Returns: {"project": "poimen", "files": [...]}

GET /memory/vault/poimen/query-123
# Returns: {"project": "...", "file": "...", "metadata": {...}, "content": "..."}

---

Frontend Integration

React/Vue Implementation

// Vault browser
async function getProjectVault(project: string, token: string) {
  const res = await fetch(
    `/memory/vault?project=${project}`,
    { headers: { 'Authorization': `Bearer ${token}` } }
  );
  const data = await res.json();
  return data.files;  // Array of {path, name, title, updated_at}
}

// Get file content
async function getFileContent(project: string, file: string, token: string) {
  const res = await fetch(
    `/memory/vault/${project}/${file}`,
    { headers: { 'Authorization': `Bearer ${token}` } }
  );
  return await res.json();
  // {metadata: {...}, content: "..."}
}

// Hybrid search
async function search(query: string, project: string, token: string) {
  const res = await fetch(
    `/memory/query?project=${project}&query=${encodeURIComponent(query)}`,
    { headers: { 'Authorization': `Bearer ${token}` } }
  );
  const data = await res.json();
  return data.results;  // Top-10 hybrid results
}

---

DNS & Ingress

DNS Records

Add to your DNS:

vault.riotpiao.com      IN A 203.x.x.x  (cluster IP)
memory.riotpiao.com     IN A 203.x.x.x  (same)

Ingress Configuration

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: memory-ingress
  namespace: poimen
spec:
  tls:
    - hosts:
        - vault.riotpiao.com
        - memory.riotpiao.com
      secretName: memory-tls
  rules:
    # Vault endpoints
    - host: vault.riotpiao.com
      http:
        paths:
          - path: /memory/vault
            pathType: Prefix
            backend:
              service:
                name: poimen-memory
                port:
                  number: 8080
    # Full API
    - host: memory.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: poimen-memory
                port:
                  number: 8080

---

Fallback Behavior

If OpenSearch is unavailable:

1. Hybrid requests fall back to semantic-only (no error)
2. Returns method: "semantic_fallback" in response
3. Lexical-specific queries not supported (return 400 Bad Request)

To require hybrid (fail if unavailable):

curl '...?query=...&method=hybrid&strict=true'
# Returns 503 Service Unavailable if OpenSearch down

---

Performance Tuning

pgvector Index Parameters

-- Current: IVFFlat with 100 lists
CREATE INDEX ON memory_vector
USING ivfflat (embedding vector_cosine_ops)
WITH (lists=100);

-- For larger datasets (>1M vectors):
-- Use lists=sqrt(rows), e.g., lists=1000 for 1M

OpenSearch Shard Configuration

# In opensearch.yaml
index:
  number_of_shards: 3
  number_of_replicas: 1
  codec: best_compression

Caching

OpenSearchClient has 1-hour query cache. Clear if needed:

curl -X POST http://opensearch:9200/vault-*/_cache/clear

---

Security Considerations

JWT Validation

✅ Memory Service validates Authentik tokens ✅ OpenSearch has JWT realm configured ⚠️ No TLS between Memory Service → OpenSearch (K8s network isolated)

Rate Limiting

/memory/vault/*: 100 req/hr per API key
/memory/query: 1000 req/hr per API key

Field-Level Access Control

⚠️ Future: row-level security per project_id (not yet implemented)

---

Metrics

Monitor these endpoints for production:

# Latency
histogram_quantile(0.95, http_request_duration_seconds{endpoint="/memory/query"})

# Cache hit rate
opensearch_query_cache_hit_count / (opensearch_query_cache_hit_count + opensearch_query_cache_miss_count)

# Cluster health
opensearch_cluster_health_status

---

Next Steps

1. ✅ Deploy OpenSearch manifests (k8s/infra/databases/opensearch.yaml)
2. ✅ Configure Memory Service env vars (OPENSEARCH_HOSTS)
3. ✅ Update ingress for vault.riotpiao.com
4. ⬜ Frontend React app (vault browser UI, search form)
5. ⬜ GRC endpoints (git + merge workflow)