poimen-memory/docs/API_VAULT_ENDPOINTS.md

# Memory Service API — Vault Endpoints & Hybrid Search

## Overview

Memory Service now exposes JSON API endpoints for vault browsing and hybrid search (semantic + lexical).

**Deployment:** vault.riotpiao.com for vault endpoints, memory.riotpiao.com for full API

## Vault Endpoints

All vault endpoints return JSON (not HTML). Authentication via JWT (Authentik).

### 1. List All Projects

**GET /memory/vault**

Returns all projects with memories.

```bash
curl -H "Authorization: Bearer $JWT" \
  http://vault.riotpiao.com/memory/vault

# Response:
{
  "projects": [
    {"name": "poimen"},
    {"name": "refcorpus"},
    {"name": "devops"}
  ]
}
```

### 2. List Files in Project

**GET /memory/vault?project=<project>**

Returns file tree for a specific project.

```bash
curl -H "Authorization: Bearer $JWT" \
  'http://vault.riotpiao.com/memory/vault?project=poimen'

# Response:
{
  "project": "poimen",
  "files": [
    {
      "path": "poimen/index.md",
      "name": "index.md",
      "title": "index",
      "updated_at": "2025-01-27T15:30:45Z"
    },
    {
      "path": "poimen/query-123.md",
      "name": "query-123.md",
      "title": "query 123",
      "updated_at": "2025-01-27T15:25:00Z"
    }
  ]
}
```

### 3. Get File Content

**GET /memory/vault/{project}/{file}**

Returns markdown file with frontmatter parsed to JSON.

```bash
curl -H "Authorization: Bearer $JWT" \
  http://vault.riotpiao.com/memory/vault/poimen/query-123

# Response:
{
  "project": "poimen",
  "file": "query-123.md",
  "path": "poimen/query-123.md",
  "title": "query 123",
  "metadata": {
    "level": "L1",
    "query_id": "query-123",
    "updated": "2025-01-27T12:00:00Z",
    "chunks_seen": "100",
    "chunks_used": "50"
  },
  "content": "This is the memory text...\n\n## Provenance\n\n- [[pi-1]] — chunk 1\n- [[claude-2]] — chunk 2"
}
```

---

## Search Endpoints

Hybrid search combines semantic (pgvector) + lexical (OpenSearch) retrieval.

### Semantic Search Only

**GET /memory/query?project=<proj>&query=<q>&method=semantic**

Uses pgvector embeddings only. Fast, but misses exact-match terms.

```bash
curl -H "Authorization: Bearer $JWT" \
  'http://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes+port+conflict&method=semantic'

# Response:
{
  "query": "kubernetes port conflict",
  "project": "poimen",
  "method": "semantic",
  "results": [
    {
      "level": "L1",
      "score": 0.92,
      "text": "To fix port conflicts in Kubernetes...",
      "source": "claude",
      "provenance": ["pi-1", "claude-2"]
    }
  ]
}
```

### Lexical Search Only (OpenSearch not required)

**GET /memory/query?project=<proj>&query=<q>&method=lexical**

Uses BM25 exact-match terms. Better for structured queries.

```bash
curl -H "Authorization: Bearer $JWT" \
  'http://memory.riotpiao.com/memory/query?project=poimen&query=fix+port&method=lexical'

# Falls back to semantic if OpenSearch not available
```

### Hybrid Search (Recommended)

**GET /memory/query?project=<proj>&query=<q>&method=hybrid** (default)

Combines semantic (60%) + lexical (40%) scores. Best accuracy.

**Requires:** OpenSearch deployment

```bash
curl -H "Authorization: Bearer $JWT" \
  'http://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes+port+conflict'

# Response (with fallback):
{
  "query": "kubernetes port conflict",
  "project": "poimen",
  "method": "hybrid",  # or "semantic_fallback" if OpenSearch unavailable
  "results": [
    {
      "level": "L1",
      "score": 0.992,
      "text": "...",
      "source": "claude",
      "provenance": ["pi-1", "claude-2"]
    }
  ]
}
```

**Score Calculation (Hybrid):**
```
final_score = 0.6 * semantic_score + 0.4 * lexical_score
```

---

## Architecture

### Semantic Path (pgvector)

```
Query → LLM Embed (768-dim) → pgvector IVFFlat search
        ↓
      Top-50 results (cosine distance)
```

**Index:** `memory_vector (kind='Text')`  
**Partial Index:** `ON (kind = 'Text') WHERE level IN ('L0', 'L1')`

### Lexical Path (OpenSearch BM25)

```
Query → Tokenize → OpenSearch BM25 search (with JWT auth)
        ↓
      Top-50 results (TF-IDF score)
```

**Index:** `vault-* indices` with `multi_match` on `content^2, breadcrumb`  
**Security:** JWT realm validates Authentik tokens

### Fusion (Hybrid Only)

```
semantic_norm[0..1] + lexical_norm[0..1]
     ↓
merge results by ID
     ↓
final_score = 0.6*sem + 0.4*lex
     ↓
sort descending → top-10
```

---

## Deployment Checklist

### Prerequisites

- [ ] Memory Service pod running (with JWT validator configured)
- [ ] pgvector database running (memory-db-0/1)
- [ ] Authentik OIDC issuer configured

### Deploy OpenSearch (Optional for Hybrid)

```bash
# 1. Apply manifests
kubectl apply -k k8s/infra/databases/

# 2. Wait for OpenSearch cluster to be ready
kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch -w

# 3. Verify health
kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &
curl http://localhost:9200/_cluster/health
# Should see: "status":"green"
```

### Configure Memory Service

Set environment variables in deployment:

```yaml
env:
  - name: OPENSEARCH_HOSTS
    value: "opensearch-internal.poimen.svc.cluster.local:9200"
  - name: MEM_AUTH_MODE
    value: "jwt"
```

Restart pods:
```bash
kubectl rollout restart deployment poimen-memory -n poimen
```

### Test API

```bash
# Get JWT from Authentik
TOKEN=$(curl -X POST http://authentik:9000/application/o/token/ \
  -d "client_id=..." \
  -d "grant_type=client_credentials" | jq -r .access_token)

# Test vault endpoint
curl -H "Authorization: Bearer $TOKEN" \
  http://vault.riotpiao.com/memory/vault

# Test hybrid search
curl -H "Authorization: Bearer $TOKEN" \
  'http://memory.riotpiao.com/memory/query?project=poimen&query=fix+port'
```

---

## Migration Guide: HTML → JSON

### Before (Old)

```bash
GET /memory/vault
# Returns: <html><body>..Project list...</body></html>

GET /memory/vault/poimen
# Returns: <html>File listing</html>

GET /memory/vault/poimen/query-123
# Returns: <html>Rendered markdown</html>
```

### After (New)

```bash
GET /memory/vault
# Returns: {"projects": [...]}

GET /memory/vault?project=poimen
# Returns: {"project": "poimen", "files": [...]}

GET /memory/vault/poimen/query-123
# Returns: {"project": "...", "file": "...", "metadata": {...}, "content": "..."}
```

---

## Frontend Integration

### React/Vue Implementation

```typescript
// Vault browser
async function getProjectVault(project: string, token: string) {
  const res = await fetch(
    `/memory/vault?project=${project}`,
    { headers: { 'Authorization': `Bearer ${token}` } }
  );
  const data = await res.json();
  return data.files;  // Array of {path, name, title, updated_at}
}

// Get file content
async function getFileContent(project: string, file: string, token: string) {
  const res = await fetch(
    `/memory/vault/${project}/${file}`,
    { headers: { 'Authorization': `Bearer ${token}` } }
  );
  return await res.json();
  // {metadata: {...}, content: "..."}
}

// Hybrid search
async function search(query: string, project: string, token: string) {
  const res = await fetch(
    `/memory/query?project=${project}&query=${encodeURIComponent(query)}`,
    { headers: { 'Authorization': `Bearer ${token}` } }
  );
  const data = await res.json();
  return data.results;  // Top-10 hybrid results
}
```

---

## DNS & Ingress

### DNS Records

Add to your DNS:

```
vault.riotpiao.com      IN A 203.x.x.x  (cluster IP)
memory.riotpiao.com     IN A 203.x.x.x  (same)
```

### Ingress Configuration

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: memory-ingress
  namespace: poimen
spec:
  tls:
    - hosts:
        - vault.riotpiao.com
        - memory.riotpiao.com
      secretName: memory-tls
  rules:
    # Vault endpoints
    - host: vault.riotpiao.com
      http:
        paths:
          - path: /memory/vault
            pathType: Prefix
            backend:
              service:
                name: poimen-memory
                port:
                  number: 8080
    # Full API
    - host: memory.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: poimen-memory
                port:
                  number: 8080
```

---

## Fallback Behavior

If OpenSearch is unavailable:

1. Hybrid requests fall back to semantic-only (no error)
2. Returns `method: "semantic_fallback"` in response
3. Lexical-specific queries not supported (return 400 Bad Request)

To require hybrid (fail if unavailable):

```bash
curl '...?query=...&method=hybrid&strict=true'
# Returns 503 Service Unavailable if OpenSearch down
```

---

## Performance Tuning

### pgvector Index Parameters

```sql
-- Current: IVFFlat with 100 lists
CREATE INDEX ON memory_vector
USING ivfflat (embedding vector_cosine_ops)
WITH (lists=100);

-- For larger datasets (>1M vectors):
-- Use lists=sqrt(rows), e.g., lists=1000 for 1M
```

### OpenSearch Shard Configuration

```yaml
# In opensearch.yaml
index:
  number_of_shards: 3
  number_of_replicas: 1
  codec: best_compression
```

### Caching

OpenSearchClient has 1-hour query cache. Clear if needed:

```bash
curl -X POST http://opensearch:9200/vault-*/_cache/clear
```

---

## Security Considerations

### JWT Validation

✅ Memory Service validates Authentik tokens
✅ OpenSearch has JWT realm configured
⚠️  No TLS between Memory Service → OpenSearch (K8s network isolated)

### Rate Limiting

```
/memory/vault/*: 100 req/hr per API key
/memory/query: 1000 req/hr per API key
```

### Field-Level Access Control

⚠️  Future: row-level security per project_id (not yet implemented)

---

## Metrics

Monitor these endpoints for production:

```prometheus
# Latency
histogram_quantile(0.95, http_request_duration_seconds{endpoint="/memory/query"})

# Cache hit rate
opensearch_query_cache_hit_count / (opensearch_query_cache_hit_count + opensearch_query_cache_miss_count)

# Cluster health
opensearch_cluster_health_status
```

---

## Next Steps

1. ✅ Deploy OpenSearch manifests (`k8s/infra/databases/opensearch.yaml`)
2. ✅ Configure Memory Service env vars (OPENSEARCH_HOSTS)
3. ✅ Update ingress for vault.riotpiao.com
4. ⬜ Frontend React app (vault browser UI, search form)
5. ⬜ GRC endpoints (git + merge workflow)