poimen-memory/README.md

# Poimen Memory

> **Poimen** (ποιμήν) — Greek for "shepherd". Guiding AI agents to grounded knowledge.

**Agent-ready Graph-RAG system with hallucination prevention and enterprise RBAC.**

Poimen Memory is a knowledge retrieval system designed for AI agents. It learns from conversations and documents, builds wiki-link knowledge graphs, and serves grounded context that reduces hallucinations. Agents cite sources instead of fabricating answers.

## Why Poimen?

| Problem | Poimen Solution |
|---------|-----------------|
| LLMs hallucinate facts | Three-tier retrieval grounds responses in verified knowledge |
| Vector search misses context | Wiki-link graph propagates relevance to connected docs |
| Agents forget across sessions | Persistent memory with provenance tracking |
| Multi-tenant data leakage | Hierarchical RBAC with project/visibility scopes |
| Context window limits | Budget-aware assembly with intelligent compression |

## Architecture

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                              AI Agents                                       │
│                    (Claude, GPT, Local LLMs, etc.)                          │
└─────────────────────────────────┬───────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         Poimen Memory API                                    │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐ │
│  │   /query    │  │  /context   │  │   /ingest   │  │      /learn         │ │
│  │ Hybrid RAG  │  │ Three-Tier  │  │  Add Facts  │  │ Chunk + Synthesize  │ │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────────┬──────────┘ │
│         │                │                │                     │           │
│         └────────────────┴────────────────┴─────────────────────┘           │
│                                   │                                          │
│                          ┌────────┴────────┐                                │
│                          │  Access Guard   │  ← JWT roles + RBAC scopes     │
│                          │  (Authentik)    │                                │
│                          └────────┬────────┘                                │
└───────────────────────────────────┼─────────────────────────────────────────┘
                                    │
        ┌───────────────────────────┼───────────────────────────┐
        │                           │                           │
        ▼                           ▼                           ▼
┌───────────────┐         ┌─────────────────┐         ┌─────────────────┐
│   pgvector    │         │   OpenSearch    │         │    Obsidian     │
│  (Semantic)   │         │   (Lexical)     │         │  (Reference)    │
│               │         │                 │         │                 │
│ HNSW cosine   │         │  BM25 ranking   │         │  Markdown docs  │
│ 768-dim vecs  │         │  Full-text      │         │  Wiki-links     │
└───────────────┘         └─────────────────┘         └─────────────────┘
        │                           │                           │
        └───────────────────────────┴───────────────────────────┘
                                    │
                          ┌─────────┴─────────┐
                          │  Wiki-Link Graph  │
                          │  PageRank boost   │
                          │  Provenance trace │
                          └───────────────────┘
```

## Core Features

### 1. Graph-RAG Retrieval

Traditional RAG retrieves isolated chunks. Poimen builds a **wiki-link graph** from `[[linked-documents]]` and propagates relevance scores to connected knowledge.

```
Document A: "Kubernetes uses [[etcd]] for state storage"
Document B: "[[etcd]] requires TLS certificates"
Document C: "Generate certs with [[cfssl]]"

Query: "Kubernetes certificate issues"
  → Finds A (direct match)
  → Boosts B (linked from A)
  → Surfaces C (2-hop connection)
```

### 2. Three-Tier Context Lookup

Agents call `/memory/context` with tool + task + failure log. Poimen returns grounded knowledge in priority order:

| Tier | Source | Latency | Use Case |
|------|--------|---------|----------|
| **Tier 1** | Exact signature match | <50ms | Known error patterns |
| **Tier 2** | Graph-boosted hybrid search | <500ms | Similar problems |
| **Tier 3** | Reference corpus fallback | <1s | Documentation |

```bash
curl -X POST /memory/context \
  -d '{"tool": "kubectl", "task": "debug-pod", "failure_log": "CrashLoopBackOff"}'

# Returns:
{
  "tier": 1,
  "lessons": [{
    "text": "CrashLoopBackOff: check container logs with kubectl logs -p",
    "seen_count": 23,
    "provenance": ["session-123", "session-456"]
  }]
}
```

### 3. Hallucination Prevention

Every retrieved chunk includes:

- **`provenance[]`** — Which sessions/documents contributed this fact
- **`source`** — Original file or conversation URI
- **`seen_count`** — How many times this pattern was observed
- **`score`** — Retrieval confidence (semantic + lexical + graph boost)

Agents can cite sources: *"Based on 23 previous occurrences (source: troubleshooting/k8s.md)..."*

### 4. Hierarchical RBAC

Two-level access control integrated with Authentik OIDC:

**Level 1: Capabilities** (HTTP endpoint access)
```
memory:read   → /query, /context, /projects, /skills
memory:write  → /ingest, /learn
*             → all endpoints (admin)
```

**Level 2: Resource Scopes** (fine-grained filtering)

| Scope | Description | Example |
|-------|-------------|--------|
| `projects` | Allowed project names | `[homelab, portfolio]` |
| `visibility` | Public or private docs | `public` |
| `owner` | Resource ownership | `self` (own only) |
| `groups` | Required group membership | `[engineering]` |

**Built-in Roles:**

```yaml
# Admin: full access
- role: admin
  rules:
    - resources: ["*"]
      verbs: [read, write, delete, query]

# Portfolio visitor: public docs only, own conversations
- role: portfolio-agent
  rules:
    - resources: [wiki, embedding]
      verbs: [read, query]
      scope:
        projects: [homelab, portfolio]
        visibility: public
    - resources: [conversation]
      verbs: [read, write]
      scope:
        owner: self  # Can only access own conversations

# Authenticated user: all docs, own conversations
- role: authenticated-user
  rules:
    - resources: [wiki, embedding, skill]
      verbs: [read, query]
      # No visibility restriction → sees public + private
    - resources: [conversation]
      verbs: [read, write, delete]
      scope:
        owner: self
```

**JWT Claims → RBAC:**
```json
{
  "sub": "alice",
  "roles": ["authenticated-user", "homelab-team"],
  "groups": ["engineering"],
  "permissions": ["memory:read", "memory:write"]
}
```

Agents only retrieve knowledge they're authorized to access. Results are filtered post-retrieval by `AccessGuard`.

### 5. Budget-Aware Context Assembly

LLM context windows are limited. Poimen optimizes what fits:

```
Budget: 8192 tokens
  │
  ├─ Tier 1 lessons (never dropped)     → 2000 tokens
  ├─ Tier 2 relevant chunks             → 4000 tokens
  ├─ Tier 3 reference excerpts          → 1500 tokens
  └─ Skills/tools                       →  500 tokens
                                        ────────────
                                          8000 tokens ✓

If over budget:
  1. Drop Tier 3 first
  2. Drop lowest-score Tier 2
  3. Compress remaining chunks
  4. Never drop Tier 1
```

## Quick Start

### Prerequisites

- Rust 1.75+
- PostgreSQL 15+ with pgvector extension
- OpenSearch 2.x
- (Optional) Authentik for OIDC

### Run Locally

```bash
# Clone
git clone https://github.com/your-org/poimen-memory.git
cd poimen-memory

# Start dependencies
docker-compose up -d postgres opensearch

# Configure
cp .env.example .env
# Edit .env with your settings

# Build and run
cargo build --release
./target/release/mem serve

# Health check
curl http://localhost:8080/health
```

### Docker

```bash
docker run -d \
  -e PGVECTOR_HOST=postgres:5432 \
  -e OPENSEARCH_HOST=opensearch:9200 \
  -p 8080:8080 \
  ghcr.io/your-org/poimen-memory:latest
```

### Kubernetes (ArgoCD)

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: poimen-memory
spec:
  source:
    repoURL: https://github.com/your-org/poimen-memory
    path: k8s/app
  destination:
    namespace: poimen
```

## API Usage

### Ingest Knowledge

```bash
# From conversation
curl -X POST http://localhost:8080/memory/ingest \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "project": "homelab",
    "source": "conversation://claude/session-123",
    "records": [
      {"text": "To fix port 8080 conflict, use: kubectl delete pod -l app=nginx"},
      {"text": "etcd backup: etcdctl snapshot save /backup/etcd.db"}
    ]
  }'
```

### Query Memory

```bash
# Hybrid search (semantic + lexical + graph)
curl "http://localhost:8080/memory/query?project=homelab&query=kubernetes%20port%20conflict" \
  -H "Authorization: Bearer $TOKEN"

# Response
{
  "results": [{
    "text": "To fix port 8080 conflict...",
    "score": 0.92,
    "source": "conversation://claude/session-123",
    "provenance": ["session-123"]
  }]
}
```

### Get Agent Context

```bash
# Tool-specific context with failure diagnosis
curl -X POST http://localhost:8080/memory/context \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project": "homelab",
    "tool": "kubectl",
    "task": "debug-pod",
    "failure_log": "Error: ImagePullBackOff",
    "budget": 4096
  }'
```

### Learn from Documents

```bash
# Chunk, embed, and synthesize
curl -X POST http://localhost:8080/memory/learn \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project": "homelab",
    "text": "# Kubernetes Networking\n\nPods communicate via [[CNI]] plugins...",
    "chunk_size": 2000
  }'
```

## Retrieval Pipeline

```
Query: "fix kubernetes certificate error"
                    │
                    ▼
        ┌───────────────────────┐
        │   Query Optimizer     │
        │  Classify: bug_fix    │
        │  Route: hybrid        │
        └───────────┬───────────┘
                    │
        ┌───────────┴───────────┐
        │                       │
        ▼                       ▼
┌───────────────┐       ┌───────────────┐
│   Semantic    │       │    Lexical    │
│   pgvector    │       │  OpenSearch   │
│  cosine sim   │       │    BM25       │
└───────┬───────┘       └───────┬───────┘
        │                       │
        └───────────┬───────────┘
                    │
                    ▼
        ┌───────────────────────┐
        │     RRF Fusion        │
        │  60% semantic         │
        │  40% lexical          │
        └───────────┬───────────┘
                    │
                    ▼
        ┌───────────────────────┐
        │  Wiki-Link Graph      │
        │  PageRank boost       │
        │  Link-distance decay  │
        └───────────┬───────────┘
                    │
                    ▼
        ┌───────────────────────┐
        │   RBAC Filter         │
        │  Project scope        │
        │  Visibility check     │
        └───────────┬───────────┘
                    │
                    ▼
        ┌───────────────────────┐
        │  Deduplication        │
        │  Shingle Jaccard      │
        │  >0.5 = duplicate     │
        └───────────┬───────────┘
                    │
                    ▼
        ┌───────────────────────┐
        │  Budget Assembly      │
        │  Rank by score        │
        │  Fit to token limit   │
        └───────────┬───────────┘
                    │
                    ▼
            Final Results
         (with provenance)
```

## Configuration

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `PGVECTOR_HOST` | PostgreSQL host | `localhost:5432` |
| `PGVECTOR_DB` | Database name | `memory` |
| `OPENSEARCH_HOST` | OpenSearch host | `localhost:9200` |
| `OBSIDIAN_URL` | Obsidian REST API | (optional) |
| `MEM_AUTH_MODE` | `jwt` or `apikey` | `jwt` |
| `AUTHENTIK_ISSUER` | OIDC issuer URL | (required for jwt) |
| `RBAC_ROLES_DIR` | Custom role definitions | (builtin only) |

### Custom Roles

```yaml
# config/roles/my-team.yaml
name: my-team
rules:
  - resources: [wiki, embedding]
    verbs: [read, write, query]
    scope:
      projects: [my-project]
      visibility: private  # Can access private docs
```

## Project Structure

```
poimen-memory/
├── crates/
│   ├── mem-cli/          # HTTP server, RBAC, handlers
│   │   └── src/
│   │       ├── http_server.rs
│   │       ├── rbac/     # Access control
│   │       ├── hybrid_retrieval.rs
│   │       └── query_optimizer.rs
│   ├── mem-core/         # Domain types, scoring
│   └── mem-ingest/       # Wiki-link parsing, chunking
├── config/
│   └── roles/            # YAML role definitions
├── docs/
│   ├── API.md            # API reference
│   └── RBAC.md           # Access control guide
├── k8s/                  # Kubernetes manifests
└── tests/                # Integration tests (670+)
```

## Performance

| Metric | Target | Actual |
|--------|--------|--------|
| Tier 1 latency | <50ms | 12ms |
| Hybrid search | <500ms | 145ms |
| NDCG@10 | >0.85 | 0.88 |
| Test coverage | >600 | 670 |

## Contributing

```bash
# Run tests
cargo test --all

# Run specific test
cargo test -p mem-cli http_server::tests

# Check formatting
cargo fmt --check
cargo clippy
```

## License

MIT