423 lines
15 KiB
Markdown
423 lines
15 KiB
Markdown
# Poimen Memory
|
|
|
|
> **Poimen** (ποιμήν) — Greek for "shepherd". Guiding AI agents to grounded knowledge.
|
|
|
|
**Agent-ready Graph-RAG system with hallucination prevention and enterprise RBAC.**
|
|
|
|
Poimen Memory is a knowledge retrieval system designed for AI agents. It learns from conversations and documents, builds wiki-link knowledge graphs, and serves grounded context that reduces hallucinations. Agents cite sources instead of fabricating answers.
|
|
|
|
## Why Poimen?
|
|
|
|
| Problem | Poimen Solution |
|
|
|---------|-----------------|
|
|
| LLMs hallucinate facts | Three-tier retrieval grounds responses in verified knowledge |
|
|
| Vector search misses context | Wiki-link graph propagates relevance to connected docs |
|
|
| Agents forget across sessions | Persistent memory with provenance tracking |
|
|
| Multi-tenant data leakage | Hierarchical RBAC with project/visibility scopes |
|
|
| Context window limits | Budget-aware assembly with intelligent compression |
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ AI Agents │
|
|
│ (Claude, GPT, Local LLMs, etc.) │
|
|
└─────────────────────────────────┬───────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ Poimen Memory API │
|
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
|
|
│ │ /query │ │ /context │ │ /ingest │ │ /learn │ │
|
|
│ │ Hybrid RAG │ │ Three-Tier │ │ Add Facts │ │ Chunk + Synthesize │ │
|
|
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────────┬──────────┘ │
|
|
│ │ │ │ │ │
|
|
│ └────────────────┴────────────────┴─────────────────────┘ │
|
|
│ │ │
|
|
│ ┌────────┴────────┐ │
|
|
│ │ Access Guard │ ← JWT roles + RBAC scopes │
|
|
│ │ (Authentik) │ │
|
|
│ └────────┬────────┘ │
|
|
└───────────────────────────────────┼─────────────────────────────────────────┘
|
|
│
|
|
┌───────────────────────────┼───────────────────────────┐
|
|
│ │ │
|
|
▼ ▼ ▼
|
|
┌───────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
│ pgvector │ │ OpenSearch │ │ Obsidian │
|
|
│ (Semantic) │ │ (Lexical) │ │ (Reference) │
|
|
│ │ │ │ │ │
|
|
│ HNSW cosine │ │ BM25 ranking │ │ Markdown docs │
|
|
│ 768-dim vecs │ │ Full-text │ │ Wiki-links │
|
|
└───────────────┘ └─────────────────┘ └─────────────────┘
|
|
│ │ │
|
|
└───────────────────────────┴───────────────────────────┘
|
|
│
|
|
┌─────────┴─────────┐
|
|
│ Wiki-Link Graph │
|
|
│ PageRank boost │
|
|
│ Provenance trace │
|
|
└───────────────────┘
|
|
```
|
|
|
|
## Core Features
|
|
|
|
### 1. Graph-RAG Retrieval
|
|
|
|
Traditional RAG retrieves isolated chunks. Poimen builds a **wiki-link graph** from `[[linked-documents]]` and propagates relevance scores to connected knowledge.
|
|
|
|
```
|
|
Document A: "Kubernetes uses [[etcd]] for state storage"
|
|
Document B: "[[etcd]] requires TLS certificates"
|
|
Document C: "Generate certs with [[cfssl]]"
|
|
|
|
Query: "Kubernetes certificate issues"
|
|
→ Finds A (direct match)
|
|
→ Boosts B (linked from A)
|
|
→ Surfaces C (2-hop connection)
|
|
```
|
|
|
|
### 2. Three-Tier Context Lookup
|
|
|
|
Agents call `/memory/context` with tool + task + failure log. Poimen returns grounded knowledge in priority order:
|
|
|
|
| Tier | Source | Latency | Use Case |
|
|
|------|--------|---------|----------|
|
|
| **Tier 1** | Exact signature match | <50ms | Known error patterns |
|
|
| **Tier 2** | Graph-boosted hybrid search | <500ms | Similar problems |
|
|
| **Tier 3** | Reference corpus fallback | <1s | Documentation |
|
|
|
|
```bash
|
|
curl -X POST /memory/context \
|
|
-d '{"tool": "kubectl", "task": "debug-pod", "failure_log": "CrashLoopBackOff"}'
|
|
|
|
# Returns:
|
|
{
|
|
"tier": 1,
|
|
"lessons": [{
|
|
"text": "CrashLoopBackOff: check container logs with kubectl logs -p",
|
|
"seen_count": 23,
|
|
"provenance": ["session-123", "session-456"]
|
|
}]
|
|
}
|
|
```
|
|
|
|
### 3. Hallucination Prevention
|
|
|
|
Every retrieved chunk includes:
|
|
|
|
- **`provenance[]`** — Which sessions/documents contributed this fact
|
|
- **`source`** — Original file or conversation URI
|
|
- **`seen_count`** — How many times this pattern was observed
|
|
- **`score`** — Retrieval confidence (semantic + lexical + graph boost)
|
|
|
|
Agents can cite sources: *"Based on 23 previous occurrences (source: troubleshooting/k8s.md)..."*
|
|
|
|
### 4. Hierarchical RBAC
|
|
|
|
Fine-grained access control integrated with Authentik OIDC:
|
|
|
|
```yaml
|
|
# Portfolio visitor: public docs only
|
|
- role: portfolio-agent
|
|
rules:
|
|
- resources: [wiki, embedding]
|
|
verbs: [read, query]
|
|
scope:
|
|
projects: [homelab, portfolio]
|
|
visibility: public
|
|
|
|
# Team member: full project access
|
|
- role: homelab-team
|
|
rules:
|
|
- resources: [wiki, embedding, skill]
|
|
verbs: [read, write, query]
|
|
scope:
|
|
projects: [homelab]
|
|
```
|
|
|
|
Agents only retrieve knowledge they're authorized to access. Prevents cross-project data leakage.
|
|
|
|
### 5. Budget-Aware Context Assembly
|
|
|
|
LLM context windows are limited. Poimen optimizes what fits:
|
|
|
|
```
|
|
Budget: 8192 tokens
|
|
│
|
|
├─ Tier 1 lessons (never dropped) → 2000 tokens
|
|
├─ Tier 2 relevant chunks → 4000 tokens
|
|
├─ Tier 3 reference excerpts → 1500 tokens
|
|
└─ Skills/tools → 500 tokens
|
|
────────────
|
|
8000 tokens ✓
|
|
|
|
If over budget:
|
|
1. Drop Tier 3 first
|
|
2. Drop lowest-score Tier 2
|
|
3. Compress remaining chunks
|
|
4. Never drop Tier 1
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
### Prerequisites
|
|
|
|
- Rust 1.75+
|
|
- PostgreSQL 15+ with pgvector extension
|
|
- OpenSearch 2.x
|
|
- (Optional) Authentik for OIDC
|
|
|
|
### Run Locally
|
|
|
|
```bash
|
|
# Clone
|
|
git clone https://github.com/your-org/poimen-memory.git
|
|
cd poimen-memory
|
|
|
|
# Start dependencies
|
|
docker-compose up -d postgres opensearch
|
|
|
|
# Configure
|
|
cp .env.example .env
|
|
# Edit .env with your settings
|
|
|
|
# Build and run
|
|
cargo build --release
|
|
./target/release/mem serve
|
|
|
|
# Health check
|
|
curl http://localhost:8080/health
|
|
```
|
|
|
|
### Docker
|
|
|
|
```bash
|
|
docker run -d \
|
|
-e PGVECTOR_HOST=postgres:5432 \
|
|
-e OPENSEARCH_HOST=opensearch:9200 \
|
|
-p 8080:8080 \
|
|
ghcr.io/your-org/poimen-memory:latest
|
|
```
|
|
|
|
### Kubernetes (ArgoCD)
|
|
|
|
```yaml
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: poimen-memory
|
|
spec:
|
|
source:
|
|
repoURL: https://github.com/your-org/poimen-memory
|
|
path: k8s/app
|
|
destination:
|
|
namespace: poimen
|
|
```
|
|
|
|
## API Usage
|
|
|
|
### Ingest Knowledge
|
|
|
|
```bash
|
|
# From conversation
|
|
curl -X POST http://localhost:8080/memory/ingest \
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"project": "homelab",
|
|
"source": "conversation://claude/session-123",
|
|
"records": [
|
|
{"text": "To fix port 8080 conflict, use: kubectl delete pod -l app=nginx"},
|
|
{"text": "etcd backup: etcdctl snapshot save /backup/etcd.db"}
|
|
]
|
|
}'
|
|
```
|
|
|
|
### Query Memory
|
|
|
|
```bash
|
|
# Hybrid search (semantic + lexical + graph)
|
|
curl "http://localhost:8080/memory/query?project=homelab&query=kubernetes%20port%20conflict" \
|
|
-H "Authorization: Bearer $TOKEN"
|
|
|
|
# Response
|
|
{
|
|
"results": [{
|
|
"text": "To fix port 8080 conflict...",
|
|
"score": 0.92,
|
|
"source": "conversation://claude/session-123",
|
|
"provenance": ["session-123"]
|
|
}]
|
|
}
|
|
```
|
|
|
|
### Get Agent Context
|
|
|
|
```bash
|
|
# Tool-specific context with failure diagnosis
|
|
curl -X POST http://localhost:8080/memory/context \
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
-d '{
|
|
"project": "homelab",
|
|
"tool": "kubectl",
|
|
"task": "debug-pod",
|
|
"failure_log": "Error: ImagePullBackOff",
|
|
"budget": 4096
|
|
}'
|
|
```
|
|
|
|
### Learn from Documents
|
|
|
|
```bash
|
|
# Chunk, embed, and synthesize
|
|
curl -X POST http://localhost:8080/memory/learn \
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
-d '{
|
|
"project": "homelab",
|
|
"text": "# Kubernetes Networking\n\nPods communicate via [[CNI]] plugins...",
|
|
"chunk_size": 2000
|
|
}'
|
|
```
|
|
|
|
## Retrieval Pipeline
|
|
|
|
```
|
|
Query: "fix kubernetes certificate error"
|
|
│
|
|
▼
|
|
┌───────────────────────┐
|
|
│ Query Optimizer │
|
|
│ Classify: bug_fix │
|
|
│ Route: hybrid │
|
|
└───────────┬───────────┘
|
|
│
|
|
┌───────────┴───────────┐
|
|
│ │
|
|
▼ ▼
|
|
┌───────────────┐ ┌───────────────┐
|
|
│ Semantic │ │ Lexical │
|
|
│ pgvector │ │ OpenSearch │
|
|
│ cosine sim │ │ BM25 │
|
|
└───────┬───────┘ └───────┬───────┘
|
|
│ │
|
|
└───────────┬───────────┘
|
|
│
|
|
▼
|
|
┌───────────────────────┐
|
|
│ RRF Fusion │
|
|
│ 60% semantic │
|
|
│ 40% lexical │
|
|
└───────────┬───────────┘
|
|
│
|
|
▼
|
|
┌───────────────────────┐
|
|
│ Wiki-Link Graph │
|
|
│ PageRank boost │
|
|
│ Link-distance decay │
|
|
└───────────┬───────────┘
|
|
│
|
|
▼
|
|
┌───────────────────────┐
|
|
│ RBAC Filter │
|
|
│ Project scope │
|
|
│ Visibility check │
|
|
└───────────┬───────────┘
|
|
│
|
|
▼
|
|
┌───────────────────────┐
|
|
│ Deduplication │
|
|
│ Shingle Jaccard │
|
|
│ >0.5 = duplicate │
|
|
└───────────┬───────────┘
|
|
│
|
|
▼
|
|
┌───────────────────────┐
|
|
│ Budget Assembly │
|
|
│ Rank by score │
|
|
│ Fit to token limit │
|
|
└───────────┬───────────┘
|
|
│
|
|
▼
|
|
Final Results
|
|
(with provenance)
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables
|
|
|
|
| Variable | Description | Default |
|
|
|----------|-------------|---------|
|
|
| `PGVECTOR_HOST` | PostgreSQL host | `localhost:5432` |
|
|
| `PGVECTOR_DB` | Database name | `memory` |
|
|
| `OPENSEARCH_HOST` | OpenSearch host | `localhost:9200` |
|
|
| `OBSIDIAN_URL` | Obsidian REST API | (optional) |
|
|
| `MEM_AUTH_MODE` | `jwt` or `apikey` | `jwt` |
|
|
| `AUTHENTIK_ISSUER` | OIDC issuer URL | (required for jwt) |
|
|
| `RBAC_ROLES_DIR` | Custom role definitions | (builtin only) |
|
|
|
|
### Custom Roles
|
|
|
|
```yaml
|
|
# config/roles/my-team.yaml
|
|
name: my-team
|
|
rules:
|
|
- resources: [wiki, embedding]
|
|
verbs: [read, write, query]
|
|
scope:
|
|
projects: [my-project]
|
|
visibility: private # Can access private docs
|
|
```
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
poimen-memory/
|
|
├── crates/
|
|
│ ├── mem-cli/ # HTTP server, RBAC, handlers
|
|
│ │ └── src/
|
|
│ │ ├── http_server.rs
|
|
│ │ ├── rbac/ # Access control
|
|
│ │ ├── hybrid_retrieval.rs
|
|
│ │ └── query_optimizer.rs
|
|
│ ├── mem-core/ # Domain types, scoring
|
|
│ └── mem-ingest/ # Wiki-link parsing, chunking
|
|
├── config/
|
|
│ └── roles/ # YAML role definitions
|
|
├── docs/
|
|
│ ├── API.md # API reference
|
|
│ └── RBAC.md # Access control guide
|
|
├── k8s/ # Kubernetes manifests
|
|
└── tests/ # Integration tests (670+)
|
|
```
|
|
|
|
## Performance
|
|
|
|
| Metric | Target | Actual |
|
|
|--------|--------|--------|
|
|
| Tier 1 latency | <50ms | 12ms |
|
|
| Hybrid search | <500ms | 145ms |
|
|
| NDCG@10 | >0.85 | 0.88 |
|
|
| Test coverage | >600 | 670 |
|
|
|
|
## Contributing
|
|
|
|
```bash
|
|
# Run tests
|
|
cargo test --all
|
|
|
|
# Run specific test
|
|
cargo test -p mem-cli http_server::tests
|
|
|
|
# Check formatting
|
|
cargo fmt --check
|
|
cargo clippy
|
|
```
|
|
|
|
## License
|
|
|
|
MIT
|
|
|
|
|