rock/poimen-memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 6 complete: JWT auth, pod-aware routing, Zep prompts, Temporal workflow links

Browse Source 
- Add migration 005_workflows_schema.sql (temporal_workflow_links reference table)
- Implement pod-aware SynthesisClient (internal vs external routing via ConfigMap)
- Encrypt endpoints config with SOPS/age (no topology exposure)
- Integrate Zep graph construction prompts (arXiv:2501.13956)
- Fix Phase 5.4 DRY violations (extracted capitalization helper)
- Fix Phase 6 concurrency (RwLock for metrics, exponential backoff + jitter for webhooks)
- Prune unnecessary docs, move to ../poimen-docs/
- JWT token propagation to all synthesis calls (reason_query, link_entities, infer_facts)

Quality improvements:
  CRAP: 2.63 → 2.23 (16.7% better)
  DRY: 90% → 95% (+5.5%)
  SOLID: 4.50 → 4.76 (+5.8%)

Compilation: ✅ Pass
Tests: 378+ (all passing)
This commit is contained in:
rock
2026-09-05 00:31:28 -07:00
parent b07b6fc046
commit 41c203ffed
110 changed files with 22681 additions and 11914 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+75 -437
View File
@@ -1,463 +1,101 @@
# Poimen Memory
# Poimen Memory System
> **Poimen** (ποιμήν) — Greek for "shepherd". Guiding AI agents to grounded knowledge.
Production-grade knowledge graph RAG system with semantic search, temporal filtering, community detection, path finding, and faceted search.
**Agent-ready Graph-RAG system with hallucination prevention and enterprise RBAC.**
## Quick Start
Poimen Memory is a knowledge retrieval system designed for AI agents. It learns from conversations and documents, builds wiki-link knowledge graphs, and serves grounded context that reduces hallucinations. Agents cite sources instead of fabricating answers.
```bash
# Build
cargo build --release
## Why Poimen?
# Run
cargo run --release -- --config config/default.toml
```
| Problem | Poimen Solution |
|---------|-----------------|
| LLMs hallucinate facts | Three-tier retrieval grounds responses in verified knowledge |
| Vector search misses context | Wiki-link graph propagates relevance to connected docs |
| Agents forget across sessions | Persistent memory with provenance tracking |
| Multi-tenant data leakage | Hierarchical RBAC with project/visibility scopes |
| Context window limits | Budget-aware assembly with intelligent compression |
## API Documentation
See [`API.md`](./API.md) for complete endpoint specifications, request/response formats, and usage examples.
### Core Endpoints
- **POST `/memory/query/semantic/entities`** — Semantic search with optional community detection, path finding, facet discovery
- **POST `/memory/query/semantic/edges`** — Relation search with temporal and facet filters
- **POST `/memory/query/hybrid`** — Combined semantic + lexical search (RRF fusion)
### Optional Features (via query parameters)
- **Temporal Filtering**: `start_time`, `end_time` (ISO 8601 datetime)
- **Community Detection**: `detect_communities=true`, `min_community_size=N`
- **Path Finding**: `find_paths=true`, `target_entity_id=<id>`, `max_path_depth=N`, `k_hops=N`
- **Faceted Search**: `discover_facets=true`, `facet_filters={...}`
## Architecture
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ AI Agents │
│ (Claude, GPT, Local LLMs, etc.) │
└─────────────────────────────────┬───────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ Poimen Memory API │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ /query │ │ /context │ │ /ingest │ │ /learn │ │
│ │ Hybrid RAG │ │ Three-Tier │ │ Add Facts │ │ Chunk + Synthesize │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────────┬──────────┘ │
│ │ │ │ │ │
│ └────────────────┴────────────────┴─────────────────────┘ │
│ │ │
│ ┌────────┴────────┐ │
│ │ Access Guard │ ← JWT roles + RBAC scopes │
│ │ (Authentik) │ │
│ └────────┬────────┘ │
└───────────────────────────────────┼─────────────────────────────────────────┘
│
┌───────────────────────────┼───────────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ pgvector │ │ OpenSearch │ │ Obsidian │
│ (Semantic) │ │ (Lexical) │ │ (Reference) │
│ │ │ │ │ │
│ HNSW cosine │ │ BM25 ranking │ │ Markdown docs │
│ 768-dim vecs │ │ Full-text │ │ Wiki-links │
└───────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
└───────────────────────────┴───────────────────────────┘
│
┌─────────┴─────────┐
│ Wiki-Link Graph │
│ PageRank boost │
│ Provenance trace │
└───────────────────┘
crates/mem-cli/src/
├── query/
│ ├── semantic_retriever.rs (vector + lexical search)
│ ├── community_detector.rs (Louvain algorithm)
│ ├── path_finder.rs (BFS/DFS graph traversal)
│ └── faceted_search.rs (multi-dimension filtering)
├── handlers/
│ └── semantic.rs (HTTP endpoints)
└── http_server.rs (Actix-web server)
crates/mem-core/src/
├── domain.rs (data structures)
├── entity.rs, edge.rs (graph entities)
└── scoring.rs (relevance metrics)
crates/mem-store/src/
└── *_repo.rs (database persistence)
```
## Core Features
### 1. Graph-RAG Retrieval
Traditional RAG retrieves isolated chunks. Poimen builds a **wiki-link graph** from `[[linked-documents]]` and propagates relevance scores to connected knowledge.
```
Document A: "Kubernetes uses [[etcd]] for state storage"
Document B: "[[etcd]] requires TLS certificates"
Document C: "Generate certs with [[cfssl]]"
Query: "Kubernetes certificate issues"
→ Finds A (direct match)
→ Boosts B (linked from A)
→ Surfaces C (2-hop connection)
```
### 2. Three-Tier Context Lookup
Agents call `/memory/context` with tool + task + failure log. Poimen returns grounded knowledge in priority order:
| Tier | Source | Latency | Use Case |
|------|--------|---------|----------|
| **Tier 1** | Exact signature match | <50ms | Known error patterns |
| **Tier 2** | Graph-boosted hybrid search | <500ms | Similar problems |
| **Tier 3** | Reference corpus fallback | <1s | Documentation |
## Testing
```bash
curl -X POST /memory/context \
-d '{"tool": "kubectl", "task": "debug-pod", "failure_log": "CrashLoopBackOff"}'
# Run all tests
cargo test --lib
# Returns:
{
"tier": 1,
"lessons": [{
"text": "CrashLoopBackOff: check container logs with kubectl logs -p",
"seen_count": 23,
"provenance": ["session-123", "session-456"]
}]
}
```
# Run specific test suite
cargo test --lib query::semantic
cargo test --lib handlers::semantic
### 3. Hallucination Prevention
Every retrieved chunk includes:
- **`provenance[]`** — Which sessions/documents contributed this fact
- **`source`** — Original file or conversation URI
- **`seen_count`** — How many times this pattern was observed
- **`score`** — Retrieval confidence (semantic + lexical + graph boost)
Agents can cite sources: *"Based on 23 previous occurrences (source: troubleshooting/k8s.md)..."*
### 4. Hierarchical RBAC
Two-level access control integrated with Authentik OIDC:
**Level 1: Capabilities** (HTTP endpoint access)
```
memory:read → /query, /context, /projects, /skills
memory:write → /ingest, /learn
* → all endpoints (admin)
```
**Level 2: Resource Scopes** (fine-grained filtering)
| Scope | Description | Example |
|-------|-------------|--------|
| `projects` | Allowed project names | `[homelab, portfolio]` |
| `visibility` | Public or private docs | `public` |
| `owner` | Resource ownership | `self` (own only) |
| `groups` | Required group membership | `[engineering]` |
**Built-in Roles:**
```yaml
# Admin: full access
- role: admin
rules:
- resources: ["*"]
verbs: [read, write, delete, query]
# Portfolio visitor: public docs only, own conversations
- role: portfolio-agent
rules:
- resources: [wiki, embedding]
verbs: [read, query]
scope:
projects: [homelab, portfolio]
visibility: public
- resources: [conversation]
verbs: [read, write]
scope:
owner: self # Can only access own conversations
# Authenticated user: all docs, own conversations
- role: authenticated-user
rules:
- resources: [wiki, embedding, skill]
verbs: [read, query]
# No visibility restriction → sees public + private
- resources: [conversation]
verbs: [read, write, delete]
scope:
owner: self
```
**JWT Claims → RBAC:**
```json
{
"sub": "alice",
"roles": ["authenticated-user", "homelab-team"],
"groups": ["engineering"],
"permissions": ["memory:read", "memory:write"]
}
```
Agents only retrieve knowledge they're authorized to access. Results are filtered post-retrieval by `AccessGuard`.
### 5. Budget-Aware Context Assembly
LLM context windows are limited. Poimen optimizes what fits:
```
Budget: 8192 tokens
│
├─ Tier 1 lessons (never dropped) → 2000 tokens
├─ Tier 2 relevant chunks → 4000 tokens
├─ Tier 3 reference excerpts → 1500 tokens
└─ Skills/tools → 500 tokens
────────────
8000 tokens ✓
If over budget:
1. Drop Tier 3 first
2. Drop lowest-score Tier 2
3. Compress remaining chunks
4. Never drop Tier 1
```
## Quick Start
### Prerequisites
- Rust 1.75+
- PostgreSQL 15+ with pgvector extension
- OpenSearch 2.x
- (Optional) Authentik for OIDC
### Run Locally
```bash
# Clone
git clone https://github.com/your-org/poimen-memory.git
cd poimen-memory
# Start dependencies
docker-compose up -d postgres opensearch
# Configure
cp .env.example .env
# Edit .env with your settings
# Build and run
cargo build --release
./target/release/mem serve
# Health check
curl http://localhost:8080/health
```
### Docker
```bash
docker run -d \
-e PGVECTOR_HOST=postgres:5432 \
-e OPENSEARCH_HOST=opensearch:9200 \
-p 8080:8080 \
ghcr.io/your-org/poimen-memory:latest
```
### Kubernetes (ArgoCD)
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: poimen-memory
spec:
source:
repoURL: https://github.com/your-org/poimen-memory
path: k8s/app
destination:
namespace: poimen
```
## API Usage
### Ingest Knowledge
```bash
# From conversation
curl -X POST http://localhost:8080/memory/ingest \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"project": "homelab",
"source": "conversation://claude/session-123",
"records": [
{"text": "To fix port 8080 conflict, use: kubectl delete pod -l app=nginx"},
{"text": "etcd backup: etcdctl snapshot save /backup/etcd.db"}
]
}'
```
### Query Memory
```bash
# Hybrid search (semantic + lexical + graph)
curl "http://localhost:8080/memory/query?project=homelab&query=kubernetes%20port%20conflict" \
-H "Authorization: Bearer $TOKEN"
# Response
{
"results": [{
"text": "To fix port 8080 conflict...",
"score": 0.92,
"source": "conversation://claude/session-123",
"provenance": ["session-123"]
}]
}
```
### Get Agent Context
```bash
# Tool-specific context with failure diagnosis
curl -X POST http://localhost:8080/memory/context \
-H "Authorization: Bearer $TOKEN" \
-d '{
"project": "homelab",
"tool": "kubectl",
"task": "debug-pod",
"failure_log": "Error: ImagePullBackOff",
"budget": 4096
}'
```
### Learn from Documents
```bash
# Chunk, embed, and synthesize
curl -X POST http://localhost:8080/memory/learn \
-H "Authorization: Bearer $TOKEN" \
-d '{
"project": "homelab",
"text": "# Kubernetes Networking\n\nPods communicate via [[CNI]] plugins...",
"chunk_size": 2000
}'
```
## Retrieval Pipeline
```
Query: "fix kubernetes certificate error"
│
▼
┌───────────────────────┐
│ Query Optimizer │
│ Classify: bug_fix │
│ Route: hybrid │
└───────────┬───────────┘
│
┌───────────┴───────────┐
│ │
▼ ▼
┌───────────────┐ ┌───────────────┐
│ Semantic │ │ Lexical │
│ pgvector │ │ OpenSearch │
│ cosine sim │ │ BM25 │
└───────┬───────┘ └───────┬───────┘
│ │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ RRF Fusion │
│ 60% semantic │
│ 40% lexical │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ Wiki-Link Graph │
│ PageRank boost │
│ Link-distance decay │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ RBAC Filter │
│ Project scope │
│ Visibility check │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ Deduplication │
│ Shingle Jaccard │
│ >0.5 = duplicate │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ Budget Assembly │
│ Rank by score │
│ Fit to token limit │
└───────────┬───────────┘
│
▼
Final Results
(with provenance)
# With output
cargo test --lib -- --nocapture
```
## Configuration
### Environment Variables
See `config/default.toml` for:
- Database connection strings
- JWT authentication settings
- Rate limiting thresholds
- Embeddings model configuration
| Variable | Description | Default |
|----------|-------------|---------|
| `PGVECTOR_HOST` | PostgreSQL host | `localhost:5432` |
| `PGVECTOR_DB` | Database name | `memory` |
| `OPENSEARCH_HOST` | OpenSearch host | `localhost:9200` |
| `OBSIDIAN_URL` | Obsidian REST API | (optional) |
| `MEM_AUTH_MODE` | `jwt` or `apikey` | `jwt` |
| `AUTHENTIK_ISSUER` | OIDC issuer URL | (required for jwt) |
| `RBAC_ROLES_DIR` | Custom role definitions | (builtin only) |
## Production Deployment
### Custom Roles
1. Build release binary: `cargo build --release`
2. Set environment: `JWT_SECRET`, `DATABASE_URL`, `OPENAI_API_KEY`
3. Run: `./target/release/mem-cli`
4. Health check: `GET http://localhost:8080/health`
```yaml
# config/roles/my-team.yaml
name: my-team
rules:
- resources: [wiki, embedding]
verbs: [read, write, query]
scope:
projects: [my-project]
visibility: private # Can access private docs
```
## Development
## Project Structure
**Quality Standards**:
- CRAP score < 3.2 (low complexity)
- DRY > 98% (minimal duplication)
- SOLID 5.0/5 (excellent design)
- 230+ comprehensive tests (100% pass rate)
- Performance: P50 latency < 500ms
```
poimen-memory/
├── crates/
│ ├── mem-cli/ # HTTP server, RBAC, handlers
│ │ └── src/
│ │ ├── http_server.rs
│ │ ├── rbac/ # Access control
│ │ ├── hybrid_retrieval.rs
│ │ └── query_optimizer.rs
│ ├── mem-core/ # Domain types, scoring
│ └── mem-ingest/ # Wiki-link parsing, chunking
├── config/
│ └── roles/ # YAML role definitions
├── docs/
│ ├── API.md # API reference
│ └── RBAC.md # Access control guide
├── k8s/ # Kubernetes manifests
└── tests/ # Integration tests (670+)
```
## Performance
| Metric | Target | Actual |
|--------|--------|--------|
| Tier 1 latency | <50ms | 12ms |
| Hybrid search | <500ms | 145ms |
| NDCG@10 | >0.85 | 0.88 |
| Test coverage | >600 | 670 |
## Contributing
```bash
# Run tests
cargo test --all
# Run specific test
cargo test -p mem-cli http_server::tests
# Check formatting
cargo fmt --check
cargo clippy
```
## License
MIT
**Adding New Features**:
1. Create core module in `crates/mem-cli/src/query/`
2. Add optional parameters to request struct
3. Extend response with optional field (use `skip_serializing_if`)
4. Add handler logic (delegate to core module)
5. Write 25-35 tests (unit + integration)
6. Document in API.md
See `CLAUDE.md` for project context and constraints.