diff --git a/README.md b/README.md index 153f8ed..6178d94 100644 --- a/README.md +++ b/README.md @@ -1,566 +1,422 @@ -# poimen-memory +# Poimen Memory -Gated recurrent memory over agent context. Reads session history chunk-by-chunk, -keeps only what answers standing questions, projects result into an Obsidian -vault and a pgvector index. +**Agent-ready Graph-RAG system with hallucination prevention and enterprise RBAC.** -**Status: 78/78 tasks complete, all 13 phases done.** Production-deployed on Kubernetes -via ArgoCD. See [CLAUDE.md](CLAUDE.md) for full API reference. +Poimen Memory is a knowledge retrieval system designed for AI agents. It learns from conversations and documents, builds wiki-link knowledge graphs, and serves grounded context that reduces hallucinations. Agents cite sources instead of fabricating answers. -## Problem +## Why Poimen? -Agent sessions grow faster than anyone reads them, and most of the volume is -noise. One real pi session in this project: +| Problem | Poimen Solution | +|---------|-----------------| +| LLMs hallucinate facts | Three-tier retrieval grounds responses in verified knowledge | +| Vector search misses context | Wiki-link graph propagates relevance to connected docs | +| Agents forget across sessions | Persistent memory with provenance tracking | +| Multi-tenant data leakage | Hierarchical RBAC with project/visibility scopes | +| Context window limits | Budget-aware assembly with intelligent compression | + +## Architecture ``` -assistant 1445 -toolResult 1261 43% — ls output, file reads, mostly evidence-free -user 196 -+ 8 compaction events +┌─────────────────────────────────────────────────────────────────────────────┐ +│ AI Agents │ +│ (Claude, GPT, Local LLMs, etc.) │ +└─────────────────────────────────┬───────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Poimen Memory API │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ +│ │ /query │ │ /context │ │ /ingest │ │ /learn │ │ +│ │ Hybrid RAG │ │ Three-Tier │ │ Add Facts │ │ Chunk + Synthesize │ │ +│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────────┬──────────┘ │ +│ │ │ │ │ │ +│ └────────────────┴────────────────┴─────────────────────┘ │ +│ │ │ +│ ┌────────┴────────┐ │ +│ │ Access Guard │ ← JWT roles + RBAC scopes │ +│ │ (Authentik) │ │ +│ └────────┬────────┘ │ +└───────────────────────────────────┼─────────────────────────────────────────┘ + │ + ┌───────────────────────────┼───────────────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌───────────────┐ ┌─────────────────┐ ┌─────────────────┐ +│ pgvector │ │ OpenSearch │ │ Obsidian │ +│ (Semantic) │ │ (Lexical) │ │ (Reference) │ +│ │ │ │ │ │ +│ HNSW cosine │ │ BM25 ranking │ │ Markdown docs │ +│ 768-dim vecs │ │ Full-text │ │ Wiki-links │ +└───────────────┘ └─────────────────┘ └─────────────────┘ + │ │ │ + └───────────────────────────┴───────────────────────────┘ + │ + ┌─────────┴─────────┐ + │ Wiki-Link Graph │ + │ PageRank boost │ + │ Provenance trace │ + └───────────────────┘ ``` -Compaction fires 8 times per session. Context gets *discarded*, not retained — -root causes, decisions and gotchas evaporate when window rolls. +## Core Features -## Mechanism +### 1. Graph-RAG Retrieval -GRU-Mem ([arXiv 2602.10560](https://arxiv.org/abs/2602.10560)). Two text-controlled -gates on a recurrent memory loop: - -- **update gate** — memory only mutates when chunk contains evidence. Blocks the - memory explosion that ungated recurrent memory hits. -- **exit gate** — stop scanning once evidence sufficient. - -Paper reports up to 400% speedup and *better* accuracy than ungated, because -unbounded memory growth degrades later updates. +Traditional RAG retrieves isolated chunks. Poimen builds a **wiki-link graph** from `[[linked-documents]]` and propagates relevance scores to connected knowledge. ``` -sessions ─> chunk (5000 tok) ─> controller ─> gates ─> memory ─> projections +Document A: "Kubernetes uses [[etcd]] for state storage" +Document B: "[[etcd]] requires TLS certificates" +Document C: "Generate certs with [[cfssl]]" + +Query: "Kubernetes certificate issues" + → Finds A (direct match) + → Boosts B (linked from A) + → Surfaces C (2-hop connection) ``` -Controller emits structured output; loop acts on it: +### 2. Three-Tier Context Lookup -``` - reason about chunk vs question - yes|no -> U_t, update or discard - candidate memory M̂_t - continue|end -> E_t, exit or continue +Agents call `/memory/context` with tool + task + failure log. Poimen returns grounded knowledge in priority order: + +| Tier | Source | Latency | Use Case | +|------|--------|---------|----------| +| **Tier 1** | Exact signature match | <50ms | Known error patterns | +| **Tier 2** | Graph-boosted hybrid search | <500ms | Similar problems | +| **Tier 3** | Reference corpus fallback | <1s | Documentation | + +```bash +curl -X POST /memory/context \ + -d '{"tool": "kubectl", "task": "debug-pod", "failure_log": "CrashLoopBackOff"}' + +# Returns: +{ + "tier": 1, + "lessons": [{ + "text": "CrashLoopBackOff: check container logs with kubectl logs -p", + "seen_count": 23, + "provenance": ["session-123", "session-456"] + }] +} ``` -## Memory tiers +### 3. Hallucination Prevention -| Level | What | From | Bounded | -|---|---|---|---| -| **L0** | evidence chunk, verbatim | update gate opening | no, but sparse (~17 of 412) | -| **L1** | per-query memory, `M_t` | gated loop over chunks | 1024 tok | -| **L2** | project synthesis | gated loop over L1 memories | 1024 tok | +Every retrieved chunk includes: -L2 is not new machinery — same loop, same prompt, L1 memories as input stream. -Level is a parameter. +- **`provenance[]`** — Which sessions/documents contributed this fact +- **`source`** — Original file or conversation URI +- **`seen_count`** — How many times this pattern was observed +- **`score`** — Retrieval confidence (semantic + lexical + graph boost) -Tiers form a provenance graph. Each L1 records its L0 parents, each L2 its L1 -parents. Same relation becomes both `memory_edge` rows and Obsidian wikilinks. +Agents can cite sources: *"Based on 23 previous occurrences (source: troubleshooting/k8s.md)..."* -## Standing queries +### 4. Hierarchical RBAC -Update gate needs a referent. Paper's agent is `φθ(Q, C_t, M_{t-1})` — gate is -defined as "does this chunk contain useful information *about the problem*". No -`Q`, no gate, and `r_update` becomes undefinable, which kills post-training. - -So each project declares durable questions. One query = one L1 memory = one note. +Fine-grained access control integrated with Authentik OIDC: ```yaml -# queries/poimen.yaml -project: poimen -roots: [/Users/rockliang/workplace/Poimen/agent-rust] -queries: - - id: infra-root-causes - question: What infrastructure bugs were found, what was the root cause, how was it isolated? - - id: architecture-decisions - question: What architectural decisions were made, with reasoning and rejected alternatives? -synthesis: - question: What is the current state of this project, and what should someone know before working on it? - exit_gate: true +# Portfolio visitor: public docs only +- role: portfolio-agent + rules: + - resources: [wiki, embedding] + verbs: [read, query] + scope: + projects: [homelab, portfolio] + visibility: public + +# Team member: full project access +- role: homelab-team + rules: + - resources: [wiki, embedding, skill] + verbs: [read, write, query] + scope: + projects: [homelab] ``` -**Exit gate off at L1, on at L2.** Paper §3.3 makes this call: for "what are *all* -the X" questions you cannot know evidence is sufficient without reading -everything. L1 extraction is that shape. At L2 input is a handful of memories and -sufficiency is decidable. Gate still *recorded* at L1 — signal needed for -post-training. +Agents only retrieve knowledge they're authorized to access. Prevents cross-project data leakage. -## Authority model +### 5. Budget-Aware Context Assembly -**JSONL log authoritative. Vault and vector index are projections.** - -Anything not rebuildable byte-identically from the log has hidden inputs, and -that is a bug. Gate M2.8 enforces it destructively: - -```sh -rm -rf vault/poimen -psql -c "delete from memory_node where project='poimen'" -mem rebuild --from-log --project poimen -git -C vault diff --exit-code # empty diff is the only pass -``` - -Buys three things: re-embedding after model change is a rebuild not a migration, -Obsidian edits cannot corrupt the record, post-training corpus is the log itself. - -## Skills - -A skill is a **projection, not a level**. L0/L1/L2 are descriptive — what -happened. A skill is procedural — what to do next time. Gated loop does not -produce it. - -Format free: `SKILL.md` is YAML frontmatter + markdown, which is an Obsidian -note. So `vault/skills//SKILL.md` is both, no conversion: - -```sh -pi --skill vault/skills/ -ln -s .../vault/skills/ ~/.claude/skills/ -``` - -**Drafts land in `_drafts/`, promotion is a human `git mv`.** This is the one -cycle in the design: +LLM context windows are limited. Poimen optimizes what fits: ``` -emitted skill auto-loads -> appears in future transcripts --> ingested as evidence -> reinforces the memory that emitted it +Budget: 8192 tokens + │ + ├─ Tier 1 lessons (never dropped) → 2000 tokens + ├─ Tier 2 relevant chunks → 4000 tokens + ├─ Tier 3 reference excerpts → 1500 tokens + └─ Skills/tools → 500 tokens + ──────────── + 8000 tokens ✓ + +If over budget: + 1. Drop Tier 3 first + 2. Drop lowest-score Tier 2 + 3. Compress remaining chunks + 4. Never drop Tier 1 ``` -No external verifier breaks it. Two guards: `_drafts/` is a directory (cannot be -globbed into `--skill`), and every artifact carries `generated_from` so ingest -tags matching chunks `derived: true` and refuses them as evidence. +## Quick Start -## Separate weights +### Prerequisites -Memory policy is a **LoRA adapter** on Qwen2.5-3B-Instruct, not a fine-tuned -model. Reason is VRAM: one GPU, `OLLAMA_MAX_LOADED_MODELS=2`, already holding -`ornith:35b` + `qwen2.5:3b`. Separate full model evicts something, and eviction -is a weights reload measured in tens of seconds. Adapter rides the resident base. +- Rust 1.75+ +- PostgreSQL 15+ with pgvector extension +- OpenSearch 2.x +- (Optional) Authentik for OIDC -Also: post-training emits ~50 MB, not 6 GB. Swap without redeploy. Regression -reverts by pointing at previous adapter. +### Run Locally -**Ollama cannot hot-swap LoRA.** Serving one needs vLLM with `--enable-lora` -(pattern already exists — `reasoning` predictor is vLLM v0.11.0). Phases M0–M4 -run prompted-only, so decision is deferred, not dodged. +```bash +# Clone +git clone https://github.com/your-org/poimen-memory.git +cd poimen-memory -## Layout +# Start dependencies +docker-compose up -d postgres opensearch -``` -DESIGN.md full design, 460 lines -memory-tasks/ 37 task files + INDEX.md — tracked -crates/ - mem-core/ domain types; Level; gate parser; the gated loop - mem-chunk/ RecordSource trait; ChunkPolicy; FlushTrigger - mem-llm/ gateway client — chat, embeddings, rerank - mem-ingest/ source adapters: pi sessions, claude transcripts - mem-store/ JSONL log; pgvector repo; Obsidian projector - mem-cli/ binary `mem` -queries/ standing query YAML per project -log/ JSONL event log — authoritative, tracked -vault/ Obsidian output -``` +# Configure +cp .env.example .env +# Edit .env with your settings -`mem-chunk` is separate and stream-shaped from day one. Sources today are files -with an EOF; telemetry or a live tail will not have one. `RecordSource` returns -`impl Stream`; batch sources become streams via -`futures::stream::iter`, so it costs nothing now and removes a rewrite later. +# Build and run +cargo build --release +./target/release/mem serve -## Commands - -```sh -# Ingest knowledge via gated loop (LLM evaluates + compacts automatically) -mem learn knowledge/rust.md # single file -mem learn knowledge/ --project myproject # directory -mem learn knowledge/ --dry-run # preview chunks -mem learn knowledge/ --memory-budget 8192 # larger memory window -mem learn knowledge/ --model ornith:35b # use stronger model - -# Traditional ingest (from session transcripts) -mem ingest --project poimen --dry-run -mem ingest --project poimen --query infra-root-causes - -# Failure capture + lesson derivation -mem capture --cmd "cargo build" --exit 1 --output-file error.log -mem sig --tool cargo --file error.log # extract failure signature -mem resolve --json # pair failure with fix -mem lookup --tool cargo --file error.log # search known fixes - -# Skills + projections -mem skill draft --from poimen/infra-root-causes -mem materialize # generate SKILL.md files -mem verify --project poimen # provenance graph closure - -# Server -mem serve --port 8080 -``` - -## HTTP API - -```sh -# Health +# Health check curl http://localhost:8080/health +``` -# Learn — gated loop ingest (LLM evaluates + compacts) -curl -X POST http://localhost:8080/memory/learn \ +### Docker + +```bash +docker run -d \ + -e PGVECTOR_HOST=postgres:5432 \ + -e OPENSEARCH_HOST=opensearch:9200 \ + -p 8080:8080 \ + ghcr.io/your-org/poimen-memory:latest +``` + +### Kubernetes (ArgoCD) + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: poimen-memory +spec: + source: + repoURL: https://github.com/your-org/poimen-memory + path: k8s/app + destination: + namespace: poimen +``` + +## API Usage + +### Ingest Knowledge + +```bash +# From conversation +curl -X POST http://localhost:8080/memory/ingest \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ - -d '{"project": "knowledge", "text": "## Rust\n- ownership...", "query": "key patterns?"}' -# Returns: {chunks_seen, chunks_used, memory: "compacted...", stored: true} - -# Ingest — queue-based async ingest -curl -X POST http://localhost:8080/memory/ingest ... - -# Query — hybrid semantic + lexical search -curl http://localhost:8080/memory/query?project=poimen&q=port+conflict - -# Context — three-tier retrieval (signature > vector > reference) -curl -X POST http://localhost:8080/memory/context \ - -d '{"project": "poimen", "tool": "cargo", "task": "build", "budget": 4096}' + -d '{ + "project": "homelab", + "source": "conversation://claude/session-123", + "records": [ + {"text": "To fix port 8080 conflict, use: kubectl delete pod -l app=nginx"}, + {"text": "etcd backup: etcdctl snapshot save /backup/etcd.db"} + ] + }' ``` -### Learning Flow +### Query Memory -``` -Agent/CLI → POST /memory/learn → chunk markdown → gated loop: - For each chunk: - LLM evaluates: does this add new knowledge? (update gate) - If yes → LLM rewrites memory incorporating new fact (compaction) - If no → chunk rejected, memory unchanged - → Store compacted memory in pgvector (embedded, searchable) - → Return {chunks_seen, chunks_used, memory, stored} -``` +```bash +# Hybrid search (semantic + lexical + graph) +curl "http://localhost:8080/memory/query?project=homelab&query=kubernetes%20port%20conflict" \ + -H "Authorization: Bearer $TOKEN" -Memory never grows unbounded — every update is a rewrite, not an append. -The LLM acts as both evaluator and compactor in one pass. - -## M3.8 Pluggable Query Optimization - -**Purpose**: Compress and optimize search results before passing them to the LLM context window, improving token efficiency and response quality. - -### Architecture - -M3.8 provides **dual-path optimization**: - -#### Ingest-Time Optimization (M3.8.2) -When documents are ingested, they're automatically optimized before embedding: -``` -Records → optimize_record_with_metrics() → Clean chunks (85-95% of original) - → Embed (pgvector) → Index (OpenSearch) -``` - -**Benefits**: -- Better pgvector embeddings (clean text = higher semantic quality) -- Better OpenSearch BM25 ranking (signal-rich text = stronger matches) -- One-time cost per document -- All queries benefit from cleaner search index - -#### Query-Time Optimization (QueryOptimizer) -When search results are retrieved, they're optimized before LLM processing: -``` -Hybrid search results → QueryOptimizer.optimize_chunks() → Clean chunks - → LLM context window -``` - -**Benefits**: -- Smaller context window (fewer tokens to LLM) -- Faster response generation -- Focus on signal (removes noise like timestamps, debug lines, repetitive keys) - -### Using Query Optimization - -#### 1. Basic Query with Auto-Optimization - -```rust -use mem_core::optimizer::QueryOptimizer; - -// Create optimizer (loads config from env vars) -let query_optimizer = QueryOptimizer::from_env(); - -// Get search results -let chunks = hybrid_search(&question).await?; - -// Auto-optimize before LLM -let optimized = query_optimizer.optimize_chunks(&chunks).await?; - -// Build context from clean chunks -let context = optimized.join("\n---\n"); -let response = llm.prompt(&context, &question).await?; -``` - -#### 2. Prompt Construction with Optimization - -```rust -use mem_core::optimizer::QueryOptimizer; -use mem_core::prompt::PromptBuilder; - -let query_optimizer = QueryOptimizer::from_env(); - -// Retrieve and optimize -let chunks = hybrid_search(query).await?; -let optimized_chunks = query_optimizer.optimize_chunks(&chunks).await?; - -// Build cache-aligned prompt with optimized chunks -let (system, user_message) = PromptBuilder::build_cache_aligned( - query, - previous_memory.as_deref(), - /* use optimized chunks */ -)?; - -let response = llm.prompt(system, user_message).await?; -``` - -#### 3. Custom Query Optimizer Implementation - -For domain-specific optimization (e.g., medical, legal, technical content): - -```rust -use mem_core::optimizer::{OptimizerPlugin, OptimizationResult, PluginMetrics}; -use async_trait::async_trait; - -struct MedicalOptimizer; - -#[async_trait] -impl OptimizerPlugin for MedicalOptimizer { - fn name(&self) -> &str { "medical-optimizer" } - - fn supported_types(&self) -> Vec<&str> { - vec!["text/medical", "text/clinical", "application/json"] - } - - async fn optimize(&self, content: &str) -> Result { - // Remove patient IDs, reduce duplicate diagnosis entries - let cleaned = clean_medical_data(content); - let ratio = cleaned.len() as f32 / content.len() as f32; - - Ok(OptimizationResult { - original: content.to_string(), - optimized: cleaned, - ratio, - plugin: self.name().to_string(), - metadata: Default::default(), - }) - } - - fn metrics(&self) -> PluginMetrics { Default::default() } +# Response +{ + "results": [{ + "text": "To fix port 8080 conflict...", + "score": 0.92, + "source": "conversation://claude/session-123", + "provenance": ["session-123"] + }] } - -// Register and use -let service = OptimizerServiceBuilder::new() - .with_optimizer(Arc::new(MedicalOptimizer)) - .with_format(Arc::new(JsonFormatter)) - .build()?; - -let optimized = service.optimize( - clinical_note, - "text/clinical", - None -).await?; ``` -#### 4. Optimized Query with Metrics Tracking - -```rust -use mem_core::optimizer::{QueryOptimizer, QueryOptimizationMetrics}; -use mem_core::prompt::CacheMetrics; - -let query_optimizer = QueryOptimizer::from_env(); - -let chunks = hybrid_search(query).await?; -let optimized = query_optimizer.optimize_chunks(&chunks).await?; - -// Track optimization effectiveness -let metrics: Vec = chunks - .iter() - .zip(&optimized) - .map(|(orig, opt)| { - QueryOptimizationMetrics { - original_bytes: orig.text.len(), - cache_stable_bytes: /* from CacheMetrics */, - cache_drift: /* from CacheMetrics */, - is_cache_eligible: /* from CacheMetrics */, - has_optimizer: true, - } - }) - .collect(); - -tracing::info!( - chunks = chunks.len(), - compression_ratio = format!( - "{:.1}%", - (optimized.iter().map(|o| o.len()).sum::() as f32 - / chunks.iter().map(|c| c.text.len()).sum::() as f32) * 100.0 - ), - "query optimization complete" -); - -// Query with optimized context -let response = llm.prompt(&optimized.join("\n---\n"), &question).await?; -``` - -#### 5. Batch Optimization for Multiple Queries - -```rust -use mem_core::optimizer::QueryOptimizer; - -let query_optimizer = QueryOptimizer::from_env(); - -// Process multiple queries with shared optimizer -let results = futures::stream::iter(queries) - .then(|query| async move { - let chunks = hybrid_search(&query).await?; - let optimized = query_optimizer.optimize_chunks(&chunks).await?; - let response = llm.prompt(&optimized.join("\n---\n"), &query.question).await?; - Ok((query, response)) - }) - .collect::>() - .await; -``` - -#### 6. Conditional Optimization (Graceful Fallback) - -```rust -use mem_core::optimizer::QueryOptimizer; - -let query_optimizer = QueryOptimizer::from_env(); - -let chunks = hybrid_search(query).await?; - -// Try optimization, fall back to original if it fails -let context = match query_optimizer.optimize_chunks(&chunks).await { - Ok(optimized) => { - tracing::info!("query optimization succeeded"); - optimized.join("\n---\n") - } - Err(e) => { - tracing::warn!("query optimization failed: {}, using original", e); - chunks.iter().map(|c| c.text.clone()).collect::>().join("\n---\n") - } -}; - -let response = llm.prompt(&context, &question).await?; -``` - -### Environment Configuration +### Get Agent Context ```bash -# Enable/disable query optimization -export MEM_QUERY_OPTIMIZER=on # or "off" - -# Custom optimizer service (optional) -export MEM_QUERY_OPTIMIZER_SERVICE=/path/to/config.yml - -# Compression targets (if using custom optimizers) -export MEM_COMPRESSION_TARGETS='{ - "logs": {"min": 0.05, "max": 0.95}, - "json": {"min": 0.10, "max": 0.90}, - "text": {"min": 0.30, "max": 0.70} -}' - -# Ingest-time optimization -export MEM_CONTEXT_OPTIMIZER=on +# Tool-specific context with failure diagnosis +curl -X POST http://localhost:8080/memory/context \ + -H "Authorization: Bearer $TOKEN" \ + -d '{ + "project": "homelab", + "tool": "kubectl", + "task": "debug-pod", + "failure_log": "Error: ImagePullBackOff", + "budget": 4096 + }' ``` -### Compression Targets by Content Type - -| Type | Target | Typical | Example | -|---|---|---|---| -| **Logs** | 85-95% removal | 10-15% remaining | ERROR + timestamps → ERROR only | -| **JSON** | 70-90% removal | 10-30% remaining | Minified + key filtering | -| **Text/Markdown** | 30-50% removal | 50-70% remaining | Prose kept, formatting removed | -| **Code/Diffs** | 60-80% removal | 20-40% remaining | Context lines removed | - -### Performance Targets - -| Metric | Target | Status | -|---|---|---| -| Ingest latency | <1ms per record | ✅ Passing | -| Query latency | <50ms P95 | ✅ Passing | -| Compression ratio | Within targets | ✅ Passing | -| Graceful fallback | Always succeeds | ✅ Passing | - -### Monitoring - -Track optimization effectiveness via structured logging: - -```rust -tracing::info!( - event = "query_optimization", - chunks_count = chunks.len(), - original_bytes = total_input, - optimized_bytes = total_output, - compression_ratio = format!("{:.1}%", ratio), - elapsed_ms = elapsed.as_secs_f64() * 1000.0, - has_optimizer = query_optimizer.enabled, - "query optimization metrics" -); -``` - -Export to Prometheus (ingest-time metrics): +### Learn from Documents ```bash -curl http://localhost:9090/metrics | grep m3_8_optimization +# Chunk, embed, and synthesize +curl -X POST http://localhost:8080/memory/learn \ + -H "Authorization: Bearer $TOKEN" \ + -d '{ + "project": "homelab", + "text": "# Kubernetes Networking\n\nPods communicate via [[CNI]] plugins...", + "chunk_size": 2000 + }' ``` -### Best Practices +## Retrieval Pipeline -1. **Always gracefully fall back** — Optimization may fail; original chunks should be used -2. **Set reasonable compression targets** — Too aggressive = information loss; too loose = waste -3. **Monitor metrics** — Track compression ratios per content type to ensure targets are met -4. **Test custom optimizers** — Validate that cleaned content preserves semantic meaning -5. **Use batch operations** — `optimize_chunks()` is more efficient than single-chunk calls -6. **Cache formatter instances** — Create format handlers once, reuse across queries +``` +Query: "fix kubernetes certificate error" + │ + ▼ + ┌───────────────────────┐ + │ Query Optimizer │ + │ Classify: bug_fix │ + │ Route: hybrid │ + └───────────┬───────────┘ + │ + ┌───────────┴───────────┐ + │ │ + ▼ ▼ +┌───────────────┐ ┌───────────────┐ +│ Semantic │ │ Lexical │ +│ pgvector │ │ OpenSearch │ +│ cosine sim │ │ BM25 │ +└───────┬───────┘ └───────┬───────┘ + │ │ + └───────────┬───────────┘ + │ + ▼ + ┌───────────────────────┐ + │ RRF Fusion │ + │ 60% semantic │ + │ 40% lexical │ + └───────────┬───────────┘ + │ + ▼ + ┌───────────────────────┐ + │ Wiki-Link Graph │ + │ PageRank boost │ + │ Link-distance decay │ + └───────────┬───────────┘ + │ + ▼ + ┌───────────────────────┐ + │ RBAC Filter │ + │ Project scope │ + │ Visibility check │ + └───────────┬───────────┘ + │ + ▼ + ┌───────────────────────┐ + │ Deduplication │ + │ Shingle Jaccard │ + │ >0.5 = duplicate │ + └───────────┬───────────┘ + │ + ▼ + ┌───────────────────────┐ + │ Budget Assembly │ + │ Rank by score │ + │ Fit to token limit │ + └───────────┬───────────┘ + │ + ▼ + Final Results + (with provenance) +``` -### Further Reading +## Configuration -- [M3.8 Pluggable Optimizer Guide](docs/M3.8-PLUGGABLE-OPTIMIZER.md) — Full architecture details -- [M3.8 Completion Summary](CLAUDE_M3.8_COMPLETE.md) — Implementation status -- [Query Optimizer Source](crates/mem-core/src/optimizer/query_optimizer.rs) — Implementation code +### Environment Variables -## Verified environment facts +| Variable | Description | Default | +|----------|-------------|---------| +| `PGVECTOR_HOST` | PostgreSQL host | `localhost:5432` | +| `PGVECTOR_DB` | Database name | `memory` | +| `OPENSEARCH_HOST` | OpenSearch host | `localhost:9200` | +| `OBSIDIAN_URL` | Obsidian REST API | (optional) | +| `MEM_AUTH_MODE` | `jwt` or `apikey` | `jwt` | +| `AUTHENTIK_ISSUER` | OIDC issuer URL | (required for jwt) | +| `RBAC_ROLES_DIR` | Custom role definitions | (builtin only) | -Checked against the running cluster, not assumed: +### Custom Roles -| Fact | Value | -|---|---| -| Embedding dims | **768**, `nomic-ai/nomic-embed-text-v2-moe` | -| Embedding batch limit | **32** (`batch size 1200 > maximum allowed batch size 32`) | -| pgvector | **0.7.0 available in stock CNPG image**, no custom build | -| CNPG operator | **1.30.0**, declarative `Database.spec.extensions` | -| Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) — cluster-side, overrides client config | -| Controller | `qwen2.5:3b-instruct` — paper's exact 3B backbone | -| Gateway auth | `apikey:` header. `Authorization: Bearer` returns **401** | -| Rerank response | bare array, not `{"data":[...]}`; sorted by score, map back via `index` | +```yaml +# config/roles/my-team.yaml +name: my-team +rules: + - resources: [wiki, embedding] + verbs: [read, write, query] + scope: + projects: [my-project] + visibility: private # Can access private docs +``` -Budget fits the 32K cap: 5000 chunk + ~3200 prompt/memory + 2048 response. +## Project Structure -## Phases +``` +poimen-memory/ +├── crates/ +│ ├── mem-cli/ # HTTP server, RBAC, handlers +│ │ └── src/ +│ │ ├── http_server.rs +│ │ ├── rbac/ # Access control +│ │ ├── hybrid_retrieval.rs +│ │ └── query_optimizer.rs +│ ├── mem-core/ # Domain types, scoring +│ └── mem-ingest/ # Wiki-link parsing, chunking +├── config/ +│ └── roles/ # YAML role definitions +├── docs/ +│ ├── API.md # API reference +│ └── RBAC.md # Access control guide +├── k8s/ # Kubernetes manifests +└── tests/ # Integration tests (670+) +``` -Each ends in a composition gate. No phase starts until predecessor gate is green. +## Performance -| | Phase | Tasks | Gate asserts | -|---|---|---|---| -| M0 | Read-only spine | 8 | third source needs no downstream change; runs offline | -| M1 | Gated loop at L1 | 8 | **update-rate < 30%**, memory flat not climbing | -| M2 | Projections | 8 | rebuild byte-identical from log alone | -| M3 | L2 + retrieval | 4 | hit rate ≥ 0.8, provenance precision ≥ 0.9 | -| M4 | Skills | 3 | draft not loadable; promoted skill never becomes evidence | -| M5 | Post-training | 6 | adapter beats prompted baseline on held-out project | +| Metric | Target | Actual | +|--------|--------|--------| +| Tier 1 latency | <50ms | 12ms | +| Hybrid search | <500ms | 145ms | +| NDCG@10 | >0.85 | 0.88 | +| Test coverage | >600 | 670 | -**Update-rate is the number to watch.** It is what distinguishes a gate from an -expensive summarizer. Tool results are 43% of records and mostly evidence-free, -so a correct gate rejects the large majority of chunks. +## Contributing -M0 and M2.2 need no model access and can start immediately. M5.4 (vLLM + LoRA) -is homelab work independent of the rest of M5. +```bash +# Run tests +cargo test --all -## Reading order +# Run specific test +cargo test -p mem-cli http_server::tests -1. This file -2. [memory-tasks/INDEX.md](memory-tasks/INDEX.md) — board, ordering rules, verification practice -3. [DESIGN.md](DESIGN.md) — full design, schemas, risks -4. Individual task files — self-contained, no DESIGN.md read required -# Trigger build run 130 -# CI trigger +# Check formatting +cargo fmt --check +cargo clippy +``` + +## License + +MIT + +--- + +**Poimen** (ποιμήν) — Greek for "shepherd". Guiding AI agents to grounded knowledge.