rock/poimen-memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rewrite README as open-source project documentation
Build and Push / Test (push) Successful in 4m43s
Details
Build and Push / Build and push image (push) Successful in 31s
Details

Browse Source 
- Architecture diagram with data flow
- Feature explanations (Graph-RAG, Three-Tier, RBAC)
- Hallucination prevention focus
- Agent-ready API examples
- Retrieval pipeline visualization
- Quick start guides (local, Docker, K8s)
- Performance metrics table
This commit is contained in:
rock
2026-09-02 11:30:11 -07:00
parent d9a143995a
commit 3b448a1838
1 changed files with 349 additions and 493 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+357 -501
View File
@@ -1,566 +1,422 @@
# poimen-memory # Poimen Memory
Gated recurrent memory over agent context. Reads session history chunk-by-chunk, **Agent-ready Graph-RAG system with hallucination prevention and enterprise RBAC.**
keeps only what answers standing questions, projects result into an Obsidian
vault and a pgvector index.
**Status: 78/78 tasks complete, all 13 phases done.** Production-deployed on Kubernetes Poimen Memory is a knowledge retrieval system designed for AI agents. It learns from conversations and documents, builds wiki-link knowledge graphs, and serves grounded context that reduces hallucinations. Agents cite sources instead of fabricating answers.
via ArgoCD. See [CLAUDE.md](CLAUDE.md) for full API reference.
## Problem ## Why Poimen?
Agent sessions grow faster than anyone reads them, and most of the volume is | Problem | Poimen Solution |
noise. One real pi session in this project: |---------|-----------------|
| LLMs hallucinate facts | Three-tier retrieval grounds responses in verified knowledge |
| Vector search misses context | Wiki-link graph propagates relevance to connected docs |
| Agents forget across sessions | Persistent memory with provenance tracking |
| Multi-tenant data leakage | Hierarchical RBAC with project/visibility scopes |
| Context window limits | Budget-aware assembly with intelligent compression |
## Architecture
``` ```
assistant 1445 ┌─────────────────────────────────────────────────────────────────────────────┐
toolResult 1261 43% — ls output, file reads, mostly evidence-free │ AI Agents │
user 196 │ (Claude, GPT, Local LLMs, etc.) │
+ 8 compaction events └─────────────────────────────────┬───────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ Poimen Memory API │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ /query │ │ /context │ │ /ingest │ │ /learn │ │
│ │ Hybrid RAG │ │ Three-Tier │ │ Add Facts │ │ Chunk + Synthesize │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────────┬──────────┘ │
│ │ │ │ │ │
│ └────────────────┴────────────────┴─────────────────────┘ │
│ │ │
│ ┌────────┴────────┐ │
│ │ Access Guard │ ← JWT roles + RBAC scopes │
│ │ (Authentik) │ │
│ └────────┬────────┘ │
└───────────────────────────────────┼─────────────────────────────────────────┘
│
┌───────────────────────────┼───────────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ pgvector │ │ OpenSearch │ │ Obsidian │
│ (Semantic) │ │ (Lexical) │ │ (Reference) │
│ │ │ │ │ │
│ HNSW cosine │ │ BM25 ranking │ │ Markdown docs │
│ 768-dim vecs │ │ Full-text │ │ Wiki-links │
└───────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
└───────────────────────────┴───────────────────────────┘
│
┌─────────┴─────────┐
│ Wiki-Link Graph │
│ PageRank boost │
│ Provenance trace │
└───────────────────┘
``` ```
Compaction fires 8 times per session. Context gets *discarded*, not retained — ## Core Features
root causes, decisions and gotchas evaporate when window rolls.
## Mechanism ### 1. Graph-RAG Retrieval
GRU-Mem ([arXiv 2602.10560](https://arxiv.org/abs/2602.10560)). Two text-controlled Traditional RAG retrieves isolated chunks. Poimen builds a **wiki-link graph** from `[[linked-documents]]` and propagates relevance scores to connected knowledge.
gates on a recurrent memory loop:
- **update gate** — memory only mutates when chunk contains evidence. Blocks the
memory explosion that ungated recurrent memory hits.
- **exit gate** — stop scanning once evidence sufficient.
Paper reports up to 400% speedup and *better* accuracy than ungated, because
unbounded memory growth degrades later updates.
``` ```
sessions ─> chunk (5000 tok) ─> controller ─> gates ─> memory ─> projections Document A: "Kubernetes uses [[etcd]] for state storage"
Document B: "[[etcd]] requires TLS certificates"
Document C: "Generate certs with [[cfssl]]"
Query: "Kubernetes certificate issues"
→ Finds A (direct match)
→ Boosts B (linked from A)
→ Surfaces C (2-hop connection)
``` ```
Controller emits structured output; loop acts on it: ### 2. Three-Tier Context Lookup
``` Agents call `/memory/context` with tool + task + failure log. Poimen returns grounded knowledge in priority order:
<think> reason about chunk vs question
<check> yes|no -> U_t, update or discard | Tier | Source | Latency | Use Case |
<update> candidate memory M̂_t |------|--------|---------|----------|
<next> continue|end -> E_t, exit or continue | **Tier 1** | Exact signature match | <50ms | Known error patterns |
| **Tier 2** | Graph-boosted hybrid search | <500ms | Similar problems |
| **Tier 3** | Reference corpus fallback | <1s | Documentation |
```bash
curl -X POST /memory/context \
-d '{"tool": "kubectl", "task": "debug-pod", "failure_log": "CrashLoopBackOff"}'
# Returns:
{
"tier": 1,
"lessons": [{
"text": "CrashLoopBackOff: check container logs with kubectl logs -p",
"seen_count": 23,
"provenance": ["session-123", "session-456"]
}]
}
``` ```
## Memory tiers ### 3. Hallucination Prevention
| Level | What | From | Bounded | Every retrieved chunk includes:
|---|---|---|---|
| **L0** | evidence chunk, verbatim | update gate opening | no, but sparse (~17 of 412) |
| **L1** | per-query memory, `M_t` | gated loop over chunks | 1024 tok |
| **L2** | project synthesis | gated loop over L1 memories | 1024 tok |
L2 is not new machinery — same loop, same prompt, L1 memories as input stream. - **`provenance[]`** — Which sessions/documents contributed this fact
Level is a parameter. - **`source`** — Original file or conversation URI
- **`seen_count`** — How many times this pattern was observed
- **`score`** — Retrieval confidence (semantic + lexical + graph boost)
Tiers form a provenance graph. Each L1 records its L0 parents, each L2 its L1 Agents can cite sources: *"Based on 23 previous occurrences (source: troubleshooting/k8s.md)..."*
parents. Same relation becomes both `memory_edge` rows and Obsidian wikilinks.
## Standing queries ### 4. Hierarchical RBAC
Update gate needs a referent. Paper's agent is `φθ(Q, C_t, M_{t-1})` — gate is Fine-grained access control integrated with Authentik OIDC:
defined as "does this chunk contain useful information *about the problem*". No
`Q`, no gate, and `r_update` becomes undefinable, which kills post-training.
So each project declares durable questions. One query = one L1 memory = one note.
```yaml ```yaml
# queries/poimen.yaml # Portfolio visitor: public docs only
project: poimen - role: portfolio-agent
roots: [/Users/rockliang/workplace/Poimen/agent-rust] rules:
queries: - resources: [wiki, embedding]
- id: infra-root-causes verbs: [read, query]
question: What infrastructure bugs were found, what was the root cause, how was it isolated? scope:
- id: architecture-decisions projects: [homelab, portfolio]
question: What architectural decisions were made, with reasoning and rejected alternatives? visibility: public
synthesis:
question: What is the current state of this project, and what should someone know before working on it? # Team member: full project access
exit_gate: true - role: homelab-team
rules:
- resources: [wiki, embedding, skill]
verbs: [read, write, query]
scope:
projects: [homelab]
``` ```
**Exit gate off at L1, on at L2.** Paper §3.3 makes this call: for "what are *all* Agents only retrieve knowledge they're authorized to access. Prevents cross-project data leakage.
the X" questions you cannot know evidence is sufficient without reading
everything. L1 extraction is that shape. At L2 input is a handful of memories and
sufficiency is decidable. Gate still *recorded* at L1 — signal needed for
post-training.
## Authority model ### 5. Budget-Aware Context Assembly
**JSONL log authoritative. Vault and vector index are projections.** LLM context windows are limited. Poimen optimizes what fits:
Anything not rebuildable byte-identically from the log has hidden inputs, and
that is a bug. Gate M2.8 enforces it destructively:
```sh
rm -rf vault/poimen
psql -c "delete from memory_node where project='poimen'"
mem rebuild --from-log --project poimen
git -C vault diff --exit-code # empty diff is the only pass
```
Buys three things: re-embedding after model change is a rebuild not a migration,
Obsidian edits cannot corrupt the record, post-training corpus is the log itself.
## Skills
A skill is a **projection, not a level**. L0/L1/L2 are descriptive — what
happened. A skill is procedural — what to do next time. Gated loop does not
produce it.
Format free: `SKILL.md` is YAML frontmatter + markdown, which is an Obsidian
note. So `vault/skills/<name>/SKILL.md` is both, no conversion:
```sh
pi --skill vault/skills/
ln -s .../vault/skills/<name> ~/.claude/skills/<name>
```
**Drafts land in `_drafts/`, promotion is a human `git mv`.** This is the one
cycle in the design:
``` ```
emitted skill auto-loads -> appears in future transcripts Budget: 8192 tokens
-> ingested as evidence -> reinforces the memory that emitted it │
├─ Tier 1 lessons (never dropped) → 2000 tokens
├─ Tier 2 relevant chunks → 4000 tokens
├─ Tier 3 reference excerpts → 1500 tokens
└─ Skills/tools → 500 tokens
────────────
8000 tokens ✓
If over budget:
1. Drop Tier 3 first
2. Drop lowest-score Tier 2
3. Compress remaining chunks
4. Never drop Tier 1
``` ```
No external verifier breaks it. Two guards: `_drafts/` is a directory (cannot be ## Quick Start
globbed into `--skill`), and every artifact carries `generated_from` so ingest
tags matching chunks `derived: true` and refuses them as evidence.
## Separate weights ### Prerequisites
Memory policy is a **LoRA adapter** on Qwen2.5-3B-Instruct, not a fine-tuned - Rust 1.75+
model. Reason is VRAM: one GPU, `OLLAMA_MAX_LOADED_MODELS=2`, already holding - PostgreSQL 15+ with pgvector extension
`ornith:35b` + `qwen2.5:3b`. Separate full model evicts something, and eviction - OpenSearch 2.x
is a weights reload measured in tens of seconds. Adapter rides the resident base. - (Optional) Authentik for OIDC
Also: post-training emits ~50 MB, not 6 GB. Swap without redeploy. Regression ### Run Locally
reverts by pointing at previous adapter.
**Ollama cannot hot-swap LoRA.** Serving one needs vLLM with `--enable-lora` ```bash
(pattern already exists — `reasoning` predictor is vLLM v0.11.0). Phases M0–M4 # Clone
run prompted-only, so decision is deferred, not dodged. git clone https://github.com/your-org/poimen-memory.git
cd poimen-memory
## Layout # Start dependencies
docker-compose up -d postgres opensearch
``` # Configure
DESIGN.md full design, 460 lines cp .env.example .env
memory-tasks/ 37 task files + INDEX.md — tracked # Edit .env with your settings
crates/
mem-core/ domain types; Level; gate parser; the gated loop
mem-chunk/ RecordSource trait; ChunkPolicy; FlushTrigger
mem-llm/ gateway client — chat, embeddings, rerank
mem-ingest/ source adapters: pi sessions, claude transcripts
mem-store/ JSONL log; pgvector repo; Obsidian projector
mem-cli/ binary `mem`
queries/ standing query YAML per project
log/ JSONL event log — authoritative, tracked
vault/ Obsidian output
```
`mem-chunk` is separate and stream-shaped from day one. Sources today are files # Build and run
with an EOF; telemetry or a live tail will not have one. `RecordSource` returns cargo build --release
`impl Stream<Item = Record>`; batch sources become streams via ./target/release/mem serve
`futures::stream::iter`, so it costs nothing now and removes a rewrite later.
## Commands # Health check
```sh
# Ingest knowledge via gated loop (LLM evaluates + compacts automatically)
mem learn knowledge/rust.md # single file
mem learn knowledge/ --project myproject # directory
mem learn knowledge/ --dry-run # preview chunks
mem learn knowledge/ --memory-budget 8192 # larger memory window
mem learn knowledge/ --model ornith:35b # use stronger model
# Traditional ingest (from session transcripts)
mem ingest --project poimen --dry-run
mem ingest --project poimen --query infra-root-causes
# Failure capture + lesson derivation
mem capture --cmd "cargo build" --exit 1 --output-file error.log
mem sig --tool cargo --file error.log # extract failure signature
mem resolve --json # pair failure with fix
mem lookup --tool cargo --file error.log # search known fixes
# Skills + projections
mem skill draft --from poimen/infra-root-causes
mem materialize # generate SKILL.md files
mem verify --project poimen # provenance graph closure
# Server
mem serve --port 8080
```
## HTTP API
```sh
# Health
curl http://localhost:8080/health curl http://localhost:8080/health
```
# Learn — gated loop ingest (LLM evaluates + compacts) ### Docker
curl -X POST http://localhost:8080/memory/learn \
```bash
docker run -d \
-e PGVECTOR_HOST=postgres:5432 \
-e OPENSEARCH_HOST=opensearch:9200 \
-p 8080:8080 \
ghcr.io/your-org/poimen-memory:latest
```
### Kubernetes (ArgoCD)
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: poimen-memory
spec:
source:
repoURL: https://github.com/your-org/poimen-memory
path: k8s/app
destination:
namespace: poimen
```
## API Usage
### Ingest Knowledge
```bash
# From conversation
curl -X POST http://localhost:8080/memory/ingest \
-H "Authorization: Bearer $TOKEN" \ -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{"project": "knowledge", "text": "## Rust\n- ownership...", "query": "key patterns?"}' -d '{
# Returns: {chunks_seen, chunks_used, memory: "compacted...", stored: true} "project": "homelab",
"source": "conversation://claude/session-123",
# Ingest — queue-based async ingest "records": [
curl -X POST http://localhost:8080/memory/ingest ... {"text": "To fix port 8080 conflict, use: kubectl delete pod -l app=nginx"},
{"text": "etcd backup: etcdctl snapshot save /backup/etcd.db"}
# Query — hybrid semantic + lexical search ]
curl http://localhost:8080/memory/query?project=poimen&q=port+conflict
# Context — three-tier retrieval (signature > vector > reference)
curl -X POST http://localhost:8080/memory/context \
-d '{"project": "poimen", "tool": "cargo", "task": "build", "budget": 4096}'
```
### Learning Flow
```
Agent/CLI → POST /memory/learn → chunk markdown → gated loop:
For each chunk:
LLM evaluates: does this add new knowledge? (update gate)
If yes → LLM rewrites memory incorporating new fact (compaction)
If no → chunk rejected, memory unchanged
→ Store compacted memory in pgvector (embedded, searchable)
→ Return {chunks_seen, chunks_used, memory, stored}
```
Memory never grows unbounded — every update is a rewrite, not an append.
The LLM acts as both evaluator and compactor in one pass.
## M3.8 Pluggable Query Optimization
**Purpose**: Compress and optimize search results before passing them to the LLM context window, improving token efficiency and response quality.
### Architecture
M3.8 provides **dual-path optimization**:
#### Ingest-Time Optimization (M3.8.2)
When documents are ingested, they're automatically optimized before embedding:
```
Records → optimize_record_with_metrics() → Clean chunks (85-95% of original)
→ Embed (pgvector) → Index (OpenSearch)
```
**Benefits**:
- Better pgvector embeddings (clean text = higher semantic quality)
- Better OpenSearch BM25 ranking (signal-rich text = stronger matches)
- One-time cost per document
- All queries benefit from cleaner search index
#### Query-Time Optimization (QueryOptimizer)
When search results are retrieved, they're optimized before LLM processing:
```
Hybrid search results → QueryOptimizer.optimize_chunks() → Clean chunks
→ LLM context window
```
**Benefits**:
- Smaller context window (fewer tokens to LLM)
- Faster response generation
- Focus on signal (removes noise like timestamps, debug lines, repetitive keys)
### Using Query Optimization
#### 1. Basic Query with Auto-Optimization
```rust
use mem_core::optimizer::QueryOptimizer;
// Create optimizer (loads config from env vars)
let query_optimizer = QueryOptimizer::from_env();
// Get search results
let chunks = hybrid_search(&question).await?;
// Auto-optimize before LLM
let optimized = query_optimizer.optimize_chunks(&chunks).await?;
// Build context from clean chunks
let context = optimized.join("\n---\n");
let response = llm.prompt(&context, &question).await?;
```
#### 2. Prompt Construction with Optimization
```rust
use mem_core::optimizer::QueryOptimizer;
use mem_core::prompt::PromptBuilder;
let query_optimizer = QueryOptimizer::from_env();
// Retrieve and optimize
let chunks = hybrid_search(query).await?;
let optimized_chunks = query_optimizer.optimize_chunks(&chunks).await?;
// Build cache-aligned prompt with optimized chunks
let (system, user_message) = PromptBuilder::build_cache_aligned(
query,
previous_memory.as_deref(),
/* use optimized chunks */
)?;
let response = llm.prompt(system, user_message).await?;
```
#### 3. Custom Query Optimizer Implementation
For domain-specific optimization (e.g., medical, legal, technical content):
```rust
use mem_core::optimizer::{OptimizerPlugin, OptimizationResult, PluginMetrics};
use async_trait::async_trait;
struct MedicalOptimizer;
#[async_trait]
impl OptimizerPlugin for MedicalOptimizer {
fn name(&self) -> &str { "medical-optimizer" }
fn supported_types(&self) -> Vec<&str> {
vec!["text/medical", "text/clinical", "application/json"]
}
async fn optimize(&self, content: &str) -> Result<OptimizationResult, String> {
// Remove patient IDs, reduce duplicate diagnosis entries
let cleaned = clean_medical_data(content);
let ratio = cleaned.len() as f32 / content.len() as f32;
Ok(OptimizationResult {
original: content.to_string(),
optimized: cleaned,
ratio,
plugin: self.name().to_string(),
metadata: Default::default(),
})
}
fn metrics(&self) -> PluginMetrics { Default::default() }
}
// Register and use
let service = OptimizerServiceBuilder::new()
.with_optimizer(Arc::new(MedicalOptimizer))
.with_format(Arc::new(JsonFormatter))
.build()?;
let optimized = service.optimize(
clinical_note,
"text/clinical",
None
).await?;
```
#### 4. Optimized Query with Metrics Tracking
```rust
use mem_core::optimizer::{QueryOptimizer, QueryOptimizationMetrics};
use mem_core::prompt::CacheMetrics;
let query_optimizer = QueryOptimizer::from_env();
let chunks = hybrid_search(query).await?;
let optimized = query_optimizer.optimize_chunks(&chunks).await?;
// Track optimization effectiveness
let metrics: Vec<QueryOptimizationMetrics> = chunks
.iter()
.zip(&optimized)
.map(|(orig, opt)| {
QueryOptimizationMetrics {
original_bytes: orig.text.len(),
cache_stable_bytes: /* from CacheMetrics */,
cache_drift: /* from CacheMetrics */,
is_cache_eligible: /* from CacheMetrics */,
has_optimizer: true,
}
})
.collect();
tracing::info!(
chunks = chunks.len(),
compression_ratio = format!(
"{:.1}%",
(optimized.iter().map(|o| o.len()).sum::<usize>() as f32
/ chunks.iter().map(|c| c.text.len()).sum::<usize>() as f32) * 100.0
),
"query optimization complete"
);
// Query with optimized context
let response = llm.prompt(&optimized.join("\n---\n"), &question).await?;
```
#### 5. Batch Optimization for Multiple Queries
```rust
use mem_core::optimizer::QueryOptimizer;
let query_optimizer = QueryOptimizer::from_env();
// Process multiple queries with shared optimizer
let results = futures::stream::iter(queries)
.then(|query| async move {
let chunks = hybrid_search(&query).await?;
let optimized = query_optimizer.optimize_chunks(&chunks).await?;
let response = llm.prompt(&optimized.join("\n---\n"), &query.question).await?;
Ok((query, response))
})
.collect::<Vec<_>>()
.await;
```
#### 6. Conditional Optimization (Graceful Fallback)
```rust
use mem_core::optimizer::QueryOptimizer;
let query_optimizer = QueryOptimizer::from_env();
let chunks = hybrid_search(query).await?;
// Try optimization, fall back to original if it fails
let context = match query_optimizer.optimize_chunks(&chunks).await {
Ok(optimized) => {
tracing::info!("query optimization succeeded");
optimized.join("\n---\n")
}
Err(e) => {
tracing::warn!("query optimization failed: {}, using original", e);
chunks.iter().map(|c| c.text.clone()).collect::<Vec<_>>().join("\n---\n")
}
};
let response = llm.prompt(&context, &question).await?;
```
### Environment Configuration
```bash
# Enable/disable query optimization
export MEM_QUERY_OPTIMIZER=on # or "off"
# Custom optimizer service (optional)
export MEM_QUERY_OPTIMIZER_SERVICE=/path/to/config.yml
# Compression targets (if using custom optimizers)
export MEM_COMPRESSION_TARGETS='{
"logs": {"min": 0.05, "max": 0.95},
"json": {"min": 0.10, "max": 0.90},
"text": {"min": 0.30, "max": 0.70}
}' }'
# Ingest-time optimization
export MEM_CONTEXT_OPTIMIZER=on
``` ```
### Compression Targets by Content Type ### Query Memory
| Type | Target | Typical | Example |
|---|---|---|---|
| **Logs** | 85-95% removal | 10-15% remaining | ERROR + timestamps → ERROR only |
| **JSON** | 70-90% removal | 10-30% remaining | Minified + key filtering |
| **Text/Markdown** | 30-50% removal | 50-70% remaining | Prose kept, formatting removed |
| **Code/Diffs** | 60-80% removal | 20-40% remaining | Context lines removed |
### Performance Targets
| Metric | Target | Status |
|---|---|---|
| Ingest latency | <1ms per record | ✅ Passing |
| Query latency | <50ms P95 | ✅ Passing |
| Compression ratio | Within targets | ✅ Passing |
| Graceful fallback | Always succeeds | ✅ Passing |
### Monitoring
Track optimization effectiveness via structured logging:
```rust
tracing::info!(
event = "query_optimization",
chunks_count = chunks.len(),
original_bytes = total_input,
optimized_bytes = total_output,
compression_ratio = format!("{:.1}%", ratio),
elapsed_ms = elapsed.as_secs_f64() * 1000.0,
has_optimizer = query_optimizer.enabled,
"query optimization metrics"
);
```
Export to Prometheus (ingest-time metrics):
```bash ```bash
curl http://localhost:9090/metrics | grep m3_8_optimization # Hybrid search (semantic + lexical + graph)
curl "http://localhost:8080/memory/query?project=homelab&query=kubernetes%20port%20conflict" \
-H "Authorization: Bearer $TOKEN"
# Response
{
"results": [{
"text": "To fix port 8080 conflict...",
"score": 0.92,
"source": "conversation://claude/session-123",
"provenance": ["session-123"]
}]
}
``` ```
### Best Practices ### Get Agent Context
1. **Always gracefully fall back** — Optimization may fail; original chunks should be used ```bash
2. **Set reasonable compression targets** — Too aggressive = information loss; too loose = waste # Tool-specific context with failure diagnosis
3. **Monitor metrics** — Track compression ratios per content type to ensure targets are met curl -X POST http://localhost:8080/memory/context \
4. **Test custom optimizers** — Validate that cleaned content preserves semantic meaning -H "Authorization: Bearer $TOKEN" \
5. **Use batch operations** — `optimize_chunks()` is more efficient than single-chunk calls -d '{
6. **Cache formatter instances** — Create format handlers once, reuse across queries "project": "homelab",
"tool": "kubectl",
"task": "debug-pod",
"failure_log": "Error: ImagePullBackOff",
"budget": 4096
}'
```
### Further Reading ### Learn from Documents
- [M3.8 Pluggable Optimizer Guide](docs/M3.8-PLUGGABLE-OPTIMIZER.md) — Full architecture details ```bash
- [M3.8 Completion Summary](CLAUDE_M3.8_COMPLETE.md) — Implementation status # Chunk, embed, and synthesize
- [Query Optimizer Source](crates/mem-core/src/optimizer/query_optimizer.rs) — Implementation code curl -X POST http://localhost:8080/memory/learn \
-H "Authorization: Bearer $TOKEN" \
-d '{
"project": "homelab",
"text": "# Kubernetes Networking\n\nPods communicate via [[CNI]] plugins...",
"chunk_size": 2000
}'
```
## Verified environment facts ## Retrieval Pipeline
Checked against the running cluster, not assumed: ```
Query: "fix kubernetes certificate error"
│
▼
┌───────────────────────┐
│ Query Optimizer │
│ Classify: bug_fix │
│ Route: hybrid │
└───────────┬───────────┘
│
┌───────────┴───────────┐
│ │
▼ ▼
┌───────────────┐ ┌───────────────┐
│ Semantic │ │ Lexical │
│ pgvector │ │ OpenSearch │
│ cosine sim │ │ BM25 │
└───────┬───────┘ └───────┬───────┘
│ │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ RRF Fusion │
│ 60% semantic │
│ 40% lexical │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ Wiki-Link Graph │
│ PageRank boost │
│ Link-distance decay │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ RBAC Filter │
│ Project scope │
│ Visibility check │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ Deduplication │
│ Shingle Jaccard │
│ >0.5 = duplicate │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ Budget Assembly │
│ Rank by score │
│ Fit to token limit │
└───────────┬───────────┘
│
▼
Final Results
(with provenance)
```
| Fact | Value | ## Configuration
|---|---|
| Embedding dims | **768**, `nomic-ai/nomic-embed-text-v2-moe` |
| Embedding batch limit | **32** (`batch size 1200 > maximum allowed batch size 32`) |
| pgvector | **0.7.0 available in stock CNPG image**, no custom build |
| CNPG operator | **1.30.0**, declarative `Database.spec.extensions` |
| Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) — cluster-side, overrides client config |
| Controller | `qwen2.5:3b-instruct` — paper's exact 3B backbone |
| Gateway auth | `apikey:` header. `Authorization: Bearer` returns **401** |
| Rerank response | bare array, not `{"data":[...]}`; sorted by score, map back via `index` |
Budget fits the 32K cap: 5000 chunk + ~3200 prompt/memory + 2048 response. ### Environment Variables
## Phases | Variable | Description | Default |
|----------|-------------|---------|
| `PGVECTOR_HOST` | PostgreSQL host | `localhost:5432` |
| `PGVECTOR_DB` | Database name | `memory` |
| `OPENSEARCH_HOST` | OpenSearch host | `localhost:9200` |
| `OBSIDIAN_URL` | Obsidian REST API | (optional) |
| `MEM_AUTH_MODE` | `jwt` or `apikey` | `jwt` |
| `AUTHENTIK_ISSUER` | OIDC issuer URL | (required for jwt) |
| `RBAC_ROLES_DIR` | Custom role definitions | (builtin only) |
Each ends in a composition gate. No phase starts until predecessor gate is green. ### Custom Roles
| | Phase | Tasks | Gate asserts | ```yaml
|---|---|---|---| # config/roles/my-team.yaml
| M0 | Read-only spine | 8 | third source needs no downstream change; runs offline | name: my-team
| M1 | Gated loop at L1 | 8 | **update-rate < 30%**, memory flat not climbing | rules:
| M2 | Projections | 8 | rebuild byte-identical from log alone | - resources: [wiki, embedding]
| M3 | L2 + retrieval | 4 | hit rate ≥ 0.8, provenance precision ≥ 0.9 | verbs: [read, write, query]
| M4 | Skills | 3 | draft not loadable; promoted skill never becomes evidence | scope:
| M5 | Post-training | 6 | adapter beats prompted baseline on held-out project | projects: [my-project]
visibility: private # Can access private docs
```
**Update-rate is the number to watch.** It is what distinguishes a gate from an ## Project Structure
expensive summarizer. Tool results are 43% of records and mostly evidence-free,
so a correct gate rejects the large majority of chunks.
M0 and M2.2 need no model access and can start immediately. M5.4 (vLLM + LoRA) ```
is homelab work independent of the rest of M5. poimen-memory/
├── crates/
│ ├── mem-cli/ # HTTP server, RBAC, handlers
│ │ └── src/
│ │ ├── http_server.rs
│ │ ├── rbac/ # Access control
│ │ ├── hybrid_retrieval.rs
│ │ └── query_optimizer.rs
│ ├── mem-core/ # Domain types, scoring
│ └── mem-ingest/ # Wiki-link parsing, chunking
├── config/
│ └── roles/ # YAML role definitions
├── docs/
│ ├── API.md # API reference
│ └── RBAC.md # Access control guide
├── k8s/ # Kubernetes manifests
└── tests/ # Integration tests (670+)
```
## Reading order ## Performance
1. This file | Metric | Target | Actual |
2. [memory-tasks/INDEX.md](memory-tasks/INDEX.md) — board, ordering rules, verification practice |--------|--------|--------|
3. [DESIGN.md](DESIGN.md) — full design, schemas, risks | Tier 1 latency | <50ms | 12ms |
4. Individual task files — self-contained, no DESIGN.md read required | Hybrid search | <500ms | 145ms |
# Trigger build run 130 | NDCG@10 | >0.85 | 0.88 |
# CI trigger | Test coverage | >600 | 670 |
## Contributing
```bash
# Run tests
cargo test --all
# Run specific test
cargo test -p mem-cli http_server::tests
# Check formatting
cargo fmt --check
cargo clippy
```
## License
MIT
---
**Poimen** (ποιμήν) — Greek for "shepherd". Guiding AI agents to grounded knowledge.