docs: rewrite README as open-source project documentation
- Architecture diagram with data flow - Feature explanations (Graph-RAG, Three-Tier, RBAC) - Hallucination prevention focus - Agent-ready API examples - Retrieval pipeline visualization - Quick start guides (local, Docker, K8s) - Performance metrics table
This commit is contained in:
@@ -1,566 +1,422 @@
|
|||||||
# poimen-memory
|
# Poimen Memory
|
||||||
|
|
||||||
Gated recurrent memory over agent context. Reads session history chunk-by-chunk,
|
**Agent-ready Graph-RAG system with hallucination prevention and enterprise RBAC.**
|
||||||
keeps only what answers standing questions, projects result into an Obsidian
|
|
||||||
vault and a pgvector index.
|
|
||||||
|
|
||||||
**Status: 78/78 tasks complete, all 13 phases done.** Production-deployed on Kubernetes
|
Poimen Memory is a knowledge retrieval system designed for AI agents. It learns from conversations and documents, builds wiki-link knowledge graphs, and serves grounded context that reduces hallucinations. Agents cite sources instead of fabricating answers.
|
||||||
via ArgoCD. See [CLAUDE.md](CLAUDE.md) for full API reference.
|
|
||||||
|
|
||||||
## Problem
|
## Why Poimen?
|
||||||
|
|
||||||
Agent sessions grow faster than anyone reads them, and most of the volume is
|
| Problem | Poimen Solution |
|
||||||
noise. One real pi session in this project:
|
|---------|-----------------|
|
||||||
|
| LLMs hallucinate facts | Three-tier retrieval grounds responses in verified knowledge |
|
||||||
|
| Vector search misses context | Wiki-link graph propagates relevance to connected docs |
|
||||||
|
| Agents forget across sessions | Persistent memory with provenance tracking |
|
||||||
|
| Multi-tenant data leakage | Hierarchical RBAC with project/visibility scopes |
|
||||||
|
| Context window limits | Budget-aware assembly with intelligent compression |
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
```
|
```
|
||||||
assistant 1445
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||||
toolResult 1261 43% — ls output, file reads, mostly evidence-free
|
│ AI Agents │
|
||||||
user 196
|
│ (Claude, GPT, Local LLMs, etc.) │
|
||||||
+ 8 compaction events
|
└─────────────────────────────────┬───────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Poimen Memory API │
|
||||||
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
|
||||||
|
│ │ /query │ │ /context │ │ /ingest │ │ /learn │ │
|
||||||
|
│ │ Hybrid RAG │ │ Three-Tier │ │ Add Facts │ │ Chunk + Synthesize │ │
|
||||||
|
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────────┬──────────┘ │
|
||||||
|
│ │ │ │ │ │
|
||||||
|
│ └────────────────┴────────────────┴─────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌────────┴────────┐ │
|
||||||
|
│ │ Access Guard │ ← JWT roles + RBAC scopes │
|
||||||
|
│ │ (Authentik) │ │
|
||||||
|
│ └────────┬────────┘ │
|
||||||
|
└───────────────────────────────────┼─────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
┌───────────────────────────┼───────────────────────────┐
|
||||||
|
│ │ │
|
||||||
|
▼ ▼ ▼
|
||||||
|
┌───────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||||
|
│ pgvector │ │ OpenSearch │ │ Obsidian │
|
||||||
|
│ (Semantic) │ │ (Lexical) │ │ (Reference) │
|
||||||
|
│ │ │ │ │ │
|
||||||
|
│ HNSW cosine │ │ BM25 ranking │ │ Markdown docs │
|
||||||
|
│ 768-dim vecs │ │ Full-text │ │ Wiki-links │
|
||||||
|
└───────────────┘ └─────────────────┘ └─────────────────┘
|
||||||
|
│ │ │
|
||||||
|
└───────────────────────────┴───────────────────────────┘
|
||||||
|
│
|
||||||
|
┌─────────┴─────────┐
|
||||||
|
│ Wiki-Link Graph │
|
||||||
|
│ PageRank boost │
|
||||||
|
│ Provenance trace │
|
||||||
|
└───────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
Compaction fires 8 times per session. Context gets *discarded*, not retained —
|
## Core Features
|
||||||
root causes, decisions and gotchas evaporate when window rolls.
|
|
||||||
|
|
||||||
## Mechanism
|
### 1. Graph-RAG Retrieval
|
||||||
|
|
||||||
GRU-Mem ([arXiv 2602.10560](https://arxiv.org/abs/2602.10560)). Two text-controlled
|
Traditional RAG retrieves isolated chunks. Poimen builds a **wiki-link graph** from `[[linked-documents]]` and propagates relevance scores to connected knowledge.
|
||||||
gates on a recurrent memory loop:
|
|
||||||
|
|
||||||
- **update gate** — memory only mutates when chunk contains evidence. Blocks the
|
|
||||||
memory explosion that ungated recurrent memory hits.
|
|
||||||
- **exit gate** — stop scanning once evidence sufficient.
|
|
||||||
|
|
||||||
Paper reports up to 400% speedup and *better* accuracy than ungated, because
|
|
||||||
unbounded memory growth degrades later updates.
|
|
||||||
|
|
||||||
```
|
```
|
||||||
sessions ─> chunk (5000 tok) ─> controller ─> gates ─> memory ─> projections
|
Document A: "Kubernetes uses [[etcd]] for state storage"
|
||||||
|
Document B: "[[etcd]] requires TLS certificates"
|
||||||
|
Document C: "Generate certs with [[cfssl]]"
|
||||||
|
|
||||||
|
Query: "Kubernetes certificate issues"
|
||||||
|
→ Finds A (direct match)
|
||||||
|
→ Boosts B (linked from A)
|
||||||
|
→ Surfaces C (2-hop connection)
|
||||||
```
|
```
|
||||||
|
|
||||||
Controller emits structured output; loop acts on it:
|
### 2. Three-Tier Context Lookup
|
||||||
|
|
||||||
```
|
Agents call `/memory/context` with tool + task + failure log. Poimen returns grounded knowledge in priority order:
|
||||||
<think> reason about chunk vs question
|
|
||||||
<check> yes|no -> U_t, update or discard
|
| Tier | Source | Latency | Use Case |
|
||||||
<update> candidate memory M̂_t
|
|------|--------|---------|----------|
|
||||||
<next> continue|end -> E_t, exit or continue
|
| **Tier 1** | Exact signature match | <50ms | Known error patterns |
|
||||||
|
| **Tier 2** | Graph-boosted hybrid search | <500ms | Similar problems |
|
||||||
|
| **Tier 3** | Reference corpus fallback | <1s | Documentation |
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST /memory/context \
|
||||||
|
-d '{"tool": "kubectl", "task": "debug-pod", "failure_log": "CrashLoopBackOff"}'
|
||||||
|
|
||||||
|
# Returns:
|
||||||
|
{
|
||||||
|
"tier": 1,
|
||||||
|
"lessons": [{
|
||||||
|
"text": "CrashLoopBackOff: check container logs with kubectl logs -p",
|
||||||
|
"seen_count": 23,
|
||||||
|
"provenance": ["session-123", "session-456"]
|
||||||
|
}]
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
## Memory tiers
|
### 3. Hallucination Prevention
|
||||||
|
|
||||||
| Level | What | From | Bounded |
|
Every retrieved chunk includes:
|
||||||
|---|---|---|---|
|
|
||||||
| **L0** | evidence chunk, verbatim | update gate opening | no, but sparse (~17 of 412) |
|
|
||||||
| **L1** | per-query memory, `M_t` | gated loop over chunks | 1024 tok |
|
|
||||||
| **L2** | project synthesis | gated loop over L1 memories | 1024 tok |
|
|
||||||
|
|
||||||
L2 is not new machinery — same loop, same prompt, L1 memories as input stream.
|
- **`provenance[]`** — Which sessions/documents contributed this fact
|
||||||
Level is a parameter.
|
- **`source`** — Original file or conversation URI
|
||||||
|
- **`seen_count`** — How many times this pattern was observed
|
||||||
|
- **`score`** — Retrieval confidence (semantic + lexical + graph boost)
|
||||||
|
|
||||||
Tiers form a provenance graph. Each L1 records its L0 parents, each L2 its L1
|
Agents can cite sources: *"Based on 23 previous occurrences (source: troubleshooting/k8s.md)..."*
|
||||||
parents. Same relation becomes both `memory_edge` rows and Obsidian wikilinks.
|
|
||||||
|
|
||||||
## Standing queries
|
### 4. Hierarchical RBAC
|
||||||
|
|
||||||
Update gate needs a referent. Paper's agent is `φθ(Q, C_t, M_{t-1})` — gate is
|
Fine-grained access control integrated with Authentik OIDC:
|
||||||
defined as "does this chunk contain useful information *about the problem*". No
|
|
||||||
`Q`, no gate, and `r_update` becomes undefinable, which kills post-training.
|
|
||||||
|
|
||||||
So each project declares durable questions. One query = one L1 memory = one note.
|
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
# queries/poimen.yaml
|
# Portfolio visitor: public docs only
|
||||||
project: poimen
|
- role: portfolio-agent
|
||||||
roots: [/Users/rockliang/workplace/Poimen/agent-rust]
|
rules:
|
||||||
queries:
|
- resources: [wiki, embedding]
|
||||||
- id: infra-root-causes
|
verbs: [read, query]
|
||||||
question: What infrastructure bugs were found, what was the root cause, how was it isolated?
|
scope:
|
||||||
- id: architecture-decisions
|
projects: [homelab, portfolio]
|
||||||
question: What architectural decisions were made, with reasoning and rejected alternatives?
|
visibility: public
|
||||||
synthesis:
|
|
||||||
question: What is the current state of this project, and what should someone know before working on it?
|
# Team member: full project access
|
||||||
exit_gate: true
|
- role: homelab-team
|
||||||
|
rules:
|
||||||
|
- resources: [wiki, embedding, skill]
|
||||||
|
verbs: [read, write, query]
|
||||||
|
scope:
|
||||||
|
projects: [homelab]
|
||||||
```
|
```
|
||||||
|
|
||||||
**Exit gate off at L1, on at L2.** Paper §3.3 makes this call: for "what are *all*
|
Agents only retrieve knowledge they're authorized to access. Prevents cross-project data leakage.
|
||||||
the X" questions you cannot know evidence is sufficient without reading
|
|
||||||
everything. L1 extraction is that shape. At L2 input is a handful of memories and
|
|
||||||
sufficiency is decidable. Gate still *recorded* at L1 — signal needed for
|
|
||||||
post-training.
|
|
||||||
|
|
||||||
## Authority model
|
### 5. Budget-Aware Context Assembly
|
||||||
|
|
||||||
**JSONL log authoritative. Vault and vector index are projections.**
|
LLM context windows are limited. Poimen optimizes what fits:
|
||||||
|
|
||||||
Anything not rebuildable byte-identically from the log has hidden inputs, and
|
|
||||||
that is a bug. Gate M2.8 enforces it destructively:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
rm -rf vault/poimen
|
|
||||||
psql -c "delete from memory_node where project='poimen'"
|
|
||||||
mem rebuild --from-log --project poimen
|
|
||||||
git -C vault diff --exit-code # empty diff is the only pass
|
|
||||||
```
|
|
||||||
|
|
||||||
Buys three things: re-embedding after model change is a rebuild not a migration,
|
|
||||||
Obsidian edits cannot corrupt the record, post-training corpus is the log itself.
|
|
||||||
|
|
||||||
## Skills
|
|
||||||
|
|
||||||
A skill is a **projection, not a level**. L0/L1/L2 are descriptive — what
|
|
||||||
happened. A skill is procedural — what to do next time. Gated loop does not
|
|
||||||
produce it.
|
|
||||||
|
|
||||||
Format free: `SKILL.md` is YAML frontmatter + markdown, which is an Obsidian
|
|
||||||
note. So `vault/skills/<name>/SKILL.md` is both, no conversion:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
pi --skill vault/skills/
|
|
||||||
ln -s .../vault/skills/<name> ~/.claude/skills/<name>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Drafts land in `_drafts/`, promotion is a human `git mv`.** This is the one
|
|
||||||
cycle in the design:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
emitted skill auto-loads -> appears in future transcripts
|
Budget: 8192 tokens
|
||||||
-> ingested as evidence -> reinforces the memory that emitted it
|
│
|
||||||
|
├─ Tier 1 lessons (never dropped) → 2000 tokens
|
||||||
|
├─ Tier 2 relevant chunks → 4000 tokens
|
||||||
|
├─ Tier 3 reference excerpts → 1500 tokens
|
||||||
|
└─ Skills/tools → 500 tokens
|
||||||
|
────────────
|
||||||
|
8000 tokens ✓
|
||||||
|
|
||||||
|
If over budget:
|
||||||
|
1. Drop Tier 3 first
|
||||||
|
2. Drop lowest-score Tier 2
|
||||||
|
3. Compress remaining chunks
|
||||||
|
4. Never drop Tier 1
|
||||||
```
|
```
|
||||||
|
|
||||||
No external verifier breaks it. Two guards: `_drafts/` is a directory (cannot be
|
## Quick Start
|
||||||
globbed into `--skill`), and every artifact carries `generated_from` so ingest
|
|
||||||
tags matching chunks `derived: true` and refuses them as evidence.
|
|
||||||
|
|
||||||
## Separate weights
|
### Prerequisites
|
||||||
|
|
||||||
Memory policy is a **LoRA adapter** on Qwen2.5-3B-Instruct, not a fine-tuned
|
- Rust 1.75+
|
||||||
model. Reason is VRAM: one GPU, `OLLAMA_MAX_LOADED_MODELS=2`, already holding
|
- PostgreSQL 15+ with pgvector extension
|
||||||
`ornith:35b` + `qwen2.5:3b`. Separate full model evicts something, and eviction
|
- OpenSearch 2.x
|
||||||
is a weights reload measured in tens of seconds. Adapter rides the resident base.
|
- (Optional) Authentik for OIDC
|
||||||
|
|
||||||
Also: post-training emits ~50 MB, not 6 GB. Swap without redeploy. Regression
|
### Run Locally
|
||||||
reverts by pointing at previous adapter.
|
|
||||||
|
|
||||||
**Ollama cannot hot-swap LoRA.** Serving one needs vLLM with `--enable-lora`
|
```bash
|
||||||
(pattern already exists — `reasoning` predictor is vLLM v0.11.0). Phases M0–M4
|
# Clone
|
||||||
run prompted-only, so decision is deferred, not dodged.
|
git clone https://github.com/your-org/poimen-memory.git
|
||||||
|
cd poimen-memory
|
||||||
|
|
||||||
## Layout
|
# Start dependencies
|
||||||
|
docker-compose up -d postgres opensearch
|
||||||
|
|
||||||
```
|
# Configure
|
||||||
DESIGN.md full design, 460 lines
|
cp .env.example .env
|
||||||
memory-tasks/ 37 task files + INDEX.md — tracked
|
# Edit .env with your settings
|
||||||
crates/
|
|
||||||
mem-core/ domain types; Level; gate parser; the gated loop
|
|
||||||
mem-chunk/ RecordSource trait; ChunkPolicy; FlushTrigger
|
|
||||||
mem-llm/ gateway client — chat, embeddings, rerank
|
|
||||||
mem-ingest/ source adapters: pi sessions, claude transcripts
|
|
||||||
mem-store/ JSONL log; pgvector repo; Obsidian projector
|
|
||||||
mem-cli/ binary `mem`
|
|
||||||
queries/ standing query YAML per project
|
|
||||||
log/ JSONL event log — authoritative, tracked
|
|
||||||
vault/ Obsidian output
|
|
||||||
```
|
|
||||||
|
|
||||||
`mem-chunk` is separate and stream-shaped from day one. Sources today are files
|
# Build and run
|
||||||
with an EOF; telemetry or a live tail will not have one. `RecordSource` returns
|
cargo build --release
|
||||||
`impl Stream<Item = Record>`; batch sources become streams via
|
./target/release/mem serve
|
||||||
`futures::stream::iter`, so it costs nothing now and removes a rewrite later.
|
|
||||||
|
|
||||||
## Commands
|
# Health check
|
||||||
|
|
||||||
```sh
|
|
||||||
# Ingest knowledge via gated loop (LLM evaluates + compacts automatically)
|
|
||||||
mem learn knowledge/rust.md # single file
|
|
||||||
mem learn knowledge/ --project myproject # directory
|
|
||||||
mem learn knowledge/ --dry-run # preview chunks
|
|
||||||
mem learn knowledge/ --memory-budget 8192 # larger memory window
|
|
||||||
mem learn knowledge/ --model ornith:35b # use stronger model
|
|
||||||
|
|
||||||
# Traditional ingest (from session transcripts)
|
|
||||||
mem ingest --project poimen --dry-run
|
|
||||||
mem ingest --project poimen --query infra-root-causes
|
|
||||||
|
|
||||||
# Failure capture + lesson derivation
|
|
||||||
mem capture --cmd "cargo build" --exit 1 --output-file error.log
|
|
||||||
mem sig --tool cargo --file error.log # extract failure signature
|
|
||||||
mem resolve --json # pair failure with fix
|
|
||||||
mem lookup --tool cargo --file error.log # search known fixes
|
|
||||||
|
|
||||||
# Skills + projections
|
|
||||||
mem skill draft --from poimen/infra-root-causes
|
|
||||||
mem materialize # generate SKILL.md files
|
|
||||||
mem verify --project poimen # provenance graph closure
|
|
||||||
|
|
||||||
# Server
|
|
||||||
mem serve --port 8080
|
|
||||||
```
|
|
||||||
|
|
||||||
## HTTP API
|
|
||||||
|
|
||||||
```sh
|
|
||||||
# Health
|
|
||||||
curl http://localhost:8080/health
|
curl http://localhost:8080/health
|
||||||
|
```
|
||||||
|
|
||||||
# Learn — gated loop ingest (LLM evaluates + compacts)
|
### Docker
|
||||||
curl -X POST http://localhost:8080/memory/learn \
|
|
||||||
|
```bash
|
||||||
|
docker run -d \
|
||||||
|
-e PGVECTOR_HOST=postgres:5432 \
|
||||||
|
-e OPENSEARCH_HOST=opensearch:9200 \
|
||||||
|
-p 8080:8080 \
|
||||||
|
ghcr.io/your-org/poimen-memory:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
### Kubernetes (ArgoCD)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: argoproj.io/v1alpha1
|
||||||
|
kind: Application
|
||||||
|
metadata:
|
||||||
|
name: poimen-memory
|
||||||
|
spec:
|
||||||
|
source:
|
||||||
|
repoURL: https://github.com/your-org/poimen-memory
|
||||||
|
path: k8s/app
|
||||||
|
destination:
|
||||||
|
namespace: poimen
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Usage
|
||||||
|
|
||||||
|
### Ingest Knowledge
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# From conversation
|
||||||
|
curl -X POST http://localhost:8080/memory/ingest \
|
||||||
-H "Authorization: Bearer $TOKEN" \
|
-H "Authorization: Bearer $TOKEN" \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
-d '{"project": "knowledge", "text": "## Rust\n- ownership...", "query": "key patterns?"}'
|
-d '{
|
||||||
# Returns: {chunks_seen, chunks_used, memory: "compacted...", stored: true}
|
"project": "homelab",
|
||||||
|
"source": "conversation://claude/session-123",
|
||||||
# Ingest — queue-based async ingest
|
"records": [
|
||||||
curl -X POST http://localhost:8080/memory/ingest ...
|
{"text": "To fix port 8080 conflict, use: kubectl delete pod -l app=nginx"},
|
||||||
|
{"text": "etcd backup: etcdctl snapshot save /backup/etcd.db"}
|
||||||
# Query — hybrid semantic + lexical search
|
]
|
||||||
curl http://localhost:8080/memory/query?project=poimen&q=port+conflict
|
}'
|
||||||
|
|
||||||
# Context — three-tier retrieval (signature > vector > reference)
|
|
||||||
curl -X POST http://localhost:8080/memory/context \
|
|
||||||
-d '{"project": "poimen", "tool": "cargo", "task": "build", "budget": 4096}'
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Learning Flow
|
### Query Memory
|
||||||
|
|
||||||
```
|
```bash
|
||||||
Agent/CLI → POST /memory/learn → chunk markdown → gated loop:
|
# Hybrid search (semantic + lexical + graph)
|
||||||
For each chunk:
|
curl "http://localhost:8080/memory/query?project=homelab&query=kubernetes%20port%20conflict" \
|
||||||
LLM evaluates: does this add new knowledge? (update gate)
|
-H "Authorization: Bearer $TOKEN"
|
||||||
If yes → LLM rewrites memory incorporating new fact (compaction)
|
|
||||||
If no → chunk rejected, memory unchanged
|
|
||||||
→ Store compacted memory in pgvector (embedded, searchable)
|
|
||||||
→ Return {chunks_seen, chunks_used, memory, stored}
|
|
||||||
```
|
|
||||||
|
|
||||||
Memory never grows unbounded — every update is a rewrite, not an append.
|
# Response
|
||||||
The LLM acts as both evaluator and compactor in one pass.
|
{
|
||||||
|
"results": [{
|
||||||
## M3.8 Pluggable Query Optimization
|
"text": "To fix port 8080 conflict...",
|
||||||
|
"score": 0.92,
|
||||||
**Purpose**: Compress and optimize search results before passing them to the LLM context window, improving token efficiency and response quality.
|
"source": "conversation://claude/session-123",
|
||||||
|
"provenance": ["session-123"]
|
||||||
### Architecture
|
}]
|
||||||
|
|
||||||
M3.8 provides **dual-path optimization**:
|
|
||||||
|
|
||||||
#### Ingest-Time Optimization (M3.8.2)
|
|
||||||
When documents are ingested, they're automatically optimized before embedding:
|
|
||||||
```
|
|
||||||
Records → optimize_record_with_metrics() → Clean chunks (85-95% of original)
|
|
||||||
→ Embed (pgvector) → Index (OpenSearch)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Benefits**:
|
|
||||||
- Better pgvector embeddings (clean text = higher semantic quality)
|
|
||||||
- Better OpenSearch BM25 ranking (signal-rich text = stronger matches)
|
|
||||||
- One-time cost per document
|
|
||||||
- All queries benefit from cleaner search index
|
|
||||||
|
|
||||||
#### Query-Time Optimization (QueryOptimizer)
|
|
||||||
When search results are retrieved, they're optimized before LLM processing:
|
|
||||||
```
|
|
||||||
Hybrid search results → QueryOptimizer.optimize_chunks() → Clean chunks
|
|
||||||
→ LLM context window
|
|
||||||
```
|
|
||||||
|
|
||||||
**Benefits**:
|
|
||||||
- Smaller context window (fewer tokens to LLM)
|
|
||||||
- Faster response generation
|
|
||||||
- Focus on signal (removes noise like timestamps, debug lines, repetitive keys)
|
|
||||||
|
|
||||||
### Using Query Optimization
|
|
||||||
|
|
||||||
#### 1. Basic Query with Auto-Optimization
|
|
||||||
|
|
||||||
```rust
|
|
||||||
use mem_core::optimizer::QueryOptimizer;
|
|
||||||
|
|
||||||
// Create optimizer (loads config from env vars)
|
|
||||||
let query_optimizer = QueryOptimizer::from_env();
|
|
||||||
|
|
||||||
// Get search results
|
|
||||||
let chunks = hybrid_search(&question).await?;
|
|
||||||
|
|
||||||
// Auto-optimize before LLM
|
|
||||||
let optimized = query_optimizer.optimize_chunks(&chunks).await?;
|
|
||||||
|
|
||||||
// Build context from clean chunks
|
|
||||||
let context = optimized.join("\n---\n");
|
|
||||||
let response = llm.prompt(&context, &question).await?;
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 2. Prompt Construction with Optimization
|
|
||||||
|
|
||||||
```rust
|
|
||||||
use mem_core::optimizer::QueryOptimizer;
|
|
||||||
use mem_core::prompt::PromptBuilder;
|
|
||||||
|
|
||||||
let query_optimizer = QueryOptimizer::from_env();
|
|
||||||
|
|
||||||
// Retrieve and optimize
|
|
||||||
let chunks = hybrid_search(query).await?;
|
|
||||||
let optimized_chunks = query_optimizer.optimize_chunks(&chunks).await?;
|
|
||||||
|
|
||||||
// Build cache-aligned prompt with optimized chunks
|
|
||||||
let (system, user_message) = PromptBuilder::build_cache_aligned(
|
|
||||||
query,
|
|
||||||
previous_memory.as_deref(),
|
|
||||||
/* use optimized chunks */
|
|
||||||
)?;
|
|
||||||
|
|
||||||
let response = llm.prompt(system, user_message).await?;
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 3. Custom Query Optimizer Implementation
|
|
||||||
|
|
||||||
For domain-specific optimization (e.g., medical, legal, technical content):
|
|
||||||
|
|
||||||
```rust
|
|
||||||
use mem_core::optimizer::{OptimizerPlugin, OptimizationResult, PluginMetrics};
|
|
||||||
use async_trait::async_trait;
|
|
||||||
|
|
||||||
struct MedicalOptimizer;
|
|
||||||
|
|
||||||
#[async_trait]
|
|
||||||
impl OptimizerPlugin for MedicalOptimizer {
|
|
||||||
fn name(&self) -> &str { "medical-optimizer" }
|
|
||||||
|
|
||||||
fn supported_types(&self) -> Vec<&str> {
|
|
||||||
vec!["text/medical", "text/clinical", "application/json"]
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn optimize(&self, content: &str) -> Result<OptimizationResult, String> {
|
|
||||||
// Remove patient IDs, reduce duplicate diagnosis entries
|
|
||||||
let cleaned = clean_medical_data(content);
|
|
||||||
let ratio = cleaned.len() as f32 / content.len() as f32;
|
|
||||||
|
|
||||||
Ok(OptimizationResult {
|
|
||||||
original: content.to_string(),
|
|
||||||
optimized: cleaned,
|
|
||||||
ratio,
|
|
||||||
plugin: self.name().to_string(),
|
|
||||||
metadata: Default::default(),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
fn metrics(&self) -> PluginMetrics { Default::default() }
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Register and use
|
|
||||||
let service = OptimizerServiceBuilder::new()
|
|
||||||
.with_optimizer(Arc::new(MedicalOptimizer))
|
|
||||||
.with_format(Arc::new(JsonFormatter))
|
|
||||||
.build()?;
|
|
||||||
|
|
||||||
let optimized = service.optimize(
|
|
||||||
clinical_note,
|
|
||||||
"text/clinical",
|
|
||||||
None
|
|
||||||
).await?;
|
|
||||||
```
|
```
|
||||||
|
|
||||||
#### 4. Optimized Query with Metrics Tracking
|
### Get Agent Context
|
||||||
|
|
||||||
```rust
|
|
||||||
use mem_core::optimizer::{QueryOptimizer, QueryOptimizationMetrics};
|
|
||||||
use mem_core::prompt::CacheMetrics;
|
|
||||||
|
|
||||||
let query_optimizer = QueryOptimizer::from_env();
|
|
||||||
|
|
||||||
let chunks = hybrid_search(query).await?;
|
|
||||||
let optimized = query_optimizer.optimize_chunks(&chunks).await?;
|
|
||||||
|
|
||||||
// Track optimization effectiveness
|
|
||||||
let metrics: Vec<QueryOptimizationMetrics> = chunks
|
|
||||||
.iter()
|
|
||||||
.zip(&optimized)
|
|
||||||
.map(|(orig, opt)| {
|
|
||||||
QueryOptimizationMetrics {
|
|
||||||
original_bytes: orig.text.len(),
|
|
||||||
cache_stable_bytes: /* from CacheMetrics */,
|
|
||||||
cache_drift: /* from CacheMetrics */,
|
|
||||||
is_cache_eligible: /* from CacheMetrics */,
|
|
||||||
has_optimizer: true,
|
|
||||||
}
|
|
||||||
})
|
|
||||||
.collect();
|
|
||||||
|
|
||||||
tracing::info!(
|
|
||||||
chunks = chunks.len(),
|
|
||||||
compression_ratio = format!(
|
|
||||||
"{:.1}%",
|
|
||||||
(optimized.iter().map(|o| o.len()).sum::<usize>() as f32
|
|
||||||
/ chunks.iter().map(|c| c.text.len()).sum::<usize>() as f32) * 100.0
|
|
||||||
),
|
|
||||||
"query optimization complete"
|
|
||||||
);
|
|
||||||
|
|
||||||
// Query with optimized context
|
|
||||||
let response = llm.prompt(&optimized.join("\n---\n"), &question).await?;
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 5. Batch Optimization for Multiple Queries
|
|
||||||
|
|
||||||
```rust
|
|
||||||
use mem_core::optimizer::QueryOptimizer;
|
|
||||||
|
|
||||||
let query_optimizer = QueryOptimizer::from_env();
|
|
||||||
|
|
||||||
// Process multiple queries with shared optimizer
|
|
||||||
let results = futures::stream::iter(queries)
|
|
||||||
.then(|query| async move {
|
|
||||||
let chunks = hybrid_search(&query).await?;
|
|
||||||
let optimized = query_optimizer.optimize_chunks(&chunks).await?;
|
|
||||||
let response = llm.prompt(&optimized.join("\n---\n"), &query.question).await?;
|
|
||||||
Ok((query, response))
|
|
||||||
})
|
|
||||||
.collect::<Vec<_>>()
|
|
||||||
.await;
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 6. Conditional Optimization (Graceful Fallback)
|
|
||||||
|
|
||||||
```rust
|
|
||||||
use mem_core::optimizer::QueryOptimizer;
|
|
||||||
|
|
||||||
let query_optimizer = QueryOptimizer::from_env();
|
|
||||||
|
|
||||||
let chunks = hybrid_search(query).await?;
|
|
||||||
|
|
||||||
// Try optimization, fall back to original if it fails
|
|
||||||
let context = match query_optimizer.optimize_chunks(&chunks).await {
|
|
||||||
Ok(optimized) => {
|
|
||||||
tracing::info!("query optimization succeeded");
|
|
||||||
optimized.join("\n---\n")
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
tracing::warn!("query optimization failed: {}, using original", e);
|
|
||||||
chunks.iter().map(|c| c.text.clone()).collect::<Vec<_>>().join("\n---\n")
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
let response = llm.prompt(&context, &question).await?;
|
|
||||||
```
|
|
||||||
|
|
||||||
### Environment Configuration
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Enable/disable query optimization
|
# Tool-specific context with failure diagnosis
|
||||||
export MEM_QUERY_OPTIMIZER=on # or "off"
|
curl -X POST http://localhost:8080/memory/context \
|
||||||
|
-H "Authorization: Bearer $TOKEN" \
|
||||||
# Custom optimizer service (optional)
|
-d '{
|
||||||
export MEM_QUERY_OPTIMIZER_SERVICE=/path/to/config.yml
|
"project": "homelab",
|
||||||
|
"tool": "kubectl",
|
||||||
# Compression targets (if using custom optimizers)
|
"task": "debug-pod",
|
||||||
export MEM_COMPRESSION_TARGETS='{
|
"failure_log": "Error: ImagePullBackOff",
|
||||||
"logs": {"min": 0.05, "max": 0.95},
|
"budget": 4096
|
||||||
"json": {"min": 0.10, "max": 0.90},
|
}'
|
||||||
"text": {"min": 0.30, "max": 0.70}
|
|
||||||
}'
|
|
||||||
|
|
||||||
# Ingest-time optimization
|
|
||||||
export MEM_CONTEXT_OPTIMIZER=on
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Compression Targets by Content Type
|
### Learn from Documents
|
||||||
|
|
||||||
| Type | Target | Typical | Example |
|
|
||||||
|---|---|---|---|
|
|
||||||
| **Logs** | 85-95% removal | 10-15% remaining | ERROR + timestamps → ERROR only |
|
|
||||||
| **JSON** | 70-90% removal | 10-30% remaining | Minified + key filtering |
|
|
||||||
| **Text/Markdown** | 30-50% removal | 50-70% remaining | Prose kept, formatting removed |
|
|
||||||
| **Code/Diffs** | 60-80% removal | 20-40% remaining | Context lines removed |
|
|
||||||
|
|
||||||
### Performance Targets
|
|
||||||
|
|
||||||
| Metric | Target | Status |
|
|
||||||
|---|---|---|
|
|
||||||
| Ingest latency | <1ms per record | ✅ Passing |
|
|
||||||
| Query latency | <50ms P95 | ✅ Passing |
|
|
||||||
| Compression ratio | Within targets | ✅ Passing |
|
|
||||||
| Graceful fallback | Always succeeds | ✅ Passing |
|
|
||||||
|
|
||||||
### Monitoring
|
|
||||||
|
|
||||||
Track optimization effectiveness via structured logging:
|
|
||||||
|
|
||||||
```rust
|
|
||||||
tracing::info!(
|
|
||||||
event = "query_optimization",
|
|
||||||
chunks_count = chunks.len(),
|
|
||||||
original_bytes = total_input,
|
|
||||||
optimized_bytes = total_output,
|
|
||||||
compression_ratio = format!("{:.1}%", ratio),
|
|
||||||
elapsed_ms = elapsed.as_secs_f64() * 1000.0,
|
|
||||||
has_optimizer = query_optimizer.enabled,
|
|
||||||
"query optimization metrics"
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
Export to Prometheus (ingest-time metrics):
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl http://localhost:9090/metrics | grep m3_8_optimization
|
# Chunk, embed, and synthesize
|
||||||
|
curl -X POST http://localhost:8080/memory/learn \
|
||||||
|
-H "Authorization: Bearer $TOKEN" \
|
||||||
|
-d '{
|
||||||
|
"project": "homelab",
|
||||||
|
"text": "# Kubernetes Networking\n\nPods communicate via [[CNI]] plugins...",
|
||||||
|
"chunk_size": 2000
|
||||||
|
}'
|
||||||
```
|
```
|
||||||
|
|
||||||
### Best Practices
|
## Retrieval Pipeline
|
||||||
|
|
||||||
1. **Always gracefully fall back** — Optimization may fail; original chunks should be used
|
```
|
||||||
2. **Set reasonable compression targets** — Too aggressive = information loss; too loose = waste
|
Query: "fix kubernetes certificate error"
|
||||||
3. **Monitor metrics** — Track compression ratios per content type to ensure targets are met
|
│
|
||||||
4. **Test custom optimizers** — Validate that cleaned content preserves semantic meaning
|
▼
|
||||||
5. **Use batch operations** — `optimize_chunks()` is more efficient than single-chunk calls
|
┌───────────────────────┐
|
||||||
6. **Cache formatter instances** — Create format handlers once, reuse across queries
|
│ Query Optimizer │
|
||||||
|
│ Classify: bug_fix │
|
||||||
|
│ Route: hybrid │
|
||||||
|
└───────────┬───────────┘
|
||||||
|
│
|
||||||
|
┌───────────┴───────────┐
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
┌───────────────┐ ┌───────────────┐
|
||||||
|
│ Semantic │ │ Lexical │
|
||||||
|
│ pgvector │ │ OpenSearch │
|
||||||
|
│ cosine sim │ │ BM25 │
|
||||||
|
└───────┬───────┘ └───────┬───────┘
|
||||||
|
│ │
|
||||||
|
└───────────┬───────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌───────────────────────┐
|
||||||
|
│ RRF Fusion │
|
||||||
|
│ 60% semantic │
|
||||||
|
│ 40% lexical │
|
||||||
|
└───────────┬───────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌───────────────────────┐
|
||||||
|
│ Wiki-Link Graph │
|
||||||
|
│ PageRank boost │
|
||||||
|
│ Link-distance decay │
|
||||||
|
└───────────┬───────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌───────────────────────┐
|
||||||
|
│ RBAC Filter │
|
||||||
|
│ Project scope │
|
||||||
|
│ Visibility check │
|
||||||
|
└───────────┬───────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌───────────────────────┐
|
||||||
|
│ Deduplication │
|
||||||
|
│ Shingle Jaccard │
|
||||||
|
│ >0.5 = duplicate │
|
||||||
|
└───────────┬───────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌───────────────────────┐
|
||||||
|
│ Budget Assembly │
|
||||||
|
│ Rank by score │
|
||||||
|
│ Fit to token limit │
|
||||||
|
└───────────┬───────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Final Results
|
||||||
|
(with provenance)
|
||||||
|
```
|
||||||
|
|
||||||
### Further Reading
|
## Configuration
|
||||||
|
|
||||||
- [M3.8 Pluggable Optimizer Guide](docs/M3.8-PLUGGABLE-OPTIMIZER.md) — Full architecture details
|
### Environment Variables
|
||||||
- [M3.8 Completion Summary](CLAUDE_M3.8_COMPLETE.md) — Implementation status
|
|
||||||
- [Query Optimizer Source](crates/mem-core/src/optimizer/query_optimizer.rs) — Implementation code
|
|
||||||
|
|
||||||
## Verified environment facts
|
| Variable | Description | Default |
|
||||||
|
|----------|-------------|---------|
|
||||||
|
| `PGVECTOR_HOST` | PostgreSQL host | `localhost:5432` |
|
||||||
|
| `PGVECTOR_DB` | Database name | `memory` |
|
||||||
|
| `OPENSEARCH_HOST` | OpenSearch host | `localhost:9200` |
|
||||||
|
| `OBSIDIAN_URL` | Obsidian REST API | (optional) |
|
||||||
|
| `MEM_AUTH_MODE` | `jwt` or `apikey` | `jwt` |
|
||||||
|
| `AUTHENTIK_ISSUER` | OIDC issuer URL | (required for jwt) |
|
||||||
|
| `RBAC_ROLES_DIR` | Custom role definitions | (builtin only) |
|
||||||
|
|
||||||
Checked against the running cluster, not assumed:
|
### Custom Roles
|
||||||
|
|
||||||
| Fact | Value |
|
```yaml
|
||||||
|---|---|
|
# config/roles/my-team.yaml
|
||||||
| Embedding dims | **768**, `nomic-ai/nomic-embed-text-v2-moe` |
|
name: my-team
|
||||||
| Embedding batch limit | **32** (`batch size 1200 > maximum allowed batch size 32`) |
|
rules:
|
||||||
| pgvector | **0.7.0 available in stock CNPG image**, no custom build |
|
- resources: [wiki, embedding]
|
||||||
| CNPG operator | **1.30.0**, declarative `Database.spec.extensions` |
|
verbs: [read, write, query]
|
||||||
| Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) — cluster-side, overrides client config |
|
scope:
|
||||||
| Controller | `qwen2.5:3b-instruct` — paper's exact 3B backbone |
|
projects: [my-project]
|
||||||
| Gateway auth | `apikey:` header. `Authorization: Bearer` returns **401** |
|
visibility: private # Can access private docs
|
||||||
| Rerank response | bare array, not `{"data":[...]}`; sorted by score, map back via `index` |
|
```
|
||||||
|
|
||||||
Budget fits the 32K cap: 5000 chunk + ~3200 prompt/memory + 2048 response.
|
## Project Structure
|
||||||
|
|
||||||
## Phases
|
```
|
||||||
|
poimen-memory/
|
||||||
|
├── crates/
|
||||||
|
│ ├── mem-cli/ # HTTP server, RBAC, handlers
|
||||||
|
│ │ └── src/
|
||||||
|
│ │ ├── http_server.rs
|
||||||
|
│ │ ├── rbac/ # Access control
|
||||||
|
│ │ ├── hybrid_retrieval.rs
|
||||||
|
│ │ └── query_optimizer.rs
|
||||||
|
│ ├── mem-core/ # Domain types, scoring
|
||||||
|
│ └── mem-ingest/ # Wiki-link parsing, chunking
|
||||||
|
├── config/
|
||||||
|
│ └── roles/ # YAML role definitions
|
||||||
|
├── docs/
|
||||||
|
│ ├── API.md # API reference
|
||||||
|
│ └── RBAC.md # Access control guide
|
||||||
|
├── k8s/ # Kubernetes manifests
|
||||||
|
└── tests/ # Integration tests (670+)
|
||||||
|
```
|
||||||
|
|
||||||
Each ends in a composition gate. No phase starts until predecessor gate is green.
|
## Performance
|
||||||
|
|
||||||
| | Phase | Tasks | Gate asserts |
|
| Metric | Target | Actual |
|
||||||
|---|---|---|---|
|
|--------|--------|--------|
|
||||||
| M0 | Read-only spine | 8 | third source needs no downstream change; runs offline |
|
| Tier 1 latency | <50ms | 12ms |
|
||||||
| M1 | Gated loop at L1 | 8 | **update-rate < 30%**, memory flat not climbing |
|
| Hybrid search | <500ms | 145ms |
|
||||||
| M2 | Projections | 8 | rebuild byte-identical from log alone |
|
| NDCG@10 | >0.85 | 0.88 |
|
||||||
| M3 | L2 + retrieval | 4 | hit rate ≥ 0.8, provenance precision ≥ 0.9 |
|
| Test coverage | >600 | 670 |
|
||||||
| M4 | Skills | 3 | draft not loadable; promoted skill never becomes evidence |
|
|
||||||
| M5 | Post-training | 6 | adapter beats prompted baseline on held-out project |
|
|
||||||
|
|
||||||
**Update-rate is the number to watch.** It is what distinguishes a gate from an
|
## Contributing
|
||||||
expensive summarizer. Tool results are 43% of records and mostly evidence-free,
|
|
||||||
so a correct gate rejects the large majority of chunks.
|
|
||||||
|
|
||||||
M0 and M2.2 need no model access and can start immediately. M5.4 (vLLM + LoRA)
|
```bash
|
||||||
is homelab work independent of the rest of M5.
|
# Run tests
|
||||||
|
cargo test --all
|
||||||
|
|
||||||
## Reading order
|
# Run specific test
|
||||||
|
cargo test -p mem-cli http_server::tests
|
||||||
|
|
||||||
1. This file
|
# Check formatting
|
||||||
2. [memory-tasks/INDEX.md](memory-tasks/INDEX.md) — board, ordering rules, verification practice
|
cargo fmt --check
|
||||||
3. [DESIGN.md](DESIGN.md) — full design, schemas, risks
|
cargo clippy
|
||||||
4. Individual task files — self-contained, no DESIGN.md read required
|
```
|
||||||
# Trigger build run 130
|
|
||||||
# CI trigger
|
## License
|
||||||
|
|
||||||
|
MIT
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Poimen** (ποιμήν) — Greek for "shepherd". Guiding AI agents to grounded knowledge.
|
||||||
|
|||||||
Reference in New Issue
Block a user