fix: gitignore log/ dir, remove tracked JSONL from repo
Event logs are runtime data, not source code. Also adds mem compact command and browser-use + memory-service knowledge.
This commit is contained in:
@@ -0,0 +1,237 @@
|
||||
# Poimen Memory Service — Agent Integration Guide
|
||||
|
||||
## What Is This
|
||||
Poimen Memory is a knowledge management system that stores, retrieves, and reasons over learned facts. Agents use it to remember solutions, patterns, and skills across sessions. It prevents re-learning the same things and surfaces relevant knowledge when needed.
|
||||
|
||||
## Architecture
|
||||
- **Event Log** (JSONL): Immutable source of truth. All knowledge written here first.
|
||||
- **pgvector** (Postgres): Vector store for semantic search (768-dim embeddings).
|
||||
- **OpenSearch**: Lexical index for BM25 keyword search.
|
||||
- **Hybrid Search**: 60% semantic + 40% lexical via RRF fusion.
|
||||
- **Three-Tier Retrieval**: Tier 1 (exact signature) → Tier 2 (hybrid search) → Tier 3 (reference docs).
|
||||
|
||||
## When to Memorize (What's Worth Storing)
|
||||
|
||||
### DO Memorize
|
||||
- **Bug fixes**: Error message → root cause → solution. Next time the same error appears, instant recall.
|
||||
- **Configuration patterns**: Port conflicts, env var requirements, service dependencies.
|
||||
- **Architecture decisions**: Why we chose X over Y, with trade-offs documented.
|
||||
- **Tool usage patterns**: Commands that work, flags that matter, common gotchas.
|
||||
- **Deployment procedures**: Steps to deploy, verify, rollback. Infra-specific knowledge.
|
||||
- **API contracts**: Request/response formats, auth requirements, rate limits.
|
||||
- **Design patterns**: Patterns that worked well in this codebase, anti-patterns to avoid.
|
||||
- **Failure signatures**: Specific error outputs mapped to known fixes.
|
||||
|
||||
### DO NOT Memorize
|
||||
- Temporary debugging output or scratch work.
|
||||
- Information already in official docs (link to docs instead).
|
||||
- Personal preferences or style opinions (put in CLAUDE.md instead).
|
||||
- Secrets, tokens, passwords, API keys — NEVER.
|
||||
- Exact code implementations (too large, changes too often — store the pattern, not the code).
|
||||
- Trivial facts that any LLM already knows (e.g., "Rust uses cargo for builds").
|
||||
|
||||
### Quality Bar
|
||||
Ask: "If I hit this problem again in 3 months, would this knowledge save me 30+ minutes?" If yes, memorize it. If not, skip it.
|
||||
|
||||
## CLI Usage — `mem learn`
|
||||
|
||||
### Ingest Markdown Knowledge
|
||||
```bash
|
||||
# Ingest a single file
|
||||
mem learn knowledge/rust-fundamentals.md
|
||||
|
||||
# Ingest entire directory
|
||||
mem learn knowledge/
|
||||
|
||||
# Preview chunks without writing (dry run)
|
||||
mem learn knowledge/ --dry-run
|
||||
|
||||
# Custom project namespace
|
||||
mem learn docs/ --project my-project
|
||||
|
||||
# Smaller chunks for fine-grained retrieval
|
||||
mem learn knowledge/ --chunk-size 500
|
||||
```
|
||||
|
||||
### How Chunking Works
|
||||
- Splits on `## ` headings — each section becomes a chunk.
|
||||
- If a section exceeds `--chunk-size`, it hard-splits at the boundary.
|
||||
- Each chunk gets a SHA256 hash for deduplication.
|
||||
- Chunks are stored as JSONL events in `log/<project>/learn/latest.jsonl`.
|
||||
|
||||
### Writing Good Knowledge Files
|
||||
```markdown
|
||||
# Topic Title
|
||||
|
||||
## Subtopic A
|
||||
- Concise bullet points with actionable information.
|
||||
- Include the WHY, not just the WHAT.
|
||||
- Code examples with context, not bare snippets.
|
||||
|
||||
## Subtopic B
|
||||
- Each ## section becomes one retrievable chunk.
|
||||
- Keep sections focused — one concept per section.
|
||||
- 200-500 chars per section is ideal for retrieval precision.
|
||||
```
|
||||
|
||||
## CLI Usage — Other Commands
|
||||
|
||||
### Capture Failures (Auto-Learn from Errors)
|
||||
```bash
|
||||
# Record a command execution and its output
|
||||
mem capture --cmd "cargo build" --exit 1 --output-file /tmp/build-error.log
|
||||
|
||||
# Extract failure signature
|
||||
mem sig --tool cargo --file /tmp/build-error.log
|
||||
```
|
||||
|
||||
### Resolve Lessons (Pair Failures with Fixes)
|
||||
```bash
|
||||
# After fixing a bug, derive the lesson
|
||||
mem resolve --json
|
||||
```
|
||||
|
||||
### Lookup Known Fixes
|
||||
```bash
|
||||
# Search memory for known fix
|
||||
echo "error[E0308]: mismatched types" | mem lookup --tool cargo
|
||||
|
||||
# With minimum confidence threshold
|
||||
mem lookup --tool kubectl --file /tmp/error.log --floor 0.7
|
||||
```
|
||||
|
||||
### Materialize Skills
|
||||
```bash
|
||||
# Generate SKILL.md files from memory
|
||||
mem materialize
|
||||
```
|
||||
|
||||
## HTTP API — Programmatic Access
|
||||
|
||||
### Health Check
|
||||
```bash
|
||||
curl -s http://localhost:8080/health | jq .
|
||||
# {"status": "ok", "timestamp": "..."}
|
||||
```
|
||||
|
||||
### Ingest Records
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8080/memory/ingest \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"project": "poimen",
|
||||
"source": "transcript://session-123",
|
||||
"kind": "L1",
|
||||
"text": "Port 8080 conflict fixed by changing to 8081 in deployment.yaml",
|
||||
"metadata": {"topic": "kubernetes", "session_id": "sess-123"}
|
||||
}' | jq .
|
||||
# Returns: 201 with chunk ID and SHA256
|
||||
```
|
||||
|
||||
### Query Memory (Hybrid Search)
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8080/memory/query \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"project": "poimen",
|
||||
"query": "how to fix kubernetes port conflict",
|
||||
"limit": 5,
|
||||
"floor": 0.6
|
||||
}' | jq '.results[] | {score, text}'
|
||||
# Returns: ranked results with semantic + lexical scores
|
||||
```
|
||||
|
||||
### Context Lookup (Three-Tier)
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8080/memory/context \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"project": "poimen",
|
||||
"tool": "cargo",
|
||||
"task": "build",
|
||||
"budget": 4096
|
||||
}' | jq .
|
||||
# Returns: tier-1 signature matches, tier-2 search results, tier-3 reference docs
|
||||
```
|
||||
|
||||
### Browse Vault
|
||||
```bash
|
||||
curl -s "http://localhost:8080/memory/vault?project=poimen" \
|
||||
-H "Authorization: Bearer $TOKEN" | jq '.files[] | .path'
|
||||
```
|
||||
|
||||
### Rebuild from Log
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8080/memory/rebuild \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"project": "poimen", "dry_run": true, "verify_parity": true}' | jq .
|
||||
```
|
||||
|
||||
## Authentication
|
||||
```bash
|
||||
# Get JWT token from Authentik
|
||||
TOKEN=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
|
||||
-d "grant_type=client_credentials" \
|
||||
-d "client_id=$CLIENT_ID" \
|
||||
-d "client_secret=$CLIENT_SECRET" | jq -r .access_token)
|
||||
|
||||
# Use token
|
||||
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8080/memory/query ...
|
||||
```
|
||||
|
||||
## Rate Limits
|
||||
- Ingest: 100 requests/hour
|
||||
- Query: 1000 requests/hour
|
||||
- Context: 100 requests/hour
|
||||
- Idempotency: 24-hour TTL (same ingest key → 409 Conflict)
|
||||
|
||||
## Agent Workflow — When to Query vs Ingest
|
||||
|
||||
### Before Starting a Task
|
||||
```
|
||||
1. Extract key terms from the task description
|
||||
2. Query memory: mem lookup --tool <tool> or POST /memory/query
|
||||
3. If results found with score > 0.7 → apply known solution
|
||||
4. If no results → proceed with normal problem-solving
|
||||
```
|
||||
|
||||
### After Completing a Task
|
||||
```
|
||||
1. Did I learn something new? (bug fix, config pattern, deployment step)
|
||||
2. Would this save 30+ minutes if encountered again?
|
||||
3. If yes → write a markdown note → mem learn note.md
|
||||
4. If it was a failure→fix pair → mem capture + mem resolve
|
||||
```
|
||||
|
||||
### During Debugging
|
||||
```
|
||||
1. Hit an error → mem sig --tool <tool> --file error.log
|
||||
2. Check if signature matches known fix → mem lookup
|
||||
3. If known → apply fix
|
||||
4. If unknown → debug normally, then memorize the solution
|
||||
```
|
||||
|
||||
## Knowledge Levels
|
||||
- **L0**: Raw evidence (transcripts, logs). Automatically extracted.
|
||||
- **L1**: Per-query memory (specific facts, fixes, patterns). Most useful for agents.
|
||||
- **L2**: Project-level synthesis (cross-cutting themes, architecture summaries).
|
||||
- **R**: Reference documents (Obsidian vault, external docs). Read-only fallback.
|
||||
|
||||
## Infrastructure
|
||||
- **Cluster**: Kubernetes (poimen namespace)
|
||||
- **Database**: CNPG pgvector (2 replicas)
|
||||
- **Search**: OpenSearch (2-node HA)
|
||||
- **Auth**: Authentik (JWT/OIDC)
|
||||
- **Deployment**: ArgoCD from `forgejo.riotpiao.com/rock/poimen-memory`
|
||||
- **Obsidian**: REST API at `obsidian-server.poimen.svc` (reference corpus)
|
||||
|
||||
## Gotchas
|
||||
- Event log is immutable — you can't delete knowledge, only supersede it with newer facts.
|
||||
- Rebuild from log is deterministic — same log → same vector store state.
|
||||
- OpenSearch is fail-soft — if it's down, semantic search still works via pgvector alone.
|
||||
- Embedding model is `sentence-transformers/all-MiniLM-L6-v2` (768-dim). Don't mix with other embedding models.
|
||||
- `mem learn` appends to `latest.jsonl` — running it twice on the same files creates duplicate chunks (use SHA256 to detect).
|
||||
Reference in New Issue
Block a user