From 594f497683f2ac98d9cb33192790b8f47e97ed47 Mon Sep 17 00:00:00 2001 From: rock Date: Thu, 10 Sep 2026 04:45:36 +0900 Subject: [PATCH] docs: update CLAUDE.md to current state --- CLAUDE.md | 136 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..d6c32b7 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,136 @@ +# Poimen Memory System + +## Project Status + +**Architecture**: Temporal Knowledge Graph for Agent Memory (Zep paper alignment — arXiv:2501.13956) + +**Current**: Ingest pipeline with LLM entity + fact extraction working E2E. Deployed to K8s. + +### What Works +- ✅ HTTP server (actix-web) with 15+ endpoints +- ✅ LLM entity extraction (LlmEntityExtractor) — extracts person/tool/concept/org entities +- ✅ LLM fact extraction (LlmFactExtractor) — extracts relationships between entities +- ✅ Reasoning model support — strips `` tags, markdown fences +- ✅ Ollama + vLLM + OpenAI-compatible API support +- ✅ Entity persistence to pgvector (memory_entity table) +- ✅ Edge persistence (memory_edge table with temporal fields) +- ✅ Graph query endpoints (entities, edges, BFS traversal) +- ✅ Visualization (React Flow JSON, force-directed layout, SSE streaming) +- ✅ JWT auth (Authentik OIDC) with RBAC +- ✅ K8s deployment (CNPG postgres, ConfigMap, SOPS secrets) +- ✅ CI: PR builds push :SHA tag, main merges retag :latest +- ✅ 781 tests passing + +### Deployment +- **Namespace**: `poimen` +- **Image**: `forgejo.riotpiao.com/riotpiao-poimen/poimen-memory:latest` +- **DB**: CNPG cluster `memory-db` (pgvector) +- **LLM**: `reasoning-predictor.llm-serving.svc.cluster.local` (ornith:35b / qwen2.5:3b) +- **Auth**: Authentik OIDC (`MEM_AUTH_MODE=none` for dev) +- **Registry**: Forgejo container registry (FORGEJO_REGISTRY_USER/TOKEN secrets) + +### Key Env Vars +``` +DATABASE_URL postgresql://... +MEM_AUTH_MODE none|jwt|apikey +LLM_ENDPOINT http://localhost:11434/v1/chat/completions (Ollama) +LLM_MODEL qwen2.5:3b | ornith:35b | reasoning +LLM_API_KEY (for authenticated LLM APIs) +MEM_API_KEY (server API key, fallback "test-key") +OPENSEARCH_HOSTS (optional, hybrid search) +GATEWAY_URL (optional, external queue) +``` + +## Rules + +1. **No progress markdown files.** Track via Forgejo issues + PRs only. +2. **Obsidian vault repo**: `ssh://git@git.riotpiao.com:2222/rock/poimen-obesdient-memory.git` +3. **Secrets via KSOPS**: Age-based SOPS encryption. Never commit plaintext. +4. **Tea CLI**: `poimen` login has API token `1f717a00134f17c9d2d656c620b955e03ea41276` + +## Architecture (Zep Paper §2) + +### Three-Tier Knowledge Graph +``` +Episode Subgraph (raw messages) + → Entity Subgraph (extracted entities + facts/edges) + → Community Subgraph (clusters, planned Phase 4) +``` + +### Ingest Pipeline (4 stages) +1. **Entity extraction** — LLM extracts named entities with type + summary +2. **Deduplication** — HashSet on normalized name +3. **Fact extraction** — LLM extracts relationships between entity pairs +4. **Contradiction detection** — pre-filter + review queue + +### Retrieval (3 methods, §3) +- Cosine semantic similarity (pgvector HNSW) +- BM25 full-text (OpenSearch, optional) +- BFS graph traversal (depth 1-3) + +### Extractors +- `LlmEntityExtractor`: calls LLM_ENDPOINT, parses JSON, handles reasoning models +- `LlmFactExtractor`: takes entity list + text, extracts edges between known entities +- `WikiLinkFallbackExtractor`: pattern-matches `[[wiki links]]` (no LLM) +- `SimpleFactExtractor`: verb pattern matching (no LLM) +- Selection: LLM extractors when `LLM_ENDPOINT` set, else fallbacks + +### LLM Response Cleaning +`clean_llm_response()` handles: +- `...` blocks (reasoning models) +- Markdown code fences (```json ... ```) +- Array responses (wrap in `{"entities": [...]}`) +- Extract first JSON object from mixed text + +## Crate Structure + +``` +crates/ + mem-core/ — Entity, Edge, domain types (174 tests) + mem-store/ — DB repos, schema, vector store + mem-ingest/ — Entity/fact extraction, contradiction detection (87 tests) + mem-llm/ — Embeddings, chat, rerank clients + mem-cli/ — HTTP server, handlers, query, ingest worker (496 tests) +``` + +## API Endpoints + +``` +GET /health +POST /memory/ingest — Queue ingest job +GET /memory/ingest/{id} — Check job status +GET /memory/query?project=&question= — Graph query +POST /memory/query — Unified query +POST /memory/context — Three-tier retrieval +POST /memory/learn — Direct learn +POST /memory/visualize — React Flow JSON +POST /memory/visualize/stream — SSE streaming +POST /memory/compact — Trigger compaction +GET /memory/projects — List projects +GET /memory/skills — List skills +GET /memory/vault — Browse vault +POST /memory/synthesis/* — Entity linking, alias detection +``` + +## Current PRs / Branches + +- **PR #48** `feat/memory-ingest-retrieval` — LLM entity + fact extraction, deployment fixes +- **PR #47** merged — Agent entity types (Phase 3.1) +- **PR #46** merged — Integration test fixes, CI + +## Next Steps + +1. Merge PR #48 → new image with LLM extraction +2. Query retrieval E2E — verify entities/edges returned in query results +3. Visualization E2E — test /memory/visualize with extracted graph +4. Restore 198 deleted tests from PR #46 +5. Community detection (Phase 4, Zep §2.3) +6. Temporal edge invalidation (Zep §2.2.3) +7. Reranker (cross-encoder, RRF, episode-mentions — Zep §3.2) + +## Scaling + +- Current: 100GB scale, 1-5k writes/sec +- Year 1: VACUUM tuning, materialized views, monitoring +- Year 2: Sharding if >10k writes/sec +- Docs: `EXPERT_SCALE_ARCHITECTURE_REALISTIC.md`