poimenandrock
fb61de6b47
## Changes
### Entity Extraction
- Switch from WikiLinkFallbackExtractor to LlmEntityExtractor when LLM_ENDPOINT set
- `clean_llm_response()`: strips `<think>` tags, markdown fences, extracts JSON
- Handle array responses (Ollama returns `[...]` not `{entities: [...]}`)
- EntityType custom Deserialize: unknown variants → Unknown (no crash)
- Increase timeout 30s→90s, max_tokens 500→1500 for reasoning models
- Graceful reflection fallback: keep entities if verification fails
### Fact Extraction (NEW)
- LlmFactExtractor: LLM-based relationship extraction between entity pairs
- Validates source/target against known entity list (drops hallucinated edges)
- Same robust JSON cleaning for reasoning models + Ollama
- IngestWorker auto-selects LLM vs Simple based on LLM_ENDPOINT env
### K8s Deployment
- Add `command: ["/app/mem"]` (fix args replacing CMD)
- Add LLM_ENDPOINT, LLM_MODEL env vars for in-cluster LLM
## E2E Tested (local Ollama qwen2.5:3b)
- 12 entities extracted (person, tool, concept, organization)
- 5 edges with relationships and facts
- 781 tests pass
## Zep Paper Alignment (§2.2)
- Entity extraction + resolution (§2.2.1)
- Fact extraction between entity pairs (§2.2.2)
- Temporal edge invalidation ready (t_valid/t_invalid schema)
- Reflection verification (§2.2.1, graceful fallback)
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #48
Co-authored-by: poimen <[email protected]>
28 lines
796 B
YAML
28 lines
796 B
YAML
# Non-sensitive environment variables for poimen-memory
|
|
# Change these without redeploying secrets.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: poimen-memory-config
|
|
namespace: poimen
|
|
labels:
|
|
app.kubernetes.io/name: poimen-memory
|
|
app.kubernetes.io/component: config
|
|
data:
|
|
# Auth mode: jwt | apikey
|
|
MEM_AUTH_MODE: "none"
|
|
# Rate limiting
|
|
MEM_RATE_LIMIT_INGEST: "100"
|
|
MEM_RATE_LIMIT_QUERY: "1000"
|
|
MEM_IDEMPOTENCY_TTL_SECS: "86400"
|
|
# Embeddings
|
|
MEM_EMBEDDING_BATCH_SIZE: "32"
|
|
# OpenSearch
|
|
OPENSEARCH_HOST: "opensearch.poimen.svc.cluster.local:9200"
|
|
# Obsidian
|
|
# LLM Configuration (for entity extraction)
|
|
LLM_ENDPOINT: "http://api-internal.riotpiao.com:8000/v1/chat/completions"
|
|
LLM_MODEL: "qwen:7b"
|
|
LLM_TIMEOUT_SECS: "30"
|
|
ENABLE_LLM_EXTRACTION: "true"
|