feat: LLM entity + fact extraction pipeline (Zep paper alignment) (#48)
## Changes
### Entity Extraction
- Switch from WikiLinkFallbackExtractor to LlmEntityExtractor when LLM_ENDPOINT set
- `clean_llm_response()`: strips `<think>` tags, markdown fences, extracts JSON
- Handle array responses (Ollama returns `[...]` not `{entities: [...]}`)
- EntityType custom Deserialize: unknown variants → Unknown (no crash)
- Increase timeout 30s→90s, max_tokens 500→1500 for reasoning models
- Graceful reflection fallback: keep entities if verification fails
### Fact Extraction (NEW)
- LlmFactExtractor: LLM-based relationship extraction between entity pairs
- Validates source/target against known entity list (drops hallucinated edges)
- Same robust JSON cleaning for reasoning models + Ollama
- IngestWorker auto-selects LLM vs Simple based on LLM_ENDPOINT env
### K8s Deployment
- Add `command: ["/app/mem"]` (fix args replacing CMD)
- Add LLM_ENDPOINT, LLM_MODEL env vars for in-cluster LLM
## E2E Tested (local Ollama qwen2.5:3b)
- 12 entities extracted (person, tool, concept, organization)
- 5 edges with relationships and facts
- 781 tests pass
## Zep Paper Alignment (§2.2)
- Entity extraction + resolution (§2.2.1)
- Fact extraction between entity pairs (§2.2.2)
- Temporal edge invalidation ready (t_valid/t_invalid schema)
- Reflection verification (§2.2.1, graceful fallback)
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #48
Co-authored-by: poimen <[email protected]>
This commit was merged in pull request #48.
This commit is contained in:
@@ -1,8 +1,6 @@
|
||||
---
|
||||
# CNPG Postgres cluster for Poimen Memory system (GitOps, declarative extensions).
|
||||
# 2 instances, pgvector 0.7.0 via spec.extensions (not manual CREATE EXTENSION).
|
||||
# Storage: 10Gi longhorn, consistent with temporal-db.yaml.
|
||||
# No manual psql needed — all via git/ArgoCD.
|
||||
# Dedicated CNPG Postgres for Poimen Memory (GitOps, wave 2).
|
||||
# Matches homelab/k8s/infra/databases/memory-db.yaml — single source of truth.
|
||||
# CNPG generates secret `memory-db-app` + service `memory-db-rw` in ns poimen.
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Cluster
|
||||
metadata:
|
||||
@@ -13,20 +11,6 @@ metadata:
|
||||
spec:
|
||||
instances: 2
|
||||
imageName: ghcr.io/cloudnative-pg/postgresql:16.2
|
||||
enableSuperuserAccess: false
|
||||
storage:
|
||||
size: 10Gi
|
||||
storageClass: longhorn
|
||||
resources:
|
||||
requests: { memory: "512Mi", cpu: "250m" }
|
||||
limits: { memory: "2Gi", cpu: "1" }
|
||||
affinity:
|
||||
podAntiAffinityType: preferred
|
||||
topologyKey: kubernetes.io/hostname
|
||||
tolerations:
|
||||
- key: node-role.kubernetes.io/control-plane
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
bootstrap:
|
||||
initdb:
|
||||
database: memory
|
||||
@@ -34,25 +18,19 @@ spec:
|
||||
encoding: UTF8
|
||||
localeCollate: C
|
||||
localeCType: C
|
||||
monitoring:
|
||||
enabled: true
|
||||
podMonitorTemplate:
|
||||
spec:
|
||||
interval: 30s
|
||||
scrapeTimeout: 10s
|
||||
---
|
||||
# Database resource with pgvector extension (declarative, git-managed).
|
||||
# CNPG 1.30.0+ supports this via spec.extensions on the Database CRD.
|
||||
# Ensures pgvector is installed and available for HNSW indexing.
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Database
|
||||
metadata:
|
||||
name: memory
|
||||
namespace: poimen
|
||||
spec:
|
||||
cluster:
|
||||
name: memory-db
|
||||
owner: app
|
||||
extensions:
|
||||
- name: vector
|
||||
ensure: present
|
||||
postInitApplicationSQL:
|
||||
- "CREATE EXTENSION vector;"
|
||||
enableSuperuserAccess: false
|
||||
resources:
|
||||
requests: { memory: "512Mi", cpu: "250m" }
|
||||
limits: { memory: "2Gi", cpu: "1" }
|
||||
storage:
|
||||
size: 20Gi
|
||||
storageClass: longhorn
|
||||
affinity:
|
||||
podAntiAffinityType: preferred
|
||||
topologyKey: kubernetes.io/hostname
|
||||
tolerations:
|
||||
- key: node-role.kubernetes.io/control-plane
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
|
||||
Reference in New Issue
Block a user