Commit Graph
7 Commits
Author SHA1 Message Date
poimenandrock fb61de6b47 feat: LLM entity + fact extraction pipeline (Zep paper alignment) (#48)
CI / CI (push) Successful in 12m9s
Deploy / Tag & Push Latest (push) Failing after 41s
DB Migration / Run Migrations (push) Failing after 18s
## Changes

### Entity Extraction
- Switch from WikiLinkFallbackExtractor to LlmEntityExtractor when LLM_ENDPOINT set
- `clean_llm_response()`: strips `<think>` tags, markdown fences, extracts JSON
- Handle array responses (Ollama returns `[...]` not `{entities: [...]}`)
- EntityType custom Deserialize: unknown variants → Unknown (no crash)
- Increase timeout 30s→90s, max_tokens 500→1500 for reasoning models
- Graceful reflection fallback: keep entities if verification fails

### Fact Extraction (NEW)
- LlmFactExtractor: LLM-based relationship extraction between entity pairs
- Validates source/target against known entity list (drops hallucinated edges)
- Same robust JSON cleaning for reasoning models + Ollama
- IngestWorker auto-selects LLM vs Simple based on LLM_ENDPOINT env

### K8s Deployment
- Add `command: ["/app/mem"]` (fix args replacing CMD)
- Add LLM_ENDPOINT, LLM_MODEL env vars for in-cluster LLM

## E2E Tested (local Ollama qwen2.5:3b)
- 12 entities extracted (person, tool, concept, organization)
- 5 edges with relationships and facts
- 781 tests pass

## Zep Paper Alignment (§2.2)
- Entity extraction + resolution (§2.2.1)
- Fact extraction between entity pairs (§2.2.2)
- Temporal edge invalidation ready (t_valid/t_invalid schema)
- Reflection verification (§2.2.1, graceful fallback)

---------

Co-authored-by: rock <[email protected]>
Reviewed-on: #48
Co-authored-by: poimen <[email protected]>
2026-09-11 01:11:15 +00:00
Story Crater Bot 57f87f494a chore: reduce memory-db cluster from 3 to 2 instances
CHANGES:
- k8s/infra/databases/memory-db.yaml: instances 3 → 2
- Updated comment from '3 instances' to '2 instances'

REASONING:
- Reduces resource overhead (high availability at 2 is sufficient)
- Maintains quorum for failover (minimum 2 for HA)
- Saves memory/CPU allocation on homelab cluster
- ArgoCD will manage rollout automatically

DEPLOYMENT:
- ArgoCD will detect spec change and reconcile
- CNPG will scale down one pod
- Data preserved (3→2 replication, no data loss)
2026-08-28 08:16:02 -07:00
Story Crater Bot b923e0ad68 feat: M2.2 CNPG memory-db with pgvector (declarative, 3 instances) 2026-08-27 20:39:49 -07:00
Story Crater Bot 9ca988aeb3 Downsize memory-db to 2 instances 2026-08-22 23:53:05 -07:00
Story Crater Bot af491564cd Fix: use default longhorn (3 replicas), increase to 20Gi 2026-08-22 23:40:08 -07:00
Story Crater Bot 753435104d Fix: use longhorn-imessage-local (WaitForFirstConsumer) for stable volume binding 2026-08-22 23:36:25 -07:00
Story Crater Bot 695e115212 Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7) 2026-08-22 23:13:42 -07:00