feat: LLM entity + fact extraction pipeline (Zep paper alignment) #48

2026-09-09T08:54:07Z

poimen commented

2026-09-09 08:54:07 +00:00

Changes

Entity Extraction

Switch from WikiLinkFallbackExtractor to LlmEntityExtractor when LLM_ENDPOINT set
clean_llm_response(): strips <think> tags, markdown fences, extracts JSON
Handle array responses (Ollama returns [...] not {entities: [...]})
EntityType custom Deserialize: unknown variants → Unknown (no crash)
Increase timeout 30s→90s, max_tokens 500→1500 for reasoning models
Graceful reflection fallback: keep entities if verification fails

Fact Extraction (NEW)

LlmFactExtractor: LLM-based relationship extraction between entity pairs
Validates source/target against known entity list (drops hallucinated edges)
Same robust JSON cleaning for reasoning models + Ollama
IngestWorker auto-selects LLM vs Simple based on LLM_ENDPOINT env

K8s Deployment

Add command: ["/app/mem"] (fix args replacing CMD)
Add LLM_ENDPOINT, LLM_MODEL env vars for in-cluster LLM

E2E Tested (local Ollama qwen2.5:3b)

12 entities extracted (person, tool, concept, organization)
5 edges with relationships and facts
781 tests pass

Zep Paper Alignment (§2.2)

Entity extraction + resolution (§2.2.1)
Fact extraction between entity pairs (§2.2.2)
Temporal edge invalidation ready (t_valid/t_invalid schema)
Reflection verification (§2.2.1, graceful fallback)

## Changes ### Entity Extraction - Switch from WikiLinkFallbackExtractor to LlmEntityExtractor when LLM_ENDPOINT set - `clean_llm_response()`: strips `<think>` tags, markdown fences, extracts JSON - Handle array responses (Ollama returns `[...]` not `{entities: [...]}`) - EntityType custom Deserialize: unknown variants → Unknown (no crash) - Increase timeout 30s→90s, max_tokens 500→1500 for reasoning models - Graceful reflection fallback: keep entities if verification fails ### Fact Extraction (NEW) - LlmFactExtractor: LLM-based relationship extraction between entity pairs - Validates source/target against known entity list (drops hallucinated edges) - Same robust JSON cleaning for reasoning models + Ollama - IngestWorker auto-selects LLM vs Simple based on LLM_ENDPOINT env ### K8s Deployment - Add `command: ["/app/mem"]` (fix args replacing CMD) - Add LLM_ENDPOINT, LLM_MODEL env vars for in-cluster LLM ## E2E Tested (local Ollama qwen2.5:3b) - 12 entities extracted (person, tool, concept, organization) - 5 edges with relationships and facts - 781 tests pass ## Zep Paper Alignment (§2.2) - Entity extraction + resolution (§2.2.1) - Fact extraction between entity pairs (§2.2.2) - Temporal edge invalidation ready (t_valid/t_invalid schema) - Reflection verification (§2.2.1, graceful fallback)

poimen added 3 commits 2026-09-09 08:54:09 +00:00

fix: add command to deployment, args replace CMD not append 99803f5ff8

K8s args without command replaces Dockerfile CMD entirely.
Container tried exec 'serve' as binary instead of '/app/mem serve'.
Add explicit command: ["/app/mem"] so args append correctly.

fix: enable LLM entity extraction + handle reasoning model output 3184c39b79

Root causes of zero entity extraction:
1. IngestWorker used WikiLinkFallbackExtractor (wiki links only)
   Fix: Use LlmEntityExtractor when LLM_ENDPOINT is set
2. ExtractedEntity.entity_type vs LLM returning "type"
   Fix: serde alias "type" -> entity_type, default confidence
3. Reasoning models output <think>...</think> before JSON
   Fix: strip_thinking_tags() extracts JSON from response
4. Reflection verification crashes pipeline on parse failure
   Fix: graceful fallback, keep all entities if reflection fails

Tested with reasoning-predictor (qwen2.5:3b) via port-forward.

feat: LLM-based fact extraction + robust entity parsing

CI / CI (pull_request) Successful in 11m41s

Details

721589d251

Entity extraction fixes:
- clean_llm_response() strips <think> tags, markdown fences, extracts JSON
- Handle array responses (wrap in {"entities": [...]})
- EntityType custom Deserialize: unknown variants map to Unknown (not crash)
- Increase timeout to 90s for reasoning models
- Increase max_tokens to 1500 for reasoning model overhead

Fact extraction (new):
- LlmFactExtractor: LLM-based relationship extraction between entities
- Validates source/target against known entity list (no hallucinated edges)
- Same clean_llm_response() for reasoning model + Ollama compatibility
- Graceful fallback: returns empty on LLM error (no pipeline crash)
- IngestWorker uses LlmFactExtractor when LLM_ENDPOINT set

K8s deployment:
- Add LLM_ENDPOINT, LLM_API_BASE, LLM_MODEL env vars
- Points to in-cluster reasoning-predictor service

Tested E2E with local Ollama (qwen2.5:3b):
- 12 entities extracted (person, tool, concept, organization)
- 5 edges with meaningful relationships and facts
- 781 tests pass

All checks were successful

CI / CI (pull_request) Successful in 11m41s

Details

You are not authorized to merge this pull request.

This pull request can be merged automatically.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/memory-ingest-retrieval:feat/memory-ingest-retrieval

git checkout feat/memory-ingest-retrieval

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/poimen-memory#48