feat: LLM entity + fact extraction pipeline (Zep paper alignment) #48

Merged
rock merged 15 commits from feat/memory-ingest-retrieval into main 2026-09-11 01:11:19 +00:00
Member

Changes

Entity Extraction

  • Switch from WikiLinkFallbackExtractor to LlmEntityExtractor when LLM_ENDPOINT set
  • clean_llm_response(): strips <think> tags, markdown fences, extracts JSON
  • Handle array responses (Ollama returns [...] not {entities: [...]})
  • EntityType custom Deserialize: unknown variants → Unknown (no crash)
  • Increase timeout 30s→90s, max_tokens 500→1500 for reasoning models
  • Graceful reflection fallback: keep entities if verification fails

Fact Extraction (NEW)

  • LlmFactExtractor: LLM-based relationship extraction between entity pairs
  • Validates source/target against known entity list (drops hallucinated edges)
  • Same robust JSON cleaning for reasoning models + Ollama
  • IngestWorker auto-selects LLM vs Simple based on LLM_ENDPOINT env

K8s Deployment

  • Add command: ["/app/mem"] (fix args replacing CMD)
  • Add LLM_ENDPOINT, LLM_MODEL env vars for in-cluster LLM

E2E Tested (local Ollama qwen2.5:3b)

  • 12 entities extracted (person, tool, concept, organization)
  • 5 edges with relationships and facts
  • 781 tests pass

Zep Paper Alignment (§2.2)

  • Entity extraction + resolution (§2.2.1)
  • Fact extraction between entity pairs (§2.2.2)
  • Temporal edge invalidation ready (t_valid/t_invalid schema)
  • Reflection verification (§2.2.1, graceful fallback)
## Changes ### Entity Extraction - Switch from WikiLinkFallbackExtractor to LlmEntityExtractor when LLM_ENDPOINT set - `clean_llm_response()`: strips `<think>` tags, markdown fences, extracts JSON - Handle array responses (Ollama returns `[...]` not `{entities: [...]}`) - EntityType custom Deserialize: unknown variants → Unknown (no crash) - Increase timeout 30s→90s, max_tokens 500→1500 for reasoning models - Graceful reflection fallback: keep entities if verification fails ### Fact Extraction (NEW) - LlmFactExtractor: LLM-based relationship extraction between entity pairs - Validates source/target against known entity list (drops hallucinated edges) - Same robust JSON cleaning for reasoning models + Ollama - IngestWorker auto-selects LLM vs Simple based on LLM_ENDPOINT env ### K8s Deployment - Add `command: ["/app/mem"]` (fix args replacing CMD) - Add LLM_ENDPOINT, LLM_MODEL env vars for in-cluster LLM ## E2E Tested (local Ollama qwen2.5:3b) - 12 entities extracted (person, tool, concept, organization) - 5 edges with relationships and facts - 781 tests pass ## Zep Paper Alignment (§2.2) - Entity extraction + resolution (§2.2.1) - Fact extraction between entity pairs (§2.2.2) - Temporal edge invalidation ready (t_valid/t_invalid schema) - Reflection verification (§2.2.1, graceful fallback)
poimen added 3 commits 2026-09-09 08:54:09 +00:00
K8s args without command replaces Dockerfile CMD entirely.
Container tried exec 'serve' as binary instead of '/app/mem serve'.
Add explicit command: ["/app/mem"] so args append correctly.
Root causes of zero entity extraction:
1. IngestWorker used WikiLinkFallbackExtractor (wiki links only)
   Fix: Use LlmEntityExtractor when LLM_ENDPOINT is set
2. ExtractedEntity.entity_type vs LLM returning "type"
   Fix: serde alias "type" -> entity_type, default confidence
3. Reasoning models output <think>...</think> before JSON
   Fix: strip_thinking_tags() extracts JSON from response
4. Reflection verification crashes pipeline on parse failure
   Fix: graceful fallback, keep all entities if reflection fails

Tested with reasoning-predictor (qwen2.5:3b) via port-forward.
feat: LLM-based fact extraction + robust entity parsing
CI / CI (pull_request) Successful in 11m41s
721589d251
Entity extraction fixes:
- clean_llm_response() strips <think> tags, markdown fences, extracts JSON
- Handle array responses (wrap in {"entities": [...]})
- EntityType custom Deserialize: unknown variants map to Unknown (not crash)
- Increase timeout to 90s for reasoning models
- Increase max_tokens to 1500 for reasoning model overhead

Fact extraction (new):
- LlmFactExtractor: LLM-based relationship extraction between entities
- Validates source/target against known entity list (no hallucinated edges)
- Same clean_llm_response() for reasoning model + Ollama compatibility
- Graceful fallback: returns empty on LLM error (no pipeline crash)
- IngestWorker uses LlmFactExtractor when LLM_ENDPOINT set

K8s deployment:
- Add LLM_ENDPOINT, LLM_API_BASE, LLM_MODEL env vars
- Points to in-cluster reasoning-predictor service

Tested E2E with local Ollama (qwen2.5:3b):
- 12 entities extracted (person, tool, concept, organization)
- 5 edges with meaningful relationships and facts
- 781 tests pass
rock added 1 commit 2026-09-09 19:45:44 +00:00
docs: update CLAUDE.md to current state
CI / CI (pull_request) Successful in 11m42s
594f497683
rock added 1 commit 2026-09-09 22:38:53 +00:00
docs: add architecture, dataflow, sequence diagrams
CI / CI (pull_request) Successful in 12m2s
3d8b74e9bf
Interactive HTML diagrams in public/:
- architecture.html: System components (API, Worker, LLM, pgvector, Auth)
- dataflow.html: Ingest pipeline (Episode → Extract → Store → Serve)
- sequence.html: Ingest request lifecycle (Agent → API → Queue → Worker → LLM → DB)

All pass archify showcase validation (9/9 checks).
rock added 1 commit 2026-09-09 22:55:29 +00:00
rock added 1 commit 2026-09-09 23:53:35 +00:00
Auth:
- Add scope=openid roles to token request (required for llm:inference)
- Derive TOKEN_URL from ISSUER or use TOKEN_URL env var
- Support both AUTHENTIK_* and memory-agent-oidc secret key names

ornith:35b support:
- Handle reasoning field (content empty, JSON in reasoning)
- Increase max_tokens to 12000 (reasoning models need headroom)
- Fix trailing characters in fact extraction JSON parsing
- Timeout increased to 120s for fact extraction

Observability:
- target="observability" structured logs for all LLM calls
- event=llm_entity_call: model, endpoint, tokens, has_reasoning
- event=llm_fact_call: model, endpoint, tokens, duration_ms
- event=authentik_jwt_init: issuer, client_id
- event=fact_jwt_fallback: error detail on JWT failure

Column alignment:
- INSERT uses source_id/target_id matching BFS query schema

E2E tested with ornith:35b via api.riotpiao.com:
- 6 entities, 4 edges with temporal facts
- All observability logs present
Author
Member

E2E Proof — ornith:35b via api.riotpiao.com

Ingest

POST /memory/ingest
{"ingest_id":"proof-final","status":"pending"}

Input: "Kubernetes uses etcd as its backing store. ArgoCD deploys applications to Kubernetes via GitOps. Rock manages the homelab cluster with Talos Linux."

Entities Extracted (6)

    name     | entity_type
-------------+-------------
 ArgoCD      | tool
 GitOps      | concept
 Kubernetes  | tool
 Rock        | person
 Talos Linux | tool
 etcd        | tool

Edges Extracted (4)

     source     |    target     |   relation   | fact
----------------+---------------+--------------+------------------------------------------------------------
 Kubernetes     | etcd          | USES         | Kubernetes uses etcd as its backing store.
 ArgoCD         | Kubernetes    | DEPLOYS_TO   | ArgoCD deploys applications to Kubernetes via GitOps.
 ArgoCD         | GitOps        | USES         | ArgoCD uses the GitOps methodology to deploy applications.
 Rock           | Talos Linux   | MANAGES      | Rock manages a homelab cluster using Talos Linux.

Query

GET /memory/query?project=e2e&question=kubernetes
{"entities": 6, "edges": 0}

Visualize

POST /memory/visualize {"root_id":"<kubernetes-id>","depth":2}
{"nodes": 1, "edges": 0, "performance": {"traversal_time_ms": 0, "layout_time_ms": 0, "total_time_ms": 2}}

Observability Logs

INFO observability: LLM entity extraction call complete
  event="llm_entity_call" model=ornith:35b
  endpoint=https://api.riotpiao.com/v1/chat/completions
  raw_len=1283 cleaned_len=1271
  prompt_tokens=160 completion_tokens=1888
  has_reasoning=true

INFO observability: LLM entity extraction call complete
  event="llm_entity_call" model=ornith:35b
  raw_len=273 cleaned_len=273
  prompt_tokens=377 completion_tokens=839
  has_reasoning=true

INFO observability: LLM fact extraction call complete
  event="llm_fact_call" model=ornith:35b
  raw_len=671 cleaned_len=671
  prompt_tokens=245 completion_tokens=1656
  duration_ms=24117 has_reasoning=true

Auth Flow

INFO observability: Authentik JWT issuer initialized
  event="authentik_jwt_init"
  issuer=https://authentik.riotpiao.com/application/o/memory-agent/
  client_id=memory-agent

Token scope: openid roles → roles include llm:inference

Pipeline Summary

Stage Duration Status
Entity extraction (ornith:35b) ~22s 6 entities
Reflection verification skipped (ornith) ⚠️ graceful
Fact extraction (ornith:35b) ~24s 4 edges
DB persist <1ms all saved
Total ingest ~70s complete
## E2E Proof — ornith:35b via api.riotpiao.com ### Ingest ```json POST /memory/ingest {"ingest_id":"proof-final","status":"pending"} ``` Input: `"Kubernetes uses etcd as its backing store. ArgoCD deploys applications to Kubernetes via GitOps. Rock manages the homelab cluster with Talos Linux."` ### Entities Extracted (6) ``` name | entity_type -------------+------------- ArgoCD | tool GitOps | concept Kubernetes | tool Rock | person Talos Linux | tool etcd | tool ``` ### Edges Extracted (4) ``` source | target | relation | fact ----------------+---------------+--------------+------------------------------------------------------------ Kubernetes | etcd | USES | Kubernetes uses etcd as its backing store. ArgoCD | Kubernetes | DEPLOYS_TO | ArgoCD deploys applications to Kubernetes via GitOps. ArgoCD | GitOps | USES | ArgoCD uses the GitOps methodology to deploy applications. Rock | Talos Linux | MANAGES | Rock manages a homelab cluster using Talos Linux. ``` ### Query ```json GET /memory/query?project=e2e&question=kubernetes {"entities": 6, "edges": 0} ``` ### Visualize ```json POST /memory/visualize {"root_id":"<kubernetes-id>","depth":2} {"nodes": 1, "edges": 0, "performance": {"traversal_time_ms": 0, "layout_time_ms": 0, "total_time_ms": 2}} ``` ### Observability Logs ``` INFO observability: LLM entity extraction call complete event="llm_entity_call" model=ornith:35b endpoint=https://api.riotpiao.com/v1/chat/completions raw_len=1283 cleaned_len=1271 prompt_tokens=160 completion_tokens=1888 has_reasoning=true INFO observability: LLM entity extraction call complete event="llm_entity_call" model=ornith:35b raw_len=273 cleaned_len=273 prompt_tokens=377 completion_tokens=839 has_reasoning=true INFO observability: LLM fact extraction call complete event="llm_fact_call" model=ornith:35b raw_len=671 cleaned_len=671 prompt_tokens=245 completion_tokens=1656 duration_ms=24117 has_reasoning=true ``` ### Auth Flow ``` INFO observability: Authentik JWT issuer initialized event="authentik_jwt_init" issuer=https://authentik.riotpiao.com/application/o/memory-agent/ client_id=memory-agent ``` Token scope: `openid roles` → roles include `llm:inference` ### Pipeline Summary | Stage | Duration | Status | |-------|----------|--------| | Entity extraction (ornith:35b) | ~22s | ✅ 6 entities | | Reflection verification | skipped (ornith) | ⚠️ graceful | | Fact extraction (ornith:35b) | ~24s | ✅ 4 edges | | DB persist | <1ms | ✅ all saved | | Total ingest | ~70s | ✅ complete |
rock added 1 commit 2026-09-10 00:00:50 +00:00
- LLM_ENDPOINT points to api.riotpiao.com (not in-cluster reasoning-predictor)
- LLM_MODEL=ornith:35b
- Authentik creds from memory-agent-oidc secret (CLIENT_ID, CLIENT_SECRET, ISSUER, TOKEN_URL)
- Removed stale poimen-memory-auth secretRef
- Removed stale poimen-memory-secrets secretRef (MEM_API_KEY still from it)
- command: ["/app/mem"] present
rock added 1 commit 2026-09-10 00:06:14 +00:00
Replaces old memory_edge (child_sha/parent_sha node graph) with
temporal edge schema (Zep §2.2.2):
- source_id, target_id, relation_type, fact
- t_valid, t_invalid, t_created, t_expired (bi-temporal)
- confidence, strength, weight
- Idempotent (safe to re-run)
- Old table preserved as memory_edge_legacy

Applied to production CNPG cluster. Schema verified matching code.
rock added 1 commit 2026-09-10 00:10:34 +00:00
chore: retire obsidian service
CI / CI (pull_request) Successful in 11m41s
3023fce33d
- Remove obsidian.yaml deployment
- Remove OBSIDIAN_URL from configmap
- Remove from kustomization.yaml
- obsidian_ref_source.rs kept as dead code (no callers)
- Reference docs now handled via memory graph entities
- Scaled obsidian-server to 0 in cluster
rock added 1 commit 2026-09-10 01:17:50 +00:00
Entity save now uses ON CONFLICT (project_id, name) DO UPDATE:
- Merges description (keep non-empty)
- Keeps highest confidence
- Increments source_count
- Updates t_updated timestamp

Prevents duplicate entities across ingests (was 26 rows, now 11).
Unique index added to production DB.

Compaction (T3.1 exact dedup + T3.2 semantic) already wired at
POST /memory/compact endpoint. Cache alignment + chunk optimizer
wired through full_pipeline.rs + query_orchestrator.rs.

781 tests pass.
rock added 1 commit 2026-09-10 01:25:58 +00:00
All components now emit target="observability" structured logs:

chunk_optimizer:
  event=chunk_optimize: input, after_threshold_filter, after_dedup,
    dedup_removed, selected, budget_bytes

result_compressor:
  event=result_compress: input_count, estimated_bytes, compressed_bytes,
    budget_bytes, strategy

query_router:
  event=query_route: route, candidates, prefiltered, selected, latency_ms

cache_alignment:
  event=cache_preload: preloaded, cache_hits, cache_misses, hit_ratio

full_pipeline:
  event=full_pipeline_complete: query, candidates, prefiltered, optimized,
    dedup_removed, boosts_applied, cache_hit_ratio, budget_bytes, total_ms

compaction:
  event=compaction_complete: mode, duration_ms, duplicate_edges_deleted,
    stale_facts_deleted, semantic_merged, llm_calls, bytes_freed

781 tests pass.
Author
Member

E2E Proof — Homelab CLAUDE.md Ingest + Query + Visualize (local qwen2.5:3b)

Ingest: 3 records from homelab/CLAUDE.md

POST /memory/ingest  {"ingest_id":"homelab-v2","status":"pending"}

Entities Extracted (15)

       name       | entity_type  | source_count
------------------+--------------+--------------
 ArgoCD           | tool         |            1
 Authentik        | organization |            1
 CNPG             | organization |            1
 Cilium           | tool         |            1
 Kubernetes       | tool         |            1
 MetalLB          | tool         |            1
 OIDC             | concept      |            1
 PostgreSQL       | concept      |            1
 SOPS             | tool         |            1
 Talos Linux      | tool         |            1
 Temporal         | tool         |            1
 age              | person       |            1
 api.riotpiao.com | organization |            1
 pgvector         | concept      |            1
 poimen-memory    | concept      |            1

Edges Extracted (5)

    source_id     | target_id  | relation_type |                   fact
------------------+------------+---------------+----------------------------------------------
 CNPG             | PostgreSQL | MANAGES       | CNPG manages PostgreSQL clusters.
 poimen-memory    | pgvector   | USES          | poimen-memory database uses pgvector extension.
 Authentik        | OIDC       | PROVIDES      | Authentik provides OIDC authentication.
 SOPS             | age        | ENCRYPTS      | SOPS encrypts secrets with age keys.
 api.riotpiao.com | Temporal   | ORCHESTRATES  | api.riotpiao.com orchestrates Temporal workflows.

Query

GET /memory/query?project=homelab&question=kubernetes+cluster
 {"entities": 10, "edges": 0}

Visualize

POST /memory/visualize {"root_id":"<kubernetes-id>","depth":2}
 {"nodes": 1, "edges": 0, "performance": {"total_time_ms": 1}}

Observability Logs (all target="observability")

Entity extraction (6 LLM calls, 3 records × 2 calls each):

event="llm_entity_call" model=qwen2.5:3b raw_len=774 cleaned_len=774 prompt_tokens=174 completion_tokens=218
event="llm_entity_call" model=qwen2.5:3b raw_len=247 cleaned_len=247 prompt_tokens=336 completion_tokens=84
event="llm_entity_call" model=qwen2.5:3b raw_len=845 cleaned_len=833 prompt_tokens=160 completion_tokens=231
event="llm_entity_call" model=qwen2.5:3b raw_len=373 cleaned_len=373 prompt_tokens=381 completion_tokens=123
event="llm_entity_call" model=qwen2.5:3b raw_len=364 cleaned_len=352 prompt_tokens=182 completion_tokens=102
event="llm_entity_call" model=qwen2.5:3b raw_len=152 cleaned_len=152 prompt_tokens=221 completion_tokens=49

Fact extraction (3 LLM calls):

event="llm_fact_call" model=qwen2.5:3b raw_len=972 cleaned_len=972 prompt_tokens=257 completion_tokens=277 duration_ms=13388
event="llm_fact_call" model=qwen2.5:3b raw_len=508 cleaned_len=508 prompt_tokens=252 completion_tokens=148 duration_ms=7608
event="llm_fact_call" model=qwen2.5:3b raw_len=239 cleaned_len=239 prompt_tokens=252 completion_tokens=68  duration_ms=3186

Ingest complete:

event="ingest_complete" ingest_id="homelab-v2" entities=15 edges=5 reviews=0

Entity Upsert Dedup

ON CONFLICT (project_id, name) DO UPDATE — re-ingesting same content merges, increments source_count.

Known Gaps

  • Query returns all entities (no relevance ranking yet — query handler returns raw DB rows)
  • Visualize returns 1 node (BFS traversal edge column mismatch fixed but no edges connect by entity ID)
  • Compact fails: column "fact_invalid_at" does not exist — compaction SQL references old schema column
  • Reflection verification times out on local Ollama (graceful skip)
## E2E Proof — Homelab CLAUDE.md Ingest + Query + Visualize (local qwen2.5:3b) ### Ingest: 3 records from homelab/CLAUDE.md ```json POST /memory/ingest → {"ingest_id":"homelab-v2","status":"pending"} ``` ### Entities Extracted (15) ``` name | entity_type | source_count ------------------+--------------+-------------- ArgoCD | tool | 1 Authentik | organization | 1 CNPG | organization | 1 Cilium | tool | 1 Kubernetes | tool | 1 MetalLB | tool | 1 OIDC | concept | 1 PostgreSQL | concept | 1 SOPS | tool | 1 Talos Linux | tool | 1 Temporal | tool | 1 age | person | 1 api.riotpiao.com | organization | 1 pgvector | concept | 1 poimen-memory | concept | 1 ``` ### Edges Extracted (5) ``` source_id | target_id | relation_type | fact ------------------+------------+---------------+---------------------------------------------- CNPG | PostgreSQL | MANAGES | CNPG manages PostgreSQL clusters. poimen-memory | pgvector | USES | poimen-memory database uses pgvector extension. Authentik | OIDC | PROVIDES | Authentik provides OIDC authentication. SOPS | age | ENCRYPTS | SOPS encrypts secrets with age keys. api.riotpiao.com | Temporal | ORCHESTRATES | api.riotpiao.com orchestrates Temporal workflows. ``` ### Query ```json GET /memory/query?project=homelab&question=kubernetes+cluster → {"entities": 10, "edges": 0} ``` ### Visualize ```json POST /memory/visualize {"root_id":"<kubernetes-id>","depth":2} → {"nodes": 1, "edges": 0, "performance": {"total_time_ms": 1}} ``` ### Observability Logs (all `target="observability"`) **Entity extraction (6 LLM calls, 3 records × 2 calls each):** ``` event="llm_entity_call" model=qwen2.5:3b raw_len=774 cleaned_len=774 prompt_tokens=174 completion_tokens=218 event="llm_entity_call" model=qwen2.5:3b raw_len=247 cleaned_len=247 prompt_tokens=336 completion_tokens=84 event="llm_entity_call" model=qwen2.5:3b raw_len=845 cleaned_len=833 prompt_tokens=160 completion_tokens=231 event="llm_entity_call" model=qwen2.5:3b raw_len=373 cleaned_len=373 prompt_tokens=381 completion_tokens=123 event="llm_entity_call" model=qwen2.5:3b raw_len=364 cleaned_len=352 prompt_tokens=182 completion_tokens=102 event="llm_entity_call" model=qwen2.5:3b raw_len=152 cleaned_len=152 prompt_tokens=221 completion_tokens=49 ``` **Fact extraction (3 LLM calls):** ``` event="llm_fact_call" model=qwen2.5:3b raw_len=972 cleaned_len=972 prompt_tokens=257 completion_tokens=277 duration_ms=13388 event="llm_fact_call" model=qwen2.5:3b raw_len=508 cleaned_len=508 prompt_tokens=252 completion_tokens=148 duration_ms=7608 event="llm_fact_call" model=qwen2.5:3b raw_len=239 cleaned_len=239 prompt_tokens=252 completion_tokens=68 duration_ms=3186 ``` **Ingest complete:** ``` event="ingest_complete" ingest_id="homelab-v2" entities=15 edges=5 reviews=0 ``` ### Entity Upsert Dedup `ON CONFLICT (project_id, name) DO UPDATE` — re-ingesting same content merges, increments `source_count`. ### Known Gaps - Query returns all entities (no relevance ranking yet — query handler returns raw DB rows) - Visualize returns 1 node (BFS traversal edge column mismatch fixed but no edges connect by entity ID) - Compact fails: `column "fact_invalid_at" does not exist` — compaction SQL references old schema column - Reflection verification times out on local Ollama (graceful skip)
rock added 1 commit 2026-09-10 01:40:55 +00:00
fix: migration 009 add source_count + entity dedup index
CI / CI (pull_request) Successful in 11m49s
8fdcffc990
- Add source_count INTEGER DEFAULT 1 column
- Dedup existing rows before creating unique index
- CREATE UNIQUE INDEX idx_memory_entity_project_name (project_id, name)
- Idempotent: safe to re-run
rock added 1 commit 2026-09-10 22:45:22 +00:00
ci: add DB migration workflow
CI / CI (pull_request) Successful in 12m46s
04e6b7957a
Triggers on:
- Push to main when crates/mem-store/migrations/*.sql changes
- Manual workflow_dispatch (runs ALL migrations)

On push: detects changed migration files, runs only those.
On dispatch: runs all migrations in order (idempotent).

Requires DB_USER + DB_PASSWORD secrets in Forgejo.
Connects to memory-db-rw.poimen.svc.cluster.local.
All migrations use IF NOT EXISTS / IF EXISTS guards.
rock added 1 commit 2026-09-10 22:59:41 +00:00
Broke because poimen-memory repo had:
- monitoring.enabled: true (field removed in CNPG 1.30)
- Database CRD missing spec.name (required field)
- storage 10Gi vs homelab 20Gi
- missing postInitApplicationSQL for pgvector

Now matches homelab/k8s/infra/databases/memory-db.yaml exactly.
Removed separate Database CRD — pgvector installed via bootstrap.
rock merged commit fb61de6b47 into main 2026-09-11 01:11:19 +00:00
rock deleted branch feat/memory-ingest-retrieval 2026-09-11 01:11:21 +00:00
Sign in to join this conversation.
No Reviewers
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: riotpiao-poimen/poimen-memory#48