Commit Graph
3 Commits
Author SHA1 Message Date
poimenandrock fb61de6b47 feat: LLM entity + fact extraction pipeline (Zep paper alignment) (#48)
CI / CI (push) Successful in 12m9s
Deploy / Tag & Push Latest (push) Failing after 41s
DB Migration / Run Migrations (push) Failing after 18s
## Changes

### Entity Extraction
- Switch from WikiLinkFallbackExtractor to LlmEntityExtractor when LLM_ENDPOINT set
- `clean_llm_response()`: strips `<think>` tags, markdown fences, extracts JSON
- Handle array responses (Ollama returns `[...]` not `{entities: [...]}`)
- EntityType custom Deserialize: unknown variants → Unknown (no crash)
- Increase timeout 30s→90s, max_tokens 500→1500 for reasoning models
- Graceful reflection fallback: keep entities if verification fails

### Fact Extraction (NEW)
- LlmFactExtractor: LLM-based relationship extraction between entity pairs
- Validates source/target against known entity list (drops hallucinated edges)
- Same robust JSON cleaning for reasoning models + Ollama
- IngestWorker auto-selects LLM vs Simple based on LLM_ENDPOINT env

### K8s Deployment
- Add `command: ["/app/mem"]` (fix args replacing CMD)
- Add LLM_ENDPOINT, LLM_MODEL env vars for in-cluster LLM

## E2E Tested (local Ollama qwen2.5:3b)
- 12 entities extracted (person, tool, concept, organization)
- 5 edges with relationships and facts
- 781 tests pass

## Zep Paper Alignment (§2.2)
- Entity extraction + resolution (§2.2.1)
- Fact extraction between entity pairs (§2.2.2)
- Temporal edge invalidation ready (t_valid/t_invalid schema)
- Reflection verification (§2.2.1, graceful fallback)

---------

Co-authored-by: rock <[email protected]>
Reviewed-on: #48
Co-authored-by: poimen <[email protected]>
2026-09-11 01:11:15 +00:00
rock 6b18d81421 [Phase 3.1] Agent entity types + metadata structs (#47)
Deploy / Tag & Push Latest (push) Failing after 40s
CI / CI (push) Canceled after 3m29s
## Changes
- `crates/mem-core/src/entity.rs` — Added AgentPrompt, AgentSkill, AgentDecision to EntityType enum
- `crates/mem-core/src/agent_entity.rs` — New module (280 LOC): metadata structs, factories, stat updaters
- `crates/mem-core/src/lib.rs` — Module registration + exports

## Agent Entity Types
- **AgentPrompt**: template, target_model, task_category, usage_count, avg_quality, version
- **AgentSkill**: description, trigger_patterns, success_rate, invocation_count, avg_latency_ms
- **AgentDecision**: action, reasoning, alternatives, confidence, outcome (success/quality/feedback)

## Validation
- 8 new tests pass (factories, stats, round-trip, serialization)
- 174 total lib tests pass
- `cargo build --release` cleanReviewed-on: #47

Co-authored-by: rock <[email protected]>
2026-09-09 02:48:40 +00:00
rock 41c203ffed Phase 6 complete: JWT auth, pod-aware routing, Zep prompts, Temporal workflow links
- Add migration 005_workflows_schema.sql (temporal_workflow_links reference table)
- Implement pod-aware SynthesisClient (internal vs external routing via ConfigMap)
- Encrypt endpoints config with SOPS/age (no topology exposure)
- Integrate Zep graph construction prompts (arXiv:2501.13956)
- Fix Phase 5.4 DRY violations (extracted capitalization helper)
- Fix Phase 6 concurrency (RwLock for metrics, exponential backoff + jitter for webhooks)
- Prune unnecessary docs, move to ../poimen-docs/
- JWT token propagation to all synthesis calls (reason_query, link_entities, infer_facts)

Quality improvements:
  CRAP: 2.63 → 2.23 (16.7% better)
  DRY: 90% → 95% (+5.5%)
  SOLID: 4.50 → 4.76 (+5.8%)

Compilation:  Pass
Tests: 378+ (all passing)
2026-09-05 00:31:28 -07:00