feat: LLM entity + fact extraction pipeline (Zep paper alignment) (#48)

## Changes ### Entity Extraction - Switch from WikiLinkFallbackExtractor to LlmEntityExtractor when LLM_ENDPOINT set - `clean_llm_response()`: strips `<think>` tags, markdown fences, extracts JSON - Handle array responses (Ollama returns `[...]` not `{entities: [...]}`) - EntityType custom Deserialize: unknown variants → Unknown (no crash) - Increase timeout 30s→90s, max_tokens 500→1500 for reasoning models - Graceful reflection fallback: keep entities if verification fails ### Fact Extraction (NEW) - LlmFactExtractor: LLM-based relationship extraction between entity pairs - Validates source/target against known entity list (drops hallucinated edges) - Same robust JSON cleaning for reasoning models + Ollama - IngestWorker auto-selects LLM vs Simple based on LLM_ENDPOINT env ### K8s Deployment - Add `command: ["/app/mem"]` (fix args replacing CMD) - Add LLM_ENDPOINT, LLM_MODEL env vars for in-cluster LLM ## E2E Tested (local Ollama qwen2.5:3b) - 12 entities extracted (person, tool, concept, organization) - 5 edges with relationships and facts - 781 tests pass ## Zep Paper Alignment (§2.2) - Entity extraction + resolution (§2.2.1) - Fact extraction between entity pairs (§2.2.2) - Temporal edge invalidation ready (t_valid/t_invalid schema) - Reflection verification (§2.2.1, graceful fallback) --------- Co-authored-by: rock <[email protected]> Reviewed-on: #48 Co-authored-by: poimen <[email protected]>
2026-09-11 01:11:15 +00:00
parent 6b18d81421
commit fb61de6b47
19 changed files with 859 additions and 268 deletions
@@ -0,0 +1,76 @@
+name: DB Migration
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'crates/mem-store/migrations/**'
+  workflow_dispatch:
+
+env:
+  DB_HOST: memory-db-rw.poimen.svc.cluster.local
+  DB_PORT: "5432"
+  DB_NAME: memory
+
+jobs:
+  migrate:
+    name: Run Migrations
+    runs-on: rust
+    steps:
+      - name: Install psql
+        run: apt-get update && apt-get install -y postgresql-client
+
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Fetch previous migrations state
+        run: |
+          git fetch origin main --depth=2
+          # List changed migration files
+          CHANGED=$(git diff --name-only HEAD~1 HEAD -- crates/mem-store/migrations/ || echo "")
+          echo "Changed migrations: $CHANGED"
+          echo "CHANGED_MIGRATIONS=$CHANGED" >> $GITHUB_ENV
+
+      - name: Run migrations
+        if: env.CHANGED_MIGRATIONS != ''
+        run: |
+          export PGPASSWORD="${DB_PASSWORD}"
+
+          echo "=== Running changed migrations ==="
+          for f in $CHANGED_MIGRATIONS; do
+            if [ -f "$f" ]; then
+              echo "--- Applying: $f ---"
+              psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -f "$f" 2>&1
+              if [ $? -ne 0 ]; then
+                echo "ERROR: Migration $f failed!"
+                exit 1
+              fi
+              echo "--- OK: $f ---"
+            fi
+          done
+
+          echo "=== Verify schema ==="
+          psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c "\dt memory*"
+        env:
+          DB_USER: ${{ secrets.DB_USER }}
+          DB_PASSWORD: ${{ secrets.DB_PASSWORD }}
+
+      - name: Run all migrations (manual trigger)
+        if: github.event_name == 'workflow_dispatch'
+        run: |
+          export PGPASSWORD="${DB_PASSWORD}"
+
+          echo "=== Running all migrations in order ==="
+          for f in $(ls crates/mem-store/migrations/*.sql | sort); do
+            echo "--- Applying: $f ---"
+            psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -f "$f" 2>&1 || true
+            echo "--- Done: $f ---"
+          done
+
+          echo "=== Final schema ==="
+          psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c "\dt memory*"
+          psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c "\d memory_entity"
+          psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c "\d memory_edge"
+        env:
+          DB_USER: ${{ secrets.DB_USER }}
+          DB_PASSWORD: ${{ secrets.DB_PASSWORD }}