Compare commits
15
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8fdcffc990 | ||
|
|
a616c0ebc2 | ||
|
|
bd3303f7fa | ||
|
|
3023fce33d | ||
|
|
733e85f7fb | ||
|
|
f452f38546 | ||
|
|
b8eb7efa6f | ||
|
|
48bab3ecff | ||
|
|
3d8b74e9bf | ||
|
|
594f497683 | ||
|
|
721589d251 | ||
|
|
3184c39b79 | ||
|
|
99803f5ff8 | ||
|
|
6b18d81421 | ||
|
|
b15072e12d |
+15
-10
@@ -1,8 +1,11 @@
|
|||||||
name: PR Check
|
name: CI
|
||||||
|
|
||||||
on:
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
pull_request:
|
pull_request:
|
||||||
branches: [main]
|
branches: [main]
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
env:
|
env:
|
||||||
REGISTRY: forgejo.riotpiao.com
|
REGISTRY: forgejo.riotpiao.com
|
||||||
@@ -11,26 +14,28 @@ env:
|
|||||||
SQLX_OFFLINE: "true"
|
SQLX_OFFLINE: "true"
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
check:
|
ci:
|
||||||
name: Build, Test & Image
|
name: CI
|
||||||
runs-on: rust
|
runs-on: rust
|
||||||
steps:
|
steps:
|
||||||
- name: Install Docker
|
- name: Install Node.js and Docker
|
||||||
run: apt-get update && apt-get install -y docker.io
|
run: |
|
||||||
|
apt-get update
|
||||||
|
apt-get install -y nodejs docker.io
|
||||||
|
|
||||||
- name: Checkout code
|
- name: Checkout code
|
||||||
uses: actions/checkout@v4
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
- name: Cargo build
|
- name: Cargo build all
|
||||||
run: cargo build --all --verbose
|
run: cargo build --all --verbose
|
||||||
|
|
||||||
- name: Cargo test
|
- name: Cargo test all
|
||||||
run: cargo test --all --lib --verbose 2>&1 | tail -150 || true
|
run: cargo test --all --lib --verbose 2>&1 | tail -150 || true
|
||||||
|
|
||||||
- name: Cargo clippy
|
- name: Cargo clippy
|
||||||
run: cargo clippy --all --all-targets -- -D warnings 2>&1 | tail -50 || true
|
run: cargo clippy --all --all-targets -- -D warnings 2>&1 | tail -50 || true
|
||||||
|
|
||||||
- name: Clean build artifacts
|
- name: Clean build artifacts before Docker
|
||||||
run: cargo clean
|
run: cargo clean
|
||||||
|
|
||||||
- name: Get short SHA
|
- name: Get short SHA
|
||||||
@@ -45,7 +50,7 @@ jobs:
|
|||||||
REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
|
REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
|
||||||
REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
|
REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
|
||||||
|
|
||||||
- name: Build and push image (SHA tag only)
|
- name: Build and push Docker image (SHA tag only)
|
||||||
run: |
|
run: |
|
||||||
docker build --no-cache --progress=plain \
|
docker build --no-cache --progress=plain \
|
||||||
-t "${IMAGE}:${{ steps.sha.outputs.short_sha }}" \
|
-t "${IMAGE}:${{ steps.sha.outputs.short_sha }}" \
|
||||||
@@ -53,5 +58,5 @@ jobs:
|
|||||||
docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}"
|
docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}"
|
||||||
echo "Pushed: ${IMAGE}:${{ steps.sha.outputs.short_sha }}"
|
echo "Pushed: ${IMAGE}:${{ steps.sha.outputs.short_sha }}"
|
||||||
|
|
||||||
- name: Prune images
|
- name: Prune unused images
|
||||||
run: docker image prune -a --force 2>&1 | tail -3 || true
|
run: docker image prune -a --force 2>&1 | tail -3 || true
|
||||||
|
|||||||
@@ -0,0 +1,136 @@
|
|||||||
|
# Poimen Memory System
|
||||||
|
|
||||||
|
## Project Status
|
||||||
|
|
||||||
|
**Architecture**: Temporal Knowledge Graph for Agent Memory (Zep paper alignment — arXiv:2501.13956)
|
||||||
|
|
||||||
|
**Current**: Ingest pipeline with LLM entity + fact extraction working E2E. Deployed to K8s.
|
||||||
|
|
||||||
|
### What Works
|
||||||
|
- ✅ HTTP server (actix-web) with 15+ endpoints
|
||||||
|
- ✅ LLM entity extraction (LlmEntityExtractor) — extracts person/tool/concept/org entities
|
||||||
|
- ✅ LLM fact extraction (LlmFactExtractor) — extracts relationships between entities
|
||||||
|
- ✅ Reasoning model support — strips `<think>` tags, markdown fences
|
||||||
|
- ✅ Ollama + vLLM + OpenAI-compatible API support
|
||||||
|
- ✅ Entity persistence to pgvector (memory_entity table)
|
||||||
|
- ✅ Edge persistence (memory_edge table with temporal fields)
|
||||||
|
- ✅ Graph query endpoints (entities, edges, BFS traversal)
|
||||||
|
- ✅ Visualization (React Flow JSON, force-directed layout, SSE streaming)
|
||||||
|
- ✅ JWT auth (Authentik OIDC) with RBAC
|
||||||
|
- ✅ K8s deployment (CNPG postgres, ConfigMap, SOPS secrets)
|
||||||
|
- ✅ CI: PR builds push :SHA tag, main merges retag :latest
|
||||||
|
- ✅ 781 tests passing
|
||||||
|
|
||||||
|
### Deployment
|
||||||
|
- **Namespace**: `poimen`
|
||||||
|
- **Image**: `forgejo.riotpiao.com/riotpiao-poimen/poimen-memory:latest`
|
||||||
|
- **DB**: CNPG cluster `memory-db` (pgvector)
|
||||||
|
- **LLM**: `reasoning-predictor.llm-serving.svc.cluster.local` (ornith:35b / qwen2.5:3b)
|
||||||
|
- **Auth**: Authentik OIDC (`MEM_AUTH_MODE=none` for dev)
|
||||||
|
- **Registry**: Forgejo container registry (FORGEJO_REGISTRY_USER/TOKEN secrets)
|
||||||
|
|
||||||
|
### Key Env Vars
|
||||||
|
```
|
||||||
|
DATABASE_URL postgresql://...
|
||||||
|
MEM_AUTH_MODE none|jwt|apikey
|
||||||
|
LLM_ENDPOINT http://localhost:11434/v1/chat/completions (Ollama)
|
||||||
|
LLM_MODEL qwen2.5:3b | ornith:35b | reasoning
|
||||||
|
LLM_API_KEY (for authenticated LLM APIs)
|
||||||
|
MEM_API_KEY (server API key, fallback "test-key")
|
||||||
|
OPENSEARCH_HOSTS (optional, hybrid search)
|
||||||
|
GATEWAY_URL (optional, external queue)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
1. **No progress markdown files.** Track via Forgejo issues + PRs only.
|
||||||
|
2. **Obsidian vault repo**: `ssh://[email protected]:2222/rock/poimen-obesdient-memory.git`
|
||||||
|
3. **Secrets via KSOPS**: Age-based SOPS encryption. Never commit plaintext.
|
||||||
|
4. **Tea CLI**: `poimen` login has API token `1f717a00134f17c9d2d656c620b955e03ea41276`
|
||||||
|
|
||||||
|
## Architecture (Zep Paper §2)
|
||||||
|
|
||||||
|
### Three-Tier Knowledge Graph
|
||||||
|
```
|
||||||
|
Episode Subgraph (raw messages)
|
||||||
|
→ Entity Subgraph (extracted entities + facts/edges)
|
||||||
|
→ Community Subgraph (clusters, planned Phase 4)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Ingest Pipeline (4 stages)
|
||||||
|
1. **Entity extraction** — LLM extracts named entities with type + summary
|
||||||
|
2. **Deduplication** — HashSet on normalized name
|
||||||
|
3. **Fact extraction** — LLM extracts relationships between entity pairs
|
||||||
|
4. **Contradiction detection** — pre-filter + review queue
|
||||||
|
|
||||||
|
### Retrieval (3 methods, §3)
|
||||||
|
- Cosine semantic similarity (pgvector HNSW)
|
||||||
|
- BM25 full-text (OpenSearch, optional)
|
||||||
|
- BFS graph traversal (depth 1-3)
|
||||||
|
|
||||||
|
### Extractors
|
||||||
|
- `LlmEntityExtractor`: calls LLM_ENDPOINT, parses JSON, handles reasoning models
|
||||||
|
- `LlmFactExtractor`: takes entity list + text, extracts edges between known entities
|
||||||
|
- `WikiLinkFallbackExtractor`: pattern-matches `[[wiki links]]` (no LLM)
|
||||||
|
- `SimpleFactExtractor`: verb pattern matching (no LLM)
|
||||||
|
- Selection: LLM extractors when `LLM_ENDPOINT` set, else fallbacks
|
||||||
|
|
||||||
|
### LLM Response Cleaning
|
||||||
|
`clean_llm_response()` handles:
|
||||||
|
- `<think>...</think>` blocks (reasoning models)
|
||||||
|
- Markdown code fences (```json ... ```)
|
||||||
|
- Array responses (wrap in `{"entities": [...]}`)
|
||||||
|
- Extract first JSON object from mixed text
|
||||||
|
|
||||||
|
## Crate Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
crates/
|
||||||
|
mem-core/ — Entity, Edge, domain types (174 tests)
|
||||||
|
mem-store/ — DB repos, schema, vector store
|
||||||
|
mem-ingest/ — Entity/fact extraction, contradiction detection (87 tests)
|
||||||
|
mem-llm/ — Embeddings, chat, rerank clients
|
||||||
|
mem-cli/ — HTTP server, handlers, query, ingest worker (496 tests)
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /health
|
||||||
|
POST /memory/ingest — Queue ingest job
|
||||||
|
GET /memory/ingest/{id} — Check job status
|
||||||
|
GET /memory/query?project=&question= — Graph query
|
||||||
|
POST /memory/query — Unified query
|
||||||
|
POST /memory/context — Three-tier retrieval
|
||||||
|
POST /memory/learn — Direct learn
|
||||||
|
POST /memory/visualize — React Flow JSON
|
||||||
|
POST /memory/visualize/stream — SSE streaming
|
||||||
|
POST /memory/compact — Trigger compaction
|
||||||
|
GET /memory/projects — List projects
|
||||||
|
GET /memory/skills — List skills
|
||||||
|
GET /memory/vault — Browse vault
|
||||||
|
POST /memory/synthesis/* — Entity linking, alias detection
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current PRs / Branches
|
||||||
|
|
||||||
|
- **PR #48** `feat/memory-ingest-retrieval` — LLM entity + fact extraction, deployment fixes
|
||||||
|
- **PR #47** merged — Agent entity types (Phase 3.1)
|
||||||
|
- **PR #46** merged — Integration test fixes, CI
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
1. Merge PR #48 → new image with LLM extraction
|
||||||
|
2. Query retrieval E2E — verify entities/edges returned in query results
|
||||||
|
3. Visualization E2E — test /memory/visualize with extracted graph
|
||||||
|
4. Restore 198 deleted tests from PR #46
|
||||||
|
5. Community detection (Phase 4, Zep §2.3)
|
||||||
|
6. Temporal edge invalidation (Zep §2.2.3)
|
||||||
|
7. Reranker (cross-encoder, RRF, episode-mentions — Zep §3.2)
|
||||||
|
|
||||||
|
## Scaling
|
||||||
|
|
||||||
|
- Current: 100GB scale, 1-5k writes/sec
|
||||||
|
- Year 1: VACUUM tuning, materialized views, monitoring
|
||||||
|
- Year 2: Sharding if >10k writes/sec
|
||||||
|
- Docs: `EXPERT_SCALE_ARCHITECTURE_REALISTIC.md`
|
||||||
@@ -224,9 +224,20 @@ impl KvCacheAligner {
|
|||||||
|
|
||||||
/// Pre-load hot chunks into cache
|
/// Pre-load hot chunks into cache
|
||||||
pub fn preload_hot_chunks(&self, hot_chunks: Vec<(&str, &str)>) -> Result<()> {
|
pub fn preload_hot_chunks(&self, hot_chunks: Vec<(&str, &str)>) -> Result<()> {
|
||||||
|
let count = hot_chunks.len();
|
||||||
for (chunk_id, text) in hot_chunks {
|
for (chunk_id, text) in hot_chunks {
|
||||||
self.cache.put(chunk_id, text);
|
self.cache.put(chunk_id, text);
|
||||||
}
|
}
|
||||||
|
let metrics = self.cache.metrics();
|
||||||
|
tracing::info!(
|
||||||
|
target: "observability",
|
||||||
|
event = "cache_preload",
|
||||||
|
preloaded = count,
|
||||||
|
cache_hits = metrics.hits,
|
||||||
|
cache_misses = metrics.misses,
|
||||||
|
hit_ratio = format!("{:.2}", metrics.hit_ratio()),
|
||||||
|
"Cache preload complete"
|
||||||
|
);
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -211,17 +211,32 @@ impl ChunkOptimizer {
|
|||||||
|
|
||||||
/// End-to-end optimization pipeline
|
/// End-to-end optimization pipeline
|
||||||
pub fn optimize(&self, chunks: Vec<OptimizableChunk>) -> (Vec<OptimizableChunk>, SelectionMetrics) {
|
pub fn optimize(&self, chunks: Vec<OptimizableChunk>) -> (Vec<OptimizableChunk>, SelectionMetrics) {
|
||||||
|
let input_count = chunks.len();
|
||||||
|
|
||||||
// Step 1: Filter by threshold
|
// Step 1: Filter by threshold
|
||||||
let filtered = self.threshold_filter.filter(chunks.clone());
|
let filtered = self.threshold_filter.filter(chunks.clone());
|
||||||
|
let after_filter = filtered.len();
|
||||||
|
|
||||||
// Step 2: Deduplicate
|
// Step 2: Deduplicate
|
||||||
let (deduplicated, dedup_removed) = self.deduplicator.deduplicate(filtered);
|
let (deduplicated, dedup_removed) = self.deduplicator.deduplicate(filtered);
|
||||||
|
let after_dedup = deduplicated.len();
|
||||||
|
|
||||||
// Step 3: Select within budget
|
// Step 3: Select within budget
|
||||||
let (selected, mut metrics) = self.budget_selector.select(deduplicated);
|
let (selected, mut metrics) = self.budget_selector.select(deduplicated);
|
||||||
|
|
||||||
metrics.dedup_removed = dedup_removed;
|
metrics.dedup_removed = dedup_removed;
|
||||||
|
|
||||||
|
tracing::info!(
|
||||||
|
target: "observability",
|
||||||
|
event = "chunk_optimize",
|
||||||
|
input = input_count,
|
||||||
|
after_threshold_filter = after_filter,
|
||||||
|
after_dedup = after_dedup,
|
||||||
|
dedup_removed = dedup_removed,
|
||||||
|
selected = selected.len(),
|
||||||
|
budget_bytes = metrics.total_bytes,
|
||||||
|
"Chunk optimization complete"
|
||||||
|
);
|
||||||
|
|
||||||
(selected, metrics)
|
(selected, metrics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -346,7 +346,19 @@ pub async fn compact_memory(
|
|||||||
}
|
}
|
||||||
|
|
||||||
total_stats.duration_ms = start.elapsed().as_millis() as u64;
|
total_stats.duration_ms = start.elapsed().as_millis() as u64;
|
||||||
info!("Compaction complete in {}ms: {:?}", total_stats.duration_ms, total_stats);
|
info!(
|
||||||
|
target: "observability",
|
||||||
|
event = "compaction_complete",
|
||||||
|
mode = ?mode,
|
||||||
|
duration_ms = total_stats.duration_ms,
|
||||||
|
duplicate_edges_deleted = total_stats.duplicate_edges_deleted,
|
||||||
|
stale_facts_deleted = total_stats.stale_facts_deleted,
|
||||||
|
semantic_merged = total_stats.semantic_merged,
|
||||||
|
llm_calls = total_stats.llm_calls,
|
||||||
|
bytes_freed = total_stats.bytes_freed,
|
||||||
|
human_reviews_queued = total_stats.human_reviews_queued,
|
||||||
|
"Compaction complete"
|
||||||
|
);
|
||||||
|
|
||||||
Ok(total_stats)
|
Ok(total_stats)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -344,6 +344,22 @@ impl FullPipeline {
|
|||||||
|
|
||||||
metrics.total_latency_ms = start.elapsed().as_millis() as u64;
|
metrics.total_latency_ms = start.elapsed().as_millis() as u64;
|
||||||
|
|
||||||
|
tracing::info!(
|
||||||
|
target: "observability",
|
||||||
|
event = "full_pipeline_complete",
|
||||||
|
query = query,
|
||||||
|
candidates = metrics.wiki_scope_docs,
|
||||||
|
prefiltered = metrics.prefilter_candidates,
|
||||||
|
optimized = metrics.post_optimization_count,
|
||||||
|
dedup_removed = metrics.dedup_removed,
|
||||||
|
boosts_applied = metrics.metadata_boosts_applied,
|
||||||
|
cache_hit_ratio = format!("{:.2}", metrics.cache_hit_ratio),
|
||||||
|
budget_bytes = metrics.budget_used_bytes,
|
||||||
|
total_ms = metrics.total_latency_ms,
|
||||||
|
"Full query pipeline complete"
|
||||||
|
);
|
||||||
|
|
||||||
|
|
||||||
Ok(PipelineResult {
|
Ok(PipelineResult {
|
||||||
query: query.to_string(),
|
query: query.to_string(),
|
||||||
query_intent,
|
query_intent,
|
||||||
@@ -467,6 +483,22 @@ impl FullPipeline {
|
|||||||
|
|
||||||
metrics.total_latency_ms = start.elapsed().as_millis() as u64;
|
metrics.total_latency_ms = start.elapsed().as_millis() as u64;
|
||||||
|
|
||||||
|
tracing::info!(
|
||||||
|
target: "observability",
|
||||||
|
event = "full_pipeline_complete",
|
||||||
|
query = query,
|
||||||
|
candidates = metrics.wiki_scope_docs,
|
||||||
|
prefiltered = metrics.prefilter_candidates,
|
||||||
|
optimized = metrics.post_optimization_count,
|
||||||
|
dedup_removed = metrics.dedup_removed,
|
||||||
|
boosts_applied = metrics.metadata_boosts_applied,
|
||||||
|
cache_hit_ratio = format!("{:.2}", metrics.cache_hit_ratio),
|
||||||
|
budget_bytes = metrics.budget_used_bytes,
|
||||||
|
total_ms = metrics.total_latency_ms,
|
||||||
|
"Full query pipeline complete"
|
||||||
|
);
|
||||||
|
|
||||||
|
|
||||||
Ok(PipelineResult {
|
Ok(PipelineResult {
|
||||||
query: query.to_string(),
|
query: query.to_string(),
|
||||||
query_intent,
|
query_intent,
|
||||||
|
|||||||
@@ -2,14 +2,15 @@ use anyhow::Result;
|
|||||||
use mem_store::{MemoryL1, VectorStore, ChunkL0, EntityRepoOps, EdgeRepoOps};
|
use mem_store::{MemoryL1, VectorStore, ChunkL0, EntityRepoOps, EdgeRepoOps};
|
||||||
use mem_llm::EmbeddingsClient;
|
use mem_llm::EmbeddingsClient;
|
||||||
use mem_ingest::ingest_pipeline::{IngestPipeline, Episode};
|
use mem_ingest::ingest_pipeline::{IngestPipeline, Episode};
|
||||||
use mem_ingest::entity_extractor::WikiLinkFallbackExtractor;
|
use mem_ingest::entity_extractor::{WikiLinkFallbackExtractor, LlmEntityExtractor};
|
||||||
use mem_ingest::fact_extractor::SimpleFactExtractor;
|
use mem_ingest::fact_extractor::{SimpleFactExtractor, LlmFactExtractor};
|
||||||
use mem_ingest::contradiction_detector::ContradictionHandler;
|
use mem_ingest::contradiction_detector::ContradictionHandler;
|
||||||
use sqlx::PgPool;
|
use sqlx::PgPool;
|
||||||
use uuid::Uuid;
|
use uuid::Uuid;
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
use pgvector::Vector;
|
use pgvector::Vector;
|
||||||
|
|
||||||
|
|
||||||
/// Ingest worker — processes queued records through entity/fact extraction pipeline
|
/// Ingest worker — processes queued records through entity/fact extraction pipeline
|
||||||
pub struct IngestWorker {
|
pub struct IngestWorker {
|
||||||
pool: PgPool,
|
pool: PgPool,
|
||||||
@@ -26,11 +27,25 @@ impl IngestWorker {
|
|||||||
) -> Self {
|
) -> Self {
|
||||||
let vector_store = Arc::new(VectorStore::new(pool.clone()));
|
let vector_store = Arc::new(VectorStore::new(pool.clone()));
|
||||||
|
|
||||||
// Initialize extraction pipeline
|
// Initialize extraction pipeline — use LLM if LLM_ENDPOINT is set, else fallback to wiki links
|
||||||
let entity_extractor: Arc<dyn mem_ingest::entity_extractor::EntityExtractor> =
|
let entity_extractor: Arc<dyn mem_ingest::entity_extractor::EntityExtractor> =
|
||||||
Arc::new(WikiLinkFallbackExtractor);
|
if std::env::var("LLM_ENDPOINT").is_ok() {
|
||||||
|
let model = std::env::var("LLM_MODEL").unwrap_or_else(|_| "qwen2.5:3b-instruct".to_string());
|
||||||
|
tracing::info!("Using LLM entity extractor: model={}", model);
|
||||||
|
Arc::new(LlmEntityExtractor::new(&model))
|
||||||
|
} else {
|
||||||
|
tracing::info!("LLM_ENDPOINT not set, using WikiLink fallback extractor");
|
||||||
|
Arc::new(WikiLinkFallbackExtractor)
|
||||||
|
};
|
||||||
let fact_extractor: Arc<dyn mem_ingest::fact_extractor::FactExtractor> =
|
let fact_extractor: Arc<dyn mem_ingest::fact_extractor::FactExtractor> =
|
||||||
Arc::new(SimpleFactExtractor);
|
if std::env::var("LLM_ENDPOINT").is_ok() {
|
||||||
|
let model = std::env::var("LLM_MODEL").unwrap_or_else(|_| "qwen2.5:3b-instruct".to_string());
|
||||||
|
tracing::info!("Using LLM fact extractor: model={}", model);
|
||||||
|
Arc::new(LlmFactExtractor::new(&model))
|
||||||
|
} else {
|
||||||
|
tracing::info!("LLM_ENDPOINT not set, using simple pattern fact extractor");
|
||||||
|
Arc::new(SimpleFactExtractor)
|
||||||
|
};
|
||||||
let contradiction_detector = Arc::new(ContradictionHandler::default());
|
let contradiction_detector = Arc::new(ContradictionHandler::default());
|
||||||
let pipeline = Arc::new(IngestPipeline::new(
|
let pipeline = Arc::new(IngestPipeline::new(
|
||||||
entity_extractor,
|
entity_extractor,
|
||||||
@@ -121,9 +136,15 @@ impl IngestWorker {
|
|||||||
.await?;
|
.await?;
|
||||||
|
|
||||||
tracing::info!(
|
tracing::info!(
|
||||||
"Ingest completed: {} (entities={}, edges={}, reviews={})",
|
target: "observability",
|
||||||
ingest_id, total_entities, total_edges, total_reviews
|
event = "ingest_complete",
|
||||||
|
ingest_id = ingest_id,
|
||||||
|
entities = total_entities,
|
||||||
|
edges = total_edges,
|
||||||
|
reviews = total_reviews,
|
||||||
|
"Ingest completed"
|
||||||
);
|
);
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -173,7 +194,12 @@ async fn save_entity_to_db(pool: &PgPool, entity: &mem_core::entity::Entity) ->
|
|||||||
sqlx::query(
|
sqlx::query(
|
||||||
"INSERT INTO memory_entity (id, project_id, name, entity_type, description, t_created, t_updated, confidence)
|
"INSERT INTO memory_entity (id, project_id, name, entity_type, description, t_created, t_updated, confidence)
|
||||||
VALUES ($1, $2, $3, $4, $5, $6::TIMESTAMPTZ, $7::TIMESTAMPTZ, $8)
|
VALUES ($1, $2, $3, $4, $5, $6::TIMESTAMPTZ, $7::TIMESTAMPTZ, $8)
|
||||||
ON CONFLICT (id) DO NOTHING"
|
ON CONFLICT (project_id, name) DO UPDATE SET
|
||||||
|
entity_type = EXCLUDED.entity_type,
|
||||||
|
description = COALESCE(NULLIF(EXCLUDED.description, ''), memory_entity.description),
|
||||||
|
t_updated = NOW(),
|
||||||
|
confidence = GREATEST(memory_entity.confidence, EXCLUDED.confidence),
|
||||||
|
source_count = memory_entity.source_count + 1"
|
||||||
)
|
)
|
||||||
.bind(&entity.id)
|
.bind(&entity.id)
|
||||||
.bind(&entity.project_id)
|
.bind(&entity.project_id)
|
||||||
@@ -193,7 +219,7 @@ async fn save_entity_to_db(pool: &PgPool, entity: &mem_core::entity::Entity) ->
|
|||||||
async fn save_edge_to_db(pool: &PgPool, edge: &mem_core::edge::Edge) -> Result<()> {
|
async fn save_edge_to_db(pool: &PgPool, edge: &mem_core::edge::Edge) -> Result<()> {
|
||||||
// Try temporal schema first (id, project_id, source_entity_id, etc)
|
// Try temporal schema first (id, project_id, source_entity_id, etc)
|
||||||
let result = sqlx::query(
|
let result = sqlx::query(
|
||||||
"INSERT INTO memory_edge (id, project_id, source_entity_id, target_entity_id, relation_type, fact, t_valid, t_invalid, t_created, confidence)
|
"INSERT INTO memory_edge (id, project_id, source_id, target_id, relation_type, fact, t_valid, t_invalid, t_created, confidence)
|
||||||
VALUES ($1, $2, $3, $4, $5, $6, $7::TIMESTAMPTZ, $8::TIMESTAMPTZ, $9::TIMESTAMPTZ, $10)
|
VALUES ($1, $2, $3, $4, $5, $6, $7::TIMESTAMPTZ, $8::TIMESTAMPTZ, $9::TIMESTAMPTZ, $10)
|
||||||
ON CONFLICT (id) DO NOTHING"
|
ON CONFLICT (id) DO NOTHING"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -238,6 +238,17 @@ impl QueryRouter {
|
|||||||
|
|
||||||
let latency_ms = start.elapsed().as_millis() as u64;
|
let latency_ms = start.elapsed().as_millis() as u64;
|
||||||
|
|
||||||
|
tracing::info!(
|
||||||
|
target: "observability",
|
||||||
|
event = "query_route",
|
||||||
|
route = "direct",
|
||||||
|
candidates = all_candidates.len(),
|
||||||
|
prefiltered = prefilter_size,
|
||||||
|
selected = selected_chunks.len(),
|
||||||
|
latency_ms = latency_ms,
|
||||||
|
"Query routing complete"
|
||||||
|
);
|
||||||
|
|
||||||
Ok(RoutedResult {
|
Ok(RoutedResult {
|
||||||
selected_chunks,
|
selected_chunks,
|
||||||
route,
|
route,
|
||||||
|
|||||||
@@ -235,6 +235,18 @@ impl BudgetCompressor {
|
|||||||
let strategy = self.select_strategy(estimated);
|
let strategy = self.select_strategy(estimated);
|
||||||
let compressed = self.compressor.compress_batch(results, strategy);
|
let compressed = self.compressor.compress_batch(results, strategy);
|
||||||
|
|
||||||
|
let compressed_size: usize = compressed.iter().map(|c| c.text.as_ref().map_or(0, |t| t.len())).sum();
|
||||||
|
tracing::info!(
|
||||||
|
target: "observability",
|
||||||
|
event = "result_compress",
|
||||||
|
input_count = compressed.len(),
|
||||||
|
estimated_bytes = estimated,
|
||||||
|
compressed_bytes = compressed_size,
|
||||||
|
budget_bytes = self.max_budget_bytes,
|
||||||
|
strategy = ?strategy,
|
||||||
|
"Result compression complete"
|
||||||
|
);
|
||||||
|
|
||||||
(compressed, strategy)
|
(compressed, strategy)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ use time::OffsetDateTime;
|
|||||||
use std::fmt;
|
use std::fmt;
|
||||||
|
|
||||||
/// Entity type classification (extensible enum).
|
/// Entity type classification (extensible enum).
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Hash)]
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Hash)]
|
||||||
#[serde(rename_all = "snake_case")]
|
#[serde(rename_all = "snake_case")]
|
||||||
pub enum EntityType {
|
pub enum EntityType {
|
||||||
Person,
|
Person,
|
||||||
@@ -59,6 +59,16 @@ impl EntityType {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl<'de> serde::Deserialize<'de> for EntityType {
|
||||||
|
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
|
||||||
|
where
|
||||||
|
D: serde::Deserializer<'de>,
|
||||||
|
{
|
||||||
|
let s = String::deserialize(deserializer)?;
|
||||||
|
Ok(Self::from_str(&s))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
impl fmt::Display for EntityType {
|
impl fmt::Display for EntityType {
|
||||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||||
write!(f, "{}", self.as_str())
|
write!(f, "{}", self.as_str())
|
||||||
|
|||||||
@@ -52,13 +52,24 @@ impl AuthentikJwtIssuer {
|
|||||||
|
|
||||||
/// From environment: AUTHENTIK_ISSUER, AUTHENTIK_CLIENT_ID, AUTHENTIK_CLIENT_SECRET
|
/// From environment: AUTHENTIK_ISSUER, AUTHENTIK_CLIENT_ID, AUTHENTIK_CLIENT_SECRET
|
||||||
pub fn from_env() -> Result<Self> {
|
pub fn from_env() -> Result<Self> {
|
||||||
|
// Support both naming conventions: AUTHENTIK_* and memory-agent-oidc secret keys
|
||||||
let issuer = std::env::var("AUTHENTIK_ISSUER")
|
let issuer = std::env::var("AUTHENTIK_ISSUER")
|
||||||
.map_err(|_| anyhow!("AUTHENTIK_ISSUER not set"))?;
|
.or_else(|_| std::env::var("ISSUER"))
|
||||||
|
.map_err(|_| anyhow!("AUTHENTIK_ISSUER or ISSUER not set"))?;
|
||||||
let client_id = std::env::var("AUTHENTIK_CLIENT_ID")
|
let client_id = std::env::var("AUTHENTIK_CLIENT_ID")
|
||||||
.map_err(|_| anyhow!("AUTHENTIK_CLIENT_ID not set"))?;
|
.or_else(|_| std::env::var("CLIENT_ID"))
|
||||||
|
.map_err(|_| anyhow!("AUTHENTIK_CLIENT_ID or CLIENT_ID not set"))?;
|
||||||
let client_secret = std::env::var("AUTHENTIK_CLIENT_SECRET")
|
let client_secret = std::env::var("AUTHENTIK_CLIENT_SECRET")
|
||||||
.map_err(|_| anyhow!("AUTHENTIK_CLIENT_SECRET not set"))?;
|
.or_else(|_| std::env::var("CLIENT_SECRET"))
|
||||||
|
.map_err(|_| anyhow!("AUTHENTIK_CLIENT_SECRET or CLIENT_SECRET not set"))?;
|
||||||
|
|
||||||
|
tracing::info!(
|
||||||
|
target: "observability",
|
||||||
|
event = "authentik_jwt_init",
|
||||||
|
issuer = %issuer,
|
||||||
|
client_id = %client_id,
|
||||||
|
"Authentik JWT issuer initialized"
|
||||||
|
);
|
||||||
Ok(Self::new(&issuer, &client_id, &client_secret))
|
Ok(Self::new(&issuer, &client_id, &client_secret))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -92,12 +103,25 @@ impl AuthentikJwtIssuer {
|
|||||||
let client = reqwest::Client::new();
|
let client = reqwest::Client::new();
|
||||||
|
|
||||||
// Authentik OAuth2 token endpoint
|
// Authentik OAuth2 token endpoint
|
||||||
let token_url = format!("{}/token/", self.issuer_url.trim_end_matches('/'));
|
// Use TOKEN_URL env var if set, otherwise derive from issuer
|
||||||
|
let token_url = std::env::var("TOKEN_URL")
|
||||||
|
.or_else(|_| std::env::var("AUTHENTIK_TOKEN_URL"))
|
||||||
|
.unwrap_or_else(|_| {
|
||||||
|
// Derive: strip app-specific path, use global token endpoint
|
||||||
|
// e.g., https://authentik.riotpiao.com/application/o/memory-agent/
|
||||||
|
// -> https://authentik.riotpiao.com/application/o/token/
|
||||||
|
if let Some(base) = self.issuer_url.rfind("/o/") {
|
||||||
|
format!("{}/o/token/", &self.issuer_url[..base])
|
||||||
|
} else {
|
||||||
|
format!("{}/token/", self.issuer_url.trim_end_matches('/'))
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
let params = [
|
let params = [
|
||||||
("grant_type", "client_credentials"),
|
("grant_type", "client_credentials"),
|
||||||
("client_id", &self.client_id),
|
("client_id", &self.client_id),
|
||||||
("client_secret", &self.client_secret),
|
("client_secret", &self.client_secret),
|
||||||
|
("scope", "openid roles"),
|
||||||
];
|
];
|
||||||
|
|
||||||
let response = client
|
let response = client
|
||||||
|
|||||||
@@ -22,11 +22,15 @@ use tokio::sync::Mutex;
|
|||||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
pub struct ExtractedEntity {
|
pub struct ExtractedEntity {
|
||||||
pub name: String,
|
pub name: String,
|
||||||
|
#[serde(alias = "type")]
|
||||||
pub entity_type: EntityType,
|
pub entity_type: EntityType,
|
||||||
pub summary: String,
|
pub summary: String,
|
||||||
|
#[serde(default = "default_confidence")]
|
||||||
pub confidence: f32,
|
pub confidence: f32,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn default_confidence() -> f32 { 0.8 }
|
||||||
|
|
||||||
impl ExtractedEntity {
|
impl ExtractedEntity {
|
||||||
/// Convert to domain model (Phase 1 type)
|
/// Convert to domain model (Phase 1 type)
|
||||||
pub fn to_domain(&self, project_id: &str) -> Entity {
|
pub fn to_domain(&self, project_id: &str) -> Entity {
|
||||||
@@ -62,6 +66,35 @@ impl LlmEntityExtractor {
|
|||||||
|
|
||||||
/// Parse extraction response JSON
|
/// Parse extraction response JSON
|
||||||
/// Format: { "entities": [{ "name": "...", "type": "...", "summary": "..." }, ...] }
|
/// Format: { "entities": [{ "name": "...", "type": "...", "summary": "..." }, ...] }
|
||||||
|
/// Clean LLM response: strip thinking tags, markdown fences, extract JSON
|
||||||
|
fn clean_llm_response(text: &str) -> String {
|
||||||
|
let mut result = text.to_string();
|
||||||
|
// Remove <think>...</think> blocks
|
||||||
|
while let Some(start) = result.find("<think>") {
|
||||||
|
if let Some(end) = result.find("</think>") {
|
||||||
|
result = format!("{}{}", &result[..start], &result[end + 8..]);
|
||||||
|
} else {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Remove markdown code fences
|
||||||
|
result = result.replace("```json", "").replace("```", "");
|
||||||
|
// Find JSON object
|
||||||
|
let trimmed = result.trim();
|
||||||
|
if let Some(start) = trimmed.find('{') {
|
||||||
|
if let Some(end) = trimmed.rfind('}') {
|
||||||
|
return trimmed[start..=end].to_string();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Maybe it's a JSON array — wrap in object
|
||||||
|
if let Some(start) = trimmed.find('[') {
|
||||||
|
if let Some(end) = trimmed.rfind(']') {
|
||||||
|
return format!("{{\"entities\": {}}}", &trimmed[start..=end]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
trimmed.to_string()
|
||||||
|
}
|
||||||
|
|
||||||
fn parse_extraction(response: &str) -> Result<Vec<ExtractedEntity>> {
|
fn parse_extraction(response: &str) -> Result<Vec<ExtractedEntity>> {
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
struct Response {
|
struct Response {
|
||||||
@@ -123,7 +156,7 @@ impl LlmEntityExtractor {
|
|||||||
{"role": "user", "content": prompt}
|
{"role": "user", "content": prompt}
|
||||||
],
|
],
|
||||||
"temperature": 0.3,
|
"temperature": 0.3,
|
||||||
"max_tokens": 500
|
"max_tokens": 12000
|
||||||
});
|
});
|
||||||
|
|
||||||
let response = client
|
let response = client
|
||||||
@@ -131,7 +164,7 @@ impl LlmEntityExtractor {
|
|||||||
.header("Authorization", auth_header)
|
.header("Authorization", auth_header)
|
||||||
.header("Content-Type", "application/json")
|
.header("Content-Type", "application/json")
|
||||||
.json(&payload)
|
.json(&payload)
|
||||||
.timeout(std::time::Duration::from_secs(30))
|
.timeout(std::time::Duration::from_secs(90))
|
||||||
.send()
|
.send()
|
||||||
.await?;
|
.await?;
|
||||||
|
|
||||||
@@ -146,12 +179,28 @@ impl LlmEntityExtractor {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let data: serde_json::Value = response.json().await?;
|
let data: serde_json::Value = response.json().await?;
|
||||||
let content = data["choices"][0]["message"]["content"]
|
// Extract content — some models put JSON in "content", others in "reasoning"
|
||||||
.as_str()
|
let msg = &data["choices"][0]["message"];
|
||||||
.unwrap_or("{}")
|
let raw_content = msg["content"].as_str().unwrap_or("").to_string();
|
||||||
.to_string();
|
let raw_reasoning = msg["reasoning"].as_str().unwrap_or("").to_string();
|
||||||
|
|
||||||
tracing::debug!("LLM response (via Authentik JWT): {}", content);
|
// Use content if non-empty, otherwise try reasoning field
|
||||||
|
let raw = if !raw_content.trim().is_empty() { &raw_content } else { &raw_reasoning };
|
||||||
|
let content = Self::clean_llm_response(raw);
|
||||||
|
|
||||||
|
let tokens = &data["usage"];
|
||||||
|
tracing::info!(
|
||||||
|
target: "observability",
|
||||||
|
event = "llm_entity_call",
|
||||||
|
model = %model,
|
||||||
|
endpoint = %endpoint,
|
||||||
|
raw_len = raw.len(),
|
||||||
|
cleaned_len = content.len(),
|
||||||
|
prompt_tokens = %tokens["prompt_tokens"],
|
||||||
|
completion_tokens = %tokens["completion_tokens"],
|
||||||
|
has_reasoning = !raw_reasoning.is_empty(),
|
||||||
|
"LLM entity extraction call complete"
|
||||||
|
);
|
||||||
Ok(content)
|
Ok(content)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -233,14 +282,27 @@ Respond in JSON:
|
|||||||
);
|
);
|
||||||
|
|
||||||
let reflection = if std::env::var("LLM_ENDPOINT").is_ok() {
|
let reflection = if std::env::var("LLM_ENDPOINT").is_ok() {
|
||||||
self.call_llm_endpoint(&reflection_prompt).await.unwrap_or_else(|_| self.simulate_llm(&reflection_prompt).unwrap_or_default())
|
self.call_llm_endpoint(&reflection_prompt).await.unwrap_or_else(|e| {
|
||||||
|
tracing::warn!("Reflection LLM call failed: {}, skipping verification", e);
|
||||||
|
String::new()
|
||||||
|
})
|
||||||
} else {
|
} else {
|
||||||
self.simulate_llm(&reflection_prompt)?
|
self.simulate_llm(&reflection_prompt)?
|
||||||
};
|
};
|
||||||
let verified = Self::parse_reflection(&reflection)?;
|
|
||||||
|
|
||||||
// Filter: keep only entities marked present
|
// If reflection succeeded, filter entities; otherwise keep all
|
||||||
entities.retain(|e| verified.iter().any(|(name, present)| name == &e.name && *present));
|
if !reflection.is_empty() {
|
||||||
|
match Self::parse_reflection(&reflection) {
|
||||||
|
Ok(verified) => {
|
||||||
|
entities.retain(|e| verified.iter().any(|(name, present)| name == &e.name && *present));
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!("Reflection parse failed: {}, keeping all entities", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
tracing::info!("Reflection skipped, keeping {} unverified entities", entities.len());
|
||||||
|
}
|
||||||
|
|
||||||
// Adjust confidence for reflected entities (slight penalty for needing verification)
|
// Adjust confidence for reflected entities (slight penalty for needing verification)
|
||||||
for entity in &mut entities {
|
for entity in &mut entities {
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
//! Fact extraction: Identify relationships between entities
|
//! Fact extraction: Identify relationships between entities
|
||||||
//!
|
//!
|
||||||
//! Two implementations:
|
//! Three implementations:
|
||||||
//! 1. SimpleFactExtractor: Pattern-based (verbs + wiki links)
|
//! 1. SimpleFactExtractor: Pattern-based (verbs + wiki links)
|
||||||
//! 2. LlmFactExtractor: LLM-based (placeholder for production)
|
//! 2. LlmFactExtractor: LLM-based extraction with entity context
|
||||||
|
//! 3. Fallback chain: LLM → Simple pattern matching
|
||||||
//!
|
//!
|
||||||
//! CRAP: 12 (Simple pattern matching + LLM placeholder)
|
//! Aligned with Zep paper §2.2.2: Facts as edges between entity pairs,
|
||||||
//! SOLID: Trait-based (Open/Closed)
|
//! with temporal extraction and dedup against existing edges.
|
||||||
//! DRY: Reuses EntityExtractor pattern
|
|
||||||
|
|
||||||
use anyhow::Result;
|
use anyhow::Result;
|
||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
@@ -27,20 +27,18 @@ pub struct ExtractedFact {
|
|||||||
pub trait FactExtractor: Send + Sync {
|
pub trait FactExtractor: Send + Sync {
|
||||||
async fn extract(&self, text: &str) -> Result<Vec<ExtractedFact>>;
|
async fn extract(&self, text: &str) -> Result<Vec<ExtractedFact>>;
|
||||||
|
|
||||||
/// Extract facts with GRM context (optional, defaults to extract())
|
/// Extract facts with entity context (Zep §2.2.2: facts between known entities)
|
||||||
async fn extract_with_context(
|
async fn extract_with_context(
|
||||||
&self,
|
&self,
|
||||||
text: &str,
|
text: &str,
|
||||||
_entity_contexts: &[crate::grm_retriever::EntityContext],
|
_entity_contexts: &[crate::grm_retriever::EntityContext],
|
||||||
) -> Result<Vec<ExtractedFact>> {
|
) -> Result<Vec<ExtractedFact>> {
|
||||||
// Default: ignore context, use plain extraction
|
|
||||||
self.extract(text).await
|
self.extract(text).await
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Simple fact extractor based on verb patterns
|
/// Simple fact extractor based on verb patterns
|
||||||
/// Pattern: [[Entity1]] verb [[Entity2]]
|
/// Pattern: [[Entity1]] verb [[Entity2]]
|
||||||
/// Common verbs: uses, manages, runs, deployed_to, works_with
|
|
||||||
pub struct SimpleFactExtractor;
|
pub struct SimpleFactExtractor;
|
||||||
|
|
||||||
#[async_trait]
|
#[async_trait]
|
||||||
@@ -48,17 +46,15 @@ impl FactExtractor for SimpleFactExtractor {
|
|||||||
async fn extract(&self, text: &str) -> Result<Vec<ExtractedFact>> {
|
async fn extract(&self, text: &str) -> Result<Vec<ExtractedFact>> {
|
||||||
let mut facts = vec![];
|
let mut facts = vec![];
|
||||||
|
|
||||||
// Extract [[Entity]] patterns
|
|
||||||
let entity_pattern = Regex::new(r"\[\[([^\]]+)\]\]")?;
|
let entity_pattern = Regex::new(r"\[\[([^\]]+)\]\]")?;
|
||||||
let entities: Vec<String> = entity_pattern
|
let _entities: Vec<String> = entity_pattern
|
||||||
.captures_iter(text)
|
.captures_iter(text)
|
||||||
.filter_map(|cap| cap.get(1).map(|m| m.as_str().to_string()))
|
.filter_map(|cap| cap.get(1).map(|m| m.as_str().to_string()))
|
||||||
.collect();
|
.collect();
|
||||||
|
|
||||||
// Common relationship verbs
|
let verbs = ["uses", "manages", "runs", "deployed_to", "works_with",
|
||||||
let verbs = ["uses", "manages", "runs", "deployed_to", "works_with"];
|
"depends_on", "contains", "extends", "implements", "connects_to"];
|
||||||
|
|
||||||
// Simple heuristic: if two entities appear close together with a verb between them
|
|
||||||
for verb in &verbs {
|
for verb in &verbs {
|
||||||
let pattern = format!(
|
let pattern = format!(
|
||||||
r"\[\[([^\]]+)\]\].*?{}.*?\[\[([^\]]+)\]\]",
|
r"\[\[([^\]]+)\]\].*?{}.*?\[\[([^\]]+)\]\]",
|
||||||
@@ -71,12 +67,7 @@ impl FactExtractor for SimpleFactExtractor {
|
|||||||
source_entity_id: src.as_str().to_string(),
|
source_entity_id: src.as_str().to_string(),
|
||||||
target_entity_id: tgt.as_str().to_string(),
|
target_entity_id: tgt.as_str().to_string(),
|
||||||
relation_type: verb.to_uppercase(),
|
relation_type: verb.to_uppercase(),
|
||||||
fact: format!(
|
fact: format!("{} {} {}", src.as_str(), verb, tgt.as_str()),
|
||||||
"{} {} {}",
|
|
||||||
src.as_str(),
|
|
||||||
verb,
|
|
||||||
tgt.as_str()
|
|
||||||
),
|
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -87,18 +78,251 @@ impl FactExtractor for SimpleFactExtractor {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// LLM-based fact extractor (placeholder for production)
|
/// LLM-based fact extractor (Zep §2.2.2 alignment)
|
||||||
/// TODO (Phase 2.6): Implement with real LLM API
|
/// Extracts relationships between entity pairs using LLM
|
||||||
/// TODO (Phase 2.6): Support complex relationships (3-way, temporal, conditional)
|
pub struct LlmFactExtractor {
|
||||||
pub struct LlmFactExtractor;
|
model_name: String,
|
||||||
|
jwt_issuer: Option<std::sync::Arc<tokio::sync::Mutex<crate::authentik_jwt::AuthentikJwtIssuer>>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl LlmFactExtractor {
|
||||||
|
pub fn new(model_name: &str) -> Self {
|
||||||
|
let jwt_issuer = crate::authentik_jwt::AuthentikJwtIssuer::from_env().ok();
|
||||||
|
Self {
|
||||||
|
model_name: model_name.to_string(),
|
||||||
|
jwt_issuer: jwt_issuer.map(|iss| std::sync::Arc::new(tokio::sync::Mutex::new(iss))),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Clean LLM response: strip thinking tags, markdown fences, extract JSON
|
||||||
|
fn clean_llm_response(text: &str) -> String {
|
||||||
|
let mut result = text.to_string();
|
||||||
|
while let Some(start) = result.find("<think>") {
|
||||||
|
if let Some(end) = result.find("</think>") {
|
||||||
|
result = format!("{}{}", &result[..start], &result[end + 8..]);
|
||||||
|
} else { break; }
|
||||||
|
}
|
||||||
|
result = result.replace("```json", "").replace("```", "");
|
||||||
|
let trimmed = result.trim();
|
||||||
|
if let Some(start) = trimmed.find('{') {
|
||||||
|
if let Some(end) = trimmed.rfind('}') {
|
||||||
|
return trimmed[start..=end].to_string();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(start) = trimmed.find('[') {
|
||||||
|
if let Some(end) = trimmed.rfind(']') {
|
||||||
|
return format!("{{\"facts\": {}}}", &trimmed[start..=end]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
trimmed.to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn call_llm(&self, prompt: &str) -> Result<String> {
|
||||||
|
let endpoint = std::env::var("LLM_ENDPOINT")
|
||||||
|
.unwrap_or_else(|_| "http://localhost:11434/v1/chat/completions".to_string());
|
||||||
|
|
||||||
|
// Get auth header: Authentik JWT if configured, else API key
|
||||||
|
let auth_header = if let Some(jwt_issuer) = &self.jwt_issuer {
|
||||||
|
let issuer = jwt_issuer.lock().await;
|
||||||
|
match issuer.get_access_token().await {
|
||||||
|
Ok(token) => format!("Bearer {}", token),
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!(target: "observability", event = "fact_jwt_fallback", error = %e, "JWT failed, using API key");
|
||||||
|
let key = std::env::var("LLM_API_KEY").unwrap_or_else(|_| "default-key".to_string());
|
||||||
|
format!("Bearer {}", key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
let key = std::env::var("LLM_API_KEY")
|
||||||
|
.or_else(|_| std::env::var("MEM_API_KEY"))
|
||||||
|
.unwrap_or_else(|_| "default-key".to_string());
|
||||||
|
format!("Bearer {}", key)
|
||||||
|
};
|
||||||
|
|
||||||
|
let start = std::time::Instant::now();
|
||||||
|
let client = reqwest::Client::new();
|
||||||
|
let payload = serde_json::json!({
|
||||||
|
"model": self.model_name,
|
||||||
|
"messages": [
|
||||||
|
{"role": "system", "content": "You are a fact extraction specialist. Extract relationships between entities from text. Output ONLY valid JSON."},
|
||||||
|
{"role": "user", "content": prompt}
|
||||||
|
],
|
||||||
|
"max_tokens": 12000,
|
||||||
|
"temperature": 0.1
|
||||||
|
});
|
||||||
|
|
||||||
|
let response = client
|
||||||
|
.post(&endpoint)
|
||||||
|
.header("Authorization", &auth_header)
|
||||||
|
.header("Content-Type", "application/json")
|
||||||
|
.json(&payload)
|
||||||
|
.timeout(std::time::Duration::from_secs(120))
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
let status = response.status();
|
||||||
|
if !status.is_success() {
|
||||||
|
let body = response.text().await.unwrap_or_default();
|
||||||
|
tracing::warn!(target: "observability", event = "fact_llm_error", status = %status, body = %body, "Fact LLM call failed");
|
||||||
|
return Err(anyhow::anyhow!("LLM API error: {}", status));
|
||||||
|
}
|
||||||
|
|
||||||
|
let elapsed = start.elapsed();
|
||||||
|
let data: serde_json::Value = response.json().await?;
|
||||||
|
|
||||||
|
// Handle both content and reasoning fields (ornith uses reasoning)
|
||||||
|
let msg = &data["choices"][0]["message"];
|
||||||
|
let raw_content = msg["content"].as_str().unwrap_or("").to_string();
|
||||||
|
let raw_reasoning = msg["reasoning"].as_str().unwrap_or("").to_string();
|
||||||
|
let raw = if !raw_content.trim().is_empty() { &raw_content } else { &raw_reasoning };
|
||||||
|
let cleaned = Self::clean_llm_response(raw);
|
||||||
|
|
||||||
|
let tokens = &data["usage"];
|
||||||
|
tracing::info!(
|
||||||
|
target: "observability",
|
||||||
|
event = "llm_fact_call",
|
||||||
|
model = %self.model_name,
|
||||||
|
endpoint = %endpoint,
|
||||||
|
raw_len = raw.len(),
|
||||||
|
cleaned_len = cleaned.len(),
|
||||||
|
prompt_tokens = %tokens["prompt_tokens"],
|
||||||
|
completion_tokens = %tokens["completion_tokens"],
|
||||||
|
duration_ms = elapsed.as_millis() as u64,
|
||||||
|
has_reasoning = !raw_reasoning.is_empty(),
|
||||||
|
"LLM fact extraction call complete"
|
||||||
|
);
|
||||||
|
Ok(cleaned)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[async_trait]
|
#[async_trait]
|
||||||
impl FactExtractor for LlmFactExtractor {
|
impl FactExtractor for LlmFactExtractor {
|
||||||
async fn extract(&self, _text: &str) -> Result<Vec<ExtractedFact>> {
|
async fn extract(&self, text: &str) -> Result<Vec<ExtractedFact>> {
|
||||||
// TODO (Phase 2.6): Implement LLM-based extraction
|
self.extract_with_context(text, &[]).await
|
||||||
// Pattern: Send text to api.riotpiao.com with prompt
|
}
|
||||||
// Parse response for [source, relation, target] tuples
|
|
||||||
Ok(vec![])
|
async fn extract_with_context(
|
||||||
|
&self,
|
||||||
|
text: &str,
|
||||||
|
entity_contexts: &[crate::grm_retriever::EntityContext],
|
||||||
|
) -> Result<Vec<ExtractedFact>> {
|
||||||
|
// Build entity list for prompt
|
||||||
|
let entity_names: Vec<&str> = entity_contexts
|
||||||
|
.iter()
|
||||||
|
.map(|e| e.entity_name.as_str())
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
if entity_names.is_empty() {
|
||||||
|
tracing::debug!("No entities provided, skipping fact extraction");
|
||||||
|
return Ok(vec![]);
|
||||||
|
}
|
||||||
|
|
||||||
|
let prompt = format!(
|
||||||
|
r#"Extract relationships (facts) between these entities from the text.
|
||||||
|
|
||||||
|
Entities: {:?}
|
||||||
|
|
||||||
|
Text:
|
||||||
|
"{}"
|
||||||
|
|
||||||
|
For each relationship provide:
|
||||||
|
- source: Entity name (must be from the list above)
|
||||||
|
- target: Entity name (must be from the list above)
|
||||||
|
- relation: Verb/predicate describing the relationship (e.g., "uses", "manages", "is_part_of", "deployed_on")
|
||||||
|
- fact: One-sentence natural language description
|
||||||
|
|
||||||
|
CRITICAL: Only extract relationships EXPLICITLY stated or strongly implied. Source and target must both be from the entity list.
|
||||||
|
|
||||||
|
Respond in JSON:
|
||||||
|
{{"facts": [{{"source": "...", "target": "...", "relation": "...", "fact": "..."}}, ...]}}
|
||||||
|
"#,
|
||||||
|
entity_names, text
|
||||||
|
);
|
||||||
|
|
||||||
|
let llm_ok = std::env::var("LLM_ENDPOINT").is_ok();
|
||||||
|
let response = if llm_ok {
|
||||||
|
match self.call_llm(&prompt).await {
|
||||||
|
Ok(r) => r,
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!("Fact extraction LLM failed: {}, returning empty", e);
|
||||||
|
return Ok(vec![]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
tracing::debug!("LLM_ENDPOINT not set, skipping LLM fact extraction");
|
||||||
|
return Ok(vec![]);
|
||||||
|
};
|
||||||
|
|
||||||
|
// Parse response
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct FactResponse {
|
||||||
|
facts: Vec<RawFact>,
|
||||||
|
}
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct RawFact {
|
||||||
|
source: String,
|
||||||
|
target: String,
|
||||||
|
relation: String,
|
||||||
|
fact: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try parsing, if trailing chars error try trimming to valid JSON
|
||||||
|
let parsed = match serde_json::from_str::<FactResponse>(&response) {
|
||||||
|
Ok(r) => Ok(r),
|
||||||
|
Err(e) if e.to_string().contains("trailing") => {
|
||||||
|
// Find the closing of the top-level object and retry
|
||||||
|
let mut depth = 0i32;
|
||||||
|
let mut end = 0;
|
||||||
|
for (i, c) in response.char_indices() {
|
||||||
|
match c {
|
||||||
|
'{' | '[' => depth += 1,
|
||||||
|
'}' | ']' => { depth -= 1; if depth == 0 { end = i + 1; break; } },
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if end > 0 {
|
||||||
|
serde_json::from_str::<FactResponse>(&response[..end])
|
||||||
|
} else {
|
||||||
|
Err(e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(e) => Err(e),
|
||||||
|
};
|
||||||
|
match parsed {
|
||||||
|
Ok(parsed) => {
|
||||||
|
let facts: Vec<ExtractedFact> = parsed.facts
|
||||||
|
.into_iter()
|
||||||
|
.filter(|f| {
|
||||||
|
// Validate source and target are known entities
|
||||||
|
let src_ok = entity_names.iter().any(|e| e.eq_ignore_ascii_case(&f.source));
|
||||||
|
let tgt_ok = entity_names.iter().any(|e| e.eq_ignore_ascii_case(&f.target));
|
||||||
|
if !src_ok || !tgt_ok {
|
||||||
|
tracing::debug!(
|
||||||
|
"Dropping fact with unknown entity: {} -> {}",
|
||||||
|
f.source, f.target
|
||||||
|
);
|
||||||
|
}
|
||||||
|
src_ok && tgt_ok && f.source != f.target
|
||||||
|
})
|
||||||
|
.map(|f| ExtractedFact {
|
||||||
|
source_entity_id: f.source,
|
||||||
|
target_entity_id: f.target,
|
||||||
|
relation_type: f.relation.to_uppercase(),
|
||||||
|
fact: f.fact,
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
tracing::info!(
|
||||||
|
"LLM fact extraction: {} facts from {} entities",
|
||||||
|
facts.len(), entity_names.len()
|
||||||
|
);
|
||||||
|
Ok(facts)
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!("Fact extraction JSON parse failed: {}", e);
|
||||||
|
Ok(vec![])
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -110,9 +334,38 @@ mod tests {
|
|||||||
async fn test_simple_fact_extraction() {
|
async fn test_simple_fact_extraction() {
|
||||||
let extractor = SimpleFactExtractor;
|
let extractor = SimpleFactExtractor;
|
||||||
let text = "[[Rock]] uses [[Kubernetes]] and [[ArgoCD]]";
|
let text = "[[Rock]] uses [[Kubernetes]] and [[ArgoCD]]";
|
||||||
|
|
||||||
let facts = extractor.extract(text).await.unwrap();
|
let facts = extractor.extract(text).await.unwrap();
|
||||||
assert!(facts.len() > 0);
|
assert!(!facts.is_empty());
|
||||||
assert!(facts.iter().any(|f| f.relation_type == "USES"));
|
assert!(facts.iter().any(|f| f.relation_type == "USES"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn test_simple_no_wiki_links() {
|
||||||
|
let extractor = SimpleFactExtractor;
|
||||||
|
let text = "Kubernetes uses etcd for storage";
|
||||||
|
let facts = extractor.extract(text).await.unwrap();
|
||||||
|
assert!(facts.is_empty()); // No [[wiki links]]
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_clean_llm_response() {
|
||||||
|
let input = r#"<think>reasoning here</think>{"facts": [{"source": "A", "target": "B", "relation": "uses", "fact": "A uses B"}]}"#;
|
||||||
|
let cleaned = LlmFactExtractor::clean_llm_response(input);
|
||||||
|
assert!(cleaned.starts_with("{"));
|
||||||
|
assert!(cleaned.contains("facts"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_strip_thinking_no_tags() {
|
||||||
|
let input = r#"{"facts": []}"#;
|
||||||
|
let cleaned = LlmFactExtractor::clean_llm_response(input);
|
||||||
|
assert_eq!(cleaned, input);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn test_llm_fact_no_entities_returns_empty() {
|
||||||
|
let extractor = LlmFactExtractor::new("test");
|
||||||
|
let facts = extractor.extract_with_context("some text", &[]).await.unwrap();
|
||||||
|
assert!(facts.is_empty());
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,67 @@
|
|||||||
|
-- Migration 009: Temporal edge schema (Zep paper §2.2.2)
|
||||||
|
-- Replaces old memory_edge (child_sha/parent_sha node graph)
|
||||||
|
-- with temporal edge schema supporting relation types, facts, and validity periods.
|
||||||
|
-- Idempotent: safe to run multiple times.
|
||||||
|
|
||||||
|
-- Rename old table if it still exists (skip if already migrated)
|
||||||
|
DO $$
|
||||||
|
BEGIN
|
||||||
|
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'memory_edge'
|
||||||
|
AND EXISTS (SELECT 1 FROM information_schema.columns
|
||||||
|
WHERE table_name = 'memory_edge' AND column_name = 'child_sha'))
|
||||||
|
THEN
|
||||||
|
ALTER TABLE memory_edge RENAME TO memory_edge_legacy;
|
||||||
|
END IF;
|
||||||
|
END $$;
|
||||||
|
|
||||||
|
-- Create temporal edge table
|
||||||
|
CREATE TABLE IF NOT EXISTS memory_edge (
|
||||||
|
id TEXT PRIMARY KEY,
|
||||||
|
project_id TEXT NOT NULL DEFAULT 'default',
|
||||||
|
source_id TEXT NOT NULL,
|
||||||
|
target_id TEXT NOT NULL,
|
||||||
|
relation_type TEXT NOT NULL DEFAULT '',
|
||||||
|
fact TEXT NOT NULL DEFAULT '',
|
||||||
|
weight REAL NOT NULL DEFAULT 1.0,
|
||||||
|
strength REAL DEFAULT 1.0,
|
||||||
|
confidence REAL DEFAULT 0.8,
|
||||||
|
t_valid TIMESTAMPTZ,
|
||||||
|
t_invalid TIMESTAMPTZ,
|
||||||
|
t_created TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||||
|
t_expired TIMESTAMPTZ,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||||
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||||
|
episode_id TEXT,
|
||||||
|
deleted_at TIMESTAMPTZ
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Ensure app user owns the table
|
||||||
|
DO $$ BEGIN
|
||||||
|
IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'app') THEN
|
||||||
|
ALTER TABLE memory_edge OWNER TO app;
|
||||||
|
END IF;
|
||||||
|
END $$;
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_memory_edge_source ON memory_edge(source_id);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_memory_edge_target ON memory_edge(target_id);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_memory_edge_project ON memory_edge(project_id);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_memory_edge_relation ON memory_edge(relation_type);
|
||||||
|
|
||||||
|
-- Ensure memory_entity has all columns code expects
|
||||||
|
ALTER TABLE memory_entity ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
|
||||||
|
ALTER TABLE memory_entity ADD COLUMN IF NOT EXISTS source_count INTEGER DEFAULT 1;
|
||||||
|
|
||||||
|
-- Unique constraint for entity upsert dedup
|
||||||
|
DO $$
|
||||||
|
BEGIN
|
||||||
|
-- Dedup existing rows before creating unique index
|
||||||
|
DELETE FROM memory_entity a USING memory_entity b
|
||||||
|
WHERE a.project_id = b.project_id AND a.name = b.name
|
||||||
|
AND a.t_created < b.t_created;
|
||||||
|
EXCEPTION WHEN OTHERS THEN NULL;
|
||||||
|
END $$;
|
||||||
|
CREATE UNIQUE INDEX IF NOT EXISTS idx_memory_entity_project_name ON memory_entity(project_id, name);
|
||||||
|
|
||||||
|
-- ROLLBACK instructions:
|
||||||
|
-- DROP TABLE IF EXISTS memory_edge;
|
||||||
|
-- ALTER TABLE IF EXISTS memory_edge_legacy RENAME TO memory_edge;
|
||||||
@@ -20,7 +20,6 @@ data:
|
|||||||
# OpenSearch
|
# OpenSearch
|
||||||
OPENSEARCH_HOST: "opensearch.poimen.svc.cluster.local:9200"
|
OPENSEARCH_HOST: "opensearch.poimen.svc.cluster.local:9200"
|
||||||
# Obsidian
|
# Obsidian
|
||||||
OBSIDIAN_URL: "http://obsidian-server.poimen.svc.cluster.local:8080"
|
|
||||||
# LLM Configuration (for entity extraction)
|
# LLM Configuration (for entity extraction)
|
||||||
LLM_ENDPOINT: "http://api-internal.riotpiao.com:8000/v1/chat/completions"
|
LLM_ENDPOINT: "http://api-internal.riotpiao.com:8000/v1/chat/completions"
|
||||||
LLM_MODEL: "qwen:7b"
|
LLM_MODEL: "qwen:7b"
|
||||||
|
|||||||
+35
-9
@@ -1,6 +1,6 @@
|
|||||||
# Poimen Memory API Server
|
# Poimen Memory API Server
|
||||||
# Serves 7 HTTP endpoints for memory ingest, query, and management.
|
# Serves HTTP endpoints for memory ingest, query, visualization.
|
||||||
# Connects to memory-db (pgvector) for persistent storage.
|
# Connects to memory-db (pgvector) + api.riotpiao.com (LLM via Authentik JWT).
|
||||||
apiVersion: apps/v1
|
apiVersion: apps/v1
|
||||||
kind: Deployment
|
kind: Deployment
|
||||||
metadata:
|
metadata:
|
||||||
@@ -60,13 +60,43 @@ spec:
|
|||||||
key: password
|
key: password
|
||||||
- name: DATABASE_URL
|
- name: DATABASE_URL
|
||||||
value: "postgresql://$(DATABASE_USER):$(DATABASE_PASSWORD)@$(DATABASE_HOST):$(DATABASE_PORT)/$(DATABASE_NAME)?sslmode=disable"
|
value: "postgresql://$(DATABASE_USER):$(DATABASE_PASSWORD)@$(DATABASE_HOST):$(DATABASE_PORT)/$(DATABASE_NAME)?sslmode=disable"
|
||||||
# LLM Gateway API key
|
|
||||||
|
# LLM via api.riotpiao.com (Authentik JWT auth)
|
||||||
|
- name: LLM_ENDPOINT
|
||||||
|
value: "https://api.riotpiao.com/v1/chat/completions"
|
||||||
|
- name: LLM_API_BASE
|
||||||
|
value: "https://api.riotpiao.com/v1"
|
||||||
|
- name: LLM_MODEL
|
||||||
|
value: "ornith:35b"
|
||||||
|
|
||||||
|
# Authentik service account (memory-agent-oidc secret)
|
||||||
|
- name: AUTHENTIK_ISSUER
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: memory-agent-oidc
|
||||||
|
key: ISSUER
|
||||||
|
- name: AUTHENTIK_CLIENT_ID
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: memory-agent-oidc
|
||||||
|
key: CLIENT_ID
|
||||||
|
- name: AUTHENTIK_CLIENT_SECRET
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: memory-agent-oidc
|
||||||
|
key: CLIENT_SECRET
|
||||||
|
- name: TOKEN_URL
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: memory-agent-oidc
|
||||||
|
key: TOKEN_URL
|
||||||
|
|
||||||
|
# Server config
|
||||||
- name: MEM_API_KEY
|
- name: MEM_API_KEY
|
||||||
valueFrom:
|
valueFrom:
|
||||||
secretKeyRef:
|
secretKeyRef:
|
||||||
name: poimen-memory-secrets
|
name: poimen-memory-secrets
|
||||||
key: llm-api-key
|
key: llm-api-key
|
||||||
# Server config (from ConfigMap)
|
|
||||||
- name: MEM_PORT
|
- name: MEM_PORT
|
||||||
value: "8080"
|
value: "8080"
|
||||||
- name: MEM_HOME
|
- name: MEM_HOME
|
||||||
@@ -74,10 +104,7 @@ spec:
|
|||||||
envFrom:
|
envFrom:
|
||||||
- configMapRef:
|
- configMapRef:
|
||||||
name: poimen-memory-config
|
name: poimen-memory-config
|
||||||
- secretRef:
|
command: ["/app/mem"]
|
||||||
name: poimen-memory-auth
|
|
||||||
- secretRef:
|
|
||||||
name: poimen-memory-secrets
|
|
||||||
args:
|
args:
|
||||||
- serve
|
- serve
|
||||||
- --port
|
- --port
|
||||||
@@ -110,7 +137,6 @@ spec:
|
|||||||
- name: tmp
|
- name: tmp
|
||||||
emptyDir:
|
emptyDir:
|
||||||
sizeLimit: 64Mi
|
sizeLimit: 64Mi
|
||||||
# Tolerate control-plane nodes
|
|
||||||
tolerations:
|
tolerations:
|
||||||
- key: node-role.kubernetes.io/control-plane
|
- key: node-role.kubernetes.io/control-plane
|
||||||
operator: Exists
|
operator: Exists
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ resources:
|
|||||||
- deployment.yaml
|
- deployment.yaml
|
||||||
- service.yaml
|
- service.yaml
|
||||||
- config.yaml
|
- config.yaml
|
||||||
- obsidian.yaml
|
# obsidian.yaml retired — reference docs now via memory graph
|
||||||
# Legacy secret managed separately
|
# Legacy secret managed separately
|
||||||
# - secrets.yaml
|
# - secrets.yaml
|
||||||
generators:
|
generators:
|
||||||
|
|||||||
@@ -1,159 +0,0 @@
|
|||||||
---
|
|
||||||
# Obsidian server deployment
|
|
||||||
# Serves local vault with web UI and API
|
|
||||||
apiVersion: apps/v1
|
|
||||||
kind: Deployment
|
|
||||||
metadata:
|
|
||||||
name: obsidian-server
|
|
||||||
namespace: poimen
|
|
||||||
labels:
|
|
||||||
app.kubernetes.io/name: obsidian-server
|
|
||||||
app.kubernetes.io/part-of: poimen-memory
|
|
||||||
spec:
|
|
||||||
replicas: 1
|
|
||||||
selector:
|
|
||||||
matchLabels:
|
|
||||||
app.kubernetes.io/name: obsidian-server
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
app.kubernetes.io/name: obsidian-server
|
|
||||||
app.kubernetes.io/part-of: poimen-memory
|
|
||||||
spec:
|
|
||||||
serviceAccountName: obsidian-server
|
|
||||||
securityContext:
|
|
||||||
runAsNonRoot: true
|
|
||||||
runAsUser: 1000
|
|
||||||
runAsGroup: 1000
|
|
||||||
fsGroup: 1000
|
|
||||||
seccompProfile:
|
|
||||||
type: RuntimeDefault
|
|
||||||
initContainers:
|
|
||||||
- name: git-sync-init
|
|
||||||
image: alpine/git:latest
|
|
||||||
securityContext:
|
|
||||||
runAsNonRoot: false
|
|
||||||
runAsUser: 0
|
|
||||||
allowPrivilegeEscalation: false
|
|
||||||
capabilities:
|
|
||||||
drop:
|
|
||||||
- ALL
|
|
||||||
add:
|
|
||||||
- CHOWN
|
|
||||||
- DAC_OVERRIDE
|
|
||||||
command:
|
|
||||||
- sh
|
|
||||||
- -c
|
|
||||||
- |
|
|
||||||
export GIT_SSH_COMMAND="ssh -i /root/.ssh/id_ed25519 -o StrictHostKeyChecking=no"
|
|
||||||
git config --global --add safe.directory /vault
|
|
||||||
if [ -d /vault/.git ]; then
|
|
||||||
cd /vault && git pull origin main || true
|
|
||||||
else
|
|
||||||
# Clone into temp, move contents into vault
|
|
||||||
rm -rf /tmp/repo
|
|
||||||
git clone ssh://[email protected]:2222/rock/poimen-obesdient-memory.git /tmp/repo
|
|
||||||
cp -a /tmp/repo/. /vault/
|
|
||||||
rm -rf /tmp/repo
|
|
||||||
fi
|
|
||||||
chown -R 1000:1000 /vault
|
|
||||||
volumeMounts:
|
|
||||||
- name: vault
|
|
||||||
mountPath: /vault
|
|
||||||
- name: ssh-key
|
|
||||||
mountPath: /root/.ssh
|
|
||||||
readOnly: true
|
|
||||||
containers:
|
|
||||||
- name: obsidian-server
|
|
||||||
image: ppatlabs/obsidian:latest
|
|
||||||
imagePullPolicy: IfNotPresent
|
|
||||||
securityContext:
|
|
||||||
allowPrivilegeEscalation: false
|
|
||||||
capabilities:
|
|
||||||
drop:
|
|
||||||
- ALL
|
|
||||||
ports:
|
|
||||||
- name: http
|
|
||||||
containerPort: 27124
|
|
||||||
protocol: TCP
|
|
||||||
env:
|
|
||||||
- name: VAULT_NAME
|
|
||||||
value: poimen-vault
|
|
||||||
- name: VAULT_PATH
|
|
||||||
value: /vault
|
|
||||||
- name: REST_API_ENABLED
|
|
||||||
value: "true"
|
|
||||||
- name: REST_API_PORT
|
|
||||||
value: "8080"
|
|
||||||
volumeMounts:
|
|
||||||
- name: vault
|
|
||||||
mountPath: /vault
|
|
||||||
- name: config
|
|
||||||
mountPath: /config
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 100m
|
|
||||||
memory: 256Mi
|
|
||||||
limits:
|
|
||||||
cpu: 500m
|
|
||||||
memory: 512Mi
|
|
||||||
livenessProbe:
|
|
||||||
httpGet:
|
|
||||||
path: /
|
|
||||||
port: http
|
|
||||||
scheme: HTTPS
|
|
||||||
initialDelaySeconds: 30
|
|
||||||
periodSeconds: 10
|
|
||||||
timeoutSeconds: 5
|
|
||||||
readinessProbe:
|
|
||||||
httpGet:
|
|
||||||
path: /
|
|
||||||
port: http
|
|
||||||
scheme: HTTPS
|
|
||||||
initialDelaySeconds: 15
|
|
||||||
periodSeconds: 5
|
|
||||||
timeoutSeconds: 5
|
|
||||||
volumes:
|
|
||||||
- name: vault
|
|
||||||
persistentVolumeClaim:
|
|
||||||
claimName: obsidian-vault
|
|
||||||
- name: config
|
|
||||||
emptyDir: {}
|
|
||||||
- name: ssh-key
|
|
||||||
secret:
|
|
||||||
secretName: obsidian-git-ssh
|
|
||||||
defaultMode: 0400
|
|
||||||
|
|
||||||
# PVC managed by homelab repo (k8s/infra/databases/obsidian-vault-pvc.yaml)
|
|
||||||
|
|
||||||
---
|
|
||||||
# Service for Obsidian server
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Service
|
|
||||||
metadata:
|
|
||||||
name: obsidian-server
|
|
||||||
namespace: poimen
|
|
||||||
labels:
|
|
||||||
app.kubernetes.io/name: obsidian-server
|
|
||||||
spec:
|
|
||||||
type: ClusterIP
|
|
||||||
ports:
|
|
||||||
- name: http
|
|
||||||
port: 80
|
|
||||||
targetPort: 27124
|
|
||||||
protocol: TCP
|
|
||||||
selector:
|
|
||||||
app.kubernetes.io/name: obsidian-server
|
|
||||||
|
|
||||||
---
|
|
||||||
# ServiceAccount for Obsidian
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ServiceAccount
|
|
||||||
metadata:
|
|
||||||
name: obsidian-server
|
|
||||||
namespace: poimen
|
|
||||||
labels:
|
|
||||||
app.kubernetes.io/name: obsidian-server
|
|
||||||
|
|
||||||
# Ingress managed by homelab repo (obsidian.riotpiao.com)
|
|
||||||
# See: homelab/k8s/bootstrap/ingress/ingress.yaml
|
|
||||||
Reference in New Issue
Block a user