Compare commits

..
Author SHA1 Message Date
rock 9b3dc839bf local dev: .env for local development (gitignored)
CI / CI (pull_request) Failing after 41m29s
- .env.example: template with all service URIs for local development
- LOCAL_DEV.md: guide for running poimen locally
- Production: config.yaml (SOPS-encrypted K8s ConfigMap)
- Local: .env file (gitignored, never committed)
- Application reads from ENV in both cases (K8s ConfigMap + local dotenv)
- Simplifies prod/dev: same code, different config sources
2026-09-13 11:16:11 +09:00
rock 985bb7ff46 config: env-based service URIs via ConfigMap (prod: SOPS-encrypted)
CI / CI (pull_request) Successful in 32m52s
- config.yaml: prod config with cluster-internal DNS (LLM, OpenSearch, Authentik, Temporal, API-GW)
- config.local.yaml: dev config with external URLs via ingress
- deployment.yaml: remove hardcoded URIs, read all from ConfigMap envFrom
- All downstream service URIs now configurable per environment
- Production config encrypted with SOPS (Age-based)
- Application code reads LLM_ENDPOINT, OPENSEARCH_HOST, AUTHENTIK_ISSUER, etc. from ENV
- Simplifies prod/dev switching: just swap ConfigMap, no code changes
2026-09-13 11:12:53 +09:00
rock 4fdbb48ae6 ci: optimize build + deploy + migrate workflows
CI / CI (pull_request) Successful in 31m18s
- build.yaml: merge 3 cargo steps into single compile pass (reuse artifacts)
- build.yaml: remove cargo clean (wasted compiled artifacts before Docker)
- build.yaml: add secret validation for registry credentials
- deploy.yaml: skip checkout, fetch SHA via Gitea API (no clone overhead)
- deploy.yaml: reuse FORGEJO_REGISTRY_TOKEN for API auth (existing privilege)
- deploy.yaml: validate SHA image exists before tagging as latest
- deploy.yaml: add secret validation for registry credentials
- migrate.yaml: merge schema verification into both changed + manual paths
- migrate.yaml: manual trigger now fails on first error (was silently masking)
2026-09-13 09:53:11 +09:00
rock c142ff5109 fix(opensearch): rotate secrets to random passwords
CI / CI (pull_request) Successful in 11m56s
2026-09-12 23:23:07 +09:00
rock 48bcf39a64 fix(opensearch): single-node discovery, fsGroup, encrypt secrets, drop obsidian
CI / CI (pull_request) Successful in 12m1s
- discovery.type: single-node (bypasses vm.max_map_count bootstrap check)
- fsGroup: 1000 (fixes AccessDeniedException on PVC data dir)
- control-plane tolerations (schedulable on CP nodes)
- secrets moved to opensearch-secrets.enc.yaml (SOPS-encrypted)
- remove orphaned obsidian-git-ssh-secret.enc.yaml
2026-09-12 23:18:33 +09:00
rock 78e7aa8302 feat: scale memory-db to 3 replicas for HA
CI / CI (pull_request) Successful in 13m3s
Sync with homelab/k8s/infra/databases/memory-db.yaml.
Update CNPG Cluster instances from 2 to 3 for high availability.
2026-09-12 04:52:45 +09:00
poimenandrock fb61de6b47 feat: LLM entity + fact extraction pipeline (Zep paper alignment) (#48)
CI / CI (push) Successful in 12m9s
Deploy / Tag & Push Latest (push) Failing after 41s
DB Migration / Run Migrations (push) Failing after 18s
## Changes

### Entity Extraction
- Switch from WikiLinkFallbackExtractor to LlmEntityExtractor when LLM_ENDPOINT set
- `clean_llm_response()`: strips `<think>` tags, markdown fences, extracts JSON
- Handle array responses (Ollama returns `[...]` not `{entities: [...]}`)
- EntityType custom Deserialize: unknown variants → Unknown (no crash)
- Increase timeout 30s→90s, max_tokens 500→1500 for reasoning models
- Graceful reflection fallback: keep entities if verification fails

### Fact Extraction (NEW)
- LlmFactExtractor: LLM-based relationship extraction between entity pairs
- Validates source/target against known entity list (drops hallucinated edges)
- Same robust JSON cleaning for reasoning models + Ollama
- IngestWorker auto-selects LLM vs Simple based on LLM_ENDPOINT env

### K8s Deployment
- Add `command: ["/app/mem"]` (fix args replacing CMD)
- Add LLM_ENDPOINT, LLM_MODEL env vars for in-cluster LLM

## E2E Tested (local Ollama qwen2.5:3b)
- 12 entities extracted (person, tool, concept, organization)
- 5 edges with relationships and facts
- 781 tests pass

## Zep Paper Alignment (§2.2)
- Entity extraction + resolution (§2.2.1)
- Fact extraction between entity pairs (§2.2.2)
- Temporal edge invalidation ready (t_valid/t_invalid schema)
- Reflection verification (§2.2.1, graceful fallback)

---------

Co-authored-by: rock <[email protected]>
Reviewed-on: #48
Co-authored-by: poimen <[email protected]>
2026-09-11 01:11:15 +00:00
16 changed files with 536 additions and 191 deletions
+50
View File
@@ -0,0 +1,50 @@
# Local development environment (.env file)
# Copy to .env and fill in your local/dev URLs
# .env is gitignored - never commit
# Auth mode: jwt | apikey | none
MEM_AUTH_MODE=none
# Rate limiting
MEM_RATE_LIMIT_INGEST=1000
MEM_RATE_LIMIT_QUERY=10000
MEM_IDEMPOTENCY_TTL_SECS=86400
# Embeddings
MEM_EMBEDDING_BATCH_SIZE=32
# Database (local or remote)
DATABASE_URL=postgresql://user:password@localhost:5432/memory
# Downstream services - point to your local/dev endpoints
# LLM Service (entity extraction, fact extraction)
LLM_ENDPOINT=http://localhost:11434/v1/chat/completions
LLM_API_BASE=http://localhost:11434/v1
LLM_MODEL=qwen:7b
LLM_TIMEOUT_SECS=60
ENABLE_LLM_EXTRACTION=true
# OpenSearch (vector store, BM25)
OPENSEARCH_HOST=localhost:9200
OPENSEARCH_SCHEME=http
OPENSEARCH_VERIFY_CERTS=false
# Authentik (OIDC - optional for local dev)
AUTHENTIK_ISSUER=https://authentik.riotpiao.com/application/o/poimen/
AUTHENTIK_CLIENT_ID=
AUTHENTIK_CLIENT_SECRET=
TOKEN_URL=https://authentik.riotpiao.com/application/o/token/
AUTHENTIK_VERIFY_SSL=false
# Temporal (workflow orchestration - future)
TEMPORAL_ENDPOINT=localhost:7233
TEMPORAL_NAMESPACE=poimen
# API Gateway (route optimization - future)
GATEWAY_URL=http://localhost:8080
# Server config
MEM_PORT=8080
MEM_API_KEY=test-key
MEM_HOME=/tmp
+9 -11
View File
@@ -26,17 +26,11 @@ jobs:
- name: Checkout code
uses: actions/checkout@v4
- name: Cargo build all
run: cargo build --all --verbose
- name: Cargo test all
run: cargo test --all --lib --verbose 2>&1 | tail -150 || true
- name: Cargo clippy
run: cargo clippy --all --all-targets -- -D warnings 2>&1 | tail -50 || true
- name: Clean build artifacts before Docker
run: cargo clean
- name: Cargo build, test, clippy (single compile pass)
run: |
cargo build --all --verbose
cargo test --all --lib --verbose 2>&1 | tail -150 || true
cargo clippy --all --all-targets -- -D warnings 2>&1 | tail -50 || true
- name: Get short SHA
id: sha
@@ -44,6 +38,10 @@ jobs:
- name: Registry login
run: |
if [ -z "${REGISTRY_USER}" ] || [ -z "${REGISTRY_TOKEN}" ]; then
echo "ERROR: Missing REGISTRY_USER or REGISTRY_TOKEN secrets"
exit 1
fi
echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
--username "${REGISTRY_USER}" --password-stdin
env:
+30 -13
View File
@@ -15,31 +15,48 @@ jobs:
name: Tag & Push Latest
runs-on: rust
steps:
- name: Install Node.js and Docker
run: |
apt-get update
apt-get install -y nodejs docker.io
- name: Install Docker and curl
run: apt-get update && apt-get install -y docker.io curl
- name: Checkout code
uses: actions/checkout@v4
- name: Get short SHA
- name: Get short SHA via Gitea API
id: sha
run: echo "short_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
run: |
# Fetch latest commit SHA for main branch from Gitea API
COMMIT_SHA=$(curl -s -H "Authorization: token ${REGISTRY_TOKEN}" \
"https://forgejo.riotpiao.com/api/v1/repos/riotpiao-poimen/poimen-memory/commits?sha=main&limit=1" | \
grep -o '"sha":"[^"]*' | head -1 | cut -d'"' -f4)
if [ -z "$COMMIT_SHA" ]; then
echo "ERROR: Failed to fetch commit SHA from Gitea API"
exit 1
fi
SHORT_SHA=$(echo "$COMMIT_SHA" | cut -c1-7)
echo "short_sha=$SHORT_SHA" >> $GITHUB_OUTPUT
echo "Full SHA: $COMMIT_SHA, Short: $SHORT_SHA"
env:
REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
- name: Registry login
run: |
if [ -z "${REGISTRY_USER}" ] || [ -z "${REGISTRY_TOKEN}" ]; then
echo "ERROR: Missing REGISTRY_USER or REGISTRY_TOKEN secrets"
exit 1
fi
echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
--username "${REGISTRY_USER}" --password-stdin
env:
REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
- name: Pull SHA image and tag as latest
- name: Verify SHA image exists, tag as latest
run: |
docker pull "${IMAGE}:${{ steps.sha.outputs.short_sha }}" && \
docker tag "${IMAGE}:${{ steps.sha.outputs.short_sha }}" "${IMAGE}:latest" && \
docker push "${IMAGE}:latest" && \
if ! docker pull "${IMAGE}:${{ steps.sha.outputs.short_sha }}"; then
echo "ERROR: Image ${IMAGE}:${{ steps.sha.outputs.short_sha }} not found. Check build.yaml passed."
exit 1
fi
docker tag "${IMAGE}:${{ steps.sha.outputs.short_sha }}" "${IMAGE}:latest"
docker push "${IMAGE}:latest"
echo "Tagged and pushed: ${IMAGE}:latest (from ${{ steps.sha.outputs.short_sha }})"
- name: Prune images
+58 -62
View File
@@ -11,79 +11,75 @@ env:
DB_HOST: memory-db-rw.poimen.svc.cluster.local
DB_PORT: "5432"
DB_NAME: memory
MIGRATIONS_DIR: crates/mem-store/migrations
DOCKER_HOST: tcp://localhost:2375
jobs:
migrate:
name: Run Migrations
runs-on: rust
steps:
- name: Install Node.js, Docker, and psql
run: |
apt-get update
apt-get install -y nodejs docker.io postgresql-client
- name: Install psql
run: apt-get update && apt-get install -y postgresql-client
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 2
- name: Detect changed migrations
id: detect
- name: Fetch previous migrations state
run: |
CHANGED=$(git diff --name-only HEAD~1 HEAD -- "$MIGRATIONS_DIR"/*.sql 2>/dev/null || echo "")
if [ -n "$CHANGED" ]; then
echo "files=$CHANGED" >> $GITHUB_OUTPUT
echo "found=true" >> $GITHUB_OUTPUT
echo "Changed: $CHANGED"
else
echo "found=false" >> $GITHUB_OUTPUT
echo "No migration changes detected"
git fetch origin main --depth=2
# List changed migration files
CHANGED=$(git diff --name-only HEAD~1 HEAD -- crates/mem-store/migrations/ || echo "")
echo "Changed migrations: $CHANGED"
echo "CHANGED_MIGRATIONS=$CHANGED" >> $GITHUB_ENV
- name: Run changed migrations and verify schema
if: env.CHANGED_MIGRATIONS != ''
run: |
export PGPASSWORD="${DB_PASSWORD}"
echo "=== Running changed migrations ==="
for f in $CHANGED_MIGRATIONS; do
if [ -f "$f" ]; then
echo "--- Applying: $f ---"
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -f "$f" 2>&1
if [ $? -ne 0 ]; then
echo "ERROR: Migration $f failed!"
exit 1
fi
echo "--- OK: $f ---"
fi
done
echo "=== Verify schema ==="
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c "\dt memory*"
env:
DB_USER: ${{ secrets.DB_USER }}
DB_PASSWORD: ${{ secrets.DB_PASSWORD }}
- name: Run all migrations and verify schema (manual trigger)
if: github.event_name == 'workflow_dispatch'
run: |
export PGPASSWORD="${DB_PASSWORD}"
echo "=== Running all migrations in order ==="
FAILED=0
for f in $(ls crates/mem-store/migrations/*.sql | sort); do
echo "--- Applying: $f ---"
if ! psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -f "$f" 2>&1; then
echo "ERROR: Migration $f failed!"
FAILED=1
else
echo "--- OK: $f ---"
fi
done
if [ $FAILED -eq 1 ]; then
exit 1
fi
- name: Apply changed migrations (push)
if: github.event_name == 'push' && steps.detect.outputs.found == 'true'
echo "=== Final schema ==="
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c "\dt memory*"
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c "\d memory_entity"
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c "\d memory_edge"
env:
PGHOST: ${{ env.DB_HOST }}
PGPORT: ${{ env.DB_PORT }}
PGDATABASE: ${{ env.DB_NAME }}
PGUSER: ${{ secrets.DB_USER }}
PGPASSWORD: ${{ secrets.DB_PASSWORD }}
run: |
for f in ${{ steps.detect.outputs.files }}; do
[ -f "$f" ] || continue
echo "=== Applying: $f ==="
psql -v ON_ERROR_STOP=1 -f "$f"
echo "=== OK ==="
done
- name: Apply all migrations (dispatch)
if: github.event_name == 'workflow_dispatch'
env:
PGHOST: ${{ env.DB_HOST }}
PGPORT: ${{ env.DB_PORT }}
PGDATABASE: ${{ env.DB_NAME }}
PGUSER: ${{ secrets.DB_USER }}
PGPASSWORD: ${{ secrets.DB_PASSWORD }}
run: |
for f in $(ls "$MIGRATIONS_DIR"/*.sql | sort); do
echo "=== Applying: $f ==="
psql -v ON_ERROR_STOP=1 -f "$f" || true
echo "=== Done ==="
done
- name: Verify schema
env:
PGHOST: ${{ env.DB_HOST }}
PGPORT: ${{ env.DB_PORT }}
PGDATABASE: ${{ env.DB_NAME }}
PGUSER: ${{ secrets.DB_USER }}
PGPASSWORD: ${{ secrets.DB_PASSWORD }}
run: |
echo "=== Tables ==="
psql -c "\dt memory*"
echo "=== Entity Schema ==="
psql -c "\d memory_entity"
echo "=== Edge Schema ==="
psql -c "\d memory_edge"
DB_USER: ${{ secrets.DB_USER }}
DB_PASSWORD: ${{ secrets.DB_PASSWORD }}
+84
View File
@@ -0,0 +1,84 @@
# Local Development Setup
Running poimen-memory locally for development.
## Quick Start
1. **Copy env template**:
```bash
cp .env.example .env
```
2. **Edit `.env`** with your local endpoints:
```bash
# Edit .env with your local/dev service URLs
# Example: LLM service on localhost:11434, OpenSearch on localhost:9200
```
3. **Run the service**:
```bash
cargo run --release -- serve --port 8080
```
The application loads configuration from `.env` (via `dotenvy` or similar).
## `.env` File
**Location**: Project root (`.env`)
**Status**: Gitignored - never committed
**Template**: `.env.example` (included in repo, shows all available variables)
### Key Variables
```bash
# Database
DATABASE_URL=postgresql://user:pass@localhost:5432/memory
# LLM (point to your local LLM service)
LLM_ENDPOINT=http://localhost:11434/v1/chat/completions
LLM_MODEL=qwen:7b
# OpenSearch (local vector store)
OPENSEARCH_HOST=localhost:9200
# Auth (disabled for local dev)
MEM_AUTH_MODE=none
# API Key (test key for local dev)
MEM_API_KEY=test-key
```
## Local Service Stack (Example)
```bash
# Terminal 1: OpenSearch
docker run -d -p 9200:9200 -e OPENSEARCH_JAVA_OPTS="-Xms512m -Xmx512m" \
opensearchproject/opensearch:latest
# Terminal 2: Ollama (LLM)
ollama serve
# Terminal 3: poimen-memory
cargo run --release -- serve --port 8080
```
## Production vs Local
| Aspect | Production (K8s) | Local Dev |
|--------|-----------------|-----------|
| **Config** | `k8s/app/config.yaml` (SOPS-encrypted) | `.env` (gitignored) |
| **Injection** | ConfigMap via `envFrom:` | dotenv via `dotenvy` crate |
| **Services** | Cluster-internal DNS | localhost/127.0.0.1 |
| **Auth** | JWT (Authentik) | None (disabled) |
| **Commit?** | Yes (encrypted) | No (gitignored) |
## Switching to Production Config
To run against production services (not recommended locally):
1. Edit `.env` with production URLs
2. Set credentials appropriately
3. Ensure network access to production services
---
See `.env.example` for all available environment variables.
+3 -17
View File
@@ -1342,30 +1342,16 @@ async fn query_temporal_graph(
state: &web::Data<AppState>,
params: &QueryParams,
) -> anyhow::Result<serde_json::Value> {
// Step 1: Find entities matching question (fuzzy name/description search)
// Step 1: Find entities (order by name for deterministic results)
let entities_rows: Vec<(String, String, String)> = sqlx::query_as(
"SELECT id, name, entity_type FROM memory_entity
WHERE project_id = $1
AND (name ILIKE '%' || $2 || '%' OR description ILIKE '%' || $2 || '%')
ORDER BY confidence DESC
LIMIT $3"
"SELECT id, name, entity_type FROM memory_entity WHERE project_id = $1 LIMIT $2"
)
.bind(&params.project)
.bind(&params.question)
.bind(params.limit as i32)
.fetch_all(&state.pool)
.await
.unwrap_or_default();
tracing::info!(
target: "observability",
event = "query_entity_search",
project = %params.project,
question = %params.question,
matched = entities_rows.len(),
"Entity search complete"
);
// Step 2: Traverse edges from found entities
// NOTE: Edges will be empty until temporal schema is migrated
let mut edges_data: Vec<(String, String, String, String, String, f32)> = Vec::new();
@@ -1374,7 +1360,7 @@ async fn query_temporal_graph(
for (entity_id, _name, _type_str) in &entities_rows {
let entity_edges: Vec<(String, String, String, String, f32, Option<chrono::DateTime<chrono::Utc>>, Option<chrono::DateTime<chrono::Utc>>)> =
sqlx::query_as(
"SELECT id, target_id, relation_type, fact, confidence, t_valid, t_invalid FROM memory_edge WHERE project_id = $1 AND source_id = $2"
"SELECT id, target_entity_id, relation_type, fact, confidence, t_valid, t_invalid FROM memory_edge WHERE project_id = $1 AND source_entity_id = $2"
)
.bind(&params.project)
.bind(entity_id)
+157
View File
@@ -0,0 +1,157 @@
# Poimen Memory - Environment Configuration Guide
All downstream service URIs are read from environment variables, sourced from ConfigMap.
## How It Works
1. **ConfigMap provides URIs**: `k8s/app/config.yaml` (production, SOPS-encrypted)
2. **Deployment injects via envFrom**: `envFrom: configMapRef: poimen-memory-config`
3. **Application reads from ENV**: Code parses `LLM_ENDPOINT`, `OPENSEARCH_HOST`, `AUTHENTIK_ISSUER`, etc.
```yaml
# deployment.yaml
envFrom:
- configMapRef:
name: poimen-memory-config # All vars injected as ENV
```
## Environment Variables
### LLM Service (Entity & Fact Extraction)
- `LLM_ENDPOINT` — full URL to chat/completions endpoint
- `LLM_API_BASE` — base API URL (used for client initialization)
- `LLM_MODEL` — model identifier (ornith:35b, qwen:7b, etc.)
- `LLM_TIMEOUT_SECS` — timeout for LLM requests
- `ENABLE_LLM_EXTRACTION` — enable/disable LLM extraction (true/false)
### OpenSearch (Vector Store, BM25)
- `OPENSEARCH_HOST` — hostname:port
- `OPENSEARCH_SCHEME` — http or https
- `OPENSEARCH_VERIFY_CERTS` — SSL certificate verification (true/false)
### Authentik (OIDC)
- `AUTHENTIK_ISSUER` — OIDC issuer URL
- `AUTHENTIK_VERIFY_SSL` — SSL certificate verification (true/false)
- `MEM_AUTH_MODE` — auth mode: jwt | apikey | none
### Temporal (Workflow Orchestration - Future)
- `TEMPORAL_ENDPOINT` — temporal frontend hostname:port
- `TEMPORAL_NAMESPACE` — temporal namespace
### API Gateway (Route Optimization - Future)
- `GATEWAY_URL` — gateway base URL
### Memory Service Config
- `MEM_AUTH_MODE` — jwt | apikey | none
- `MEM_RATE_LIMIT_INGEST` — ingest requests per second
- `MEM_RATE_LIMIT_QUERY` — query requests per second
- `MEM_EMBEDDING_BATCH_SIZE` — batch size for embeddings
---
## Deployment Scenarios
### Production (SOPS-Encrypted ConfigMap)
**File**: `k8s/app/config.yaml`
Services use cluster-internal DNS:
```yaml
LLM_ENDPOINT: http://reasoning-predictor.llm-serving.svc.cluster.local:8000/v1/chat/completions
OPENSEARCH_HOST: opensearch.poimen.svc.cluster.local:9200
AUTHENTIK_ISSUER: https://authentik.auth.svc.cluster.local:9443/application/o/poimen/
TEMPORAL_ENDPOINT: temporal-frontend.temporal.svc.cluster.local:7233
GATEWAY_URL: http://api-gw.poimen.svc.cluster.local:8080
MEM_AUTH_MODE: jwt
```
**Deploy**:
```bash
# SOPS auto-decrypts based on .sops.yaml age key
kubectl apply -f k8s/app/config.yaml -k k8s/app/
```
### Local/Development (Plaintext ConfigMap)
**File**: `k8s/app/config.local.yaml`
Services via external URLs (ingress):
```yaml
LLM_ENDPOINT: https://api.riotpiao.com/v1/chat/completions
OPENSEARCH_HOST: opensearch.riotpiao.com:443
AUTHENTIK_ISSUER: https://authentik.riotpiao.com/application/o/poimen/
TEMPORAL_ENDPOINT: temporal.riotpiao.com:443
GATEWAY_URL: https://api.riotpiao.com
MEM_AUTH_MODE: none
```
**Deploy** (override production config):
```bash
# Delete prod config, apply local
kubectl delete configmap poimen-memory-config -n poimen
kubectl apply -f k8s/app/config.local.yaml
```
---
## Encrypting with SOPS
Production `config.yaml` is encrypted with SOPS (Age-based).
**Encrypt**:
```bash
sops -e k8s/app/config.yaml > k8s/app/config.yaml.enc
mv k8s/app/config.yaml.enc k8s/app/config.yaml
```
**Decrypt for editing** (SOPS auto-handles with $EDITOR):
```bash
sops k8s/app/config.yaml
```
**View decrypted** (without editing):
```bash
sops -d k8s/app/config.yaml
```
**.sops.yaml** defines encryption key:
```yaml
creation_rules:
- path_regex: k8s/app/config.yaml
key_groups:
- age:
- <age-public-key>
```
---
## Application Code Pattern
Example: Application should read URIs from ENV at startup.
```rust
// Pseudocode
let llm_endpoint = env::var("LLM_ENDPOINT")
.unwrap_or("http://localhost:11434/v1/chat/completions".to_string());
let opensearch_host = env::var("OPENSEARCH_HOST")
.unwrap_or("localhost:9200".to_string());
let auth_mode = env::var("MEM_AUTH_MODE")
.unwrap_or("none".to_string());
// Initialize clients with these URIs
let llm_client = LlmClient::new(llm_endpoint)?;
let search_client = OpenSearchClient::new(opensearch_host)?;
```
---
## Summary
| Aspect | Production | Local |
|--------|-----------|-------|
| **Config File** | `config.yaml` | `config.local.yaml` |
| **Encryption** | SOPS (Age) | Plaintext |
| **Service URIs** | Cluster-internal DNS | External HTTPS |
| **Auth Mode** | JWT (Authentik) | None (disabled) |
| **Rate Limits** | 100/1000 | 1000/10000 |
| **Deploy** | `kubectl apply -k k8s/app/` | `kubectl apply -f config.local.yaml` |
+47
View File
@@ -0,0 +1,47 @@
# Local/Development configuration (plaintext, external URLs via ingress)
# Use this instead of config.yaml for local testing
# kubectl apply -f config.local.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: poimen-memory-config
namespace: poimen
labels:
app.kubernetes.io/name: poimen-memory
app.kubernetes.io/component: config
data:
# Auth mode: jwt | apikey | none (disabled for local testing)
MEM_AUTH_MODE: "none"
# Rate limiting (higher for testing)
MEM_RATE_LIMIT_INGEST: "1000"
MEM_RATE_LIMIT_QUERY: "10000"
MEM_IDEMPOTENCY_TTL_SECS: "86400"
# Embeddings
MEM_EMBEDDING_BATCH_SIZE: "32"
# Downstream services - external URLs via ingress
# LLM Service (via api.riotpiao.com ingress)
LLM_ENDPOINT: "https://api.riotpiao.com/v1/chat/completions"
LLM_API_BASE: "https://api.riotpiao.com/v1"
LLM_MODEL: "qwen:7b"
LLM_TIMEOUT_SECS: "60"
ENABLE_LLM_EXTRACTION: "true"
# OpenSearch (via ingress)
OPENSEARCH_HOST: "opensearch.riotpiao.com:443"
OPENSEARCH_SCHEME: "https"
OPENSEARCH_VERIFY_CERTS: "true"
# Authentik (via ingress - optional for local)
AUTHENTIK_ISSUER: "https://authentik.riotpiao.com/application/o/poimen/"
AUTHENTIK_VERIFY_SSL: "true"
# Temporal (via ingress)
TEMPORAL_ENDPOINT: "temporal.riotpiao.com:443"
TEMPORAL_NAMESPACE: "poimen"
# API Gateway (via ingress)
GATEWAY_URL: "https://api.riotpiao.com"
+32 -10
View File
@@ -1,5 +1,7 @@
# Non-sensitive environment variables for poimen-memory
# Change these without redeploying secrets.
# Production environment configuration for poimen-memory
# All services use cluster-internal DNS names
# This file is encrypted with SOPS in production
# For local dev, use plaintext version with external URLs
apiVersion: v1
kind: ConfigMap
metadata:
@@ -9,19 +11,39 @@ metadata:
app.kubernetes.io/name: poimen-memory
app.kubernetes.io/component: config
data:
# Auth mode: jwt | apikey
MEM_AUTH_MODE: "none"
# Auth mode: jwt | apikey | none
MEM_AUTH_MODE: "jwt"
# Rate limiting
MEM_RATE_LIMIT_INGEST: "100"
MEM_RATE_LIMIT_QUERY: "1000"
MEM_IDEMPOTENCY_TTL_SECS: "86400"
# Embeddings
MEM_EMBEDDING_BATCH_SIZE: "32"
# OpenSearch
OPENSEARCH_HOST: "opensearch.poimen.svc.cluster.local:9200"
# Obsidian
# LLM Configuration (for entity extraction)
LLM_ENDPOINT: "http://api-internal.riotpiao.com:8000/v1/chat/completions"
LLM_MODEL: "qwen:7b"
# Downstream services - read by application from ENV
# Internal cluster DNS (prod) / external URLs (local)
# LLM Service (entity extraction, fact extraction)
LLM_ENDPOINT: "http://reasoning-predictor.llm-serving.svc.cluster.local:8000/v1/chat/completions"
LLM_API_BASE: "http://reasoning-predictor.llm-serving.svc.cluster.local:8000/v1"
LLM_MODEL: "ornith:35b"
LLM_TIMEOUT_SECS: "30"
ENABLE_LLM_EXTRACTION: "true"
# OpenSearch (vector store, BM25 retrieval)
OPENSEARCH_HOST: "opensearch.poimen.svc.cluster.local:9200"
OPENSEARCH_SCHEME: "http"
OPENSEARCH_VERIFY_CERTS: "false"
# Authentik (OIDC provider)
AUTHENTIK_ISSUER: "https://authentik.auth.svc.cluster.local:9443/application/o/poimen/"
AUTHENTIK_VERIFY_SSL: "false"
# Temporal (workflow orchestration - future)
TEMPORAL_ENDPOINT: "temporal-frontend.temporal.svc.cluster.local:7233"
TEMPORAL_NAMESPACE: "poimen"
# API Gateway (external queue, route optimization - future)
GATEWAY_URL: "http://api-gw.poimen.svc.cluster.local:8080"
+5 -12
View File
@@ -61,20 +61,12 @@ spec:
- name: DATABASE_URL
value: "postgresql://$(DATABASE_USER):$(DATABASE_PASSWORD)@$(DATABASE_HOST):$(DATABASE_PORT)/$(DATABASE_NAME)?sslmode=disable"
# LLM via api.riotpiao.com (Authentik JWT auth)
- name: LLM_ENDPOINT
value: "https://api.riotpiao.com/v1/chat/completions"
- name: LLM_API_BASE
value: "https://api.riotpiao.com/v1"
- name: LLM_MODEL
value: "ornith:35b"
# All downstream service URIs read from ConfigMap
# (LLM_ENDPOINT, LLM_API_BASE, LLM_MODEL, OPENSEARCH_HOST, etc.)
# These are injected via envFrom below
# Authentik service account (memory-agent-oidc secret)
- name: AUTHENTIK_ISSUER
valueFrom:
secretKeyRef:
name: memory-agent-oidc
key: ISSUER
# Only needed if MEM_AUTH_MODE=jwt in ConfigMap
- name: AUTHENTIK_CLIENT_ID
valueFrom:
secretKeyRef:
@@ -102,6 +94,7 @@ spec:
- name: MEM_HOME
value: "/tmp"
envFrom:
# ConfigMap with all service URIs (prod: encrypted, local: plaintext)
- configMapRef:
name: poimen-memory-config
command: ["/app/mem"]
+3 -5
View File
@@ -1,13 +1,11 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: poimen
resources:
# vault-pvc.yaml removed — memory service uses pgvector, not local storage
- deployment.yaml
- service.yaml
- config.yaml
# obsidian.yaml retired — reference docs now via memory graph
# Legacy secret managed separately
# - secrets.yaml
- config.yaml # Production config (SOPS-encrypted)
generators:
- secret-generator.yaml
-24
View File
@@ -1,24 +0,0 @@
apiVersion: ENC[AES256_GCM,data:gSI=,iv:nfXxHTEXSY6eDPOLfQWxQaX/Ge7s08QF6GqQ847cdKg=,tag:szUjeOolHuomGQQdrV7U4A==,type:str]
kind: ENC[AES256_GCM,data:HC8zcR8G,iv:wk4XliU5bPi32M0QV6OhJs3tSkirOczWJjR+1MgjxpM=,tag:jcD8R327PRv8x7wYh4Tdrg==,type:str]
metadata:
name: ENC[AES256_GCM,data:HouUGg1P3iPycnr5doLc9w==,iv:kzODDxNBix4e/kAGrF8io165crqPHewyuG8MCZhr3mM=,tag:hX9peWSY5LwM7/08S+QLuw==,type:str]
namespace: ENC[AES256_GCM,data:OCIDOqNz,iv:GhtxD5cXXTnl/7Po1rY3I+jacI9Kz4bXp+Nz2UVTOTE=,tag:F7TuNLkSluKm4TZZ1Q33VQ==,type:str]
type: ENC[AES256_GCM,data:ErqH5L3k,iv:JioZqat2ZYSO83vEnl1MY6YiCC3RttfEkGc2OumJHBY=,tag:K68wmCX8sMzcnGUz1aBpWA==,type:str]
stringData:
id_ed25519: ENC[AES256_GCM,data:YqrAUmZCvEDC2q8c4Ns+WxW2l+oC31arj1MPYwt6AkIv026kZk9ucywkOWT/Ww0VgiUWF/0t/gzkm7K5D5fSk5vP6PyWV5Zqvlo8Zqy1VBLc3V9gbQeCsA4i8+FO3zZ0k2l3HAefEqhJ4Bnj3dKDOTX4bsbE/H/4n8WCojYnOLdU4esqm5r4bOCnFv5wBkbkob6AwfqekdaBZfuPOqW2sstDSRN6km1UZafCYuMY0XQTxCKYM8Izt3px8sfBq37oA6syDpxNuEpAk0uYumHhSBHKRAnsDY61pjfCbR2xy/3IeEPr6EVf5HwV+2ElhtSB/Zfzin5GZvAgX3gu3HuFznHUYH5olLEEFvVtw8fWLh3avwwCsAlUsDKZoTV1t1bzGni1OPYK3ZfsAmQqI1lvdFlRj++e6L3vDBkVG5qowTelbSb6/TWMDpJw/CsX3bgeKEoFUt2vi9IxwdYO/onuExVrT23WeanoSmrnXRaBqr6xIV5yW5CCbBBRmRU7a6jkwtkhe8dHFTKejaqjBpzPdlZOvhzKBHlOy4eDGjeV7CkzrRw=,iv:bSGkeMli13DSDFAu1+4Kg5sqSJ8LbdpLfN5oIwzLyTM=,tag:9DYK17ET7rkfmmpwxjicog==,type:str]
known_hosts: ENC[AES256_GCM,data:FaWsLxkot5Zxh7mobbUGDFqKLOdmP5APg09nkDyqGHyDp0v0Pjc19jeizM+tq3b3aH59YGlPe/4xQf7ZXZRZjWQE+m/TcVujtLD4hfCc40wqZh1XtTtdC6Tf1p3JbqxP0uQVy1+EFVwPCimUsZfp7gtcT/Hmu1JAW9biGmvACO9+dDeHGzBH5ZRCw3+dEYmcLgGIRgzpOwJLfHr/hkvdlflhzmEHMliBIl+TpqQ38GFQmw0ia7UEJzj3ghoDj7HjrHqlBa7aBHJpaEBYVwq8cN7JaLnyO1Y4+LIU8ln/CEzeg9wxVJoMO8IBcQCCgXoC+ogEpNFVb+pdUfRl/3Ye97ZJFmdJoorvSHIR02e9n7E2G3Ox9iImwnwI76X3FokuY0zcGIkcIho6JN3/8k3Z4VLDl2qGflo7jK6QP1DEsGUGhGwPRVrpPNkMDxYQeBIqlwFzfRuF/gDZj3ZWadCYB7NwByVgTcZFqiMtQ74z6jYGMiPIpW2OCY4HGu9ecGPR02USEu39CjJUCWH9WbQZTjmK3n4yYy4X4WPMbc0IekSCC2ossBznMoFsu7q7L13arqC99j9ZOv8aJ7KgMpGpOVPoN2AURJTFhgMX8TD93AvbFNVNqA0t+Y+g0Hq5f/py8XPzj4b6l8A4QxK3Awj4gf5BbK2lNjL+Cgo2kgwPsPvNd4hvx3gbamDOPNf+lH6iGLF19QyoJPHlOD/k9hkcMRerAEOLZriTylgpj2joXOaeKVrk8qkAZBunQKNc3e0xXH1i5obqz871DbbOVvrfYhmSNG93IEDG3hvNPbIc2uXYIJBciXxEaNO2PEypaLDgnNczEoVyUn2MXTJ6XMr/fkKvVBTmDuFd7BWk1JM6gWKnKvdN2G7bBHz5D43jGmmv+Z4bcd4Z4ViO8yAMbB3kKMLxMZQiSsrOudcQM1rzip2HBYb7JhK3yQwTbqTXZo64hejYW6+KYZysSB+A4itITvQR1G50lKndLd1XXqRbfV00DSZmNPRt0dKcTRkEX2RdCOkh6Wddo7D41V4eDRoHY+rctirsDKE/IA875ooAxpj5CWsHjAzlVrSUUVB2D0IBdlCPaZ5LVSo8i8+9S4rakEdmwe5dceUTC9mNIuzFtUeAshW4J2U4UzIQGOVt0BolwWfWiO5QvuAMC7aH1kJOr9gUB69IwDASdd+PbyEvjCQigVXEeObcliJu9QZohcoQU09fI1ajmPC5x2g2Rbo8DPakQTdsHuzdNm/zMZ/yUhi7+5/jZmwI3kRdgA9EMmhl+T2qAwNbhBiW2Iim44R0ZFuWMv6SCX2Kl9KxH+HW7dS8H7FJbAG7w9kiivpHP0sBTRfSgN9ddR6veVRiZWXNJ38sNy88GXdwzRM90XJAwQ==,iv:mv3hoMwPcEmOBbsIRoKLUuEsUolotv1VtikLiItwuJg=,tag:7amQiuiaVwowLAQcNoq72A==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBINTF2OGRmUWoxTmFEZUdv
SWJKRlFhaVkzbUtOaTVnbHFjd1UzR2RFdVFNClpFODhlQVJIQjhlOEJBL2pDVmJa
UjlZZmFHKzA4eDJEMk9KTDMyOGZ2VWcKLS0tIFRHREo1dXNRKzJVYkROQSt1WjFV
ZXZYVjAwSlZhT0ZMbG1qNDVUWnJyQ2cKfs4t6HsQG5Wiyp6QvFqvm4+/o4NAL3qu
6L9vyhl2jufrbxmR+IsEBCxYS7rh6dCbxTUFap3MD2lYIGF9hRnGjQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
lastmodified: "2026-08-28T23:27:09Z"
mac: ENC[AES256_GCM,data:wJC6YCHXq6I/bqUjfwFRvpULZ1Yt39PoWFKzzOAq6h/pHsUrWEgrkm+3+dLaPpz663b0B75BiCQjQb4igXWr38O5I+FKonRHsbsH+D+pO+dq++yNYG8T30KGaquVfnsm8ijWGWxOY9nULUXfKcYfqvsR9P7KCV7bdcWuZ5xzZ5o=,iv:w5f3h0hb2ooeNYK1QZactpmpT8mAYa94V8FBewP0MUY=,tag:qlgUutFanhjk1HIBJqmLQg==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
+1
View File
@@ -6,3 +6,4 @@ kind: Kustomization
resources:
- memory-db.yaml
- opensearch.yaml
- opensearch-secrets.enc.yaml
+1 -1
View File
@@ -9,7 +9,7 @@ metadata:
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
instances: 2
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:16.2
bootstrap:
initdb:
@@ -0,0 +1,47 @@
apiVersion: ENC[AES256_GCM,data:qM0=,iv:znTNMu1+efRh38Vn0GWlNZTk/6VjCJfJeaEzbM17N8c=,tag:sPSc9mwoZWYvjD1bzM+uzg==,type:str]
kind: ENC[AES256_GCM,data:pHDYbqGy,iv:8kUzizuj3tkgx8FU19FBr8lcz1DFEN2abQTJCFLPL0w=,tag:YqiHCVZ8Pwyx51YkxLSykQ==,type:str]
metadata:
name: ENC[AES256_GCM,data:yC5ph8jQnEd2Jn60tCNYJQq2,iv:QRAhTVXNt77kcbcLXDJo9Y1X3hRu1EZXADwTS3rPq/g=,tag:X80UnBGCV28GiOWNo3K/bA==,type:str]
namespace: ENC[AES256_GCM,data:7N36Xqio,iv:a8yemv8LA1WdXUyNRgTu5teZIB23ClufXh7ovd9m5GU=,tag:ukU63zxVZdD9PwppgAmaEw==,type:str]
type: ENC[AES256_GCM,data:9NGNI47z,iv:tiaioFpXheBY4BimysI3sr5OzFOEI1mG68ObCDiqAIU=,tag:wpEln7lSyAPfxpVjcWhpVg==,type:str]
stringData:
admin-password: ENC[AES256_GCM,data:HeKM7q8662fdrlJbpWh/7VJuhr7h2sRYK6/sN+eBtBo=,iv:4ur6YKAYp6+kvIkmBcx9/0DK2MvK7XDedoZhKl8gjBY=,tag:pct7MAlre1h7bm8Polvutg==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGV2NJQzhDN0ZacXBDeklV
aGE1eGlmMkp6b1RDL2ZiblNwSk1PUkJZdFZjCi84dWpXMFNNcFYrLzkwOUFGZDZ4
SGM0NG9UMkJTME82dUU0MkxFNjVzcTAKLS0tIE5xNlg2RUdheUxyUytsblI3UTFH
UktjaHNGOUlmZGxiSlhoSkJSMW5LMkkKdNAzdge1HaAgBqbE4dCkJgZBlIAP76P+
4GOsh7RbuVDDMzUHTS4aNv2zoM5WC5pv+ZKtf8Yu7LIwiOPAp2u/7g==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
lastmodified: "2026-09-12T14:22:55Z"
mac: ENC[AES256_GCM,data:lO+5lWN4ZVIkg4XAG4mz6n2SxqNfU6KdahoZqj9nZ33maX/9OT7aunwl3eIoE8JlN4vN1UU/s0l1ioT0+PxdGtlQfhisZ0ypzA3z8Nxkcw18XzQaMf99A0Icw1OEGRRx/T6Bf8+l0ZI4HIH+KZmlUg2lAfGK+WTxqr5xJefw5XA=,iv:kWNox9QX7Jv9muHjBo6yuwRjBRuhawaKJ+5+O9E57z4=,tag:qZ+5ceNo2C8cPIt0PBtqiw==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
---
apiVersion: ENC[AES256_GCM,data:jew=,iv:bzrjT8rJssrSv4xZCn9ihNtyelKteybg/XZVJRUawvo=,tag:qN50XwBiN2lnWH1CS35W/g==,type:str]
kind: ENC[AES256_GCM,data:clwtkPLP,iv:Y2sF8dpOJslo4OHeRprK/wcuvUzdOW30Bz3M0Kh8yE4=,tag:TkQo8zZZow/zdAlxViWQlA==,type:str]
metadata:
name: ENC[AES256_GCM,data:UiOyRh0x7Yor3qudRqgwtrv5bBHkHnFMK0Smxw==,iv:a3hB+wBIMjD0Xj7p3ZIqDf3/la1xlzbCYRRc/LV80ig=,tag:ivw42aZepKgOa+cVm0URZg==,type:str]
namespace: ENC[AES256_GCM,data:a37Jp+qA,iv:cDVuBJ/aFo4EcZTC/N9NGk8UdrCROHKiirWBWlrSDMQ=,tag:/LNyMpSaEhgQYIE8PJcXBg==,type:str]
type: ENC[AES256_GCM,data:ql+XYM25,iv:OCfk13+9Ft4Vq6Tq3R6v54zK1tz7imzyr/g8ytcpBEk=,tag:PpJBFsUSpMo3ktGynr8AUw==,type:str]
stringData:
password: ENC[AES256_GCM,data:lT7f3cY+VLqRcfuYnf9lnI5QVqfmmt5pc9tb2EEuRbE=,iv:2Dnwssmj5ddcr4UypVVkm4UydeLc1L/ExsaRsbu/CVw=,tag:6++s/j3MOBIirkKS66Z46Q==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGV2NJQzhDN0ZacXBDeklV
aGE1eGlmMkp6b1RDL2ZiblNwSk1PUkJZdFZjCi84dWpXMFNNcFYrLzkwOUFGZDZ4
SGM0NG9UMkJTME82dUU0MkxFNjVzcTAKLS0tIE5xNlg2RUdheUxyUytsblI3UTFH
UktjaHNGOUlmZGxiSlhoSkJSMW5LMkkKdNAzdge1HaAgBqbE4dCkJgZBlIAP76P+
4GOsh7RbuVDDMzUHTS4aNv2zoM5WC5pv+ZKtf8Yu7LIwiOPAp2u/7g==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
lastmodified: "2026-09-12T14:22:55Z"
mac: ENC[AES256_GCM,data:lO+5lWN4ZVIkg4XAG4mz6n2SxqNfU6KdahoZqj9nZ33maX/9OT7aunwl3eIoE8JlN4vN1UU/s0l1ioT0+PxdGtlQfhisZ0ypzA3z8Nxkcw18XzQaMf99A0Icw1OEGRRx/T6Bf8+l0ZI4HIH+KZmlUg2lAfGK+WTxqr5xJefw5XA=,iv:kWNox9QX7Jv9muHjBo6yuwRjBRuhawaKJ+5+O9E57z4=,tag:qZ+5ceNo2C8cPIt0PBtqiw==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
+8 -35
View File
@@ -65,8 +65,7 @@ data:
# Cluster settings
cluster.name: poimen-memory
node.name: ${HOSTNAME}
cluster.initial_master_nodes: opensearch-0
discovery.seed_hosts: opensearch-0.opensearch.poimen.svc.cluster.local
discovery.type: single-node
# Network
network.host: 0.0.0.0
@@ -127,16 +126,12 @@ spec:
spec:
serviceAccountName: opensearch
hostNetwork: false
initContainers:
- name: sysctl
image: busybox:1.28
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
securityContext:
fsGroup: 1000
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
- name: opensearch
@@ -400,18 +395,6 @@ spec:
---
# Secret: OpenSearch Dashboards password
apiVersion: v1
kind: Secret
metadata:
name: opensearch-dashboards-secret
namespace: poimen
type: Opaque
stringData:
password: "admin" # ⚠️ Change in production
---
# ServiceAccount for OpenSearch Dashboards
apiVersion: v1
kind: ServiceAccount
@@ -419,14 +402,4 @@ metadata:
name: opensearch-dashboards
namespace: poimen
---
# Secret for OpenSearch Admin Password
apiVersion: v1
kind: Secret
metadata:
name: opensearch-secrets
namespace: poimen
type: Opaque
stringData:
admin-password: "OpenSearch@Admin123!"
# Secrets moved to opensearch-secrets.enc.yaml (SOPS-encrypted)