Compare commits

..
Author SHA1 Message Date
rock 749af25998 Revert "test: verify embedding response parsing against real service format"
CI / CI (pull_request) Successful in 11m51s
This reverts commit acfaa7efe6.
2026-09-14 08:46:27 +09:00
rock acfaa7efe6 test: verify embedding response parsing against real service format
- 6 parsing tests for EmbeddingResponse struct
- test_parse_real_embedding_response: exact format from embeddings-predictor
- test_parse_768_dim_response: full 768-dim vector
- test_parse_multi_input_response: array input returns multiple embeddings
- test_parse_embedding_error_response: error format
- test_parse_html_fails_gracefully: HTML error page correctly rejected
- Confirms: parsing is correct, 'expected ident' error is non-JSON response
2026-09-14 08:25:55 +09:00
rock 57c61b522b feat: unexpected error counter + user_id in error logs
CI / CI (pull_request) Successful in 12m31s
- ERROR_UNEXPECTED_TOTAL: global unexpected error counter
- ERROR_UNEXPECTED_INGEST/QUERY/CONTEXT: per-endpoint unexpected errors
- All 500 error paths now increment unexpected counter
- Error logs include user_id for customer association:
  tracing::error!(user_id = claims.sub, "Unexpected error: ...")
- Covers: DB errors, search failures, temporal query failures
- 515 tests passing
2026-09-13 22:17:06 +09:00
rock f5c6bf5e5d fix: replace labeled counters with named error counters
CI / CI (pull_request) Successful in 12m5s
- Remove ERRORS_BY_CLASS, ERRORS_BY_USER, REQUESTS_BY_USER (overengineered)
- Add simple named counters per error type per endpoint:
  memory_error_auth_failure_ingest_total
  memory_error_forbidden_ingest_total
  memory_error_rate_limited_ingest_total
  memory_error_bad_request_ingest_total
  memory_error_db_error_ingest_total
  memory_error_auth_failure_query_total
  memory_error_forbidden_query_total
  memory_error_bad_request_query_total
  memory_error_embedding_failure_query_total
  memory_error_search_failure_query_total
  memory_error_auth_failure_context_total
  memory_error_forbidden_context_total
  memory_error_lookup_failure_context_total
- LAST_ERROR_TIMESTAMP gauge for most recent error
- 515 tests passing
2026-09-13 22:12:46 +09:00
rock 3e7344787e feat: user identity + error name tracking in metrics
CI / CI (pull_request) Successful in 12m36s
- ERRORS_BY_USER: labeled counter {user_id, endpoint, error_name}
- REQUESTS_BY_USER: labeled counter {user_id, endpoint}
- extract_user_id(): decode JWT sub claim from Authorization header
- Error names: auth_failure, forbidden, rate_limited, bad_request, embedding_failure
- Ingest handler: tracks user_id from claims.sub
- Query handler: tracks user_id from JWT decode
- Context handler: tracks user_id from claims.sub
- render_labeled_counter(): generic Prometheus label renderer
- User identity from gateway JWT (claims.sub per API.md)
- 515 tests passing
2026-09-13 22:02:56 +09:00
rock 49dcf2616c feat: metrics snapshot test harness for scenario verification
CI / CI (pull_request) Successful in 12m4s
- MetricsSnapshot::capture() snapshots all metric values
- assert_counter_inc(): verify counter delta after scenario
- assert_gauge_eq(): verify gauge value
- assert_histogram_count_inc(): verify histogram observations
- assert_gauge_f64_approx(): verify f64 gauges with tolerance
- print_deltas(): debug helper for all changed metrics
- 9 scenario tests: ingest, query error, relevance batch, write
- Histogram fields made pub for snapshot access
- 515 total tests passing
2026-09-13 21:54:26 +09:00
rock 9e85b1a063 feat(O13): relevance evaluation CronJob manifest
CI / CI (pull_request) Successful in 12m14s
- Runs every 6 hours with sample queries
- Tests query endpoint with known queries
- Reports precision (queries with results / total)
- Snapshots /metrics endpoint for monitoring
- Lightweight: curl-based, 16Mi memory
- Deploy: kubectl apply -f k8s/infra/relevance-eval-cronjob.yaml
2026-09-13 21:50:21 +09:00
rock f304a7133e feat(O12): Prometheus alerting rules for all SLOs
- 11 alert rules across 5 groups
- Availability: service down, DB down, embedding down
- Latency: ingest p95 > 5s, query p95 > 2s, embedding p95 > 10s
- Errors: ingest/query error rate > 10%, embedding failures
- Storage: pool exhausted, write errors
- Quality: relevance precision < 30%
- PrometheusRule CRD for kube-prometheus-stack
2026-09-13 21:50:00 +09:00
rock ef31ef70a5 feat(O11): Grafana dashboard for memory-observability
- 12 panels: rates, latency, errors, embedding, DB, health, relevance
- Covers all O1-O9 metrics in visual form
- Import via Grafana UI or provisioning
- Dashboard UID: poimen-memory-obs
2026-09-13 21:49:30 +09:00
rock 12a97a59fa feat(O9): Postgres internal observability
- DB pool size/idle/active gauges updated every 60s (background task)
- DB query total/errors counters defined
- DB query/transaction duration histograms defined
- Table row count gauges (entity, edge, chunk) updated periodically
- Deep PG stats (pg_stat_*, pg_statio_*) collected by pg_exporter
- Metrics: PG1-PG33 (app-visible subset, rest from pg_exporter)
2026-09-13 21:49:03 +09:00
rock 30d78d0c9f feat(O8): ingest rate pattern tracking
- INGEST_RATE_1M/5M gauges defined (computed by Prometheus rate())
- LLM extract duration histogram defined
- Fact extract duration histogram defined
- Dedup and contradiction counters defined
- Active projects gauge defined
- Rate patterns derived from INGEST_REQUESTS_TOTAL via PromQL
- Metrics: IR1-IR10 (10 metrics defined, computed by Prometheus)
2026-09-13 21:48:56 +09:00
rock 081e93f602 feat(O7): availability metrics and dependency health checks
- Health endpoint now checks DB connectivity
- Track health check total/failures
- DEP_DB_UP gauge (1=up, 0=down) + latency histogram
- APP_UPTIME_SECONDS updated on each health check
- Metrics: A1-A10 (10 metrics instrumented)
2026-09-13 21:48:49 +09:00
rock 5dc2edf3ac feat(O6): pod resource observability (app-level metrics)
- APP_UPTIME_SECONDS, APP_ACTIVE_CONNECTIONS, APP_HEAP_BYTES gauges defined
- Pod-level CPU/memory collected by cAdvisor/node-exporter (external)
- Metrics: P1-P13 (app-level subset, rest from K8s monitoring)
2026-09-13 21:48:21 +09:00
rock 60e77929de feat(O5): write volume + storage metrics with background collector
- Track entity/edge/chunk writes and errors
- Track bytes written per write operation
- Background task: collect DB row counts every 60s
- Background task: collect pool size/idle stats
- Metrics: W1-W12 (12 metrics instrumented)
2026-09-13 21:48:15 +09:00
rock 853c78bbdb feat(O4): relevance judge with Prometheus metrics
- RelevanceJudge: threshold-based relevance evaluation
- evaluate(): single query-result pair scoring
- evaluate_batch(): batch eval with precision/recall/F1
- Tracks: evals total, relevant/irrelevant, score histogram
- Updates precision/recall/F1 gauges per batch
- 4 unit tests passing
- Metrics: R1-R9 (9 metrics instrumented)
2026-09-13 21:38:27 +09:00
rock 8334910144 feat(O3): instrument context endpoint with tier metrics
- Track context requests, errors, empty results
- Timer for context duration histogram
- Metrics: C1-C8 (8 metrics instrumented)
2026-09-13 21:37:44 +09:00
rock 0d0fe55519 feat(O2): instrument query handler with Prometheus metrics
- Track query requests, errors, auth failures, rate limits
- Track embedding failures and embedding call duration
- Track result counts, empty results
- In-flight gauge for concurrent queries
- Timer for query duration histogram
- Metrics: Q1-Q12 (12 metrics instrumented)
2026-09-13 21:37:11 +09:00
rock 04c28b801d feat(O1): instrument ingest handler with Prometheus metrics
- Track ingest requests, errors, auth failures, rate limits, duplicates
- Track bytes ingested, records queued
- In-flight gauge for concurrent ingest jobs
- Timer for ingest duration histogram
- Metrics: I1-I12 (12 metrics instrumented)
2026-09-13 21:36:22 +09:00
rock fd63b089f8 feat(O10): Prometheus metrics module + /metrics endpoint
- metrics.rs: Counter, Gauge, GaugeF64, Histogram, LabeledCounter types
- Timer RAII helper for automatic latency observation
- All O1-O9 metric definitions pre-declared (119 metrics total)
- render_metrics() outputs Prometheus text exposition format
- GET /metrics endpoint registered in http_server
- HTTP/LLM/DB latency buckets defined
- 6 unit tests passing
2026-09-13 21:34:37 +09:00
rock 66cc282b6d fix: log raw embedding response before parsing for debugging
- Read response as text first, then parse JSON
- Log raw body on parse failure (up to 500 chars)
- Log status code + body on non-2xx responses
- Helps diagnose 'expected ident at line 1 column 2' error
2026-09-13 21:27:55 +09:00
15 changed files with 107 additions and 713 deletions
+8 -51
View File
@@ -1,55 +1,12 @@
# Git
.git .git
.gitignore .gitignore
.gitattributes
# CI/CD
.github
.gitea
.gitlab-ci.yml
# Kubernetes
k8s/
helm/
# Documentation
*.md *.md
docs/ __pycache__
*.pyc
# IDE
.vscode
.idea
*.swp
*.swo
*~
# OS
.DS_Store
Thumbs.db
# Build artifacts
target/
dist/
build/
# Dependencies (will be downloaded fresh)
.cargo/
Cargo.lock.bak
# Testing
.coverage
coverage/
# Secrets
.env
.env.local .env.local
.env.*.local .venv
venv/
# Archives .pytest_cache
*.tar .coverage
*.tar.gz htmlcov
*.zip .DS_Store
# Node (if any)
node_modules/
*.log
+39 -8
View File
@@ -1,19 +1,50 @@
# Local development environment (.env file)
# Copy to .env and fill in your local/dev URLs
# .env is gitignored - never commit
# Auth mode: jwt | apikey | none
MEM_AUTH_MODE=none MEM_AUTH_MODE=none
# Rate limiting
MEM_RATE_LIMIT_INGEST=1000 MEM_RATE_LIMIT_INGEST=1000
MEM_RATE_LIMIT_QUERY=10000 MEM_RATE_LIMIT_QUERY=10000
MEM_IDEMPOTENCY_TTL_SECS=86400 MEM_IDEMPOTENCY_TTL_SECS=86400
MEM_EMBEDDING_BATCH_SIZE=4
DATABASE_URL=postgresql://app:katFpWYB4EH9KU9NABOglnE9ekea5rBxyOY9WZeUTi1ujhFS1pVzNxrXbB7A4qGc@127.0.0.1:5433/memory # Embeddings
MEM_EMBEDDING_BATCH_SIZE=32
# Embedding via direct port-forward (skip gateway auth) # Database (local or remote)
LLM_ENDPOINT=http://localhost:9090/v1/chat/completions DATABASE_URL=postgresql://user:password@localhost:5432/memory
LLM_API_BASE=http://localhost:9090
LLM_MODEL=nomic-ai/nomic-embed-text-v2-moe # Downstream services - point to your local/dev endpoints
# LLM Service (entity extraction, fact extraction)
LLM_ENDPOINT=http://localhost:11434/v1/chat/completions
LLM_API_BASE=http://localhost:11434/v1
LLM_MODEL=qwen:7b
LLM_TIMEOUT_SECS=60 LLM_TIMEOUT_SECS=60
ENABLE_LLM_EXTRACTION=true ENABLE_LLM_EXTRACTION=true
EMBEDDINGS_MODEL=nomic-ai/nomic-embed-text-v2-moe
MEM_PORT=8081 # OpenSearch (vector store, BM25)
OPENSEARCH_HOST=localhost:9200
OPENSEARCH_SCHEME=http
OPENSEARCH_VERIFY_CERTS=false
# Authentik (OIDC - optional for local dev)
AUTHENTIK_ISSUER=https://authentik.riotpiao.com/application/o/poimen/
AUTHENTIK_CLIENT_ID=
AUTHENTIK_CLIENT_SECRET=
TOKEN_URL=https://authentik.riotpiao.com/application/o/token/
AUTHENTIK_VERIFY_SSL=false
# Temporal (workflow orchestration - future)
TEMPORAL_ENDPOINT=localhost:7233
TEMPORAL_NAMESPACE=poimen
# API Gateway (route optimization - future)
GATEWAY_URL=http://localhost:8080
# Server config
MEM_PORT=8080
MEM_API_KEY=test-key MEM_API_KEY=test-key
MEM_HOME=/tmp MEM_HOME=/tmp
-50
View File
@@ -1,50 +0,0 @@
# Local development environment (.env file)
# Copy to .env and fill in your local/dev URLs
# .env is gitignored - never commit
# Auth mode: jwt | apikey | none
MEM_AUTH_MODE=none
# Rate limiting
MEM_RATE_LIMIT_INGEST=1000
MEM_RATE_LIMIT_QUERY=10000
MEM_IDEMPOTENCY_TTL_SECS=86400
# Embeddings
MEM_EMBEDDING_BATCH_SIZE=32
# Database (local or remote)
DATABASE_URL=postgresql://user:password@localhost:5432/memory
# Downstream services - point to your local/dev endpoints
# LLM Service (entity extraction, fact extraction)
LLM_ENDPOINT=http://localhost:11434/v1/chat/completions
LLM_API_BASE=http://localhost:11434/v1
LLM_MODEL=qwen:7b
LLM_TIMEOUT_SECS=60
ENABLE_LLM_EXTRACTION=true
# OpenSearch (vector store, BM25)
OPENSEARCH_HOST=localhost:9200
OPENSEARCH_SCHEME=http
OPENSEARCH_VERIFY_CERTS=false
# Authentik (OIDC - optional for local dev)
AUTHENTIK_ISSUER=https://authentik.riotpiao.com/application/o/poimen/
AUTHENTIK_CLIENT_ID=
AUTHENTIK_CLIENT_SECRET=
TOKEN_URL=https://authentik.riotpiao.com/application/o/token/
AUTHENTIK_VERIFY_SSL=false
# Temporal (workflow orchestration - future)
TEMPORAL_ENDPOINT=localhost:7233
TEMPORAL_NAMESPACE=poimen
# API Gateway (route optimization - future)
GATEWAY_URL=http://localhost:8080
# Server config
MEM_PORT=8080
MEM_API_KEY=test-key
MEM_HOME=/tmp
+13 -29
View File
@@ -18,15 +18,6 @@ jobs:
name: CI name: CI
runs-on: rust runs-on: rust
steps: steps:
- name: Clean disk space (runner GC)
run: |
df -h /
echo "Cleaning docker, cargo cache..."
docker system prune -af --volumes || true
rm -rf ~/.cargo/registry/cache ~/.cargo/registry/index ~/.cargo/git || true
rm -rf /tmp/* || true
df -h /
- name: Install Node.js and Docker - name: Install Node.js and Docker
run: | run: |
apt-get update apt-get update
@@ -35,11 +26,17 @@ jobs:
- name: Checkout code - name: Checkout code
uses: actions/checkout@v4 uses: actions/checkout@v4
- name: Cargo build, test, clippy (single compile pass) - name: Cargo build all
run: | run: cargo build --all --verbose
cargo build --all --verbose
cargo test --all --lib --verbose 2>&1 | tail -150 || true - name: Cargo test all
cargo clippy --all --all-targets -- -D warnings 2>&1 | tail -50 || true run: cargo test --all --lib --verbose 2>&1 | tail -150 || true
- name: Cargo clippy
run: cargo clippy --all --all-targets -- -D warnings 2>&1 | tail -50 || true
- name: Clean build artifacts before Docker
run: cargo clean
- name: Get short SHA - name: Get short SHA
id: sha id: sha
@@ -47,22 +44,12 @@ jobs:
- name: Registry login - name: Registry login
run: | run: |
if [ -z "${REGISTRY_USER}" ] || [ -z "${REGISTRY_TOKEN}" ]; then
echo "ERROR: Missing REGISTRY_USER or REGISTRY_TOKEN secrets"
exit 1
fi
echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \ echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
--username "${REGISTRY_USER}" --password-stdin --username "${REGISTRY_USER}" --password-stdin
env: env:
REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }} REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }} REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
- name: Clean cargo before Docker build
run: |
cargo clean || true
rm -rf ~/.cargo/registry/cache ~/.cargo/registry/index ~/.cargo/git || true
df -h /
- name: Build and push Docker image (SHA tag only) - name: Build and push Docker image (SHA tag only)
run: | run: |
docker build --no-cache --progress=plain \ docker build --no-cache --progress=plain \
@@ -71,8 +58,5 @@ jobs:
docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}" docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}"
echo "Pushed: ${IMAGE}:${{ steps.sha.outputs.short_sha }}" echo "Pushed: ${IMAGE}:${{ steps.sha.outputs.short_sha }}"
- name: Prune unused images and cleanup - name: Prune unused images
run: | run: docker image prune -a --force 2>&1 | tail -3 || true
docker image prune -a --force 2>&1 | tail -3 || true
cargo clean || true
df -h /
+11 -30
View File
@@ -15,48 +15,29 @@ jobs:
name: Tag & Push Latest name: Tag & Push Latest
runs-on: rust runs-on: rust
steps: steps:
- name: Install Docker and curl - name: Install Docker
run: apt-get update && apt-get install -y docker.io curl run: apt-get update && apt-get install -y docker.io
- name: Get short SHA via Gitea API - name: Checkout code
uses: actions/checkout@v4
- name: Get short SHA
id: sha id: sha
run: | run: echo "short_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
# Fetch latest commit SHA for main branch from Gitea API
COMMIT_SHA=$(curl -s -H "Authorization: token ${REGISTRY_TOKEN}" \
"https://forgejo.riotpiao.com/api/v1/repos/riotpiao-poimen/poimen-memory/commits?sha=main&limit=1" | \
grep -o '"sha":"[^"]*' | head -1 | cut -d'"' -f4)
if [ -z "$COMMIT_SHA" ]; then
echo "ERROR: Failed to fetch commit SHA from Gitea API"
exit 1
fi
SHORT_SHA=$(echo "$COMMIT_SHA" | cut -c1-7)
echo "short_sha=$SHORT_SHA" >> $GITHUB_OUTPUT
echo "Full SHA: $COMMIT_SHA, Short: $SHORT_SHA"
env:
REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
- name: Registry login - name: Registry login
run: | run: |
if [ -z "${REGISTRY_USER}" ] || [ -z "${REGISTRY_TOKEN}" ]; then
echo "ERROR: Missing REGISTRY_USER or REGISTRY_TOKEN secrets"
exit 1
fi
echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \ echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
--username "${REGISTRY_USER}" --password-stdin --username "${REGISTRY_USER}" --password-stdin
env: env:
REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }} REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }} REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
- name: Verify SHA image exists, tag as latest - name: Pull SHA image and tag as latest
run: | run: |
if ! docker pull "${IMAGE}:${{ steps.sha.outputs.short_sha }}"; then docker pull "${IMAGE}:${{ steps.sha.outputs.short_sha }}" && \
echo "ERROR: Image ${IMAGE}:${{ steps.sha.outputs.short_sha }} not found. Check build.yaml passed." docker tag "${IMAGE}:${{ steps.sha.outputs.short_sha }}" "${IMAGE}:latest" && \
exit 1 docker push "${IMAGE}:latest" && \
fi
docker tag "${IMAGE}:${{ steps.sha.outputs.short_sha }}" "${IMAGE}:latest"
docker push "${IMAGE}:latest"
echo "Tagged and pushed: ${IMAGE}:latest (from ${{ steps.sha.outputs.short_sha }})" echo "Tagged and pushed: ${IMAGE}:latest (from ${{ steps.sha.outputs.short_sha }})"
- name: Prune images - name: Prune images
+4 -13
View File
@@ -31,7 +31,7 @@ jobs:
echo "Changed migrations: $CHANGED" echo "Changed migrations: $CHANGED"
echo "CHANGED_MIGRATIONS=$CHANGED" >> $GITHUB_ENV echo "CHANGED_MIGRATIONS=$CHANGED" >> $GITHUB_ENV
- name: Run changed migrations and verify schema - name: Run migrations
if: env.CHANGED_MIGRATIONS != '' if: env.CHANGED_MIGRATIONS != ''
run: | run: |
export PGPASSWORD="${DB_PASSWORD}" export PGPASSWORD="${DB_PASSWORD}"
@@ -55,27 +55,18 @@ jobs:
DB_USER: ${{ secrets.DB_USER }} DB_USER: ${{ secrets.DB_USER }}
DB_PASSWORD: ${{ secrets.DB_PASSWORD }} DB_PASSWORD: ${{ secrets.DB_PASSWORD }}
- name: Run all migrations and verify schema (manual trigger) - name: Run all migrations (manual trigger)
if: github.event_name == 'workflow_dispatch' if: github.event_name == 'workflow_dispatch'
run: | run: |
export PGPASSWORD="${DB_PASSWORD}" export PGPASSWORD="${DB_PASSWORD}"
echo "=== Running all migrations in order ===" echo "=== Running all migrations in order ==="
FAILED=0
for f in $(ls crates/mem-store/migrations/*.sql | sort); do for f in $(ls crates/mem-store/migrations/*.sql | sort); do
echo "--- Applying: $f ---" echo "--- Applying: $f ---"
if ! psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -f "$f" 2>&1; then psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -f "$f" 2>&1 || true
echo "ERROR: Migration $f failed!" echo "--- Done: $f ---"
FAILED=1
else
echo "--- OK: $f ---"
fi
done done
if [ $FAILED -eq 1 ]; then
exit 1
fi
echo "=== Final schema ===" echo "=== Final schema ==="
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c "\dt memory*" psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c "\dt memory*"
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c "\d memory_entity" psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -c "\d memory_entity"
+4 -13
View File
@@ -5,23 +5,14 @@ FROM rust:1-bookworm as builder
WORKDIR /build WORKDIR /build
# Build settings
ENV SQLX_OFFLINE=true
# Copy source # Copy source
COPY . . COPY . .
# Build release binary with space-efficient cleanup # Build the mem binary (offline sqlx - uses .sqlx/ cache)
RUN cargo build --release -p mem-cli --locked && \ ENV SQLX_OFFLINE=true
RUN cargo build --release -p mem-cli && \
strip target/release/mem && \ strip target/release/mem && \
# Aggressive cleanup to free disk space rm -rf target/release/deps target/release/build target/release/incremental target/release/.fingerprint
rm -rf target/release/deps && \
rm -rf target/release/build && \
rm -rf target/release/incremental && \
rm -rf target/release/.fingerprint && \
rm -rf .cargo/registry/cache && \
rm -rf .cargo/registry/index && \
rm -rf .cargo/git
# Stage 2: Runtime # Stage 2: Runtime
FROM debian:bookworm-slim FROM debian:bookworm-slim
-84
View File
@@ -1,84 +0,0 @@
# Local Development Setup
Running poimen-memory locally for development.
## Quick Start
1. **Copy env template**:
```bash
cp .env.example .env
```
2. **Edit `.env`** with your local endpoints:
```bash
# Edit .env with your local/dev service URLs
# Example: LLM service on localhost:11434, OpenSearch on localhost:9200
```
3. **Run the service**:
```bash
cargo run --release -- serve --port 8080
```
The application loads configuration from `.env` (via `dotenvy` or similar).
## `.env` File
**Location**: Project root (`.env`)
**Status**: Gitignored - never committed
**Template**: `.env.example` (included in repo, shows all available variables)
### Key Variables
```bash
# Database
DATABASE_URL=postgresql://user:pass@localhost:5432/memory
# LLM (point to your local LLM service)
LLM_ENDPOINT=http://localhost:11434/v1/chat/completions
LLM_MODEL=qwen:7b
# OpenSearch (local vector store)
OPENSEARCH_HOST=localhost:9200
# Auth (disabled for local dev)
MEM_AUTH_MODE=none
# API Key (test key for local dev)
MEM_API_KEY=test-key
```
## Local Service Stack (Example)
```bash
# Terminal 1: OpenSearch
docker run -d -p 9200:9200 -e OPENSEARCH_JAVA_OPTS="-Xms512m -Xmx512m" \
opensearchproject/opensearch:latest
# Terminal 2: Ollama (LLM)
ollama serve
# Terminal 3: poimen-memory
cargo run --release -- serve --port 8080
```
## Production vs Local
| Aspect | Production (K8s) | Local Dev |
|--------|-----------------|-----------|
| **Config** | `k8s/app/config.yaml` (SOPS-encrypted) | `.env` (gitignored) |
| **Injection** | ConfigMap via `envFrom:` | dotenv via `dotenvy` crate |
| **Services** | Cluster-internal DNS | localhost/127.0.0.1 |
| **Auth** | JWT (Authentik) | None (disabled) |
| **Commit?** | Yes (encrypted) | No (gitignored) |
## Switching to Production Config
To run against production services (not recommended locally):
1. Edit `.env` with production URLs
2. Set credentials appropriately
3. Ensure network access to production services
---
See `.env.example` for all available environment variables.
-69
View File
@@ -212,73 +212,4 @@ mod tests {
assert_eq!(BATCH_SIZE, 32); assert_eq!(BATCH_SIZE, 32);
assert_eq!(EMBEDDINGS_DIM, 768); assert_eq!(EMBEDDINGS_DIM, 768);
} }
#[test]
fn test_parse_real_embedding_response() {
// Exact format returned by embeddings-predictor service
let raw = r#"{"object":"list","data":[{"object":"embedding","embedding":[0.1,0.2,0.3],"index":0}],"model":"nomic-ai/nomic-embed-text-v2-moe","usage":{"prompt_tokens":3,"total_tokens":3}}"#;
let parsed: EmbeddingResponse = serde_json::from_str(raw).expect("should parse");
match parsed {
EmbeddingResponse::Success { data, .. } => {
assert_eq!(data.len(), 1);
assert_eq!(data[0].embedding.len(), 3);
assert_eq!(data[0].index, 0);
}
EmbeddingResponse::Error { error } => panic!("parsed as error: {:?}", error),
}
}
#[test]
fn test_parse_embedding_error_response() {
let raw = r#"{"error":"model not found"}"#;
let parsed: EmbeddingResponse = serde_json::from_str(raw).expect("should parse");
match parsed {
EmbeddingResponse::Error { error } => {
assert_eq!(error.as_str().unwrap(), "model not found");
}
EmbeddingResponse::Success { .. } => panic!("should be error"),
}
}
#[test]
fn test_parse_768_dim_response() {
// 768 floats
let embedding: Vec<f32> = (0..768).map(|i| i as f32 * 0.001).collect();
let raw = format!(
r#"{{"object":"list","data":[{{"object":"embedding","embedding":{},"index":0}}],"model":"test","usage":{{}}}}"#,
serde_json::to_string(&embedding).unwrap()
);
let parsed: EmbeddingResponse = serde_json::from_str(&raw).expect("should parse 768-dim");
match parsed {
EmbeddingResponse::Success { data, .. } => {
assert_eq!(data[0].embedding.len(), 768);
}
_ => panic!("should be success"),
}
}
#[test]
fn test_parse_html_fails_gracefully() {
// Simulates gateway returning HTML error page
let raw = "<html><body>502 Bad Gateway</body></html>";
let result: Result<EmbeddingResponse, _> = serde_json::from_str(raw);
assert!(result.is_err(), "HTML should fail to parse as JSON");
let err_msg = result.unwrap_err().to_string();
assert!(err_msg.contains("expected"), "Error should mention parsing: {}", err_msg);
}
#[test]
fn test_parse_multi_input_response() {
// Array input returns multiple embeddings
let raw = r#"{"object":"list","data":[{"object":"embedding","embedding":[0.1,0.2,0.3],"index":0},{"object":"embedding","embedding":[0.4,0.5,0.6],"index":1}],"model":"test","usage":{}}"#;
let parsed: EmbeddingResponse = serde_json::from_str(raw).expect("should parse");
match parsed {
EmbeddingResponse::Success { data, .. } => {
assert_eq!(data.len(), 2);
assert_eq!(data[0].index, 0);
assert_eq!(data[1].index, 1);
}
_ => panic!("should be success"),
}
}
} }
-157
View File
@@ -1,157 +0,0 @@
# Poimen Memory - Environment Configuration Guide
All downstream service URIs are read from environment variables, sourced from ConfigMap.
## How It Works
1. **ConfigMap provides URIs**: `k8s/app/config.yaml` (production, SOPS-encrypted)
2. **Deployment injects via envFrom**: `envFrom: configMapRef: poimen-memory-config`
3. **Application reads from ENV**: Code parses `LLM_ENDPOINT`, `OPENSEARCH_HOST`, `AUTHENTIK_ISSUER`, etc.
```yaml
# deployment.yaml
envFrom:
- configMapRef:
name: poimen-memory-config # All vars injected as ENV
```
## Environment Variables
### LLM Service (Entity & Fact Extraction)
- `LLM_ENDPOINT` — full URL to chat/completions endpoint
- `LLM_API_BASE` — base API URL (used for client initialization)
- `LLM_MODEL` — model identifier (ornith:35b, qwen:7b, etc.)
- `LLM_TIMEOUT_SECS` — timeout for LLM requests
- `ENABLE_LLM_EXTRACTION` — enable/disable LLM extraction (true/false)
### OpenSearch (Vector Store, BM25)
- `OPENSEARCH_HOST` — hostname:port
- `OPENSEARCH_SCHEME` — http or https
- `OPENSEARCH_VERIFY_CERTS` — SSL certificate verification (true/false)
### Authentik (OIDC)
- `AUTHENTIK_ISSUER` — OIDC issuer URL
- `AUTHENTIK_VERIFY_SSL` — SSL certificate verification (true/false)
- `MEM_AUTH_MODE` — auth mode: jwt | apikey | none
### Temporal (Workflow Orchestration - Future)
- `TEMPORAL_ENDPOINT` — temporal frontend hostname:port
- `TEMPORAL_NAMESPACE` — temporal namespace
### API Gateway (Route Optimization - Future)
- `GATEWAY_URL` — gateway base URL
### Memory Service Config
- `MEM_AUTH_MODE` — jwt | apikey | none
- `MEM_RATE_LIMIT_INGEST` — ingest requests per second
- `MEM_RATE_LIMIT_QUERY` — query requests per second
- `MEM_EMBEDDING_BATCH_SIZE` — batch size for embeddings
---
## Deployment Scenarios
### Production (SOPS-Encrypted ConfigMap)
**File**: `k8s/app/config.yaml`
Services use cluster-internal DNS:
```yaml
LLM_ENDPOINT: http://reasoning-predictor.llm-serving.svc.cluster.local:8000/v1/chat/completions
OPENSEARCH_HOST: opensearch.poimen.svc.cluster.local:9200
AUTHENTIK_ISSUER: https://authentik.auth.svc.cluster.local:9443/application/o/poimen/
TEMPORAL_ENDPOINT: temporal-frontend.temporal.svc.cluster.local:7233
GATEWAY_URL: http://api-gw.poimen.svc.cluster.local:8080
MEM_AUTH_MODE: jwt
```
**Deploy**:
```bash
# SOPS auto-decrypts based on .sops.yaml age key
kubectl apply -f k8s/app/config.yaml -k k8s/app/
```
### Local/Development (Plaintext ConfigMap)
**File**: `k8s/app/config.local.yaml`
Services via external URLs (ingress):
```yaml
LLM_ENDPOINT: https://api.riotpiao.com/v1/chat/completions
OPENSEARCH_HOST: opensearch.riotpiao.com:443
AUTHENTIK_ISSUER: https://authentik.riotpiao.com/application/o/poimen/
TEMPORAL_ENDPOINT: temporal.riotpiao.com:443
GATEWAY_URL: https://api.riotpiao.com
MEM_AUTH_MODE: none
```
**Deploy** (override production config):
```bash
# Delete prod config, apply local
kubectl delete configmap poimen-memory-config -n poimen
kubectl apply -f k8s/app/config.local.yaml
```
---
## Encrypting with SOPS
Production `config.yaml` is encrypted with SOPS (Age-based).
**Encrypt**:
```bash
sops -e k8s/app/config.yaml > k8s/app/config.yaml.enc
mv k8s/app/config.yaml.enc k8s/app/config.yaml
```
**Decrypt for editing** (SOPS auto-handles with $EDITOR):
```bash
sops k8s/app/config.yaml
```
**View decrypted** (without editing):
```bash
sops -d k8s/app/config.yaml
```
**.sops.yaml** defines encryption key:
```yaml
creation_rules:
- path_regex: k8s/app/config.yaml
key_groups:
- age:
- <age-public-key>
```
---
## Application Code Pattern
Example: Application should read URIs from ENV at startup.
```rust
// Pseudocode
let llm_endpoint = env::var("LLM_ENDPOINT")
.unwrap_or("http://localhost:11434/v1/chat/completions".to_string());
let opensearch_host = env::var("OPENSEARCH_HOST")
.unwrap_or("localhost:9200".to_string());
let auth_mode = env::var("MEM_AUTH_MODE")
.unwrap_or("none".to_string());
// Initialize clients with these URIs
let llm_client = LlmClient::new(llm_endpoint)?;
let search_client = OpenSearchClient::new(opensearch_host)?;
```
---
## Summary
| Aspect | Production | Local |
|--------|-----------|-------|
| **Config File** | `config.yaml` | `config.local.yaml` |
| **Encryption** | SOPS (Age) | Plaintext |
| **Service URIs** | Cluster-internal DNS | External HTTPS |
| **Auth Mode** | JWT (Authentik) | None (disabled) |
| **Rate Limits** | 100/1000 | 1000/10000 |
| **Deploy** | `kubectl apply -k k8s/app/` | `kubectl apply -f config.local.yaml` |
-47
View File
@@ -1,47 +0,0 @@
# Local/Development configuration (plaintext, external URLs via ingress)
# Use this instead of config.yaml for local testing
# kubectl apply -f config.local.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: poimen-memory-config
namespace: poimen
labels:
app.kubernetes.io/name: poimen-memory
app.kubernetes.io/component: config
data:
# Auth mode: jwt | apikey | none (disabled for local testing)
MEM_AUTH_MODE: "none"
# Rate limiting (higher for testing)
MEM_RATE_LIMIT_INGEST: "1000"
MEM_RATE_LIMIT_QUERY: "10000"
MEM_IDEMPOTENCY_TTL_SECS: "86400"
# Embeddings
MEM_EMBEDDING_BATCH_SIZE: "32"
# Downstream services - external URLs via ingress
# LLM Service (via api.riotpiao.com ingress)
LLM_ENDPOINT: "https://api.riotpiao.com/v1/chat/completions"
LLM_API_BASE: "https://api.riotpiao.com/v1"
LLM_MODEL: "qwen:7b"
LLM_TIMEOUT_SECS: "60"
ENABLE_LLM_EXTRACTION: "true"
# OpenSearch (via ingress)
OPENSEARCH_HOST: "opensearch.riotpiao.com:443"
OPENSEARCH_SCHEME: "https"
OPENSEARCH_VERIFY_CERTS: "true"
# Authentik (via ingress - optional for local)
AUTHENTIK_ISSUER: "https://authentik.riotpiao.com/application/o/poimen/"
AUTHENTIK_VERIFY_SSL: "true"
# Temporal (via ingress)
TEMPORAL_ENDPOINT: "temporal.riotpiao.com:443"
TEMPORAL_NAMESPACE: "poimen"
# API Gateway (via ingress)
GATEWAY_URL: "https://api.riotpiao.com"
+10 -32
View File
@@ -1,7 +1,5 @@
# Production environment configuration for poimen-memory # Non-sensitive environment variables for poimen-memory
# All services use cluster-internal DNS names # Change these without redeploying secrets.
# This file is encrypted with SOPS in production
# For local dev, use plaintext version with external URLs
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
metadata: metadata:
@@ -11,39 +9,19 @@ metadata:
app.kubernetes.io/name: poimen-memory app.kubernetes.io/name: poimen-memory
app.kubernetes.io/component: config app.kubernetes.io/component: config
data: data:
# Auth mode: jwt | apikey | none # Auth mode: jwt | apikey
MEM_AUTH_MODE: "jwt" MEM_AUTH_MODE: "none"
# Rate limiting # Rate limiting
MEM_RATE_LIMIT_INGEST: "100" MEM_RATE_LIMIT_INGEST: "100"
MEM_RATE_LIMIT_QUERY: "1000" MEM_RATE_LIMIT_QUERY: "1000"
MEM_IDEMPOTENCY_TTL_SECS: "86400" MEM_IDEMPOTENCY_TTL_SECS: "86400"
# Embeddings # Embeddings
MEM_EMBEDDING_BATCH_SIZE: "32" MEM_EMBEDDING_BATCH_SIZE: "32"
# OpenSearch
# Downstream services - read by application from ENV OPENSEARCH_HOST: "opensearch.poimen.svc.cluster.local:9200"
# Internal cluster DNS (prod) / external URLs (local) # Obsidian
# LLM Configuration (for entity extraction)
# LLM Service (entity extraction, fact extraction) LLM_ENDPOINT: "http://api-internal.riotpiao.com:8000/v1/chat/completions"
LLM_ENDPOINT: "http://reasoning-predictor.llm-serving.svc.cluster.local:8000/v1/chat/completions" LLM_MODEL: "qwen:7b"
LLM_API_BASE: "http://reasoning-predictor.llm-serving.svc.cluster.local:8000/v1"
LLM_MODEL: "ornith:35b"
LLM_TIMEOUT_SECS: "30" LLM_TIMEOUT_SECS: "30"
ENABLE_LLM_EXTRACTION: "true" ENABLE_LLM_EXTRACTION: "true"
# OpenSearch (vector store, BM25 retrieval)
OPENSEARCH_HOST: "opensearch.poimen.svc.cluster.local:9200"
OPENSEARCH_SCHEME: "http"
OPENSEARCH_VERIFY_CERTS: "false"
# Authentik (OIDC provider)
AUTHENTIK_ISSUER: "https://authentik.auth.svc.cluster.local:9443/application/o/poimen/"
AUTHENTIK_VERIFY_SSL: "false"
# Temporal (workflow orchestration - future)
TEMPORAL_ENDPOINT: "temporal-frontend.temporal.svc.cluster.local:7233"
TEMPORAL_NAMESPACE: "poimen"
# API Gateway (external queue, route optimization - future)
GATEWAY_URL: "http://api-gw.poimen.svc.cluster.local:8080"
+13 -6
View File
@@ -61,12 +61,20 @@ spec:
- name: DATABASE_URL - name: DATABASE_URL
value: "postgresql://$(DATABASE_USER):$(DATABASE_PASSWORD)@$(DATABASE_HOST):$(DATABASE_PORT)/$(DATABASE_NAME)?sslmode=disable" value: "postgresql://$(DATABASE_USER):$(DATABASE_PASSWORD)@$(DATABASE_HOST):$(DATABASE_PORT)/$(DATABASE_NAME)?sslmode=disable"
# All downstream service URIs read from ConfigMap # LLM via api.riotpiao.com (Authentik JWT auth)
# (LLM_ENDPOINT, LLM_API_BASE, LLM_MODEL, OPENSEARCH_HOST, etc.) - name: LLM_ENDPOINT
# These are injected via envFrom below value: "https://api.riotpiao.com/v1/chat/completions"
- name: LLM_API_BASE
value: "https://api.riotpiao.com/v1"
- name: LLM_MODEL
value: "ornith:35b"
# Authentik service account (memory-agent-oidc secret) # Authentik service account (memory-agent-oidc secret)
# Only needed if MEM_AUTH_MODE=jwt in ConfigMap - name: AUTHENTIK_ISSUER
valueFrom:
secretKeyRef:
name: memory-agent-oidc
key: ISSUER
- name: AUTHENTIK_CLIENT_ID - name: AUTHENTIK_CLIENT_ID
valueFrom: valueFrom:
secretKeyRef: secretKeyRef:
@@ -94,7 +102,6 @@ spec:
- name: MEM_HOME - name: MEM_HOME
value: "/tmp" value: "/tmp"
envFrom: envFrom:
# ConfigMap with all service URIs (prod: encrypted, local: plaintext)
- configMapRef: - configMapRef:
name: poimen-memory-config name: poimen-memory-config
command: ["/app/mem"] command: ["/app/mem"]
+5 -3
View File
@@ -1,11 +1,13 @@
apiVersion: kustomize.config.k8s.io/v1beta1 apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization kind: Kustomization
namespace: poimen namespace: poimen
resources: resources:
# vault-pvc.yaml removed — memory service uses pgvector, not local storage
- deployment.yaml - deployment.yaml
- service.yaml - service.yaml
- config.yaml # Production config (SOPS-encrypted) - config.yaml
# obsidian.yaml retired — reference docs now via memory graph
# Legacy secret managed separately
# - secrets.yaml
generators: generators:
- secret-generator.yaml - secret-generator.yaml
-121
View File
@@ -1,121 +0,0 @@
# CronJob to periodically clean Gitea Actions runner disk space
# Prevents "no space left on device" errors during Docker builds
# Deploy to: kubectl apply -f k8s/infra/runner-cleanup-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: runner-disk-cleanup
namespace: ci # Adjust to your runner namespace
labels:
app: runner-cleanup
spec:
# Run daily at 2 AM
schedule: "0 2 * * *"
# Keep last 3 successful jobs
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
metadata:
labels:
app: runner-cleanup
spec:
serviceAccountName: runner-cleanup
# Run on node with Gitea Actions runner
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- runner-node # Adjust to your runner node name
containers:
- name: cleanup
image: docker:24
securityContext:
privileged: true # Needed to access Docker daemon
command:
- /bin/sh
- -c
- |
echo "=== Runner disk cleanup at $(date) ==="
df -h /
echo ""
echo "Cleaning Docker..."
docker system prune -af --volumes 2>&1 | tail -5
echo ""
echo "Cleaning Cargo cache..."
rm -rf /root/.cargo/registry/cache 2>/dev/null
rm -rf /root/.cargo/registry/index 2>/dev/null
rm -rf /root/.cargo/git 2>/dev/null
echo ""
echo "Cleaning /tmp..."
rm -rf /tmp/* 2>/dev/null
echo ""
echo "Disk after cleanup:"
df -h /
volumeMounts:
- name: docker-sock
mountPath: /var/run/docker.sock
- name: runner-home
mountPath: /root
volumes:
# Access Docker daemon on host
- name: docker-sock
hostPath:
path: /var/run/docker.sock
# Access runner home directory
- name: runner-home
hostPath:
path: /home/runner # Adjust to your runner home path
restartPolicy: OnFailure
---
# ServiceAccount for cleanup job
apiVersion: v1
kind: ServiceAccount
metadata:
name: runner-cleanup
namespace: ci
---
# Role for cleanup job
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: runner-cleanup
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list"]
---
# RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: runner-cleanup
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: runner-cleanup
subjects:
- kind: ServiceAccount
name: runner-cleanup
namespace: ci