## Summary
Production testing of ingest + embedding pipeline with api-gw integration.
## Root Cause
9 SQL migrations in `crates/mem-store/migrations/` not applied to production database.
Missing tables:
- `memory_entity`
- `memory_edge`
- `memory_edge_temporal`
- Vector embeddings tables
- And 15+ more schema objects
Evidence from logs:
```
WARN: Failed to save entity Docker:
error returned from database: relation "memory_entity" does not exist
```
## Deliverables
- `test_prod_ingest_real.sh` - Full E2E test against K8s + api-gw
- `apply_migrations.sh` - Manual schema migration (backup)
- `collect_prod_logs.sh` - Pod log collection before/after
- `run_production_test.sh` - Test orchestrator
- `tests/integration_ingest_with_gw.rs` - Integration test
- `tests/unit_ingest_logging.rs` - Unit tests for extraction
- Enhanced logging in `ingest_worker.rs` - Per-record event tracking
## Next Steps
1. Trigger "DB Migration" workflow in Forgejo Actions
2. This applies all 9 migrations from `crates/mem-store/migrations/`
3. Pod restart (automatic)
4. Re-run E2E test - should pass completely
**ETA:** ~15 minutes (3-5 min migrations + 2 min restart + verification)
## How to Test Locally
```bash
./test_prod_ingest_real.sh --verbose
```
Requires:
- kubectl access to poimen namespace
- Port-forwarding to memory-service
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #55
Co-authored-by: poimen <[email protected]>
## Problem
JWT validation failing with `error decoding response body: expected value at line 6 column 1`.
Root cause: `AUTHENTIK_ISSUER` pointed to slug `poimen-memory` which returns 404 on OIDC discovery. Slug was renamed to `poimen` in Authentik.
Secondary issue: auth env vars were set via `kubectl set env` (not in git), so every ArgoCD sync reverted them.
## Changes
- **k8s/app/config.yaml** — ConfigMap for non-sensitive env (auth mode, rate limits, OpenSearch/Obsidian URLs)
- **k8s/app/auth.enc.yaml** — SOPS-encrypted Secret with `AUTHENTIK_ISSUER`, `AUTHENTIK_AUDIENCE`, `JWT_CACHE_TTL_SECS`
- **k8s/app/secret-generator.yaml** — KSOPS generator for ArgoCD decryption
- **k8s/app/deployment.yaml** — `envFrom` referencing ConfigMap + Secret
- **k8s/app/kustomization.yaml** — Added config.yaml + KSOPS generator
- **k8s/app/opensearch-deployment.yaml** — Updated JWKS/issuer URLs to `poimen` slug
## Rollout
Reloader (`--auto-reload-all=true`) triggers rolling restart when ConfigMap/Secret change. Merge and ArgoCD sync handles everything.Reviewed-on: rock/poimen-memory#40
Co-authored-by: rock <[email protected]>
Memory service stores in pgvector, not local files.
PVC was RWO causing multi-node scheduling failures with 2 replicas.
MEM_HOME points to /tmp (emptyDir) for any scratch needs.
poimenandrock
Story Crater Bot