poimenandrock
a88ea918bf
## Summary
Production testing of ingest + embedding pipeline with api-gw integration.
## Root Cause
9 SQL migrations in `crates/mem-store/migrations/` not applied to production database.
Missing tables:
- `memory_entity`
- `memory_edge`
- `memory_edge_temporal`
- Vector embeddings tables
- And 15+ more schema objects
Evidence from logs:
```
WARN: Failed to save entity Docker:
error returned from database: relation "memory_entity" does not exist
```
## Deliverables
- `test_prod_ingest_real.sh` - Full E2E test against K8s + api-gw
- `apply_migrations.sh` - Manual schema migration (backup)
- `collect_prod_logs.sh` - Pod log collection before/after
- `run_production_test.sh` - Test orchestrator
- `tests/integration_ingest_with_gw.rs` - Integration test
- `tests/unit_ingest_logging.rs` - Unit tests for extraction
- Enhanced logging in `ingest_worker.rs` - Per-record event tracking
## Next Steps
1. Trigger "DB Migration" workflow in Forgejo Actions
2. This applies all 9 migrations from `crates/mem-store/migrations/`
3. Pod restart (automatic)
4. Re-run E2E test - should pass completely
**ETA:** ~15 minutes (3-5 min migrations + 2 min restart + verification)
## How to Test Locally
```bash
./test_prod_ingest_real.sh --verbose
```
Requires:
- kubectl access to poimen namespace
- Port-forwarding to memory-service
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #55
Co-authored-by: poimen <[email protected]>
50 lines
1.2 KiB
TOML
50 lines
1.2 KiB
TOML
[package]
|
|
name = "mem-cli"
|
|
version = "0.1.0"
|
|
edition = "2021"
|
|
|
|
[lib]
|
|
name = "mem_cli"
|
|
path = "src/lib.rs"
|
|
|
|
[[bin]]
|
|
name = "mem"
|
|
path = "src/main.rs"
|
|
|
|
[dependencies]
|
|
mem-core = { path = "../mem-core" }
|
|
mem-chunk = { path = "../mem-chunk" }
|
|
mem-llm = { path = "../mem-llm" }
|
|
mem-ingest = { path = "../mem-ingest" }
|
|
mem-store = { path = "../mem-store" }
|
|
tokio = { workspace = true }
|
|
futures = { workspace = true }
|
|
serde = { workspace = true }
|
|
serde_json = { workspace = true }
|
|
serde_yaml = { workspace = true }
|
|
redis = { version = "0.25", optional = true }
|
|
anyhow = { workspace = true }
|
|
thiserror = { workspace = true }
|
|
clap = { workspace = true }
|
|
tracing = { workspace = true }
|
|
tracing-subscriber = { workspace = true, features = ["json"] }
|
|
time = { workspace = true }
|
|
actix-web = { workspace = true }
|
|
actix-rt = { workspace = true }
|
|
uuid = { workspace = true }
|
|
chrono = { workspace = true }
|
|
sqlx = { workspace = true }
|
|
pgvector = { workspace = true }
|
|
base64 = { workspace = true }
|
|
sha2 = { workspace = true }
|
|
jsonwebtoken = { workspace = true }
|
|
reqwest = { workspace = true }
|
|
async-trait = { workspace = true }
|
|
urlencoding = { workspace = true }
|
|
walkdir = "2.5"
|
|
futures-util = "0.3"
|
|
async-stream = "0.3"
|
|
rand = "0.8"
|
|
lru = "0.12"
|
|
once_cell = { workspace = true }
|