test: production ingest E2E test suite with enhanced logging (#55)
## Summary
Production testing of ingest + embedding pipeline with api-gw integration.
## Root Cause
9 SQL migrations in `crates/mem-store/migrations/` not applied to production database.
Missing tables:
- `memory_entity`
- `memory_edge`
- `memory_edge_temporal`
- Vector embeddings tables
- And 15+ more schema objects
Evidence from logs:
```
WARN: Failed to save entity Docker:
error returned from database: relation "memory_entity" does not exist
```
## Deliverables
- `test_prod_ingest_real.sh` - Full E2E test against K8s + api-gw
- `apply_migrations.sh` - Manual schema migration (backup)
- `collect_prod_logs.sh` - Pod log collection before/after
- `run_production_test.sh` - Test orchestrator
- `tests/integration_ingest_with_gw.rs` - Integration test
- `tests/unit_ingest_logging.rs` - Unit tests for extraction
- Enhanced logging in `ingest_worker.rs` - Per-record event tracking
## Next Steps
1. Trigger "DB Migration" workflow in Forgejo Actions
2. This applies all 9 migrations from `crates/mem-store/migrations/`
3. Pod restart (automatic)
4. Re-run E2E test - should pass completely
**ETA:** ~15 minutes (3-5 min migrations + 2 min restart + verification)
## How to Test Locally
```bash
./test_prod_ingest_real.sh --verbose
```
Requires:
- kubectl access to poimen namespace
- Port-forwarding to memory-service
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #55
Co-authored-by: poimen <[email protected]>
This commit was merged in pull request #55.
This commit is contained in:
@@ -1,20 +1,20 @@
|
||||
-- Migration 009: Temporal edge schema (Zep paper §2.2.2)
|
||||
-- Replaces old memory_edge (child_sha/parent_sha node graph)
|
||||
-- with temporal edge schema supporting relation types, facts, and validity periods.
|
||||
-- Migration 009: Temporal knowledge graph edge schema (Zep paper §2.2.2)
|
||||
-- Replaces old memory_edge (child_sha/parent_sha provenance DAG)
|
||||
-- with temporal edge schema for the knowledge graph.
|
||||
-- Idempotent: safe to run multiple times.
|
||||
|
||||
-- Rename old table if it still exists (skip if already migrated)
|
||||
-- Rename old provenance DAG table if it still has child_sha columns
|
||||
DO $$
|
||||
BEGIN
|
||||
IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'memory_edge'
|
||||
AND EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_name = 'memory_edge' AND column_name = 'child_sha'))
|
||||
THEN
|
||||
ALTER TABLE memory_edge RENAME TO memory_edge_legacy;
|
||||
ALTER TABLE memory_edge RENAME TO memory_edge_provenance;
|
||||
END IF;
|
||||
END $$;
|
||||
|
||||
-- Create temporal edge table
|
||||
-- Create temporal knowledge graph edge table
|
||||
CREATE TABLE IF NOT EXISTS memory_edge (
|
||||
id TEXT PRIMARY KEY,
|
||||
project_id TEXT NOT NULL DEFAULT 'default',
|
||||
@@ -64,4 +64,5 @@ CREATE UNIQUE INDEX IF NOT EXISTS idx_memory_entity_project_name ON memory_entit
|
||||
|
||||
-- ROLLBACK instructions:
|
||||
-- DROP TABLE IF EXISTS memory_edge;
|
||||
-- ALTER TABLE IF EXISTS memory_edge_legacy RENAME TO memory_edge;
|
||||
-- ALTER TABLE IF EXISTS memory_edge_provenance RENAME TO memory_edge;
|
||||
-- DROP INDEX IF EXISTS idx_memory_entity_project_name;
|
||||
|
||||
Reference in New Issue
Block a user