test: production ingest E2E test suite with enhanced logging (#55)
CI / CI (push) Successful in 25m7s
Deploy / Tag & Push Latest (push) Successful in 3m49s

## Summary
Production testing of ingest + embedding pipeline with api-gw integration.

## Root Cause
9 SQL migrations in `crates/mem-store/migrations/` not applied to production database.

Missing tables:
- `memory_entity`
- `memory_edge`
- `memory_edge_temporal`
- Vector embeddings tables
- And 15+ more schema objects

Evidence from logs:
```
WARN: Failed to save entity Docker:
      error returned from database: relation "memory_entity" does not exist
```

## Deliverables
- `test_prod_ingest_real.sh` - Full E2E test against K8s + api-gw
- `apply_migrations.sh` - Manual schema migration (backup)
- `collect_prod_logs.sh` - Pod log collection before/after
- `run_production_test.sh` - Test orchestrator
- `tests/integration_ingest_with_gw.rs` - Integration test
- `tests/unit_ingest_logging.rs` - Unit tests for extraction
- Enhanced logging in `ingest_worker.rs` - Per-record event tracking

## Next Steps
1. Trigger "DB Migration" workflow in Forgejo Actions
2. This applies all 9 migrations from `crates/mem-store/migrations/`
3. Pod restart (automatic)
4. Re-run E2E test - should pass completely

**ETA:** ~15 minutes (3-5 min migrations + 2 min restart + verification)

## How to Test Locally
```bash
./test_prod_ingest_real.sh --verbose
```

Requires:
- kubectl access to poimen namespace
- Port-forwarding to memory-service

---------

Co-authored-by: rock <[email protected]>
Reviewed-on: #55
Co-authored-by: poimen <[email protected]>
This commit was merged in pull request #55.
This commit is contained in:
2026-09-16 00:10:58 +00:00
committed by rock
co-authored by rock
parent 4169effd8a
commit a88ea918bf
137 changed files with 4628 additions and 7227 deletions
+2 -4
View File
@@ -267,11 +267,9 @@ fn test_compression_handles_large_content() {
fn test_multi_chunk_search_consistency() {
let optimizer = ContextOptimizer::new().expect("optimizer init");
let chunks = vec![
"ERROR: connection failed\nDEBUG: thread id=100",
let chunks = ["ERROR: connection failed\nDEBUG: thread id=100",
"ERROR: timeout after 5000ms\nTRACE: stack unwinding",
"ERROR: retry attempt 2\nDEBUG: backoff delay=200ms",
];
"ERROR: retry attempt 2\nDEBUG: backoff delay=200ms"];
let optimized_chunks: Vec<_> = chunks
.iter()
+1 -3
View File
@@ -196,7 +196,6 @@ fn gate_memory_bounded() {
// Should not panic from memory exhaustion
// If we get here, we passed the gate
assert!(true, "memory usage bounded");
}
#[test]
@@ -231,7 +230,7 @@ fn gate_compression_targets_met() {
];
for (content, name, min_compression) in fixtures.iter() {
let optimized = optimizer.optimize(content).expect(&format!("optimize {}", name));
let optimized = optimizer.optimize(content).unwrap_or_else(|_| panic!("optimize {}", name));
let ratio = optimized.compressed.len() as f32 / content.len() as f32;
// At least some compression should happen
@@ -332,5 +331,4 @@ fn gate_summary_report() {
println!("\n🚀 STATUS: M3.8 READY FOR PRODUCTION");
assert!(true); // Just for testing framework
}