## Summary
Production testing of ingest + embedding pipeline with api-gw integration.
## Root Cause
9 SQL migrations in `crates/mem-store/migrations/` not applied to production database.
Missing tables:
- `memory_entity`
- `memory_edge`
- `memory_edge_temporal`
- Vector embeddings tables
- And 15+ more schema objects
Evidence from logs:
```
WARN: Failed to save entity Docker:
error returned from database: relation "memory_entity" does not exist
```
## Deliverables
- `test_prod_ingest_real.sh` - Full E2E test against K8s + api-gw
- `apply_migrations.sh` - Manual schema migration (backup)
- `collect_prod_logs.sh` - Pod log collection before/after
- `run_production_test.sh` - Test orchestrator
- `tests/integration_ingest_with_gw.rs` - Integration test
- `tests/unit_ingest_logging.rs` - Unit tests for extraction
- Enhanced logging in `ingest_worker.rs` - Per-record event tracking
## Next Steps
1. Trigger "DB Migration" workflow in Forgejo Actions
2. This applies all 9 migrations from `crates/mem-store/migrations/`
3. Pod restart (automatic)
4. Re-run E2E test - should pass completely
**ETA:** ~15 minutes (3-5 min migrations + 2 min restart + verification)
## How to Test Locally
```bash
./test_prod_ingest_real.sh --verbose
```
Requires:
- kubectl access to poimen namespace
- Port-forwarding to memory-service
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #55
Co-authored-by: poimen <[email protected]>
132 lines
4.1 KiB
YAML
132 lines
4.1 KiB
YAML
apiVersion: tekton.dev/v1
|
|
kind: Task
|
|
metadata:
|
|
name: agent-memory-migration
|
|
namespace: tekton-pipelines
|
|
spec:
|
|
description: Apply agent memory schema migration (004) to production database
|
|
params:
|
|
- name: migration-version
|
|
description: Migration version number
|
|
default: "004"
|
|
- name: database-name
|
|
description: Database name
|
|
default: "memory"
|
|
workspaces:
|
|
- name: source
|
|
description: Git source with migrations
|
|
- name: db-credentials
|
|
description: Database credentials secret
|
|
steps:
|
|
- name: apply-migration
|
|
image: postgres:16-alpine
|
|
workingDir: $(workspaces.source.path)
|
|
env:
|
|
- name: PGPASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: memory-db-app
|
|
key: password
|
|
- name: PGHOST
|
|
value: memory-db-rw.poimen.svc.cluster.local
|
|
- name: PGUSER
|
|
value: app
|
|
- name: PGDATABASE
|
|
value: $(params.database-name)
|
|
script: |
|
|
#!/bin/sh
|
|
set -e
|
|
|
|
echo "Applying migration $(params.migration-version)_agent_memory_schema.sql"
|
|
|
|
# Wait for database to be ready
|
|
until pg_isready -h $PGHOST -U $PGUSER -d $PGDATABASE; do
|
|
echo "Waiting for database..."
|
|
sleep 2
|
|
done
|
|
|
|
# Apply migration
|
|
psql -h $PGHOST -U $PGUSER -d $PGDATABASE \
|
|
-f migrations/$(params.migration-version)_agent_memory_schema.sql
|
|
|
|
# Verify tables created
|
|
TABLES=$(psql -h $PGHOST -U $PGUSER -d $PGDATABASE -t -c \
|
|
"SELECT count(*) FROM information_schema.tables WHERE table_schema='public' AND table_name IN ('agent_prompt', 'agent_skill', 'agent_decision', 'role_prompt_mapping', 'agent_metrics')")
|
|
|
|
if [ "$TABLES" -eq 5 ]; then
|
|
echo "✓ All agent memory tables created successfully"
|
|
exit 0
|
|
else
|
|
echo "✗ Migration failed: expected 5 tables, found $TABLES"
|
|
exit 1
|
|
fi
|
|
|
|
- name: verify-indexes
|
|
image: postgres:16-alpine
|
|
env:
|
|
- name: PGPASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: memory-db-app
|
|
key: password
|
|
- name: PGHOST
|
|
value: memory-db-rw.poimen.svc.cluster.local
|
|
- name: PGUSER
|
|
value: app
|
|
- name: PGDATABASE
|
|
value: $(params.database-name)
|
|
script: |
|
|
#!/bin/sh
|
|
set -e
|
|
|
|
echo "Verifying indexes..."
|
|
|
|
INDEXES=$(psql -h $PGHOST -U $PGUSER -d $PGDATABASE -t -c \
|
|
"SELECT count(*) FROM pg_indexes WHERE schemaname='public' AND tablename LIKE 'agent_%'")
|
|
|
|
if [ "$INDEXES" -gt 0 ]; then
|
|
echo "✓ Found $INDEXES indexes on agent tables"
|
|
psql -h $PGHOST -U $PGUSER -d $PGDATABASE -c \
|
|
"SELECT indexname FROM pg_indexes WHERE schemaname='public' AND tablename LIKE 'agent_%' ORDER BY indexname;"
|
|
else
|
|
echo "✗ No indexes found on agent tables"
|
|
exit 1
|
|
fi
|
|
|
|
- name: verify-schemas
|
|
image: postgres:16-alpine
|
|
env:
|
|
- name: PGPASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: memory-db-app
|
|
key: password
|
|
- name: PGHOST
|
|
value: memory-db-rw.poimen.svc.cluster.local
|
|
- name: PGUSER
|
|
value: app
|
|
- name: PGDATABASE
|
|
value: $(params.database-name)
|
|
script: |
|
|
#!/bin/sh
|
|
set -e
|
|
|
|
echo "Verifying table schemas..."
|
|
|
|
# Verify agent_prompt table
|
|
psql -h $PGHOST -U $PGUSER -d $PGDATABASE -c "
|
|
SELECT column_name, data_type, is_nullable
|
|
FROM information_schema.columns
|
|
WHERE table_name='agent_prompt'
|
|
ORDER BY ordinal_position;"
|
|
|
|
echo "✓ Agent prompt schema verified"
|
|
|
|
# Verify role_prompt_mapping has foreign key
|
|
psql -h $PGHOST -U $PGUSER -d $PGDATABASE -c "
|
|
SELECT constraint_name, constraint_type
|
|
FROM information_schema.table_constraints
|
|
WHERE table_name='role_prompt_mapping';"
|
|
|
|
echo "✓ All table schemas verified"
|