poimenandrock
a88ea918bf
## Summary
Production testing of ingest + embedding pipeline with api-gw integration.
## Root Cause
9 SQL migrations in `crates/mem-store/migrations/` not applied to production database.
Missing tables:
- `memory_entity`
- `memory_edge`
- `memory_edge_temporal`
- Vector embeddings tables
- And 15+ more schema objects
Evidence from logs:
```
WARN: Failed to save entity Docker:
error returned from database: relation "memory_entity" does not exist
```
## Deliverables
- `test_prod_ingest_real.sh` - Full E2E test against K8s + api-gw
- `apply_migrations.sh` - Manual schema migration (backup)
- `collect_prod_logs.sh` - Pod log collection before/after
- `run_production_test.sh` - Test orchestrator
- `tests/integration_ingest_with_gw.rs` - Integration test
- `tests/unit_ingest_logging.rs` - Unit tests for extraction
- Enhanced logging in `ingest_worker.rs` - Per-record event tracking
## Next Steps
1. Trigger "DB Migration" workflow in Forgejo Actions
2. This applies all 9 migrations from `crates/mem-store/migrations/`
3. Pod restart (automatic)
4. Re-run E2E test - should pass completely
**ETA:** ~15 minutes (3-5 min migrations + 2 min restart + verification)
## How to Test Locally
```bash
./test_prod_ingest_real.sh --verbose
```
Requires:
- kubectl access to poimen namespace
- Port-forwarding to memory-service
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #55
Co-authored-by: poimen <[email protected]>
77 lines
1.7 KiB
YAML
77 lines
1.7 KiB
YAML
---
|
|
# PipelineRun: Agent Memory Feature Testing
|
|
# Tests role-to-prompt mapping with API Platform Engineer role requirements
|
|
# Runs migrations, integration tests, and validates all constraints
|
|
|
|
apiVersion: tekton.dev/v1
|
|
kind: PipelineRun
|
|
metadata:
|
|
name: agent-memory-test-run
|
|
namespace: poimen
|
|
generateName: agent-memory-test-
|
|
spec:
|
|
pipelineRef:
|
|
name: poimen-ci
|
|
|
|
params:
|
|
- name: image
|
|
value: "forgejo.riotpiao.com/riotpiao-poimen/poimen-memory:latest"
|
|
- name: registry-user
|
|
value: "riotpiao-poimen"
|
|
- name: registry-token
|
|
value: "${FORGEJO_REGISTRY_TOKEN}" # Injected by ArgoCD/SOPS
|
|
|
|
workspaces:
|
|
- name: source
|
|
emptyDir: {} # Or use PVC for persistent builds
|
|
|
|
serviceAccountName: tekton-builder
|
|
|
|
timeouts:
|
|
pipeline: "1h"
|
|
tasks: "30m"
|
|
|
|
---
|
|
# ServiceAccount for Tekton Pipeline (builder with DB access)
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: tekton-builder
|
|
namespace: poimen
|
|
|
|
---
|
|
# ClusterRoleBinding: Allow pipeline to query database via pod exec
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRoleBinding
|
|
metadata:
|
|
name: tekton-builder-db-access
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: tekton-builder-db-access
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: tekton-builder
|
|
namespace: poimen
|
|
|
|
---
|
|
# ClusterRole: Database access for migrations
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRole
|
|
metadata:
|
|
name: tekton-builder-db-access
|
|
rules:
|
|
- apiGroups: [""]
|
|
resources: ["pods"]
|
|
verbs: ["get", "list"]
|
|
- apiGroups: [""]
|
|
resources: ["pods/exec"]
|
|
verbs: ["create"]
|
|
- apiGroups: [""]
|
|
resources: ["secrets"]
|
|
resourceNames: ["memory-db-app"]
|
|
verbs: ["get"]
|
|
- apiGroups: [""]
|
|
resources: ["services"]
|
|
verbs: ["get", "list"]
|