poimenandrock
a88ea918bf
## Summary
Production testing of ingest + embedding pipeline with api-gw integration.
## Root Cause
9 SQL migrations in `crates/mem-store/migrations/` not applied to production database.
Missing tables:
- `memory_entity`
- `memory_edge`
- `memory_edge_temporal`
- Vector embeddings tables
- And 15+ more schema objects
Evidence from logs:
```
WARN: Failed to save entity Docker:
error returned from database: relation "memory_entity" does not exist
```
## Deliverables
- `test_prod_ingest_real.sh` - Full E2E test against K8s + api-gw
- `apply_migrations.sh` - Manual schema migration (backup)
- `collect_prod_logs.sh` - Pod log collection before/after
- `run_production_test.sh` - Test orchestrator
- `tests/integration_ingest_with_gw.rs` - Integration test
- `tests/unit_ingest_logging.rs` - Unit tests for extraction
- Enhanced logging in `ingest_worker.rs` - Per-record event tracking
## Next Steps
1. Trigger "DB Migration" workflow in Forgejo Actions
2. This applies all 9 migrations from `crates/mem-store/migrations/`
3. Pod restart (automatic)
4. Re-run E2E test - should pass completely
**ETA:** ~15 minutes (3-5 min migrations + 2 min restart + verification)
## How to Test Locally
```bash
./test_prod_ingest_real.sh --verbose
```
Requires:
- kubectl access to poimen namespace
- Port-forwarding to memory-service
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #55
Co-authored-by: poimen <[email protected]>
137 lines
3.1 KiB
YAML
137 lines
3.1 KiB
YAML
---
|
|
# ClusterRole for CI/Tekton triggers to deploy and manage resources across cluster
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRole
|
|
metadata:
|
|
name: ci-tekton-trigger
|
|
rules:
|
|
# Tekton resources
|
|
- apiGroups: ["tekton.dev"]
|
|
resources: ["pipelineruns", "taskruns", "pipelines", "tasks"]
|
|
verbs: ["create", "get", "list", "watch", "patch", "update", "delete"]
|
|
|
|
# Deployments and pods
|
|
- apiGroups: ["apps"]
|
|
resources: ["deployments", "statefulsets", "daemonsets"]
|
|
verbs: ["get", "list", "watch", "create", "patch", "update"]
|
|
- apiGroups: [""]
|
|
resources: ["pods", "pods/log", "pods/status"]
|
|
verbs: ["get", "list", "watch"]
|
|
|
|
# Services and networking
|
|
- apiGroups: [""]
|
|
resources: ["services", "endpoints"]
|
|
verbs: ["get", "list", "watch"]
|
|
|
|
# ConfigMaps and Secrets
|
|
- apiGroups: [""]
|
|
resources: ["configmaps", "secrets"]
|
|
verbs: ["get", "list", "watch"]
|
|
|
|
# Events
|
|
- apiGroups: [""]
|
|
resources: ["events"]
|
|
verbs: ["create", "patch"]
|
|
|
|
# Namespaces
|
|
- apiGroups: [""]
|
|
resources: ["namespaces"]
|
|
verbs: ["get", "list", "watch"]
|
|
|
|
# Persistent volumes
|
|
- apiGroups: [""]
|
|
resources: ["persistentvolumeclaims", "persistentvolumes"]
|
|
verbs: ["get", "list", "watch"]
|
|
|
|
---
|
|
# ClusterRoleBinding for ci-tekton-trigger service account
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRoleBinding
|
|
metadata:
|
|
name: ci-tekton-trigger
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: ci-tekton-trigger
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: ci-tekton-trigger
|
|
namespace: api
|
|
|
|
---
|
|
# Additional ClusterRole for poimen namespace operations
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: Role
|
|
metadata:
|
|
name: ci-tekton-trigger-poimen
|
|
namespace: poimen
|
|
rules:
|
|
# Allow full access in poimen namespace for CI
|
|
- apiGroups: ["*"]
|
|
resources: ["*"]
|
|
verbs: ["*"]
|
|
|
|
---
|
|
# RoleBinding in poimen namespace
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: RoleBinding
|
|
metadata:
|
|
name: ci-tekton-trigger-poimen
|
|
namespace: poimen
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: Role
|
|
name: ci-tekton-trigger-poimen
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: ci-tekton-trigger
|
|
namespace: api
|
|
|
|
---
|
|
# RoleBinding in tekton-pipelines namespace
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: RoleBinding
|
|
metadata:
|
|
name: ci-tekton-trigger
|
|
namespace: tekton-pipelines
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: ci-tekton-trigger
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: ci-tekton-trigger
|
|
namespace: api
|
|
|
|
---
|
|
# RoleBinding in llm-serving namespace (for LLM checks)
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: RoleBinding
|
|
metadata:
|
|
name: ci-tekton-trigger
|
|
namespace: llm-serving
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: ci-tekton-trigger
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: ci-tekton-trigger
|
|
namespace: api
|
|
|
|
---
|
|
# RoleBinding in kube-system namespace (for cluster info)
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: RoleBinding
|
|
metadata:
|
|
name: ci-tekton-trigger
|
|
namespace: kube-system
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: ci-tekton-trigger
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: ci-tekton-trigger
|
|
namespace: api
|