test: production ingest E2E test suite with enhanced logging (#55)

## Summary
Production testing of ingest + embedding pipeline with api-gw integration.

## Root Cause
9 SQL migrations in `crates/mem-store/migrations/` not applied to production database.

Missing tables:
- `memory_entity`
- `memory_edge`
- `memory_edge_temporal`
- Vector embeddings tables
- And 15+ more schema objects

Evidence from logs:
```
WARN: Failed to save entity Docker:
      error returned from database: relation "memory_entity" does not exist
```

## Deliverables
- `test_prod_ingest_real.sh` - Full E2E test against K8s + api-gw
- `apply_migrations.sh` - Manual schema migration (backup)
- `collect_prod_logs.sh` - Pod log collection before/after
- `run_production_test.sh` - Test orchestrator
- `tests/integration_ingest_with_gw.rs` - Integration test
- `tests/unit_ingest_logging.rs` - Unit tests for extraction
- Enhanced logging in `ingest_worker.rs` - Per-record event tracking

## Next Steps
1. Trigger "DB Migration" workflow in Forgejo Actions
2. This applies all 9 migrations from `crates/mem-store/migrations/`
3. Pod restart (automatic)
4. Re-run E2E test - should pass completely

**ETA:** ~15 minutes (3-5 min migrations + 2 min restart + verification)

## How to Test Locally
```bash
./test_prod_ingest_real.sh --verbose
```

Requires:
- kubectl access to poimen namespace
- Port-forwarding to memory-service

---------

Co-authored-by: rock <[email protected]>
Reviewed-on: #55
Co-authored-by: poimen <[email protected]>

k8s/app/kustomization.yaml

@@ -6,6 +6,7 @@ resources:
   - deployment.yaml
   - service.yaml
   - config.yaml        # Production config (SOPS-encrypted)
+  - tekton-pipeline.yaml
 
 generators:
   - secret-generator.yaml

k8s/app/service-monitor.yaml

@@ -0,0 +1,15 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: poimen-memory
+  namespace: poimen
+  labels:
+    app.kubernetes.io/name: poimen-memory
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: poimen-memory
+  endpoints:
+  - port: http
+    path: /metrics
+    interval: 30s

k8s/app/tekton-pipeline.yaml

@@ -0,0 +1,121 @@
+---
+# Tekton Pipeline: poimen-ci
+# Runs smoke tests against the live service after image push.
+# Referenced by .gitea/workflows/build.yaml CI.
+apiVersion: tekton.dev/v1
+kind: Pipeline
+metadata:
+  name: poimen-ci
+  namespace: poimen
+spec:
+  params:
+  - name: image
+    type: string
+  - name: registry-user
+    type: string
+    default: ""
+  - name: registry-token
+    type: string
+    default: ""
+  tasks:
+  - name: integration-tests
+    taskRef:
+      name: poimen-integration-test
+    params:
+    - name: image
+      value: $(params.image)
+  - name: gate-on-tests
+    runAfter:
+      - integration-tests
+    taskSpec:
+      steps:
+      - name: check
+        image: alpine:latest
+        script: |
+          echo "Integration tests passed"
+---
+# Tekton Task: smoke test against live poimen-memory service
+apiVersion: tekton.dev/v1
+kind: Task
+metadata:
+  name: poimen-integration-test
+  namespace: poimen
+spec:
+  params:
+  - name: image
+    type: string
+  results:
+  - name: summary
+    type: string
+  steps:
+  - name: test
+    image: curlimages/curl:latest
+    env:
+    - name: DB_PASSWORD
+      valueFrom:
+        secretKeyRef:
+          key: password
+          name: memory-db-app
+    script: |
+      #!/bin/sh
+      set -e
+      SVC="http://poimen-memory.poimen.svc.cluster.local:8080"
+      PASS=0; FAIL=0
+
+      echo "=== Smoke Test: poimen-memory ==="
+
+      # 1. Health
+      echo "1. GET /health"
+      if curl -sf "$SVC/health" | grep -q '"status":"ok"'; then
+        echo "   PASS"; PASS=$((PASS+1))
+      else
+        echo "   FAIL"; FAIL=$((FAIL+1))
+      fi
+
+      # 2. Projects
+      echo "2. GET /memory/projects"
+      if curl -sf "$SVC/memory/projects" | grep -q '"projects"'; then
+        echo "   PASS"; PASS=$((PASS+1))
+      else
+        echo "   FAIL"; FAIL=$((FAIL+1))
+      fi
+
+      # 3. Ingest
+      echo "3. POST /memory/ingest"
+      ID="smoke-$(date +%s)"
+      RESP=$(curl -sf -X POST "$SVC/memory/ingest" \
+        -H "Content-Type: application/json" \
+        -d "{\"project\":\"ci-smoke\",\"source\":\"tekton\",\"ingest_id\":\"$ID\",\"records\":[{\"role\":\"user\",\"text\":\"[[Kubernetes]] uses [[Docker]]\",\"timestamp\":\"2026-01-01T00:00:00Z\",\"source_position\":0}]}")
+      if echo "$RESP" | grep -q '"status"'; then
+        echo "   PASS"; PASS=$((PASS+1))
+      else
+        echo "   FAIL"; FAIL=$((FAIL+1))
+      fi
+
+      sleep 3
+
+      # 4. Poll status
+      echo "4. GET /memory/ingest/$ID"
+      if curl -sf "$SVC/memory/ingest/$ID" | grep -q '"status"'; then
+        echo "   PASS"; PASS=$((PASS+1))
+      else
+        echo "   FAIL"; FAIL=$((FAIL+1))
+      fi
+
+      # 5. Query
+      echo "5. GET /memory/query"
+      if curl -sf "$SVC/memory/query?project=ci-smoke&question=Docker" | grep -q '"project"'; then
+        echo "   PASS"; PASS=$((PASS+1))
+      else
+        echo "   FAIL"; FAIL=$((FAIL+1))
+      fi
+
+      echo ""
+      echo "Result: $PASS passed, $FAIL failed"
+
+      if [ $FAIL -eq 0 ]; then
+        echo "PASS: $PASS/$((PASS+FAIL))" > /tekton/results/summary
+      else
+        echo "FAIL: $FAIL/$((PASS+FAIL))" > /tekton/results/summary
+        exit 1
+      fi