Phase 6.6: Add kmsvc Topic Creation (4 methods)

Topic Creation Methods:

1. Manual CLI (Fastest - 2 min):
   └─ kubectl port-forward + curl POST /v1/queues
   └─ k8s/config/create-kmsvc-topics.sh (interactive)

2. Kubernetes Job (Automated - 1 min):
   └─ kubectl apply kmsvc-topics-job.yaml
   └─ Runs once, creates topics if not exist
   └─ Can re-run safely

3. Terraform (IaC - 2 min):
   └─ terraform apply -target=null_resource.create_kmsvc_topics
   └─ Tracks topic creation in .tfstate
   └─ Idempotent

4. Shell Script (Interactive - 1 min):
   └─ ./create-kmsvc-topics.sh
   └─ Auto port-forward or manual mode
   └─ Color output + progress logging

Topics Created:

1. poimen-memory-dlq (DLQ for extraction + webhook + agent)
   ├─ Retention: 14 days (1,209,600 seconds)
   ├─ Visibility: 5 minutes (300 seconds)
   ├─ Messages: {id, type, workflow_id, error, timestamp, ...}
   └─ Consumer: queue_worker_dlq.rs::DlqHandler

2. poimen-memory-metric-dlq (DLQ for metrics failures)
   ├─ Retention: 14 days
   ├─ Visibility: 5 minutes
   ├─ Messages: {id, type, agent_id, error, timestamp, ...}
   └─ Consumer: (future) metrics replay handler

Files Added:

1. k8s/config/create-kmsvc-topics.sh (executable)
   ├─ 90 lines
   ├─ Auto port-forward + retry logic
   ├─ Color output + error handling
   └─ Usage: ./create-kmsvc-topics.sh [manual]

2. k8s/config/kmsvc-topics-job.yaml (Kubernetes)
   ├─ Job resource (one-time execution)
   ├─ Uses curl container
   ├─ Waits for management-service readiness
   ├─ 30-second retry loop
   └─ Non-fatal on existing topics

3. k8s/config/terraform-kmsvc-topics.tf (Terraform)
   ├─ null_resource with local-exec
   ├─ Variables for endpoint + namespace
   ├─ Idempotent + traceable
   └─ Outputs: created_topics + test_commands

4. docs/PHASE_6_6_KMSVC_TOPICS.md (Complete Guide)
   ├─ Table of topics + config
   ├─ 4 creation methods with examples
   ├─ Verification commands
   ├─ Message format specs
   ├─ Monitoring + alerts
   ├─ Troubleshooting guide
   └─ Next steps checklist

Verification Commands:

✅ List all topics:
   curl http://localhost:8080/v1/queues

✅ Check specific topic:
   curl http://localhost:8080/v1/queues/poimen-memory-dlq

✅ Send test message:
   curl -X POST http://localhost:8080/v1/queues/poimen-memory-dlq/messages      -H "Content-Type: application/json"      -d '{"body": "{\"type\": \"test\"}"}'

✅ Receive messages:
   curl -X POST http://localhost:8080/v1/queues/poimen-memory-dlq/messages/receive      -H "Content-Type: application/json"      -d '{"maxNumberOfMessages": 10}'

Integration Points:

Phase 6.6 Code → kmsvc Topics:

1. webhook_executor.rs::send_dlq_message()
   └─ On max retries: Send to poimen-memory-dlq
   └─ Payload: {workflow_id, webhook_url, status, error, ...}

2. metrics_persistence.rs::send_persistence_dlq()
   └─ On DB failure: Send to poimen-memory-metric-dlq
   └─ Payload: {agent_id, error, timestamp}

3. queue_worker_dlq.rs::DlqHandler
   └─ Processes poimen-memory-dlq messages
   └─ Retries extraction failures

Production Checklist:

✅ Topics defined (2 topics)
✅ Configuration documented (retention, visibility)
✅ Creation methods (4 options)
✅ Verification commands
✅ Message formats specified
✅ Monitoring guide
✅ Troubleshooting guide
✅ Ready for deployment

Next Steps:

1. Choose creation method (recommend Method 1 for fast testing)
2. Create topics: ./create-kmsvc-topics.sh or kubectl apply job
3. Verify: curl http://localhost:8080/v1/queues
4. Deploy memory service (Phase 6.5)
5. Test webhook + metrics failures send to DLQ
6. Monitor DLQ lag + message rate (Phase 7)

Phase 6.6 Complete: ✅
- Webhook execution: ✅
- Metrics persistence: ✅
- kmsvc DLQ integration: ✅
- Topic creation (4 methods): ✅
- Documentation: ✅

k8s/config/kmsvc-topics-job.yaml

@@ -0,0 +1,99 @@
+# Phase 6.6: Kubernetes Job to Create kmsvc Topics
+# This Job runs once to create required Kafka topics
+# Can be re-run if topics need to be recreated
+
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: create-poimen-memory-kmsvc-topics
+  namespace: poimen
+  labels:
+    app: poimen-memory
+    phase: "6.6"
+spec:
+  backoffLimit: 3
+  activeDeadlineSeconds: 600
+  template:
+    metadata:
+      labels:
+        app: poimen-memory
+        job: create-kmsvc-topics
+    spec:
+      serviceAccountName: memory-app
+      restartPolicy: Never
+      containers:
+      - name: create-topics
+        image: curlimages/curl:latest
+        imagePullPolicy: IfNotPresent
+        
+        command:
+        - /bin/sh
+        - -c
+        - |
+          set -e
+          
+          echo "Creating kmsvc topics for Poimen Memory..."
+          
+          # Wait for management-service to be ready
+          echo "Waiting for management-service to be ready..."
+          for i in {1..30}; do
+            if curl -s http://management-service.sqs.svc.cluster.local:8080/v1/queues > /dev/null 2>&1; then
+              echo "management-service is ready"
+              break
+            fi
+            echo "Attempt $i/30: Waiting for management-service..."
+            sleep 2
+          done
+          
+          # Create poimen-memory-dlq
+          echo "Creating topic: poimen-memory-dlq"
+          curl -X POST http://management-service.sqs.svc.cluster.local:8080/v1/queues \
+            -H "Content-Type: application/json" \
+            -d '{
+              "name": "poimen-memory-dlq",
+              "fifoQueue": false,
+              "visibilityTimeoutSeconds": 300,
+              "messageRetentionPeriodSeconds": 1209600
+            }' || echo "Topic may already exist"
+          
+          # Create poimen-memory-metric-dlq
+          echo "Creating topic: poimen-memory-metric-dlq"
+          curl -X POST http://management-service.sqs.svc.cluster.local:8080/v1/queues \
+            -H "Content-Type: application/json" \
+            -d '{
+              "name": "poimen-memory-metric-dlq",
+              "fifoQueue": false,
+              "visibilityTimeoutSeconds": 300,
+              "messageRetentionPeriodSeconds": 1209600
+            }' || echo "Topic may already exist"
+          
+          # List queues
+          echo "Listing all queues:"
+          curl http://management-service.sqs.svc.cluster.local:8080/v1/queues
+          
+          echo "Done!"
+        
+        resources:
+          requests:
+            cpu: "50m"
+            memory: "64Mi"
+          limits:
+            cpu: "100m"
+            memory: "128Mi"
+        
+        securityContext:
+          allowPrivilegeEscalation: false
+          readOnlyRootFilesystem: true
+          runAsNonRoot: true
+          runAsUser: 1000
+
+---
+# ServiceAccount for the job (reuse memory-app SA)
+# If not exists, create it:
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: memory-app
+  namespace: poimen
+  labels:
+    app: poimen-memory