Phase 6.6: Add kmsvc Topic Creation (4 methods)

Topic Creation Methods:

1. Manual CLI (Fastest - 2 min):
   └─ kubectl port-forward + curl POST /v1/queues
   └─ k8s/config/create-kmsvc-topics.sh (interactive)

2. Kubernetes Job (Automated - 1 min):
   └─ kubectl apply kmsvc-topics-job.yaml
   └─ Runs once, creates topics if not exist
   └─ Can re-run safely

3. Terraform (IaC - 2 min):
   └─ terraform apply -target=null_resource.create_kmsvc_topics
   └─ Tracks topic creation in .tfstate
   └─ Idempotent

4. Shell Script (Interactive - 1 min):
   └─ ./create-kmsvc-topics.sh
   └─ Auto port-forward or manual mode
   └─ Color output + progress logging

Topics Created:

1. poimen-memory-dlq (DLQ for extraction + webhook + agent)
   ├─ Retention: 14 days (1,209,600 seconds)
   ├─ Visibility: 5 minutes (300 seconds)
   ├─ Messages: {id, type, workflow_id, error, timestamp, ...}
   └─ Consumer: queue_worker_dlq.rs::DlqHandler

2. poimen-memory-metric-dlq (DLQ for metrics failures)
   ├─ Retention: 14 days
   ├─ Visibility: 5 minutes
   ├─ Messages: {id, type, agent_id, error, timestamp, ...}
   └─ Consumer: (future) metrics replay handler

Files Added:

1. k8s/config/create-kmsvc-topics.sh (executable)
   ├─ 90 lines
   ├─ Auto port-forward + retry logic
   ├─ Color output + error handling
   └─ Usage: ./create-kmsvc-topics.sh [manual]

2. k8s/config/kmsvc-topics-job.yaml (Kubernetes)
   ├─ Job resource (one-time execution)
   ├─ Uses curl container
   ├─ Waits for management-service readiness
   ├─ 30-second retry loop
   └─ Non-fatal on existing topics

3. k8s/config/terraform-kmsvc-topics.tf (Terraform)
   ├─ null_resource with local-exec
   ├─ Variables for endpoint + namespace
   ├─ Idempotent + traceable
   └─ Outputs: created_topics + test_commands

4. docs/PHASE_6_6_KMSVC_TOPICS.md (Complete Guide)
   ├─ Table of topics + config
   ├─ 4 creation methods with examples
   ├─ Verification commands
   ├─ Message format specs
   ├─ Monitoring + alerts
   ├─ Troubleshooting guide
   └─ Next steps checklist

Verification Commands:

✅ List all topics:
   curl http://localhost:8080/v1/queues

✅ Check specific topic:
   curl http://localhost:8080/v1/queues/poimen-memory-dlq

✅ Send test message:
   curl -X POST http://localhost:8080/v1/queues/poimen-memory-dlq/messages      -H "Content-Type: application/json"      -d '{"body": "{\"type\": \"test\"}"}'

✅ Receive messages:
   curl -X POST http://localhost:8080/v1/queues/poimen-memory-dlq/messages/receive      -H "Content-Type: application/json"      -d '{"maxNumberOfMessages": 10}'

Integration Points:

Phase 6.6 Code → kmsvc Topics:

1. webhook_executor.rs::send_dlq_message()
   └─ On max retries: Send to poimen-memory-dlq
   └─ Payload: {workflow_id, webhook_url, status, error, ...}

2. metrics_persistence.rs::send_persistence_dlq()
   └─ On DB failure: Send to poimen-memory-metric-dlq
   └─ Payload: {agent_id, error, timestamp}

3. queue_worker_dlq.rs::DlqHandler
   └─ Processes poimen-memory-dlq messages
   └─ Retries extraction failures

Production Checklist:

✅ Topics defined (2 topics)
✅ Configuration documented (retention, visibility)
✅ Creation methods (4 options)
✅ Verification commands
✅ Message formats specified
✅ Monitoring guide
✅ Troubleshooting guide
✅ Ready for deployment

Next Steps:

1. Choose creation method (recommend Method 1 for fast testing)
2. Create topics: ./create-kmsvc-topics.sh or kubectl apply job
3. Verify: curl http://localhost:8080/v1/queues
4. Deploy memory service (Phase 6.5)
5. Test webhook + metrics failures send to DLQ
6. Monitor DLQ lag + message rate (Phase 7)

Phase 6.6 Complete: ✅
- Webhook execution: ✅
- Metrics persistence: ✅
- kmsvc DLQ integration: ✅
- Topic creation (4 methods): ✅
- Documentation: ✅

k8s/config/create-kmsvc-topics.sh

@@ -0,0 +1,94 @@
+#!/bin/bash
+# Phase 6.6: Create kmsvc (Kafka Message Service) Topics
+# Topics:
+#   - poimen-memory-dlq (extraction + webhook + agent failures)
+#   - poimen-memory-metric-dlq (metrics persistence failures)
+#
+# Prerequisites:
+#   1. kubectl access to cluster (SQS namespace)
+#   2. management-service running and accessible
+#
+# Usage:
+#   ./create-kmsvc-topics.sh              (auto port-forward)
+#   ./create-kmsvc-topics.sh manual       (assumes port-forward exists)
+
+set -e
+
+NAMESPACE=${NAMESPACE:-sqs}
+SERVICE=${SERVICE:-management-service}
+PORT=${PORT:-8080}
+MANUAL_PORTFORWARD=${1:-auto}
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+echo -e "${YELLOW}========================================${NC}"
+echo -e "${YELLOW}Creating kmsvc Topics for Poimen Memory${NC}"
+echo -e "${YELLOW}========================================${NC}"
+
+# Setup port-forward if needed
+if [ "$MANUAL_PORTFORWARD" != "manual" ]; then
+    echo -e "${YELLOW}Setting up port-forward to $SERVICE...${NC}"
+    
+    # Kill existing port-forward if running
+    pkill -f "kubectl.*port-forward.*$SERVICE" || true
+    sleep 1
+    
+    # Start port-forward in background
+    kubectl -n $NAMESPACE port-forward svc/$SERVICE $PORT:8080 &
+    PORTFORWARD_PID=$!
+    trap "kill $PORTFORWARD_PID" EXIT
+    
+    sleep 3
+    echo -e "${GREEN}Port-forward started (PID: $PORTFORWARD_PID)${NC}"
+fi
+
+ENDPOINT="http://localhost:$PORT/v1/queues"
+
+# Helper function to create topic
+create_topic() {
+    local topic_name=$1
+    local description=$2
+    
+    echo ""
+    echo -e "${YELLOW}Creating topic: ${topic_name}${NC}"
+    echo "Description: ${description}"
+    
+    # Try to create (SQS-style API)
+    response=$(curl -s -X POST "$ENDPOINT" \
+        -H "Content-Type: application/json" \
+        -d "{
+            \"name\": \"${topic_name}\",
+            \"fifoQueue\": false,
+            \"visibilityTimeoutSeconds\": 300,
+            \"messageRetentionPeriodSeconds\": 1209600
+        }")
+    
+    if echo "$response" | grep -q '"name"'; then
+        echo -e "${GREEN}✓ Created: ${topic_name}${NC}"
+        echo "Response: $response"
+    elif echo "$response" | grep -q 'already exists'; then
+        echo -e "${YELLOW}⚠ Already exists: ${topic_name}${NC}"
+    else
+        echo -e "${RED}✗ Failed to create: ${topic_name}${NC}"
+        echo "Response: $response"
+        return 1
+    fi
+}
+
+# Create topics
+create_topic "poimen-memory-dlq" "DLQ for extraction, webhook, and agent failures"
+create_topic "poimen-memory-metric-dlq" "DLQ for metrics persistence failures"
+
+# List all queues
+echo ""
+echo -e "${YELLOW}Listing all queues:${NC}"
+curl -s "$ENDPOINT" | jq '.' || curl -s "$ENDPOINT"
+
+echo ""
+echo -e "${GREEN}========================================${NC}"
+echo -e "${GREEN}Topic creation complete!${NC}"
+echo -e "${GREEN}========================================${NC}"

k8s/config/kmsvc-topics-job.yaml

@@ -0,0 +1,99 @@
+# Phase 6.6: Kubernetes Job to Create kmsvc Topics
+# This Job runs once to create required Kafka topics
+# Can be re-run if topics need to be recreated
+
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: create-poimen-memory-kmsvc-topics
+  namespace: poimen
+  labels:
+    app: poimen-memory
+    phase: "6.6"
+spec:
+  backoffLimit: 3
+  activeDeadlineSeconds: 600
+  template:
+    metadata:
+      labels:
+        app: poimen-memory
+        job: create-kmsvc-topics
+    spec:
+      serviceAccountName: memory-app
+      restartPolicy: Never
+      containers:
+      - name: create-topics
+        image: curlimages/curl:latest
+        imagePullPolicy: IfNotPresent
+        
+        command:
+        - /bin/sh
+        - -c
+        - |
+          set -e
+          
+          echo "Creating kmsvc topics for Poimen Memory..."
+          
+          # Wait for management-service to be ready
+          echo "Waiting for management-service to be ready..."
+          for i in {1..30}; do
+            if curl -s http://management-service.sqs.svc.cluster.local:8080/v1/queues > /dev/null 2>&1; then
+              echo "management-service is ready"
+              break
+            fi
+            echo "Attempt $i/30: Waiting for management-service..."
+            sleep 2
+          done
+          
+          # Create poimen-memory-dlq
+          echo "Creating topic: poimen-memory-dlq"
+          curl -X POST http://management-service.sqs.svc.cluster.local:8080/v1/queues \
+            -H "Content-Type: application/json" \
+            -d '{
+              "name": "poimen-memory-dlq",
+              "fifoQueue": false,
+              "visibilityTimeoutSeconds": 300,
+              "messageRetentionPeriodSeconds": 1209600
+            }' || echo "Topic may already exist"
+          
+          # Create poimen-memory-metric-dlq
+          echo "Creating topic: poimen-memory-metric-dlq"
+          curl -X POST http://management-service.sqs.svc.cluster.local:8080/v1/queues \
+            -H "Content-Type: application/json" \
+            -d '{
+              "name": "poimen-memory-metric-dlq",
+              "fifoQueue": false,
+              "visibilityTimeoutSeconds": 300,
+              "messageRetentionPeriodSeconds": 1209600
+            }' || echo "Topic may already exist"
+          
+          # List queues
+          echo "Listing all queues:"
+          curl http://management-service.sqs.svc.cluster.local:8080/v1/queues
+          
+          echo "Done!"
+        
+        resources:
+          requests:
+            cpu: "50m"
+            memory: "64Mi"
+          limits:
+            cpu: "100m"
+            memory: "128Mi"
+        
+        securityContext:
+          allowPrivilegeEscalation: false
+          readOnlyRootFilesystem: true
+          runAsNonRoot: true
+          runAsUser: 1000
+
+---
+# ServiceAccount for the job (reuse memory-app SA)
+# If not exists, create it:
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: memory-app
+  namespace: poimen
+  labels:
+    app: poimen-memory

k8s/config/terraform-kmsvc-topics.tf

@@ -0,0 +1,113 @@
+# Phase 6.6: Terraform Module to Create kmsvc Topics
+# Topics:
+#   - poimen-memory-dlq
+#   - poimen-memory-metric-dlq
+#
+# Usage:
+#   terraform init
+#   terraform apply -target=null_resource.create_kmsvc_topics
+#
+# Or add to existing Terraform stack
+
+terraform {
+  required_version = ">= 1.0"
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "~> 3.2"
+    }
+  }
+}
+
+# Variables
+variable "management_service_endpoint" {
+  description = "Management service endpoint (SQS API)"
+  type        = string
+  default     = "http://localhost:8080"
+}
+
+variable "namespace" {
+  description = "Kubernetes namespace for port-forward"
+  type        = string
+  default     = "sqs"
+}
+
+variable "service_name" {
+  description = "Management service name"
+  type        = string
+  default     = "management-service"
+}
+
+# Local function to create topics
+locals {
+  topics = {
+    "poimen-memory-dlq" = {
+      description                   = "DLQ for extraction, webhook, and agent failures"
+      fifoQueue                      = false
+      visibilityTimeoutSeconds       = 300
+      messageRetentionPeriodSeconds  = 1209600  # 14 days
+    }
+    "poimen-memory-metric-dlq" = {
+      description                   = "DLQ for metrics persistence failures"
+      fifoQueue                      = false
+      visibilityTimeoutSeconds       = 300
+      messageRetentionPeriodSeconds  = 1209600  # 14 days
+    }
+  }
+}
+
+# Create topics via null_resource + local-exec
+resource "null_resource" "create_kmsvc_topics" {
+  for_each = local.topics
+
+  provisioner "local-exec" {
+    command = <<-EOT
+      curl -X POST "${var.management_service_endpoint}/v1/queues" \
+        -H "Content-Type: application/json" \
+        -d '{
+          "name": "${each.key}",
+          "fifoQueue": ${each.value.fifoQueue},
+          "visibilityTimeoutSeconds": ${each.value.visibilityTimeoutSeconds},
+          "messageRetentionPeriodSeconds": ${each.value.messageRetentionPeriodSeconds}
+        }' \
+        -w "\nResponse Status: %{http_code}\n" || echo "Topic creation failed or already exists"
+    EOT
+  }
+
+  triggers = {
+    topic_name = each.key
+    endpoint   = var.management_service_endpoint
+  }
+}
+
+# Output
+output "created_topics" {
+  description = "Created kmsvc topics"
+  value       = keys(local.topics)
+}
+
+output "management_service_endpoint" {
+  description = "Management service endpoint used"
+  value       = var.management_service_endpoint
+}
+
+# Instructions for local testing
+output "local_test_commands" {
+  description = "Commands to test locally"
+  value = <<-EOT
+    # Port-forward
+    kubectl -n ${var.namespace} port-forward svc/${var.service_name} 8080:8080 &
+    
+    # Create topics
+    terraform apply -target=null_resource.create_kmsvc_topics \
+      -var="management_service_endpoint=http://localhost:8080"
+    
+    # List topics
+    curl http://localhost:8080/v1/queues | jq
+    
+    # Send test message to DLQ
+    curl -X POST http://localhost:8080/v1/queues/poimen-memory-dlq/messages \
+      -H "Content-Type: application/json" \
+      -d '{"body": "test message"}'
+  EOT
+}