rock
31aafa53e4
Topic Creation Methods:
1. Manual CLI (Fastest - 2 min):
└─ kubectl port-forward + curl POST /v1/queues
└─ k8s/config/create-kmsvc-topics.sh (interactive)
2. Kubernetes Job (Automated - 1 min):
└─ kubectl apply kmsvc-topics-job.yaml
└─ Runs once, creates topics if not exist
└─ Can re-run safely
3. Terraform (IaC - 2 min):
└─ terraform apply -target=null_resource.create_kmsvc_topics
└─ Tracks topic creation in .tfstate
└─ Idempotent
4. Shell Script (Interactive - 1 min):
└─ ./create-kmsvc-topics.sh
└─ Auto port-forward or manual mode
└─ Color output + progress logging
Topics Created:
1. poimen-memory-dlq (DLQ for extraction + webhook + agent)
├─ Retention: 14 days (1,209,600 seconds)
├─ Visibility: 5 minutes (300 seconds)
├─ Messages: {id, type, workflow_id, error, timestamp, ...}
└─ Consumer: queue_worker_dlq.rs::DlqHandler
2. poimen-memory-metric-dlq (DLQ for metrics failures)
├─ Retention: 14 days
├─ Visibility: 5 minutes
├─ Messages: {id, type, agent_id, error, timestamp, ...}
└─ Consumer: (future) metrics replay handler
Files Added:
1. k8s/config/create-kmsvc-topics.sh (executable)
├─ 90 lines
├─ Auto port-forward + retry logic
├─ Color output + error handling
└─ Usage: ./create-kmsvc-topics.sh [manual]
2. k8s/config/kmsvc-topics-job.yaml (Kubernetes)
├─ Job resource (one-time execution)
├─ Uses curl container
├─ Waits for management-service readiness
├─ 30-second retry loop
└─ Non-fatal on existing topics
3. k8s/config/terraform-kmsvc-topics.tf (Terraform)
├─ null_resource with local-exec
├─ Variables for endpoint + namespace
├─ Idempotent + traceable
└─ Outputs: created_topics + test_commands
4. docs/PHASE_6_6_KMSVC_TOPICS.md (Complete Guide)
├─ Table of topics + config
├─ 4 creation methods with examples
├─ Verification commands
├─ Message format specs
├─ Monitoring + alerts
├─ Troubleshooting guide
└─ Next steps checklist
Verification Commands:
✅ List all topics:
curl http://localhost:8080/v1/queues
✅ Check specific topic:
curl http://localhost:8080/v1/queues/poimen-memory-dlq
✅ Send test message:
curl -X POST http://localhost:8080/v1/queues/poimen-memory-dlq/messages -H "Content-Type: application/json" -d '{"body": "{\"type\": \"test\"}"}'
✅ Receive messages:
curl -X POST http://localhost:8080/v1/queues/poimen-memory-dlq/messages/receive -H "Content-Type: application/json" -d '{"maxNumberOfMessages": 10}'
Integration Points:
Phase 6.6 Code → kmsvc Topics:
1. webhook_executor.rs::send_dlq_message()
└─ On max retries: Send to poimen-memory-dlq
└─ Payload: {workflow_id, webhook_url, status, error, ...}
2. metrics_persistence.rs::send_persistence_dlq()
└─ On DB failure: Send to poimen-memory-metric-dlq
└─ Payload: {agent_id, error, timestamp}
3. queue_worker_dlq.rs::DlqHandler
└─ Processes poimen-memory-dlq messages
└─ Retries extraction failures
Production Checklist:
✅ Topics defined (2 topics)
✅ Configuration documented (retention, visibility)
✅ Creation methods (4 options)
✅ Verification commands
✅ Message formats specified
✅ Monitoring guide
✅ Troubleshooting guide
✅ Ready for deployment
Next Steps:
1. Choose creation method (recommend Method 1 for fast testing)
2. Create topics: ./create-kmsvc-topics.sh or kubectl apply job
3. Verify: curl http://localhost:8080/v1/queues
4. Deploy memory service (Phase 6.5)
5. Test webhook + metrics failures send to DLQ
6. Monitor DLQ lag + message rate (Phase 7)
Phase 6.6 Complete: ✅
- Webhook execution: ✅
- Metrics persistence: ✅
- kmsvc DLQ integration: ✅
- Topic creation (4 methods): ✅
- Documentation: ✅
100 lines
2.9 KiB
YAML
100 lines
2.9 KiB
YAML
# Phase 6.6: Kubernetes Job to Create kmsvc Topics
|
|
# This Job runs once to create required Kafka topics
|
|
# Can be re-run if topics need to be recreated
|
|
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: create-poimen-memory-kmsvc-topics
|
|
namespace: poimen
|
|
labels:
|
|
app: poimen-memory
|
|
phase: "6.6"
|
|
spec:
|
|
backoffLimit: 3
|
|
activeDeadlineSeconds: 600
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: poimen-memory
|
|
job: create-kmsvc-topics
|
|
spec:
|
|
serviceAccountName: memory-app
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: create-topics
|
|
image: curlimages/curl:latest
|
|
imagePullPolicy: IfNotPresent
|
|
|
|
command:
|
|
- /bin/sh
|
|
- -c
|
|
- |
|
|
set -e
|
|
|
|
echo "Creating kmsvc topics for Poimen Memory..."
|
|
|
|
# Wait for management-service to be ready
|
|
echo "Waiting for management-service to be ready..."
|
|
for i in {1..30}; do
|
|
if curl -s http://management-service.sqs.svc.cluster.local:8080/v1/queues > /dev/null 2>&1; then
|
|
echo "management-service is ready"
|
|
break
|
|
fi
|
|
echo "Attempt $i/30: Waiting for management-service..."
|
|
sleep 2
|
|
done
|
|
|
|
# Create poimen-memory-dlq
|
|
echo "Creating topic: poimen-memory-dlq"
|
|
curl -X POST http://management-service.sqs.svc.cluster.local:8080/v1/queues \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"name": "poimen-memory-dlq",
|
|
"fifoQueue": false,
|
|
"visibilityTimeoutSeconds": 300,
|
|
"messageRetentionPeriodSeconds": 1209600
|
|
}' || echo "Topic may already exist"
|
|
|
|
# Create poimen-memory-metric-dlq
|
|
echo "Creating topic: poimen-memory-metric-dlq"
|
|
curl -X POST http://management-service.sqs.svc.cluster.local:8080/v1/queues \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"name": "poimen-memory-metric-dlq",
|
|
"fifoQueue": false,
|
|
"visibilityTimeoutSeconds": 300,
|
|
"messageRetentionPeriodSeconds": 1209600
|
|
}' || echo "Topic may already exist"
|
|
|
|
# List queues
|
|
echo "Listing all queues:"
|
|
curl http://management-service.sqs.svc.cluster.local:8080/v1/queues
|
|
|
|
echo "Done!"
|
|
|
|
resources:
|
|
requests:
|
|
cpu: "50m"
|
|
memory: "64Mi"
|
|
limits:
|
|
cpu: "100m"
|
|
memory: "128Mi"
|
|
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
|
|
---
|
|
# ServiceAccount for the job (reuse memory-app SA)
|
|
# If not exists, create it:
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: memory-app
|
|
namespace: poimen
|
|
labels:
|
|
app: poimen-memory
|