31aafa53e4a899bd81e69582a9c87f50af9e246b
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 rock
|
31aafa53e4 |
Phase 6.6: Add kmsvc Topic Creation (4 methods)
Topic Creation Methods:
1. Manual CLI (Fastest - 2 min):
└─ kubectl port-forward + curl POST /v1/queues
└─ k8s/config/create-kmsvc-topics.sh (interactive)
2. Kubernetes Job (Automated - 1 min):
└─ kubectl apply kmsvc-topics-job.yaml
└─ Runs once, creates topics if not exist
└─ Can re-run safely
3. Terraform (IaC - 2 min):
└─ terraform apply -target=null_resource.create_kmsvc_topics
└─ Tracks topic creation in .tfstate
└─ Idempotent
4. Shell Script (Interactive - 1 min):
└─ ./create-kmsvc-topics.sh
└─ Auto port-forward or manual mode
└─ Color output + progress logging
Topics Created:
1. poimen-memory-dlq (DLQ for extraction + webhook + agent)
├─ Retention: 14 days (1,209,600 seconds)
├─ Visibility: 5 minutes (300 seconds)
├─ Messages: {id, type, workflow_id, error, timestamp, ...}
└─ Consumer: queue_worker_dlq.rs::DlqHandler
2. poimen-memory-metric-dlq (DLQ for metrics failures)
├─ Retention: 14 days
├─ Visibility: 5 minutes
├─ Messages: {id, type, agent_id, error, timestamp, ...}
└─ Consumer: (future) metrics replay handler
Files Added:
1. k8s/config/create-kmsvc-topics.sh (executable)
├─ 90 lines
├─ Auto port-forward + retry logic
├─ Color output + error handling
└─ Usage: ./create-kmsvc-topics.sh [manual]
2. k8s/config/kmsvc-topics-job.yaml (Kubernetes)
├─ Job resource (one-time execution)
├─ Uses curl container
├─ Waits for management-service readiness
├─ 30-second retry loop
└─ Non-fatal on existing topics
3. k8s/config/terraform-kmsvc-topics.tf (Terraform)
├─ null_resource with local-exec
├─ Variables for endpoint + namespace
├─ Idempotent + traceable
└─ Outputs: created_topics + test_commands
4. docs/PHASE_6_6_KMSVC_TOPICS.md (Complete Guide)
├─ Table of topics + config
├─ 4 creation methods with examples
├─ Verification commands
├─ Message format specs
├─ Monitoring + alerts
├─ Troubleshooting guide
└─ Next steps checklist
Verification Commands:
✅ List all topics:
curl http://localhost:8080/v1/queues
✅ Check specific topic:
curl http://localhost:8080/v1/queues/poimen-memory-dlq
✅ Send test message:
curl -X POST http://localhost:8080/v1/queues/poimen-memory-dlq/messages -H "Content-Type: application/json" -d '{"body": "{\"type\": \"test\"}"}'
✅ Receive messages:
curl -X POST http://localhost:8080/v1/queues/poimen-memory-dlq/messages/receive -H "Content-Type: application/json" -d '{"maxNumberOfMessages": 10}'
Integration Points:
Phase 6.6 Code → kmsvc Topics:
1. webhook_executor.rs::send_dlq_message()
└─ On max retries: Send to poimen-memory-dlq
└─ Payload: {workflow_id, webhook_url, status, error, ...}
2. metrics_persistence.rs::send_persistence_dlq()
└─ On DB failure: Send to poimen-memory-metric-dlq
└─ Payload: {agent_id, error, timestamp}
3. queue_worker_dlq.rs::DlqHandler
└─ Processes poimen-memory-dlq messages
└─ Retries extraction failures
Production Checklist:
✅ Topics defined (2 topics)
✅ Configuration documented (retention, visibility)
✅ Creation methods (4 options)
✅ Verification commands
✅ Message formats specified
✅ Monitoring guide
✅ Troubleshooting guide
✅ Ready for deployment
Next Steps:
1. Choose creation method (recommend Method 1 for fast testing)
2. Create topics: ./create-kmsvc-topics.sh or kubectl apply job
3. Verify: curl http://localhost:8080/v1/queues
4. Deploy memory service (Phase 6.5)
5. Test webhook + metrics failures send to DLQ
6. Monitor DLQ lag + message rate (Phase 7)
Phase 6.6 Complete: ✅
- Webhook execution: ✅
- Metrics persistence: ✅
- kmsvc DLQ integration: ✅
- Topic creation (4 methods): ✅
- Documentation: ✅
|