refactor(k8s): Reorganize into 5-layer structure with production kustomizations
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,237 @@
|
||||
# Phase 2: Namespace-Scoped Auto-Provisioning Testing Guide
|
||||
|
||||
## Overview
|
||||
|
||||
Phase 2 implements **namespace-scoped automatic TemporalWorker provisioning** (Kafka broker model).
|
||||
|
||||
One TemporalWorker per Temporal namespace processes ALL task queues in that namespace. When multiple Queues share the same `temporal.io/namespace` label, they trigger creation of a single TemporalWorker that handles all of them.
|
||||
|
||||
```
|
||||
Queues (labeled temporal.io/namespace: "production")
|
||||
├── orders-fifo
|
||||
├── payments
|
||||
└── notifications
|
||||
↓
|
||||
queue-operator creates 1 TemporalWorker (worker-production)
|
||||
↓
|
||||
TemporalWorker controller creates 1 Deployment
|
||||
↓
|
||||
Worker pod(s) connect to Temporal namespace "production"
|
||||
↓
|
||||
Process ALL task queues in that namespace (scale horizontally by replicas)
|
||||
```
|
||||
|
||||
## Implementation Changes
|
||||
|
||||
### 1. TemporalWorker CRD (`apis/kmsvc/v1/temporalworker_types.go`)
|
||||
- New Kubernetes resource type to manage namespace-scoped workers
|
||||
- Fields: Namespace (required), Image, Replicas, Resources, NodeSelector, Affinity, Tolerations
|
||||
- Status: Phase (Pending/Ready/Failed), Replicas, ReadyReplicas, Conditions
|
||||
- Model: 1 TemporalWorker per Temporal namespace (not per queue)
|
||||
|
||||
### 2. QueueReconciler Extension (`internal/operator/queue_controller.go`)
|
||||
- New method: `reconcileTemporalWorker()`
|
||||
- Logic: If Queue has `temporal.io/namespace` label, create TemporalWorker for that namespace
|
||||
- Idempotent: multiple queues with same namespace label create same TemporalWorker (no duplicates)
|
||||
|
||||
### 3. TemporalWorkerReconciler (`internal/operator/temporal_worker_controller.go`)
|
||||
- New controller watching TemporalWorker objects
|
||||
- Creates/updates Kubernetes Deployment with:
|
||||
- Pod spec: container image, env vars (TEMPORAL_FRONTEND_ADDRESS, TEMPORAL_TASK_QUEUE)
|
||||
- Replicas, resources, node selector, affinity, tolerations from TemporalWorker spec
|
||||
- Updates TemporalWorker status with deployment replica counts and phase
|
||||
|
||||
### 4. Operator Main (`cmd/queue-operator/main.go`)
|
||||
- Registers TemporalWorker CRD in scheme
|
||||
- Registers TemporalWorkerReconciler controller
|
||||
- Controller watches TemporalWorker objects; owns Deployment objects
|
||||
|
||||
## Testing Procedure
|
||||
|
||||
### Prerequisites
|
||||
- kmsvc queue-operator must be running (built and deployed)
|
||||
- Temporal cluster must be ready (temporal-frontend service available at `temporal-frontend.temporal.svc.cluster.local:7233`)
|
||||
- story-crater-backend Docker image must exist (used as default worker image)
|
||||
|
||||
### Step 1: Build and Deploy kmsvc Operator
|
||||
```bash
|
||||
cd /Users/rockliang/workplace/kmsvc-manage
|
||||
make build # builds queue-operator binary
|
||||
make docker-build # builds Docker image
|
||||
make deploy # deploys to cluster (requires Helm chart)
|
||||
```
|
||||
|
||||
Or manually:
|
||||
```bash
|
||||
cd /Users/rockliang/workplace/kmsvc-manage
|
||||
go build -o bin/queue-operator ./cmd/queue-operator
|
||||
kubectl apply -f k8s/queue-operator-rbac.yaml
|
||||
kubectl apply -f k8s/queue-operator-deployment.yaml
|
||||
```
|
||||
|
||||
### Step 2: Create Queues with Temporal Namespace Labels
|
||||
```bash
|
||||
kubectl apply -f /Users/rockliang/workplace/homelab/k8s/temporal/queues/example-queue.yaml
|
||||
```
|
||||
|
||||
Verify Queues are Ready:
|
||||
```bash
|
||||
kubectl get queue -n sqs -l temporal.io/namespace=production
|
||||
kubectl describe queue -n sqs story-crater-tasks
|
||||
```
|
||||
|
||||
Expected:
|
||||
```
|
||||
NAME FIFO PHASE AGE
|
||||
story-crater-tasks false Ready 5s
|
||||
story-crater-notifications false Ready 5s
|
||||
```
|
||||
|
||||
### Step 3: Verify TemporalWorker CRD Auto-Created (1 per namespace)
|
||||
```bash
|
||||
kubectl get temporalworker -n temporal
|
||||
kubectl describe temporalworker -n temporal worker-production
|
||||
```
|
||||
|
||||
Expected:
|
||||
```
|
||||
NAME PHASE READY DESIRED AGE
|
||||
worker-production Pending 0 1 5s
|
||||
```
|
||||
|
||||
Only ONE TemporalWorker for all queues in "production" namespace!
|
||||
|
||||
### Step 4: Verify Deployment Auto-Created
|
||||
```bash
|
||||
kubectl get deploy -n temporal -l app.kubernetes.io/managed-by=kmsvc-temporal-operator
|
||||
kubectl get pods -n temporal -l app.kubernetes.io/instance=worker-production
|
||||
```
|
||||
|
||||
Expected:
|
||||
```
|
||||
NAME READY UP-TO-DATE AVAILABLE AGE
|
||||
worker-production 1/1 1 1 10s
|
||||
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
worker-production-5f8b4c... 1/1 Running 0 10s
|
||||
```
|
||||
|
||||
### Step 5: Verify Worker Connected to Temporal Namespace
|
||||
Check Temporal UI for namespace "production":
|
||||
```bash
|
||||
open https://temporal.riotpiao.homelab.com/namespaces/production/task-queues
|
||||
```
|
||||
|
||||
Look for all task queues with worker count > 0:
|
||||
- `story-crater-tasks`
|
||||
- `story-crater-notifications`
|
||||
- (worker processes all of them)
|
||||
|
||||
Or via CLI:
|
||||
```bash
|
||||
kubectl port-forward -n temporal svc/temporal-frontend 7233 &
|
||||
curl http://localhost:7233/api/v1/task-queues?namespace=production
|
||||
```
|
||||
|
||||
### Step 6: Verify TemporalWorker Status Updated
|
||||
```bash
|
||||
kubectl get temporalworker -n temporal
|
||||
kubectl describe temporalworker -n temporal worker-production
|
||||
```
|
||||
|
||||
Expected:
|
||||
```
|
||||
NAME PHASE READY DESIRED AGE
|
||||
worker-production Ready 1 1 15s
|
||||
|
||||
Status:
|
||||
Phase: Ready
|
||||
Ready Replicas: 1
|
||||
Replicas: 1
|
||||
```
|
||||
|
||||
### Step 7: Test Namespace-Level Scaling
|
||||
Create more queues in the same namespace:
|
||||
```yaml
|
||||
apiVersion: kmsvc.io/v1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: story-crater-llm-processing
|
||||
namespace: sqs
|
||||
labels:
|
||||
temporal.io/namespace: "production" # same namespace
|
||||
```
|
||||
|
||||
Verify: No new TemporalWorker created (same worker handles all 3 queues):
|
||||
```bash
|
||||
kubectl get temporalworker -n temporal # still just 1 worker-production
|
||||
kubectl get deploy -n temporal worker-production # same deployment
|
||||
```
|
||||
|
||||
Worker auto-discovers new task queue in namespace and processes it.
|
||||
|
||||
### Step 8: Test Cascading Deletion
|
||||
Delete a Queue; worker should remain (other queues still need it):
|
||||
```bash
|
||||
kubectl delete queue -n sqs story-crater-notifications
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
kubectl get temporalworker -n temporal # worker-production still exists
|
||||
kubectl get pods -n temporal worker-production # still running
|
||||
```
|
||||
|
||||
Delete all queues in namespace:
|
||||
```bash
|
||||
kubectl delete queue -n sqs -l temporal.io/namespace=production
|
||||
```
|
||||
|
||||
Verify: TemporalWorker now has no owner (not cascade-deleted; manual cleanup needed):
|
||||
```bash
|
||||
kubectl get temporalworker -n temporal # worker-production still there (manual cleanup)
|
||||
kubectl delete temporalworker -n temporal worker-production # cleanup manually
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
### Queue stuck in Pending
|
||||
Check queue-operator logs:
|
||||
```bash
|
||||
kubectl logs -n sqs deploy/kmsvc-queue-operator -f
|
||||
kubectl logs -n sqs deploy/kmsvc-queue-operator --tail=50 | grep -i error
|
||||
```
|
||||
|
||||
### TemporalWorker not created
|
||||
- Verify Queue has the label: `kubectl get queue -o yaml | grep temporal.io`
|
||||
- Check queue-operator logs for "reconcileTemporalWorker" errors
|
||||
|
||||
### Deployment not created
|
||||
- Check TemporalWorker controller logs: `kubectl logs -n sqs deploy/kmsvc-queue-operator -f`
|
||||
- Verify TemporalWorker exists: `kubectl get temporalworker -n temporal`
|
||||
- Check Deployment errors: `kubectl describe deploy -n temporal worker-story-crater-tasks`
|
||||
|
||||
### Worker not showing in Temporal UI
|
||||
- Check pod logs: `kubectl logs -n temporal deploy/worker-story-crater-tasks`
|
||||
- Verify env vars: `kubectl set env pod -n temporal <pod-name> --list | grep TEMPORAL`
|
||||
- Test connectivity: `kubectl exec -n temporal <pod-name> -- nc -zv temporal-frontend.temporal.svc.cluster.local 7233`
|
||||
|
||||
## Next Steps
|
||||
|
||||
Once Phase 2 is working:
|
||||
1. **Phase 3 (Future):** Implement autoscaling based on queue depth metrics
|
||||
2. **Production Hardening:**
|
||||
- Add QueueRef validation (ensure Queue exists in sqs namespace)
|
||||
- Add image validation/defaults from ConfigMap
|
||||
- Add worker readiness probe configuration
|
||||
- Add graceful shutdown/drain behavior
|
||||
|
||||
## Files Modified/Created
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `apis/kmsvc/v1/temporalworker_types.go` | NEW: CRD type definitions |
|
||||
| `internal/operator/queue_controller.go` | MODIFIED: Added reconcileTemporalWorker() |
|
||||
| `internal/operator/temporal_worker_controller.go` | NEW: TemporalWorker → Deployment reconciler |
|
||||
| `cmd/queue-operator/main.go` | MODIFIED: Register TemporalWorker CRD + controller |
|
||||
| `k8s/temporal/queues/example-queue.yaml` | NEW: Example Queue with label |
|
||||
Reference in New Issue
Block a user