homelab/k8s/temporal/PHASE_2_TESTING.md

Phase 2: Namespace-Scoped Auto-Provisioning Testing Guide

Overview

Phase 2 implements namespace-scoped automatic TemporalWorker provisioning (Kafka broker model).

One TemporalWorker per Temporal namespace processes ALL task queues in that namespace. When multiple Queues share the same temporal.io/namespace label, they trigger creation of a single TemporalWorker that handles all of them.

Queues (labeled temporal.io/namespace: "production")
  ├── orders-fifo
  ├── payments
  └── notifications
         ↓
queue-operator creates 1 TemporalWorker (worker-production)
         ↓
TemporalWorker controller creates 1 Deployment
         ↓
Worker pod(s) connect to Temporal namespace "production"
         ↓
Process ALL task queues in that namespace (scale horizontally by replicas)

Implementation Changes

1. TemporalWorker CRD (apis/kmsvc/v1/temporalworker_types.go)

• New Kubernetes resource type to manage namespace-scoped workers
• Fields: Namespace (required), Image, Replicas, Resources, NodeSelector, Affinity, Tolerations
• Status: Phase (Pending/Ready/Failed), Replicas, ReadyReplicas, Conditions
• Model: 1 TemporalWorker per Temporal namespace (not per queue)

2. QueueReconciler Extension (internal/operator/queue_controller.go)

• New method: reconcileTemporalWorker()
• Logic: If Queue has temporal.io/namespace label, create TemporalWorker for that namespace
• Idempotent: multiple queues with same namespace label create same TemporalWorker (no duplicates)

3. TemporalWorkerReconciler (internal/operator/temporal_worker_controller.go)

• New controller watching TemporalWorker objects
• Creates/updates Kubernetes Deployment with:
  • Pod spec: container image, env vars (TEMPORAL_FRONTEND_ADDRESS, TEMPORAL_TASK_QUEUE)
  • Replicas, resources, node selector, affinity, tolerations from TemporalWorker spec
• Updates TemporalWorker status with deployment replica counts and phase

4. Operator Main (cmd/queue-operator/main.go)

• Registers TemporalWorker CRD in scheme
• Registers TemporalWorkerReconciler controller
• Controller watches TemporalWorker objects; owns Deployment objects

Testing Procedure

Prerequisites

• kmsvc queue-operator must be running (built and deployed)
• Temporal cluster must be ready (temporal-frontend service available at temporal-frontend.temporal.svc.cluster.local:7233)
• story-crater-backend Docker image must exist (used as default worker image)

Step 1: Build and Deploy kmsvc Operator

cd /Users/rockliang/workplace/kmsvc-manage
make build                    # builds queue-operator binary
make docker-build            # builds Docker image
make deploy                  # deploys to cluster (requires Helm chart)

Or manually:

cd /Users/rockliang/workplace/kmsvc-manage
go build -o bin/queue-operator ./cmd/queue-operator
kubectl apply -f k8s/queue-operator-rbac.yaml
kubectl apply -f k8s/queue-operator-deployment.yaml

Step 2: Create Queues with Temporal Namespace Labels

kubectl apply -f /Users/rockliang/workplace/homelab/k8s/temporal/queues/example-queue.yaml

Verify Queues are Ready:

kubectl get queue -n sqs -l temporal.io/namespace=production
kubectl describe queue -n sqs story-crater-tasks

Expected:

NAME                          FIFO   PHASE   AGE
story-crater-tasks            false  Ready   5s
story-crater-notifications    false  Ready   5s

Step 3: Verify TemporalWorker CRD Auto-Created (1 per namespace)

kubectl get temporalworker -n temporal
kubectl describe temporalworker -n temporal worker-production

Expected:

NAME                  PHASE     READY   DESIRED   AGE
worker-production     Pending   0       1         5s

Only ONE TemporalWorker for all queues in "production" namespace!

Step 4: Verify Deployment Auto-Created

kubectl get deploy -n temporal -l app.kubernetes.io/managed-by=kmsvc-temporal-operator
kubectl get pods -n temporal -l app.kubernetes.io/instance=worker-production

Expected:

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
worker-production         1/1     1            1           10s

NAME                           READY   STATUS    RESTARTS   AGE
worker-production-5f8b4c...    1/1     Running   0          10s

Step 5: Verify Worker Connected to Temporal Namespace

Check Temporal UI for namespace "production":

open https://temporal.riotpiao.homelab.com/namespaces/production/task-queues

Look for all task queues with worker count > 0:

• story-crater-tasks
• story-crater-notifications
• (worker processes all of them)

Or via CLI:

kubectl port-forward -n temporal svc/temporal-frontend 7233 &
curl http://localhost:7233/api/v1/task-queues?namespace=production

Step 6: Verify TemporalWorker Status Updated

kubectl get temporalworker -n temporal
kubectl describe temporalworker -n temporal worker-production

Expected:

NAME                  PHASE   READY   DESIRED   AGE
worker-production     Ready   1       1         15s

Status:
  Phase: Ready
  Ready Replicas: 1
  Replicas: 1

Step 7: Test Namespace-Level Scaling

Create more queues in the same namespace:

apiVersion: kmsvc.io/v1
kind: Queue
metadata:
  name: story-crater-llm-processing
  namespace: sqs
  labels:
    temporal.io/namespace: "production"  # same namespace

Verify: No new TemporalWorker created (same worker handles all 3 queues):

kubectl get temporalworker -n temporal         # still just 1 worker-production
kubectl get deploy -n temporal worker-production # same deployment

Worker auto-discovers new task queue in namespace and processes it.

Step 8: Test Cascading Deletion

Delete a Queue; worker should remain (other queues still need it):

kubectl delete queue -n sqs story-crater-notifications

Verify:

kubectl get temporalworker -n temporal         # worker-production still exists
kubectl get pods -n temporal worker-production   # still running

Delete all queues in namespace:

kubectl delete queue -n sqs -l temporal.io/namespace=production

Verify: TemporalWorker now has no owner (not cascade-deleted; manual cleanup needed):

kubectl get temporalworker -n temporal    # worker-production still there (manual cleanup)
kubectl delete temporalworker -n temporal worker-production  # cleanup manually

Debugging

Queue stuck in Pending

Check queue-operator logs:

kubectl logs -n sqs deploy/kmsvc-queue-operator -f
kubectl logs -n sqs deploy/kmsvc-queue-operator --tail=50 | grep -i error

TemporalWorker not created

• Verify Queue has the label: kubectl get queue -o yaml | grep temporal.io
• Check queue-operator logs for "reconcileTemporalWorker" errors

Deployment not created

• Check TemporalWorker controller logs: kubectl logs -n sqs deploy/kmsvc-queue-operator -f
• Verify TemporalWorker exists: kubectl get temporalworker -n temporal
• Check Deployment errors: kubectl describe deploy -n temporal worker-story-crater-tasks

Worker not showing in Temporal UI

• Check pod logs: kubectl logs -n temporal deploy/worker-story-crater-tasks
• Verify env vars: kubectl set env pod -n temporal <pod-name> --list | grep TEMPORAL
• Test connectivity: kubectl exec -n temporal <pod-name> -- nc -zv temporal-frontend.temporal.svc.cluster.local 7233

Next Steps

Once Phase 2 is working:

1. Phase 3 (Future): Implement autoscaling based on queue depth metrics
2. Production Hardening:
   • Add QueueRef validation (ensure Queue exists in sqs namespace)
   • Add image validation/defaults from ConfigMap
   • Add worker readiness probe configuration
   • Add graceful shutdown/drain behavior

Files Modified/Created

File	Change
apis/kmsvc/v1/temporalworker_types.go	NEW: CRD type definitions
internal/operator/queue_controller.go	MODIFIED: Added reconcileTemporalWorker()
internal/operator/temporal_worker_controller.go	NEW: TemporalWorker → Deployment reconciler
cmd/queue-operator/main.go	MODIFIED: Register TemporalWorker CRD + controller
k8s/temporal/queues/example-queue.yaml	NEW: Example Queue with label