k8s/messaging: add kafka kmsvc and temporal workflows
- Kafka 3-broker cluster (RF=3, min-ISR=2) - kmsvc SQS-like API on Kafka - Redis dedup (standalone, can extend to HA) - Temporal workflow orchestration (Cassandra backend)
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,237 @@
|
||||
# Phase 2: Namespace-Scoped Auto-Provisioning Testing Guide
|
||||
|
||||
## Overview
|
||||
|
||||
Phase 2 implements **namespace-scoped automatic TemporalWorker provisioning** (Kafka broker model).
|
||||
|
||||
One TemporalWorker per Temporal namespace processes ALL task queues in that namespace. When multiple Queues share the same `temporal.io/namespace` label, they trigger creation of a single TemporalWorker that handles all of them.
|
||||
|
||||
```
|
||||
Queues (labeled temporal.io/namespace: "production")
|
||||
├── orders-fifo
|
||||
├── payments
|
||||
└── notifications
|
||||
↓
|
||||
queue-operator creates 1 TemporalWorker (worker-production)
|
||||
↓
|
||||
TemporalWorker controller creates 1 Deployment
|
||||
↓
|
||||
Worker pod(s) connect to Temporal namespace "production"
|
||||
↓
|
||||
Process ALL task queues in that namespace (scale horizontally by replicas)
|
||||
```
|
||||
|
||||
## Implementation Changes
|
||||
|
||||
### 1. TemporalWorker CRD (`apis/kmsvc/v1/temporalworker_types.go`)
|
||||
- New Kubernetes resource type to manage namespace-scoped workers
|
||||
- Fields: Namespace (required), Image, Replicas, Resources, NodeSelector, Affinity, Tolerations
|
||||
- Status: Phase (Pending/Ready/Failed), Replicas, ReadyReplicas, Conditions
|
||||
- Model: 1 TemporalWorker per Temporal namespace (not per queue)
|
||||
|
||||
### 2. QueueReconciler Extension (`internal/operator/queue_controller.go`)
|
||||
- New method: `reconcileTemporalWorker()`
|
||||
- Logic: If Queue has `temporal.io/namespace` label, create TemporalWorker for that namespace
|
||||
- Idempotent: multiple queues with same namespace label create same TemporalWorker (no duplicates)
|
||||
|
||||
### 3. TemporalWorkerReconciler (`internal/operator/temporal_worker_controller.go`)
|
||||
- New controller watching TemporalWorker objects
|
||||
- Creates/updates Kubernetes Deployment with:
|
||||
- Pod spec: container image, env vars (TEMPORAL_FRONTEND_ADDRESS, TEMPORAL_TASK_QUEUE)
|
||||
- Replicas, resources, node selector, affinity, tolerations from TemporalWorker spec
|
||||
- Updates TemporalWorker status with deployment replica counts and phase
|
||||
|
||||
### 4. Operator Main (`cmd/queue-operator/main.go`)
|
||||
- Registers TemporalWorker CRD in scheme
|
||||
- Registers TemporalWorkerReconciler controller
|
||||
- Controller watches TemporalWorker objects; owns Deployment objects
|
||||
|
||||
## Testing Procedure
|
||||
|
||||
### Prerequisites
|
||||
- kmsvc queue-operator must be running (built and deployed)
|
||||
- Temporal cluster must be ready (temporal-frontend service available at `temporal-frontend.temporal.svc.cluster.local:7233`)
|
||||
- story-crater-backend Docker image must exist (used as default worker image)
|
||||
|
||||
### Step 1: Build and Deploy kmsvc Operator
|
||||
```bash
|
||||
cd /Users/rockliang/workplace/kmsvc-manage
|
||||
make build # builds queue-operator binary
|
||||
make docker-build # builds Docker image
|
||||
make deploy # deploys to cluster (requires Helm chart)
|
||||
```
|
||||
|
||||
Or manually:
|
||||
```bash
|
||||
cd /Users/rockliang/workplace/kmsvc-manage
|
||||
go build -o bin/queue-operator ./cmd/queue-operator
|
||||
kubectl apply -f k8s/queue-operator-rbac.yaml
|
||||
kubectl apply -f k8s/queue-operator-deployment.yaml
|
||||
```
|
||||
|
||||
### Step 2: Create Queues with Temporal Namespace Labels
|
||||
```bash
|
||||
kubectl apply -f /Users/rockliang/workplace/homelab/k8s/temporal/queues/example-queue.yaml
|
||||
```
|
||||
|
||||
Verify Queues are Ready:
|
||||
```bash
|
||||
kubectl get queue -n sqs -l temporal.io/namespace=production
|
||||
kubectl describe queue -n sqs story-crater-tasks
|
||||
```
|
||||
|
||||
Expected:
|
||||
```
|
||||
NAME FIFO PHASE AGE
|
||||
story-crater-tasks false Ready 5s
|
||||
story-crater-notifications false Ready 5s
|
||||
```
|
||||
|
||||
### Step 3: Verify TemporalWorker CRD Auto-Created (1 per namespace)
|
||||
```bash
|
||||
kubectl get temporalworker -n temporal
|
||||
kubectl describe temporalworker -n temporal worker-production
|
||||
```
|
||||
|
||||
Expected:
|
||||
```
|
||||
NAME PHASE READY DESIRED AGE
|
||||
worker-production Pending 0 1 5s
|
||||
```
|
||||
|
||||
Only ONE TemporalWorker for all queues in "production" namespace!
|
||||
|
||||
### Step 4: Verify Deployment Auto-Created
|
||||
```bash
|
||||
kubectl get deploy -n temporal -l app.kubernetes.io/managed-by=kmsvc-temporal-operator
|
||||
kubectl get pods -n temporal -l app.kubernetes.io/instance=worker-production
|
||||
```
|
||||
|
||||
Expected:
|
||||
```
|
||||
NAME READY UP-TO-DATE AVAILABLE AGE
|
||||
worker-production 1/1 1 1 10s
|
||||
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
worker-production-5f8b4c... 1/1 Running 0 10s
|
||||
```
|
||||
|
||||
### Step 5: Verify Worker Connected to Temporal Namespace
|
||||
Check Temporal UI for namespace "production":
|
||||
```bash
|
||||
open https://temporal.riotpiao.homelab.com/namespaces/production/task-queues
|
||||
```
|
||||
|
||||
Look for all task queues with worker count > 0:
|
||||
- `story-crater-tasks`
|
||||
- `story-crater-notifications`
|
||||
- (worker processes all of them)
|
||||
|
||||
Or via CLI:
|
||||
```bash
|
||||
kubectl port-forward -n temporal svc/temporal-frontend 7233 &
|
||||
curl http://localhost:7233/api/v1/task-queues?namespace=production
|
||||
```
|
||||
|
||||
### Step 6: Verify TemporalWorker Status Updated
|
||||
```bash
|
||||
kubectl get temporalworker -n temporal
|
||||
kubectl describe temporalworker -n temporal worker-production
|
||||
```
|
||||
|
||||
Expected:
|
||||
```
|
||||
NAME PHASE READY DESIRED AGE
|
||||
worker-production Ready 1 1 15s
|
||||
|
||||
Status:
|
||||
Phase: Ready
|
||||
Ready Replicas: 1
|
||||
Replicas: 1
|
||||
```
|
||||
|
||||
### Step 7: Test Namespace-Level Scaling
|
||||
Create more queues in the same namespace:
|
||||
```yaml
|
||||
apiVersion: kmsvc.io/v1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: story-crater-llm-processing
|
||||
namespace: sqs
|
||||
labels:
|
||||
temporal.io/namespace: "production" # same namespace
|
||||
```
|
||||
|
||||
Verify: No new TemporalWorker created (same worker handles all 3 queues):
|
||||
```bash
|
||||
kubectl get temporalworker -n temporal # still just 1 worker-production
|
||||
kubectl get deploy -n temporal worker-production # same deployment
|
||||
```
|
||||
|
||||
Worker auto-discovers new task queue in namespace and processes it.
|
||||
|
||||
### Step 8: Test Cascading Deletion
|
||||
Delete a Queue; worker should remain (other queues still need it):
|
||||
```bash
|
||||
kubectl delete queue -n sqs story-crater-notifications
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
kubectl get temporalworker -n temporal # worker-production still exists
|
||||
kubectl get pods -n temporal worker-production # still running
|
||||
```
|
||||
|
||||
Delete all queues in namespace:
|
||||
```bash
|
||||
kubectl delete queue -n sqs -l temporal.io/namespace=production
|
||||
```
|
||||
|
||||
Verify: TemporalWorker now has no owner (not cascade-deleted; manual cleanup needed):
|
||||
```bash
|
||||
kubectl get temporalworker -n temporal # worker-production still there (manual cleanup)
|
||||
kubectl delete temporalworker -n temporal worker-production # cleanup manually
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
### Queue stuck in Pending
|
||||
Check queue-operator logs:
|
||||
```bash
|
||||
kubectl logs -n sqs deploy/kmsvc-queue-operator -f
|
||||
kubectl logs -n sqs deploy/kmsvc-queue-operator --tail=50 | grep -i error
|
||||
```
|
||||
|
||||
### TemporalWorker not created
|
||||
- Verify Queue has the label: `kubectl get queue -o yaml | grep temporal.io`
|
||||
- Check queue-operator logs for "reconcileTemporalWorker" errors
|
||||
|
||||
### Deployment not created
|
||||
- Check TemporalWorker controller logs: `kubectl logs -n sqs deploy/kmsvc-queue-operator -f`
|
||||
- Verify TemporalWorker exists: `kubectl get temporalworker -n temporal`
|
||||
- Check Deployment errors: `kubectl describe deploy -n temporal worker-story-crater-tasks`
|
||||
|
||||
### Worker not showing in Temporal UI
|
||||
- Check pod logs: `kubectl logs -n temporal deploy/worker-story-crater-tasks`
|
||||
- Verify env vars: `kubectl set env pod -n temporal <pod-name> --list | grep TEMPORAL`
|
||||
- Test connectivity: `kubectl exec -n temporal <pod-name> -- nc -zv temporal-frontend.temporal.svc.cluster.local 7233`
|
||||
|
||||
## Next Steps
|
||||
|
||||
Once Phase 2 is working:
|
||||
1. **Phase 3 (Future):** Implement autoscaling based on queue depth metrics
|
||||
2. **Production Hardening:**
|
||||
- Add QueueRef validation (ensure Queue exists in sqs namespace)
|
||||
- Add image validation/defaults from ConfigMap
|
||||
- Add worker readiness probe configuration
|
||||
- Add graceful shutdown/drain behavior
|
||||
|
||||
## Files Modified/Created
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `apis/kmsvc/v1/temporalworker_types.go` | NEW: CRD type definitions |
|
||||
| `internal/operator/queue_controller.go` | MODIFIED: Added reconcileTemporalWorker() |
|
||||
| `internal/operator/temporal_worker_controller.go` | NEW: TemporalWorker → Deployment reconciler |
|
||||
| `cmd/queue-operator/main.go` | MODIFIED: Register TemporalWorker CRD + controller |
|
||||
| `k8s/temporal/queues/example-queue.yaml` | NEW: Example Queue with label |
|
||||
@@ -0,0 +1,220 @@
|
||||
# Temporal OAuth2-Proxy Setup (Authentik OIDC)
|
||||
|
||||
## Overview
|
||||
|
||||
Protects Temporal UI with Authentik OIDC authentication. Traffic flow:
|
||||
|
||||
```
|
||||
Browser → Ingress (TLS) → oauth2-proxy (OIDC check) → temporal-web (internal)
|
||||
↓
|
||||
Redirects to Authentik login
|
||||
↓
|
||||
JWT cookie issued
|
||||
↓
|
||||
Forwards to temporal-web
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
✅ Authentik OIDC provider `temporal` already exists with:
|
||||
- Client ID: `temporal`
|
||||
- Client Secret: stored in Kubernetes secret `temporal-oidc` (key: `clientSecret`)
|
||||
- Redirect URI: `https://temporal.riotpiao.homelab.com/oauth2/callback`
|
||||
|
||||
## Secrets
|
||||
|
||||
The `temporal-oidc` secret must contain:
|
||||
|
||||
| Key | Value | Source |
|
||||
|-----|-------|--------|
|
||||
| `clientSecret` | OAuth2 client secret from Authentik | Authentik → Applications → temporal |
|
||||
| `cookieSecret` | Session encryption key (base64 32-byte) | Generate: `openssl rand -base64 32` |
|
||||
|
||||
### Check existing secret:
|
||||
|
||||
```bash
|
||||
kubectl get secret -n temporal temporal-oidc
|
||||
kubectl describe secret -n temporal temporal-oidc
|
||||
```
|
||||
|
||||
### If missing, create it:
|
||||
|
||||
```bash
|
||||
# Get client secret from Authentik UI
|
||||
# Applications → temporal → copy "Client Secret"
|
||||
CLIENT_SECRET="..."
|
||||
|
||||
# Generate cookie secret
|
||||
COOKIE_SECRET=$(openssl rand -base64 32)
|
||||
|
||||
# Create secret
|
||||
kubectl create secret generic temporal-oidc \
|
||||
-n temporal \
|
||||
--from-literal=clientSecret="${CLIENT_SECRET}" \
|
||||
--from-literal=cookieSecret="${COOKIE_SECRET}"
|
||||
```
|
||||
|
||||
## Deployment Steps
|
||||
|
||||
### Step 1: Apply OAuth2-Proxy Manifests
|
||||
```bash
|
||||
kubectl apply -f k8s/temporal/oauth2-proxy.yaml
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
kubectl get deploy -n temporal oauth2-proxy
|
||||
kubectl logs -n temporal deploy/oauth2-proxy
|
||||
```
|
||||
|
||||
Expected log:
|
||||
```
|
||||
[<timestamp>] [oauthproxy.go:...] Listening on 0.0.0.0:4180
|
||||
```
|
||||
|
||||
### Step 2: Apply OAuth2-Proxy Ingress
|
||||
```bash
|
||||
kubectl apply -f k8s/temporal/temporal-ingress-oauth2.yaml
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
kubectl get ingress -n temporal
|
||||
```
|
||||
|
||||
Expected:
|
||||
```
|
||||
NAME CLASS HOSTS ADDRESS PORTS AGE
|
||||
temporal nginx temporal.riotpiao.homelab.com ... 80, 443 10s
|
||||
```
|
||||
|
||||
### Step 3: Test Access
|
||||
|
||||
1. **Open Temporal UI (unauthenticated):**
|
||||
```bash
|
||||
open https://temporal.riotpiao.homelab.com
|
||||
```
|
||||
|
||||
Expected: Redirects to Authentik login page
|
||||
|
||||
2. **Login with Authentik credentials**
|
||||
- Username/email
|
||||
- Password
|
||||
- Should redirect back to `temporal.riotpiao.homelab.com` and display UI
|
||||
|
||||
3. **Verify auth:**
|
||||
```bash
|
||||
# Check for oauth2_proxy cookie
|
||||
curl -v https://temporal.riotpiao.homelab.com 2>&1 | grep -i cookie
|
||||
```
|
||||
|
||||
4. **Check oauth2-proxy logs:**
|
||||
```bash
|
||||
kubectl logs -n temporal deploy/oauth2-proxy -f
|
||||
```
|
||||
|
||||
Look for:
|
||||
```
|
||||
[timestamp] [auth_test.go:...] Authentication successful
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Redirect URI mismatch
|
||||
Error in oauth2-proxy logs:
|
||||
```
|
||||
redirect_uri_mismatch: The redirect_uri does not match the one registered in Authentik
|
||||
```
|
||||
|
||||
Fix:
|
||||
- Verify Authentik application (Applications → temporal) has redirect URI: `https://temporal.riotpiao.homelab.com/oauth2/callback`
|
||||
- Ensure HTTPS (not HTTP)
|
||||
|
||||
### Missing secret
|
||||
Error:
|
||||
```
|
||||
clientSecret: key not found in temporal-oidc secret
|
||||
```
|
||||
|
||||
Fix:
|
||||
```bash
|
||||
kubectl get secret -n temporal temporal-oidc -o yaml
|
||||
# If missing, create per "Secrets" section above
|
||||
```
|
||||
|
||||
### Cookie secret expiration
|
||||
OAuth2-Proxy won't start if `cookieSecret` is empty or invalid.
|
||||
|
||||
Fix:
|
||||
```bash
|
||||
COOKIE_SECRET=$(openssl rand -base64 32)
|
||||
kubectl patch secret temporal-oidc -n temporal \
|
||||
-p "{\"data\":{\"cookieSecret\":\"$(echo -n $COOKIE_SECRET | base64)\"}}}"
|
||||
kubectl rollout restart deploy/oauth2-proxy -n temporal
|
||||
```
|
||||
|
||||
### oauth2-proxy crashes with "connection refused"
|
||||
Error in logs:
|
||||
```
|
||||
upstream connect error or disconnect/reset before headers
|
||||
```
|
||||
|
||||
Likely cause: `temporal-web` service not accessible.
|
||||
|
||||
Check:
|
||||
```bash
|
||||
kubectl get svc -n temporal temporal-web
|
||||
kubectl exec -n temporal deploy/oauth2-proxy -- curl http://temporal-web:8080
|
||||
```
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
k8s/temporal/
|
||||
├── oauth2-proxy.yaml # oauth2-proxy Deployment + Service + SA
|
||||
├── temporal-ingress-oauth2.yaml # Ingress routing to oauth2-proxy
|
||||
├── oauth2-proxy-values.yaml # Helm values (reference only)
|
||||
└── temporal-values.yaml # Modified: ingress.enabled=false
|
||||
```
|
||||
|
||||
## Next: Add to Helmfile
|
||||
|
||||
If integrating with helmfile.yaml.gotmpl:
|
||||
|
||||
```yaml
|
||||
releases:
|
||||
- name: temporal
|
||||
# ... existing config ...
|
||||
hooks:
|
||||
postSync:
|
||||
- events: ["success"]
|
||||
showlogs: true
|
||||
command: "sh"
|
||||
args:
|
||||
- -c
|
||||
- |
|
||||
kubectl apply -f k8s/temporal/oauth2-proxy.yaml
|
||||
kubectl apply -f k8s/temporal/temporal-ingress-oauth2.yaml
|
||||
```
|
||||
|
||||
Or add separate releases:
|
||||
|
||||
```yaml
|
||||
- name: oauth2-proxy-temporal
|
||||
namespace: temporal
|
||||
chart: oauth2-proxy/oauth2-proxy
|
||||
version: "6.x.x"
|
||||
values:
|
||||
- k8s/temporal/oauth2-proxy-values.yaml
|
||||
set:
|
||||
- name: config.clientSecret
|
||||
value: "{{ (env "TEMPORAL_OIDC_CLIENT_SECRET") }}"
|
||||
- name: config.cookieSecret
|
||||
value: "{{ (env "TEMPORAL_OIDC_COOKIE_SECRET") }}"
|
||||
```
|
||||
|
||||
Then add to `.env`:
|
||||
```bash
|
||||
TEMPORAL_OIDC_CLIENT_SECRET=<from Authentik>
|
||||
TEMPORAL_OIDC_COOKIE_SECRET=$(openssl rand -base64 32)
|
||||
```
|
||||
@@ -0,0 +1,102 @@
|
||||
# Elasticsearch 7.17.0 for Temporal visibility store
|
||||
# Deployed to worker nodes (not control plane to save CP resources for LLM work)
|
||||
# 2Gi heap + 4Gi memory limit for stable operation
|
||||
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: elasticsearch-config
|
||||
namespace: temporal
|
||||
data:
|
||||
elasticsearch.yml: |
|
||||
cluster.name: temporal-elasticsearch
|
||||
node.name: temporal-elasticsearch-0
|
||||
discovery.type: single-node
|
||||
network.host: 0.0.0.0
|
||||
http.host: 0.0.0.0
|
||||
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: temporal-elasticsearch
|
||||
namespace: temporal
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: temporal-elasticsearch
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: temporal-elasticsearch
|
||||
spec:
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: node-role.kubernetes.io/worker
|
||||
operator: Exists
|
||||
containers:
|
||||
- name: elasticsearch
|
||||
image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0
|
||||
env:
|
||||
- name: discovery.type
|
||||
value: single-node
|
||||
- name: "ES_JAVA_OPTS"
|
||||
value: "-Xms2g -Xmx2g"
|
||||
ports:
|
||||
- containerPort: 9200
|
||||
name: http
|
||||
- containerPort: 9300
|
||||
name: transport
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /_cluster/health
|
||||
port: 9200
|
||||
initialDelaySeconds: 180
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 10
|
||||
failureThreshold: 5
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /_cluster/health?local=true
|
||||
port: 9200
|
||||
initialDelaySeconds: 150
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 10
|
||||
failureThreshold: 5
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 2Gi
|
||||
limits:
|
||||
cpu: 2000m
|
||||
memory: 4Gi
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
|
||||
subPath: elasticsearch.yml
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: elasticsearch-config
|
||||
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: temporal-elasticsearch
|
||||
namespace: temporal
|
||||
spec:
|
||||
selector:
|
||||
app: temporal-elasticsearch
|
||||
ports:
|
||||
- port: 9200
|
||||
targetPort: 9200
|
||||
name: http
|
||||
- port: 9300
|
||||
targetPort: 9300
|
||||
name: transport
|
||||
type: ClusterIP
|
||||
@@ -0,0 +1,45 @@
|
||||
# OAuth2-Proxy for Temporal UI — protects with Authentik OIDC
|
||||
# Deployed via Helm: oauth2-proxy/oauth2-proxy chart
|
||||
|
||||
config:
|
||||
clientID: temporal
|
||||
clientSecret: "" # injected from temporal-oidc secret
|
||||
cookieSecret: "" # generated; helm --set will override
|
||||
configFile: ""
|
||||
|
||||
auth:
|
||||
enabled: true
|
||||
|
||||
extraArgs:
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://authentik.riotpiao.homelab.com/application/o/temporal/
|
||||
- --redirect-url=https://temporal.riotpiao.homelab.com/oauth2/callback
|
||||
- --upstream=http://temporal-web:8080
|
||||
- --cookie-secure=true
|
||||
- --cookie-httponly=true
|
||||
- --cookie-samesite=Lax
|
||||
- --email-domain=*
|
||||
- --skip-auth-regex=^/health
|
||||
- --pass-authorization-header=true
|
||||
- --skip-auth-preflight=true
|
||||
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
|
||||
ingress:
|
||||
enabled: false # we'll keep temporal's ingress, just route to oauth2-proxy
|
||||
|
||||
replicaCount: 1
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 256Mi
|
||||
|
||||
podAnnotations:
|
||||
secret.reloader.stakater.com/reload: "temporal-oidc"
|
||||
@@ -0,0 +1,108 @@
|
||||
# OAuth2-Proxy deployment for Temporal UI
|
||||
# Requires: temporal-oidc secret with clientSecret and cookieSecret
|
||||
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: temporal
|
||||
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: temporal
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: oauth2-proxy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: oauth2-proxy
|
||||
annotations:
|
||||
secret.reloader.stakater.com/reload: "temporal-oidc"
|
||||
spec:
|
||||
serviceAccountName: oauth2-proxy
|
||||
containers:
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.5.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
env:
|
||||
- name: OAUTH2_PROXY_PROVIDER
|
||||
value: "oidc"
|
||||
- name: OAUTH2_PROXY_OIDC_ISSUER_URL
|
||||
value: "https://authentik.riotpiao.homelab.com/application/o/temporal/"
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
value: "temporal"
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: temporal-oidc
|
||||
key: clientSecret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: temporal-oidc
|
||||
key: cookieSecret
|
||||
- name: OAUTH2_PROXY_REDIRECT_URL
|
||||
value: "https://temporal.riotpiao.homelab.com/oauth2/callback"
|
||||
- name: OAUTH2_PROXY_UPSTREAM
|
||||
value: "http://temporal-web:8080"
|
||||
- name: OAUTH2_PROXY_COOKIE_SECURE
|
||||
value: "true"
|
||||
- name: OAUTH2_PROXY_COOKIE_HTTPONLY
|
||||
value: "true"
|
||||
- name: OAUTH2_PROXY_COOKIE_SAMESITE
|
||||
value: "Lax"
|
||||
- name: OAUTH2_PROXY_EMAIL_DOMAIN
|
||||
value: "*"
|
||||
- name: OAUTH2_PROXY_SKIP_AUTH_REGEX
|
||||
value: "^/health"
|
||||
- name: OAUTH2_PROXY_PASS_AUTHORIZATION_HEADER
|
||||
value: "true"
|
||||
- name: OAUTH2_PROXY_SKIP_AUTH_PREFLIGHT
|
||||
value: "true"
|
||||
- name: OAUTH2_PROXY_REVERSE_PROXY
|
||||
value: "true"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 256Mi
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: temporal
|
||||
spec:
|
||||
type: ClusterIP
|
||||
ports:
|
||||
- port: 4180
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
name: http
|
||||
selector:
|
||||
app: oauth2-proxy
|
||||
@@ -0,0 +1,38 @@
|
||||
# Example Queues for "production" Temporal namespace
|
||||
# When applied, queue-operator creates 1 TemporalWorker (worker-production)
|
||||
# that processes ALL queues in the "production" namespace
|
||||
---
|
||||
apiVersion: kmsvc.io/v1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: story-crater-tasks
|
||||
namespace: sqs
|
||||
labels:
|
||||
temporal.io/namespace: "production"
|
||||
spec:
|
||||
fifoQueue: false
|
||||
visibilityTimeoutSeconds: 30
|
||||
messageRetentionPeriodSeconds: 345600
|
||||
maxReceiveCount: 5
|
||||
partitionsPerShard: 6
|
||||
minShards: 1
|
||||
maxShards: 8
|
||||
shardSplitThresholdBytesPerSec: 5242880
|
||||
|
||||
---
|
||||
apiVersion: kmsvc.io/v1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: story-crater-notifications
|
||||
namespace: sqs
|
||||
labels:
|
||||
temporal.io/namespace: "production"
|
||||
spec:
|
||||
fifoQueue: false
|
||||
visibilityTimeoutSeconds: 60
|
||||
messageRetentionPeriodSeconds: 345600
|
||||
maxReceiveCount: 3
|
||||
partitionsPerShard: 3
|
||||
minShards: 1
|
||||
maxShards: 4
|
||||
shardSplitThresholdBytesPerSec: 2621440
|
||||
@@ -0,0 +1,27 @@
|
||||
# Ingress for Temporal UI — routes to OAuth2-Proxy, which proxies to temporal-web
|
||||
# TLS terminated here; oauth2-proxy handles OIDC auth before forwarding to backend
|
||||
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: temporal
|
||||
namespace: temporal
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: "letsencrypt-prod"
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
tls:
|
||||
- secretName: temporal-tls
|
||||
hosts:
|
||||
- temporal.riotpiao.homelab.com
|
||||
rules:
|
||||
- host: temporal.riotpiao.homelab.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: oauth2-proxy
|
||||
port:
|
||||
number: 4180
|
||||
@@ -0,0 +1,85 @@
|
||||
# k8s/temporal/temporal-values.yaml
|
||||
# Temporal — workflow engine for story-crater backend async task orchestration.
|
||||
# Chart: temporal/temporal from https://go.temporal.io/helm-charts
|
||||
#
|
||||
# Uses Cassandra for default store (workflow history/events)
|
||||
# Uses Elasticsearch for visibility store (namespace/workflow queries)
|
||||
# This is the chart's native, well-tested configuration.
|
||||
|
||||
# ── Datastores configuration ────
|
||||
# Disable auto-deployed PostgreSQL (we use external ddb for other services)
|
||||
postgresql:
|
||||
enabled: false
|
||||
|
||||
# Enable Elasticsearch for visibility store (deployed to worker node, 2Gi/4Gi memory)
|
||||
elasticsearch:
|
||||
enabled: true
|
||||
scheme: http
|
||||
host: temporal-elasticsearch
|
||||
port: 9200
|
||||
version: v7
|
||||
logLevel: error
|
||||
auth:
|
||||
enabled: false
|
||||
indices:
|
||||
visibility: temporal_visibility_v1
|
||||
|
||||
# Cassandra enabled for template validation; server.config overrides with actual hosts
|
||||
# Schema job template requires cassandra config to exist at top level
|
||||
cassandra:
|
||||
enabled: true
|
||||
replicas: 3
|
||||
cluster:
|
||||
seedSize: 1
|
||||
port: 9042
|
||||
|
||||
# ── Disable schema auto-setup (will initialize manually) ─────────
|
||||
jobs:
|
||||
autoSetup:
|
||||
enabled: false
|
||||
|
||||
# ── Temporal server config (Cassandra + Elasticsearch persistence) ──────────────────────────────
|
||||
server:
|
||||
replicaCount: 1
|
||||
jobService:
|
||||
enabled: false
|
||||
config:
|
||||
logLevel: "info"
|
||||
persistence:
|
||||
defaultStore: default
|
||||
visibilityStore: visibility
|
||||
numHistoryShards: 512
|
||||
datastores:
|
||||
default:
|
||||
# Cassandra for workflow history and events
|
||||
driver: cassandra
|
||||
cassandra:
|
||||
hosts: "temporal-cassandra"
|
||||
port: 9042
|
||||
keyspace: temporal
|
||||
user: user
|
||||
password: "" # Cassandra auth disabled in deployment
|
||||
replicationFactor: 3
|
||||
consistency:
|
||||
default:
|
||||
consistency: local_quorum
|
||||
serialConsistency: local_serial
|
||||
service:
|
||||
type: ClusterIP
|
||||
|
||||
# ── Temporal Web UI ────────────────────────────────────────────────────────
|
||||
web:
|
||||
replicaCount: 1
|
||||
service:
|
||||
type: ClusterIP
|
||||
|
||||
# ── Ingress ────────────────────────────────────────────────────────
|
||||
# Note: ingress is disabled here. Instead, we route via oauth2-proxy.
|
||||
# The ingress is applied separately as k8s/temporal/temporal-ingress-oauth2.yaml
|
||||
# which terminates TLS and routes to oauth2-proxy service.
|
||||
ingress:
|
||||
enabled: false
|
||||
|
||||
# ── Monitoring ────────────────────────────────────────────────────────
|
||||
prometheus:
|
||||
enabled: false
|
||||
Reference in New Issue
Block a user