Compare commits

..
10 Commits
Author SHA1 Message Date
Story Crater Bot 370b5a894f (chore) add worker queue
build-push / build-push (push) Canceled after 0s
2026-08-17 12:17:39 -07:00
Story Crater BotandClaude Sonnet 5 3e81b4454d fix: grant queue-operator create/delete RBAC on TemporalWorker and Deployment
Deployed ClusterRole only had get/list/watch/update/patch on temporalworkers,
missing create/delete needed by reconcileTemporalWorker's cross-namespace
(sqs -> temporal) CreateOrUpdate call, and never granted apps/deployments at
all -- both required for the auto-provisioned TemporalWorker + backing
Deployment to reconcile successfully.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-17 09:33:16 -07:00
Story Crater Bot f599d2d897 fix: drop dead Authentik env-var validation now that auth interceptors are unwired 2026-07-13 16:45:53 -07:00
Story Crater Bot 52b86ab8e8 feat: queue-operator auto-registers Temporal namespace before creating TemporalWorker
A Queue's temporal.io/namespace label was trusted as-is -- if the referenced
Temporal namespace was never registered (or typo'd), the failure only
surfaced as a worker pod silently polling a namespace that doesn't exist.
Now reconcileTemporalWorker calls RegisterNamespace (idempotent, ignores
AlreadyExists) via a direct WorkflowService gRPC client before creating the
TemporalWorker, so namespace and worker always come into existence together.

Also grant queue-operator's ClusterRole create/delete on temporalworkers
(previously missing, causing forbidden errors on the create-then-delete path).
2026-07-13 13:02:40 -07:00
Story Crater Bot a584fb4462 fix: don't set cross-namespace owner ref on TemporalWorker
Queue lives in the sqs namespace while its TemporalWorker is created
in the Temporal namespace (KMSVC_TEMPORAL_NAMESPACE), so
SetControllerReference always failed with "cross-namespace owner
references are disallowed". Drop the owner ref (lifecycle already
handled explicitly in reconcileDelete) and move Spec population into
the CreateOrUpdate mutate closure so updates to an existing
TemporalWorker actually stick.

Also commit the generated TemporalWorker CRD and RBAC rules
(temporalworkers, deployments) that were previously untracked.
2026-07-13 11:32:04 -07:00
Story Crater Bot a8060444c1 feat: disable OIDC/JWT auth interceptors on gRPC+REST server
Server was crash-looping on TLS trust failures fetching Authentik's OIDC
discovery document (private-CA cert not trusted by the container image).
Drop the auth wiring for now to unblock the deployment; internal/auth and
internal/api/interceptors packages are left intact for when auth comes back.
2026-07-13 10:57:28 -07:00
Story Crater Bot b4cb3a7255 feat: build queue-operator binary alongside kmsvc-server in same image 2026-07-13 10:24:58 -07:00
Story Crater Bot 2f99f8d3d6 fix: cross-compile natively via BUILDPLATFORM/TARGETARCH instead of QEMU-emulating go build 2026-07-13 10:20:14 -07:00
Story Crater Bot 9c01076092 feat: switch to public GitHub kmsvc-proto dependency and GHCR image builds
Forgejo registry unreachable from cluster nodes (WireGuard overlay vs LAN
network isolation, plus host-to-ClusterIP routing gaps). Move to public
GitHub dependency and GHCR image hosting to remove the private-network
dependency entirely.
2026-07-13 10:09:41 -07:00
Story Crater Bot c354876178 test: add comprehensive TemporalWorker tests
Queue reconciliation:
- TemporalWorker creation when label present
- Namespace label validation
- Kubernetes name validation
- Cleanup on Queue deletion

Validation helpers:
- isValidTemporalNamespace (8 cases)
- validateKubernetesName (9 cases)

TemporalWorker controller:
- Deployment creation and updates
- Env var injection including TEMPORAL_TASK_QUEUE
- Status tracking
- Delete handling
2026-07-11 07:30:33 -07:00
21 changed files with 1935 additions and 53 deletions
+48
View File
@@ -0,0 +1,48 @@
name: build-push
on:
push:
branches: [main]
tags: ["v*"]
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository_owner }}/kmsvc-management-service
jobs:
build-push:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=raw,value=latest,enable={{is_default_branch}}
type=semver,pattern={{version}}
type=sha
- uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
+12
View File
@@ -33,3 +33,15 @@ Client Service Pod N ----/ ↓
- Horizontal scaling: add more `kmsvc` service replicas—Kafka rebalances automatically - Horizontal scaling: add more `kmsvc` service replicas—Kafka rebalances automatically
- HA: Redis Sentinel/Cluster recommended for production (design.md §9) — currently standalone - HA: Redis Sentinel/Cluster recommended for production (design.md §9) — currently standalone
- Monitoring: Kafka consumer-group lag, Redis pending/inflight keys, visibility timeouts - Monitoring: Kafka consumer-group lag, Redis pending/inflight keys, visibility timeouts
## Temporal Namespace Registration — Always Automatic, Never Manual
**Never manually run `temporal operator namespace create` (or the CLI/UI equivalent) for a namespace that a Queue's `temporal.io/namespace` label will reference.** `queue-operator`'s `reconcileTemporalWorker` (`internal/operator/queue_controller.go`) registers the Temporal namespace itself — idempotently, via `TemporalNamespaceRegisterer.RegisterNamespace` (`internal/operator/temporal_namespace.go`, real impl in `internal/temporal/client.go`) — before creating the `TemporalWorker` CR. This runs on every reconcile of every Queue carrying that label, so the namespace and its worker always exist together.
**The only steps to bring up a new Temporal namespace + worker are:**
1. Apply a `Queue` CR with `metadata.labels["temporal.io/namespace"] = "<namespace>"`.
2. That's it. `queue-operator` registers the namespace, creates `TemporalWorker/worker-<namespace>` in the `temporal` namespace, and its backing Deployment.
**Why this matters:** before this existed, a Queue's `temporal.io/namespace` label was trusted as-is with no verification — a typo'd or never-registered namespace silently produced a worker pod polling a namespace that doesn't exist, with no error surfaced anywhere until someone noticed workflows never executing. Manually pre-creating the namespace masks this — don't do it, let the operator own it.
One `TemporalWorker` per Temporal namespace serves *all* Queues labeled with that namespace (not one worker per Queue) — see the type doc on `TemporalWorkerSpec` in `apis/kmsvc/v1/temporalworker_types.go`.
+6 -7
View File
@@ -1,17 +1,16 @@
FROM golang:1.26 AS build FROM --platform=$BUILDPLATFORM golang:1.26 AS build
ARG TARGETOS
ARG TARGETARCH
WORKDIR /src WORKDIR /src
ENV GOPRIVATE=forgejo.riotpiao.homelab.com
# forgejo.riotpiao.homelab.com's cert is signed by a homelab-private CA, not
# a public one -- without this, `go mod download` can't fetch kmsvc-proto.
COPY hack/homelab-ca.pem /usr/local/share/ca-certificates/homelab-ca.crt
RUN update-ca-certificates
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -o /out/kmsvc-server ./cmd/server RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -trimpath -o /out/kmsvc-server ./cmd/server
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -trimpath -o /out/queue-operator ./cmd/queue-operator
FROM gcr.io/distroless/static-debian12:nonroot FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=build /out/kmsvc-server /kmsvc-server COPY --from=build /out/kmsvc-server /kmsvc-server
COPY --from=build /out/queue-operator /queue-operator
USER nonroot:nonroot USER nonroot:nonroot
ENTRYPOINT ["/kmsvc-server"] ENTRYPOINT ["/kmsvc-server"]
+7
View File
@@ -23,6 +23,7 @@ import (
kmsvcv1 "github.com/rockliang/kafka-management-service/apis/kmsvc/v1" kmsvcv1 "github.com/rockliang/kafka-management-service/apis/kmsvc/v1"
"github.com/rockliang/kafka-management-service/internal/kafka" "github.com/rockliang/kafka-management-service/internal/kafka"
"github.com/rockliang/kafka-management-service/internal/operator" "github.com/rockliang/kafka-management-service/internal/operator"
kmsvctemporal "github.com/rockliang/kafka-management-service/internal/temporal"
) )
func main() { func main() {
@@ -66,11 +67,17 @@ func main() {
exitf("connecting to redis at %s: %v", redisAddr, err) exitf("connecting to redis at %s: %v", redisAddr, err)
} }
temporalClient, err := kmsvctemporal.NewClient(getEnv("KMSVC_TEMPORAL_FRONTEND_ADDRESS", "temporal-frontend.temporal.svc.cluster.local:7233"))
if err != nil {
exitf("creating temporal client: %v", err)
}
reconciler := &operator.QueueReconciler{ reconciler := &operator.QueueReconciler{
Client: mgr.GetClient(), Client: mgr.GetClient(),
Admin: admin, Admin: admin,
Redis: rdb, Redis: rdb,
Now: time.Now, Now: time.Now,
Temporal: temporalClient,
Zones: &operator.ZoneLocator{ Zones: &operator.ZoneLocator{
// GetAPIReader(), not GetClient(): the latter is cache-backed and // GetAPIReader(), not GetClient(): the latter is cache-backed and
// would make controller-runtime List+Watch all Pods/Nodes // would make controller-runtime List+Watch all Pods/Nodes
+2 -12
View File
@@ -16,14 +16,12 @@ import (
"syscall" "syscall"
"time" "time"
kafkamgmtv1 "forgejo.riotpiao.homelab.com/rock/kmsvc-proto/gen/kafkamgmt/v1" kafkamgmtv1 "github.com/Riotpiaole/kmsvc-proto/gen/kafkamgmt/v1"
"github.com/grpc-ecosystem/grpc-gateway/v2/runtime" "github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
goredis "github.com/redis/go-redis/v9" goredis "github.com/redis/go-redis/v9"
"google.golang.org/grpc" "google.golang.org/grpc"
"github.com/rockliang/kafka-management-service/internal/api/handlers" "github.com/rockliang/kafka-management-service/internal/api/handlers"
"github.com/rockliang/kafka-management-service/internal/api/interceptors"
"github.com/rockliang/kafka-management-service/internal/auth"
"github.com/rockliang/kafka-management-service/internal/config" "github.com/rockliang/kafka-management-service/internal/config"
"github.com/rockliang/kafka-management-service/internal/core/queue" "github.com/rockliang/kafka-management-service/internal/core/queue"
"github.com/rockliang/kafka-management-service/internal/core/reaper" "github.com/rockliang/kafka-management-service/internal/core/reaper"
@@ -66,11 +64,6 @@ func run() error {
} }
defer producer.Close() defer producer.Close()
validator, err := auth.NewValidator(ctx, cfg.AuthentikIssuerURL, cfg.AuthentikAudience)
if err != nil {
return err
}
router := &queue.ShardRouter{Redis: rdb} router := &queue.ShardRouter{Redis: rdb}
svc := &handlers.QueueService{ svc := &handlers.QueueService{
@@ -88,10 +81,7 @@ func run() error {
} }
defer svc.Consumers.Close() defer svc.Consumers.Close()
grpcSrv := grpc.NewServer( grpcSrv := grpc.NewServer()
grpc.ChainUnaryInterceptor(interceptors.UnaryServerInterceptor(validator)),
grpc.ChainStreamInterceptor(interceptors.StreamServerInterceptor(validator)),
)
kafkamgmtv1.RegisterQueueServiceServer(grpcSrv, svc) kafkamgmtv1.RegisterQueueServiceServer(grpcSrv, svc)
mux := runtime.NewServeMux() mux := runtime.NewServeMux()
File diff suppressed because it is too large Load Diff
+12
View File
@@ -12,6 +12,18 @@ rules:
- apiGroups: ["kmsvc.io"] - apiGroups: ["kmsvc.io"]
resources: ["queues/finalizers"] resources: ["queues/finalizers"]
verbs: ["update"] verbs: ["update"]
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers/status"]
verbs: ["get", "update", "patch"]
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers/finalizers"]
verbs: ["update"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["coordination.k8s.io"] - apiGroups: ["coordination.k8s.io"]
resources: ["leases"] resources: ["leases"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+3 -1
View File
@@ -3,7 +3,7 @@ module github.com/rockliang/kafka-management-service
go 1.26.0 go 1.26.0
require ( require (
forgejo.riotpiao.homelab.com/homelab/kmsvc-proto v1.1.0 github.com/Riotpiaole/kmsvc-proto v1.3.0
github.com/alicebob/miniredis/v2 v2.38.0 github.com/alicebob/miniredis/v2 v2.38.0
github.com/google/uuid v1.6.0 github.com/google/uuid v1.6.0
github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0
@@ -12,6 +12,7 @@ require (
github.com/twmb/franz-go v1.21.3 github.com/twmb/franz-go v1.21.3
github.com/twmb/franz-go/pkg/kadm v1.18.0 github.com/twmb/franz-go/pkg/kadm v1.18.0
github.com/twmb/franz-go/pkg/kfake v0.0.0-20260615024848-f17c00130060 github.com/twmb/franz-go/pkg/kfake v0.0.0-20260615024848-f17c00130060
go.temporal.io/api v1.63.3
google.golang.org/grpc v1.81.1 google.golang.org/grpc v1.81.1
google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af
k8s.io/api v0.36.2 k8s.io/api v0.36.2
@@ -48,6 +49,7 @@ require (
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nexus-rpc/nexus-proto-annotations v0.1.0 // indirect
github.com/pierrec/lz4/v4 v4.1.26 // indirect github.com/pierrec/lz4/v4 v4.1.26 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect github.com/prometheus/client_golang v1.23.2 // indirect
+6 -2
View File
@@ -1,7 +1,7 @@
forgejo.riotpiao.homelab.com/rock/kmsvc-proto v1.1.0 h1:T7Y0aWFucwtwoOKy0XTCb+MAtay8U5j4SHCgi6PCrQo=
forgejo.riotpiao.homelab.com/rock/kmsvc-proto v1.1.0/go.mod h1:pKNhLE2KPpNUeu2BL5BFDgXmPlbyB9LHCfi0G4aE38s=
github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0= github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM= github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
github.com/Riotpiaole/kmsvc-proto v1.3.0 h1:W+Nlbk5Rj9FmEctk9fPgDhR82xhkx9yRAw3EAnJ+UxY=
github.com/Riotpiaole/kmsvc-proto v1.3.0/go.mod h1:20/7jwP4wRxbNGq+qeFeZfTQs7Vtxaoqok6t+3Sc0U8=
github.com/alicebob/miniredis/v2 v2.38.0 h1:nZAzCR+Lj+Vxk4ZXzm2NuKq2O33RXj1XxJ2e2uP9jiw= github.com/alicebob/miniredis/v2 v2.38.0 h1:nZAzCR+Lj+Vxk4ZXzm2NuKq2O33RXj1XxJ2e2uP9jiw=
github.com/alicebob/miniredis/v2 v2.38.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM= github.com/alicebob/miniredis/v2 v2.38.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
@@ -101,6 +101,8 @@ github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee h1:W5t00kpgFd
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/nexus-rpc/nexus-proto-annotations v0.1.0 h1:2fELd+9sqUtNu6Fg//pw8YFsxOvp8vZ8hfP0nHhNI80=
github.com/nexus-rpc/nexus-proto-annotations v0.1.0/go.mod h1:n3UjF1bPCW8llR8tHvbxJ+27yPWrhpo8w/Yg1IOuY0Y=
github.com/onsi/ginkgo/v2 v2.27.4 h1:fcEcQW/A++6aZAZQNUmNjvA9PSOzefMJBerHJ4t8v8Y= github.com/onsi/ginkgo/v2 v2.27.4 h1:fcEcQW/A++6aZAZQNUmNjvA9PSOzefMJBerHJ4t8v8Y=
github.com/onsi/ginkgo/v2 v2.27.4/go.mod h1:ArE1D/XhNXBXCBkKOLkbsb2c81dQHCRcF5zwn/ykDRo= github.com/onsi/ginkgo/v2 v2.27.4/go.mod h1:ArE1D/XhNXBXCBkKOLkbsb2c81dQHCRcF5zwn/ykDRo=
github.com/onsi/gomega v1.39.0 h1:y2ROC3hKFmQZJNFeGAMeHZKkjBL65mIZcvrLQBF9k6Q= github.com/onsi/gomega v1.39.0 h1:y2ROC3hKFmQZJNFeGAMeHZKkjBL65mIZcvrLQBF9k6Q=
@@ -166,6 +168,8 @@ go.opentelemetry.io/otel/sdk/metric v1.43.0 h1:S88dyqXjJkuBNLeMcVPRFXpRw2fuwdvfC
go.opentelemetry.io/otel/sdk/metric v1.43.0/go.mod h1:C/RJtwSEJ5hzTiUz5pXF1kILHStzb9zFlIEe85bhj6A= go.opentelemetry.io/otel/sdk/metric v1.43.0/go.mod h1:C/RJtwSEJ5hzTiUz5pXF1kILHStzb9zFlIEe85bhj6A=
go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09nk+3A= go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09nk+3A=
go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0= go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0=
go.temporal.io/api v1.63.3 h1:09yoemfjnk1YHV6g402lMW1vZccUd9Au/NfQBEZC0Eo=
go.temporal.io/api v1.63.3/go.mod h1:0k75tRljEuELWGeXjEZZO7zYqBln4+1FrG6+IMOMy7Q=
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE= go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
+111
View File
@@ -0,0 +1,111 @@
#!/usr/bin/env bash
set -euo pipefail
# Smoke test for the queue-operator -> TemporalWorker pipeline:
# 1. send/receive/delete a message through each Queue in
# k8s/temporal/queues/example-queue.yaml (story-crater-tasks,
# story-crater-notifications) via kmsvc-cli.
# 2. start a helloworld workflow via the temporal CLI against the
# Temporal namespace those queues point at (temporal.io/namespace
# label, "production"), on the task queue the auto-created
# TemporalWorker polls ("worker-production").
#
# Workflow execution will not complete -- there is no real worker image
# registering a "HelloWorldWorkflow" handler yet (TemporalWorker's Deployment
# is running the KMSVC_TEMPORAL_WORKER_IMAGE placeholder). This only proves
# the control-plane path: namespace exists, task queue routing works, and
# StartWorkflowExecution succeeds end-to-end through the operator-managed
# pipeline. `temporal workflow describe` afterward should show it stuck in
# Running/WorkflowTaskScheduled, which is the expected signal that this half
# of the pipe is wired correctly.
#
# Env overrides:
# NAMESPACE k8s namespace the Queue CRDs + management-service live in (default: sqs)
# TEMPORAL_NAMESPACE Temporal namespace the queues are labeled with (default: production)
# TASK_QUEUE Temporal task queue the TemporalWorker polls (default: worker-production)
# LOCAL_PORT local port for the management-service port-forward (default: 9090)
# TEMPORAL_LOCAL_PORT local port for the temporal-frontend port-forward (default: 7233)
# AUTHENTIK_TOKEN_URL Authentik OAuth2 token endpoint
QUEUES=("story-crater-tasks" "story-crater-notifications")
NAMESPACE="${NAMESPACE:-sqs}"
TEMPORAL_NAMESPACE="${TEMPORAL_NAMESPACE:-production}"
TASK_QUEUE="${TASK_QUEUE:-worker-production}"
LOCAL_PORT="${LOCAL_PORT:-9090}"
TEMPORAL_LOCAL_PORT="${TEMPORAL_LOCAL_PORT:-7233}"
SERVER="127.0.0.1:${LOCAL_PORT}"
AUTHENTIK_TOKEN_URL="${AUTHENTIK_TOKEN_URL:-https://authentik.riotpiao.homelab.com/application/o/token/}"
CLIENT_ID=$(talos get cluster/AUTHENTIK_KAFAKA_CLIENT_ID --key AUTHENTIK_KAFAKA_CLIENT_ID)
CLIENT_SECRET=$(talos get cluster/AUTHENTIK_KAFAKA_CLIENT_SECRET --key AUTHENTIK_KAFAKA_CLIENT_SECRET)
TOKEN=$(curl -sf -X POST "${AUTHENTIK_TOKEN_URL}" \
-d grant_type=client_credentials \
-d client_id="${CLIENT_ID}" \
-d client_secret="${CLIENT_SECRET}" \
| jq -r '.access_token')
if [[ -z "${TOKEN}" || "${TOKEN}" == "null" ]]; then
echo "FAIL: could not obtain access token from Authentik" >&2
exit 1
fi
kubectl port-forward -n "${NAMESPACE}" svc/management-service "${LOCAL_PORT}:9090" >/tmp/kmsvc-port-forward.log 2>&1 &
KMSVC_PF_PID=$!
kubectl port-forward -n temporal svc/temporal-frontend "${TEMPORAL_LOCAL_PORT}:7233" >/tmp/temporal-port-forward.log 2>&1 &
TEMPORAL_PF_PID=$!
sleep 2
cleanup() {
kill "${KMSVC_PF_PID}" "${TEMPORAL_PF_PID}" 2>/dev/null || true
}
trap cleanup EXIT
KMSVC=(kmsvc --server "${SERVER}" --token "${TOKEN}" --insecure)
for QUEUE_NAME in "${QUEUES[@]}"; do
echo "=== ${QUEUE_NAME} ==="
PHASE=$(kubectl get queue "${QUEUE_NAME}" -n "${NAMESPACE}" -o jsonpath='{.status.phase}' 2>/dev/null || true)
if [[ "${PHASE}" != "Ready" ]]; then
echo "FAIL: queue ${QUEUE_NAME} phase=${PHASE}, want Ready" >&2
exit 1
fi
echo "--- send ---"
"${KMSVC[@]}" message send --queue "${QUEUE_NAME}" --body "hello from ${QUEUE_NAME}"
echo "--- receive ---"
RECEIVE_OUT=$("${KMSVC[@]}" message receive --queue "${QUEUE_NAME}" --max-messages 1 --wait 10 --output json)
echo "${RECEIVE_OUT}"
RECEIPT_HANDLE=$(echo "${RECEIVE_OUT}" | jq -r '.[0].receipt_handle // .[0].ReceiptHandle')
if [[ -z "${RECEIPT_HANDLE}" || "${RECEIPT_HANDLE}" == "null" ]]; then
echo "FAIL: no message received from ${QUEUE_NAME}" >&2
exit 1
fi
echo "--- delete (ack) ---"
"${KMSVC[@]}" message delete --queue "${QUEUE_NAME}" --receipt-handle "${RECEIPT_HANDLE}"
echo "OK: ${QUEUE_NAME} round-trip succeeded"
done
echo "=== temporal: start helloworld workflow ==="
WORKFLOW_ID="hack-helloworld-$(date +%s)"
temporal workflow start \
--address "127.0.0.1:${TEMPORAL_LOCAL_PORT}" \
--namespace "${TEMPORAL_NAMESPACE}" \
--task-queue "${TASK_QUEUE}" \
--type HelloWorldWorkflow \
--workflow-id "${WORKFLOW_ID}" \
--input '"hack smoke test"'
echo "--- describe ---"
temporal workflow describe \
--address "127.0.0.1:${TEMPORAL_LOCAL_PORT}" \
--namespace "${TEMPORAL_NAMESPACE}" \
--workflow-id "${WORKFLOW_ID}"
echo "OK: workflow ${WORKFLOW_ID} started on namespace=${TEMPORAL_NAMESPACE} task-queue=${TASK_QUEUE}"
echo "NOTE: it will not complete -- no worker is registering HelloWorldWorkflow yet (placeholder image on worker-production)."
+1 -1
View File
@@ -11,7 +11,7 @@ import (
"strings" "strings"
"time" "time"
kafkamgmtv1 "forgejo.riotpiao.homelab.com/rock/kmsvc-proto/gen/kafkamgmt/v1" kafkamgmtv1 "github.com/Riotpiaole/kmsvc-proto/gen/kafkamgmt/v1"
goredis "github.com/redis/go-redis/v9" goredis "github.com/redis/go-redis/v9"
"google.golang.org/grpc/codes" "google.golang.org/grpc/codes"
"google.golang.org/grpc/status" "google.golang.org/grpc/status"
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"testing" "testing"
"time" "time"
kafkamgmtv1 "forgejo.riotpiao.homelab.com/rock/kmsvc-proto/gen/kafkamgmt/v1" kafkamgmtv1 "github.com/Riotpiaole/kmsvc-proto/gen/kafkamgmt/v1"
"github.com/alicebob/miniredis/v2" "github.com/alicebob/miniredis/v2"
goredis "github.com/redis/go-redis/v9" goredis "github.com/redis/go-redis/v9"
"github.com/twmb/franz-go/pkg/kfake" "github.com/twmb/franz-go/pkg/kfake"
-7
View File
@@ -50,13 +50,6 @@ func Load() (*Config, error) {
cfg.RedisDB = n cfg.RedisDB = n
} }
if cfg.AuthentikIssuerURL == "" {
return nil, fmt.Errorf("KMSVC_AUTHENTIK_ISSUER_URL is required")
}
if cfg.AuthentikAudience == "" {
return nil, fmt.Errorf("KMSVC_AUTHENTIK_AUDIENCE is required")
}
return cfg, nil return cfg, nil
} }
+41
View File
@@ -0,0 +1,41 @@
package operator
import (
"context"
"sync"
)
// fakeTemporal is an in-memory TemporalNamespaceRegisterer for reconciler
// tests — avoids needing a real Temporal frontend just to exercise reconcile
// logic.
type fakeTemporal struct {
mu sync.Mutex
registered map[string]int
err error
}
func newFakeTemporal() *fakeTemporal {
return &fakeTemporal{registered: map[string]int{}}
}
func (f *fakeTemporal) RegisterNamespace(_ context.Context, namespace string) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.err != nil {
return f.err
}
f.registered[namespace]++
return nil
}
func (f *fakeTemporal) setErr(err error) {
f.mu.Lock()
defer f.mu.Unlock()
f.err = err
}
func (f *fakeTemporal) count(namespace string) int {
f.mu.Lock()
defer f.mu.Unlock()
return f.registered[namespace]
}
+11 -9
View File
@@ -40,6 +40,7 @@ type QueueReconciler struct {
Admin TopicAdmin Admin TopicAdmin
Redis *goredis.Client Redis *goredis.Client
Now func() time.Time Now func() time.Time
Temporal TemporalNamespaceRegisterer
// Zones resolves shard topics' broker placement to availability zones // Zones resolves shard topics' broker placement to availability zones
// (design.md §2a AZ-awareness). Nil disables zone annotation entirely -- // (design.md §2a AZ-awareness). Nil disables zone annotation entirely --
@@ -248,6 +249,10 @@ func (r *QueueReconciler) reconcileTemporalWorker(ctx context.Context, queue *km
return fmt.Errorf("invalid kubernetes name %q: %w", workerName, err) return fmt.Errorf("invalid kubernetes name %q: %w", workerName, err)
} }
if err := r.Temporal.RegisterNamespace(ctx, namespace); err != nil {
return fmt.Errorf("register temporal namespace %s: %w", namespace, err)
}
replicas := int32(1) replicas := int32(1)
workerNamespace := getEnvOrDefault("KMSVC_TEMPORAL_NAMESPACE", "temporal") workerNamespace := getEnvOrDefault("KMSVC_TEMPORAL_NAMESPACE", "temporal")
workerImage := getEnvOrDefault("KMSVC_TEMPORAL_WORKER_IMAGE", "story-crater-backend:latest") workerImage := getEnvOrDefault("KMSVC_TEMPORAL_WORKER_IMAGE", "story-crater-backend:latest")
@@ -257,18 +262,15 @@ func (r *QueueReconciler) reconcileTemporalWorker(ctx context.Context, queue *km
Name: workerName, Name: workerName,
Namespace: workerNamespace, Namespace: workerNamespace,
}, },
Spec: kmsvcv1.TemporalWorkerSpec{
Namespace: namespace,
Image: workerImage,
Replicas: &replicas,
},
}
if err := controllerutil.SetControllerReference(queue, worker, r.Client.Scheme()); err != nil {
return fmt.Errorf("set controller reference: %w", err)
} }
// Queue and TemporalWorker live in different namespaces (sqs vs. the Temporal
// namespace), so a controller owner reference is disallowed by the API server.
// Lifecycle is instead managed explicitly in reconcileDelete.
if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, worker, func() error { if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, worker, func() error {
worker.Spec.Namespace = namespace
worker.Spec.Image = workerImage
worker.Spec.Replicas = &replicas
return nil return nil
}); err != nil { }); err != nil {
return fmt.Errorf("create or update TemporalWorker %s: %w", workerName, err) return fmt.Errorf("create or update TemporalWorker %s: %w", workerName, err)
+183 -1
View File
@@ -2,6 +2,8 @@ package operator
import ( import (
"context" "context"
"fmt"
"strings"
"testing" "testing"
"time" "time"
@@ -40,7 +42,23 @@ func newTestRedis(t *testing.T) *goredis.Client {
return goredis.NewClient(&goredis.Options{Addr: mr.Addr()}) return goredis.NewClient(&goredis.Options{Addr: mr.Addr()})
} }
func newTestSchemeWithAppsV1(t *testing.T) *runtime.Scheme {
t.Helper()
scheme := newTestScheme(t)
appsv1 := runtime.NewScheme()
if err := clientgoscheme.AddToScheme(appsv1); err != nil {
t.Fatalf("add client-go scheme: %v", err)
}
return scheme
}
func newTestReconciler(t *testing.T, objs ...client.Object) (*QueueReconciler, *fakeAdmin) { func newTestReconciler(t *testing.T, objs ...client.Object) (*QueueReconciler, *fakeAdmin) {
t.Helper()
r, admin, _ := newTestReconcilerWithTemporal(t, objs...)
return r, admin
}
func newTestReconcilerWithTemporal(t *testing.T, objs ...client.Object) (*QueueReconciler, *fakeAdmin, *fakeTemporal) {
t.Helper() t.Helper()
scheme := newTestScheme(t) scheme := newTestScheme(t)
cl := fake.NewClientBuilder(). cl := fake.NewClientBuilder().
@@ -49,12 +67,14 @@ func newTestReconciler(t *testing.T, objs ...client.Object) (*QueueReconciler, *
WithObjects(objs...). WithObjects(objs...).
Build() Build()
admin := newFakeAdmin() admin := newFakeAdmin()
temporal := newFakeTemporal()
return &QueueReconciler{ return &QueueReconciler{
Client: cl, Client: cl,
Admin: admin, Admin: admin,
Redis: newTestRedis(t), Redis: newTestRedis(t),
Now: time.Now, Now: time.Now,
}, admin Temporal: temporal,
}, admin, temporal
} }
func baseQueue(name string) *kmsvcv1.Queue { func baseQueue(name string) *kmsvcv1.Queue {
@@ -273,3 +293,165 @@ func TestReconcileDrainsClosingShardWhenLagZero(t *testing.T) {
t.Errorf("expected drained topic %q to be deleted", topic) t.Errorf("expected drained topic %q to be deleted", topic)
} }
} }
func TestReconcileTemporalWorkerCreatesWhenLabelPresent(t *testing.T) {
queue := baseQueue("orders")
queue.Labels = map[string]string{"temporal.io/namespace": "default"}
r, _, temporal := newTestReconcilerWithTemporal(t, queue)
ctx := context.Background()
if err := r.Reconcile(ctx, "", "orders"); err != nil {
t.Fatalf("Reconcile: %v", err)
}
var worker kmsvcv1.TemporalWorker
if err := r.Client.Get(ctx, client.ObjectKey{Name: "worker-default", Namespace: "temporal"}, &worker); err != nil {
t.Fatalf("expected TemporalWorker to be created: %v", err)
}
if worker.Spec.Namespace != "default" {
t.Errorf("worker namespace = %q, want default", worker.Spec.Namespace)
}
if got := temporal.count("default"); got != 1 {
t.Errorf("RegisterNamespace(%q) called %d times, want 1", "default", got)
}
}
func TestReconcileTemporalWorkerRegistersNamespaceBeforeCreating(t *testing.T) {
queue := baseQueue("orders")
queue.Labels = map[string]string{"temporal.io/namespace": "checkout"}
r, _, temporal := newTestReconcilerWithTemporal(t, queue)
ctx := context.Background()
if err := r.Reconcile(ctx, "", "orders"); err != nil {
t.Fatalf("Reconcile: %v", err)
}
if got := temporal.count("checkout"); got != 1 {
t.Errorf("RegisterNamespace(%q) called %d times, want 1", "checkout", got)
}
}
func TestReconcileTemporalWorkerFailsWhenNamespaceRegistrationFails(t *testing.T) {
queue := baseQueue("orders")
queue.Labels = map[string]string{"temporal.io/namespace": "checkout"}
r, _, temporal := newTestReconcilerWithTemporal(t, queue)
temporal.setErr(fmt.Errorf("frontend unreachable"))
ctx := context.Background()
err := r.Reconcile(ctx, "", "orders")
if err == nil {
t.Fatal("expected Reconcile to fail when namespace registration fails")
}
var worker kmsvcv1.TemporalWorker
getErr := r.Client.Get(ctx, client.ObjectKey{Name: "worker-checkout", Namespace: "temporal"}, &worker)
if getErr == nil {
t.Error("expected no TemporalWorker to be created when namespace registration fails")
}
}
func TestReconcileTemporalWorkerValidatesNamespaceLabel(t *testing.T) {
queue := baseQueue("orders")
queue.Labels = map[string]string{"temporal.io/namespace": "Foo@Bar"}
r, _ := newTestReconciler(t, queue)
ctx := context.Background()
err := r.Reconcile(ctx, "", "orders")
if err == nil {
t.Fatal("expected Reconcile to fail with invalid namespace label")
}
if err.Error() == "" || err.Error() == "invalid temporal namespace" {
t.Errorf("error message not descriptive: %v", err)
}
}
func TestReconcileTemporalWorkerValidatesKubernetesName(t *testing.T) {
queue := baseQueue("orders")
queue.Labels = map[string]string{"temporal.io/namespace": strings.Repeat("a", 250)}
r, _ := newTestReconciler(t, queue)
ctx := context.Background()
err := r.Reconcile(ctx, "", "orders")
if err == nil {
t.Fatal("expected Reconcile to fail with too-long kubernetes name")
}
}
func TestReconcileDeleteRemovesTemporalWorker(t *testing.T) {
queue := baseQueue("orders")
queue.Labels = map[string]string{"temporal.io/namespace": "default"}
r, _ := newTestReconciler(t, queue)
ctx := context.Background()
if err := r.Reconcile(ctx, "", "orders"); err != nil {
t.Fatalf("initial reconcile: %v", err)
}
var got kmsvcv1.Queue
if err := r.Client.Get(ctx, client.ObjectKey{Name: "orders"}, &got); err != nil {
t.Fatalf("get queue: %v", err)
}
if err := r.Client.Delete(ctx, &got); err != nil {
t.Fatalf("delete queue: %v", err)
}
if err := r.Reconcile(ctx, "", "orders"); err != nil {
t.Fatalf("delete reconcile: %v", err)
}
var worker kmsvcv1.TemporalWorker
err := r.Client.Get(ctx, client.ObjectKey{Name: "worker-default", Namespace: "temporal"}, &worker)
if err == nil {
t.Error("expected TemporalWorker to be deleted")
}
}
func TestIsValidTemporalNamespace(t *testing.T) {
tests := []struct {
name string
ns string
want bool
}{
{"valid lowercase", "default", true},
{"valid with underscore", "my_namespace", true},
{"valid with hyphen", "my-namespace", true},
{"valid with digits", "ns123", true},
{"invalid uppercase", "MyNamespace", false},
{"invalid special chars", "my@namespace", false},
{"empty", "", false},
{"too long", strings.Repeat("a", 256), false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := isValidTemporalNamespace(tt.ns); got != tt.want {
t.Errorf("isValidTemporalNamespace(%q) = %v, want %v", tt.ns, got, tt.want)
}
})
}
}
func TestValidateKubernetesName(t *testing.T) {
tests := []struct {
name string
input string
wantOk bool
}{
{"valid", "worker-default", true},
{"valid lowercase digits", "worker-123", true},
{"invalid uppercase", "Worker-default", false},
{"invalid starts with hyphen", "-worker-default", false},
{"invalid ends with hyphen", "worker-default-", false},
{"invalid special char", "worker@default", false},
{"too long", "worker-" + strings.Repeat("a", 250), false},
{"starts with digit", "1worker-default", false},
{"empty", "", false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := validateKubernetesName(tt.input)
got := err == nil
if got != tt.wantOk {
t.Errorf("validateKubernetesName(%q) error = %v, want ok=%v", tt.input, err, tt.wantOk)
}
})
}
}
+14
View File
@@ -0,0 +1,14 @@
package operator
import "context"
// TemporalNamespaceRegisterer registers a Temporal namespace. reconcileTemporalWorker
// calls this before creating a TemporalWorker so a Queue's temporal.io/namespace
// label always has a real namespace behind it — previously that label was
// trusted as-is, and a typo'd or never-registered namespace would only
// surface as a silently-stuck worker pod polling a namespace that doesn't
// exist.
type TemporalNamespaceRegisterer interface {
// RegisterNamespace registers namespace, treating AlreadyExists as success.
RegisterNamespace(ctx context.Context, namespace string) error
}
@@ -0,0 +1,193 @@
package operator
import (
"context"
"testing"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/client/fake"
kmsvcv1 "github.com/rockliang/kafka-management-service/apis/kmsvc/v1"
)
func newTemporalWorkerTestScheme(t *testing.T) *runtime.Scheme {
t.Helper()
scheme := runtime.NewScheme()
if err := clientgoscheme.AddToScheme(scheme); err != nil {
t.Fatalf("add client-go scheme: %v", err)
}
if err := kmsvcv1.AddToScheme(scheme); err != nil {
t.Fatalf("add kmsvc scheme: %v", err)
}
if err := appsv1.AddToScheme(scheme); err != nil {
t.Fatalf("add apps scheme: %v", err)
}
return scheme
}
func newTemporalWorkerTestReconciler(t *testing.T, objs ...client.Object) (*TemporalWorkerReconciler, client.Client) {
t.Helper()
scheme := newTemporalWorkerTestScheme(t)
cl := fake.NewClientBuilder().
WithScheme(scheme).
WithStatusSubresource(&kmsvcv1.TemporalWorker{}).
WithObjects(objs...).
Build()
return &TemporalWorkerReconciler{Client: cl}, cl
}
func baseTemporalWorker(name, namespace string) *kmsvcv1.TemporalWorker {
replicas := int32(2)
return &kmsvcv1.TemporalWorker{
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace},
Spec: kmsvcv1.TemporalWorkerSpec{
Namespace: "default",
Image: "story-crater-backend:v1.0.0",
Replicas: &replicas,
},
}
}
func TestTemporalWorkerReconcileCreatesDeployment(t *testing.T) {
worker := baseTemporalWorker("worker-default", "temporal")
r, cl := newTemporalWorkerTestReconciler(t, worker)
ctx := context.Background()
if err := r.Reconcile(ctx, "temporal", "worker-default"); err != nil {
t.Fatalf("Reconcile: %v", err)
}
var deploy appsv1.Deployment
if err := cl.Get(ctx, client.ObjectKey{Name: "worker-default", Namespace: "temporal"}, &deploy); err != nil {
t.Fatalf("expected Deployment to be created: %v", err)
}
if *deploy.Spec.Replicas != 2 {
t.Errorf("replicas = %d, want 2", *deploy.Spec.Replicas)
}
if deploy.Spec.Template.Spec.Containers[0].Image != "story-crater-backend:v1.0.0" {
t.Errorf("image = %q, want story-crater-backend:v1.0.0", deploy.Spec.Template.Spec.Containers[0].Image)
}
}
func TestTemporalWorkerReconcileInjectsEnvVars(t *testing.T) {
worker := baseTemporalWorker("worker-default", "temporal")
r, cl := newTemporalWorkerTestReconciler(t, worker)
ctx := context.Background()
if err := r.Reconcile(ctx, "temporal", "worker-default"); err != nil {
t.Fatalf("Reconcile: %v", err)
}
var deploy appsv1.Deployment
if err := cl.Get(ctx, client.ObjectKey{Name: "worker-default", Namespace: "temporal"}, &deploy); err != nil {
t.Fatalf("get Deployment: %v", err)
}
envVars := deploy.Spec.Template.Spec.Containers[0].Env
envMap := make(map[string]string)
for _, ev := range envVars {
envMap[ev.Name] = ev.Value
}
if envMap["TEMPORAL_FRONTEND_ADDRESS"] != "temporal-frontend.temporal.svc.cluster.local:7233" {
t.Errorf("TEMPORAL_FRONTEND_ADDRESS = %q, want temporal-frontend.temporal.svc.cluster.local:7233", envMap["TEMPORAL_FRONTEND_ADDRESS"])
}
if envMap["TEMPORAL_NAMESPACE"] != "default" {
t.Errorf("TEMPORAL_NAMESPACE = %q, want default", envMap["TEMPORAL_NAMESPACE"])
}
if envMap["TEMPORAL_TASK_QUEUE"] != "worker-default" {
t.Errorf("TEMPORAL_TASK_QUEUE = %q, want worker-default", envMap["TEMPORAL_TASK_QUEUE"])
}
}
func TestTemporalWorkerReconcileUpdateDeployment(t *testing.T) {
replicas := int32(2)
worker := baseTemporalWorker("worker-default", "temporal")
deploy := &appsv1.Deployment{
ObjectMeta: metav1.ObjectMeta{Name: "worker-default", Namespace: "temporal"},
Spec: appsv1.DeploymentSpec{
Replicas: &replicas,
Selector: &metav1.LabelSelector{
MatchLabels: map[string]string{
"app.kubernetes.io/name": "temporal-worker",
"app.kubernetes.io/instance": "worker-default",
"app.kubernetes.io/managed-by": "kmsvc-temporal-operator",
},
},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{
"app.kubernetes.io/name": "temporal-worker",
"app.kubernetes.io/instance": "worker-default",
"app.kubernetes.io/managed-by": "kmsvc-temporal-operator",
},
},
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{
Name: "worker",
Image: "story-crater-backend:v0.9.0",
},
},
},
},
},
}
r, cl := newTemporalWorkerTestReconciler(t, worker, deploy)
ctx := context.Background()
if err := r.Reconcile(ctx, "temporal", "worker-default"); err != nil {
t.Fatalf("Reconcile: %v", err)
}
var updated appsv1.Deployment
if err := cl.Get(ctx, client.ObjectKey{Name: "worker-default", Namespace: "temporal"}, &updated); err != nil {
t.Fatalf("get Deployment: %v", err)
}
if updated.Spec.Template.Spec.Containers[0].Image != "story-crater-backend:v1.0.0" {
t.Errorf("image updated to %q, want story-crater-backend:v1.0.0", updated.Spec.Template.Spec.Containers[0].Image)
}
}
func TestTemporalWorkerReconcileUpdatesStatus(t *testing.T) {
worker := baseTemporalWorker("worker-default", "temporal")
r, cl := newTemporalWorkerTestReconciler(t, worker)
ctx := context.Background()
if err := r.Reconcile(ctx, "temporal", "worker-default"); err != nil {
t.Fatalf("Reconcile: %v", err)
}
var updated kmsvcv1.TemporalWorker
if err := cl.Get(ctx, client.ObjectKey{Name: "worker-default", Namespace: "temporal"}, &updated); err != nil {
t.Fatalf("get TemporalWorker: %v", err)
}
if updated.Status.Phase != kmsvcv1.TemporalWorkerPhasePending {
t.Errorf("phase = %v, want Pending", updated.Status.Phase)
}
if updated.Status.Replicas != 2 {
t.Errorf("status.replicas = %d, want 2", updated.Status.Replicas)
}
}
func TestTemporalWorkerReconcileDeleteHandlesMarkedForDeletion(t *testing.T) {
worker := baseTemporalWorker("worker-default", "temporal")
now := metav1.Now()
worker.ObjectMeta.DeletionTimestamp = &now
worker.ObjectMeta.Finalizers = []string{temporalWorkerFinalizerName}
r, _ := newTemporalWorkerTestReconciler(t, worker)
ctx := context.Background()
if err := r.Reconcile(ctx, "temporal", "worker-default"); err != nil {
t.Fatalf("Reconcile delete should not error: %v", err)
}
}
+55
View File
@@ -0,0 +1,55 @@
// Package temporal wraps the narrow slice of Temporal's WorkflowService that
// queue-operator needs (namespace registration) directly over gRPC, instead
// of pulling in the full Temporal Go SDK for one RPC.
package temporal
import (
"context"
"fmt"
"time"
"go.temporal.io/api/workflowservice/v1"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/status"
"google.golang.org/protobuf/types/known/durationpb"
)
// defaultRetentionPeriod is applied to namespaces queue-operator registers on
// a Queue's behalf. Namespaces created deliberately by an operator (e.g. via
// the temporal CLI) can still override this by re-registering with different
// settings — RegisterNamespace on an existing namespace is a no-op here.
const defaultRetentionPeriod = 72 * time.Hour
// Client wraps a Temporal frontend's WorkflowService.
type Client struct {
svc workflowservice.WorkflowServiceClient
}
// NewClient dials a Temporal frontend at address (e.g.
// "temporal-frontend.temporal.svc.cluster.local:7233"). The connection is
// plaintext, matching how TemporalWorkerReconciler's worker pods already
// talk to the same frontend (see temporal_worker_controller.go).
func NewClient(address string) (*Client, error) {
conn, err := grpc.NewClient(address, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
return nil, fmt.Errorf("dial temporal frontend %s: %w", address, err)
}
return &Client{svc: workflowservice.NewWorkflowServiceClient(conn)}, nil
}
// RegisterNamespace registers namespace, treating AlreadyExists as success.
func (c *Client) RegisterNamespace(ctx context.Context, namespace string) error {
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
_, err := c.svc.RegisterNamespace(ctx, &workflowservice.RegisterNamespaceRequest{
Namespace: namespace,
WorkflowExecutionRetentionPeriod: durationpb.New(defaultRetentionPeriod),
})
if err != nil && status.Code(err) != codes.AlreadyExists {
return fmt.Errorf("register temporal namespace %s: %w", namespace, err)
}
return nil
}
+12
View File
@@ -18,6 +18,15 @@ rules:
- apiGroups: ["kmsvc.io"] - apiGroups: ["kmsvc.io"]
resources: ["queues/finalizers"] resources: ["queues/finalizers"]
verbs: ["update"] verbs: ["update"]
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers/status"]
verbs: ["get", "update", "patch"]
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers/finalizers"]
verbs: ["update"]
- apiGroups: ["coordination.k8s.io"] - apiGroups: ["coordination.k8s.io"]
resources: ["leases"] resources: ["leases"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
@@ -27,6 +36,9 @@ rules:
- apiGroups: [""] - apiGroups: [""]
resources: ["pods", "nodes"] resources: ["pods", "nodes"]
verbs: ["get"] verbs: ["get"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
--- ---
apiVersion: rbac.authorization.k8s.io/v1 apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding kind: ClusterRoleBinding
+14
View File
@@ -0,0 +1,14 @@
apiVersion: kmsvc.io/v1
kind: Queue
metadata:
name: agent-worker-queue
namespace: sqs
labels:
temporal.io/namespace: production
spec:
visibilityTimeoutSeconds: 30
messageRetentionPeriodSeconds: 345600
maxReceiveCount: 5
partitionsPerShard: 1
minShards: 1
maxShards: 1