Deployed ClusterRole only had get/list/watch/update/patch on temporalworkers,
missing create/delete needed by reconcileTemporalWorker's cross-namespace
(sqs -> temporal) CreateOrUpdate call, and never granted apps/deployments at
all -- both required for the auto-provisioned TemporalWorker + backing
Deployment to reconcile successfully.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
A Queue's temporal.io/namespace label was trusted as-is -- if the referenced
Temporal namespace was never registered (or typo'd), the failure only
surfaced as a worker pod silently polling a namespace that doesn't exist.
Now reconcileTemporalWorker calls RegisterNamespace (idempotent, ignores
AlreadyExists) via a direct WorkflowService gRPC client before creating the
TemporalWorker, so namespace and worker always come into existence together.
Also grant queue-operator's ClusterRole create/delete on temporalworkers
(previously missing, causing forbidden errors on the create-then-delete path).
Queue lives in the sqs namespace while its TemporalWorker is created
in the Temporal namespace (KMSVC_TEMPORAL_NAMESPACE), so
SetControllerReference always failed with "cross-namespace owner
references are disallowed". Drop the owner ref (lifecycle already
handled explicitly in reconcileDelete) and move Spec population into
the CreateOrUpdate mutate closure so updates to an existing
TemporalWorker actually stick.
Also commit the generated TemporalWorker CRD and RBAC rules
(temporalworkers, deployments) that were previously untracked.
Server was crash-looping on TLS trust failures fetching Authentik's OIDC
discovery document (private-CA cert not trusted by the container image).
Drop the auth wiring for now to unblock the deployment; internal/auth and
internal/api/interceptors packages are left intact for when auth comes back.
Forgejo registry unreachable from cluster nodes (WireGuard overlay vs LAN
network isolation, plus host-to-ClusterIP routing gaps). Move to public
GitHub dependency and GHCR image hosting to remove the private-network
dependency entirely.
- Validate temporal namespace and Kubernetes names
- Configurable via env vars: KMSVC_TEMPORAL_NAMESPACE, KMSVC_TEMPORAL_WORKER_IMAGE
- Set ownerReference for cascade deletion and lifecycle management
- Use CreateOrUpdate for spec propagation (idempotent)
- Clean up TemporalWorker on Queue deletion
Resolves each shard topic's replica broker IDs (internal/kafka.Admin.
ReplicaBrokerIDs) to the topology.kubernetes.io/zone labels of the nodes
hosting those brokers (ZoneLocator), and writes the result into
ShardStatus.AvailabilityZones each reconcile. Uses mgr.GetAPIReader()
rather than the cached client for the Pod/Node lookups, since the cached
client would otherwise require cluster-wide list/watch RBAC on Pods just
to serve occasional point Gets.
Creates a Queue CRD, port-forwards to management-service, fetches an
Authentik client_credentials token, then exercises send/receive/delete
through kmsvc-cli before tearing the queue down.
main.go called Reconcile(ctx, req.Name) without req.Namespace, so the
Get against the namespaced Queue CRD always 404'd and was silently
swallowed as success -- no shard topics or Redis state were ever created.
Separately, ShardStatus.HashRangeStart/End were uint32, but controller-gen
maps that to OpenAPI format:int32, whose max (2147483647) is smaller than
FullHashRangeEnd (0xFFFFFFFF), so the apiserver rejected every status
update with the (misleadingly empty-looking) "must be of type integer with
format int32" error. Widened to int64, regenerated the CRD, and synced the
chart's bundled copy.
The default "longhorn" StorageClass requests 3 replicas, but the homelab
cluster currently has only 2 schedulable nodes, so the 3rd replica could
never be scheduled and volumes stayed permanently degraded. Adds
longhorn-kafka (numberOfReplicas: 2) and reduces broker PVC size so 3 broker
volumes' replicas fit within each node's remaining Longhorn scheduling
headroom. Revert to "longhorn" once a 3rd node joins.
Go's GC doesn't respect cgroup memory limits on its own, risking OOMKill
under load; set GOMEMLIMIT to ~90% of each deployment's resources.limits.memory.
Also switch imagePullPolicy to Always so :latest tags aren't served stale
from node-local cache after a new push.
Building on arm64 Mac without GOOS/GOARCH produced amd64 node "exec format
error". The homelab CA was only trusted in the build stage, so the final
distroless runtime couldn't verify Authentik's TLS cert during OIDC discovery.
Cross-file `needs:` across separate nested helmfiles didn't resolve in
Helmfile v1 (releases defined in sibling files weren't visible to each
other's dependency graph) -- confirmed live against the homelab cluster:
kafka-cluster failed with "depend(s) on an undefined release" even though
strimzi-operator had just been installed successfully by a sibling file.
Collapsed releases.d/*.gotmpl into a single helmfile.yaml.gotmpl so the
whole release graph is resolved together.
Also add a second Ingress (management-service-grpc, backend-protocol: GRPC)
scoped to the QueueService gRPC path prefix on the same host/port as the
REST ingress. kmsvc-cli dials --server directly via gRPC (default
kmsvc.homelab.internal:443 per its README), so raw gRPC needs an external
path too, not just REST -- the original "gRPC stays internal" default
didn't account for the CLI's own connection model.
Add Dockerfile.queue-operator (existed for cmd/server only before).
Local charts: kafka-cluster (Strimzi Kafka+KafkaNodePool CRs, 3-replica KRaft
topology, 5Gi memory cap, Longhorn storage, pod anti-affinity), queue-crd
(Queue CRD + queue-operator Deployment/RBAC), management-service
(Deployment/Service/ConfigMap/HPA/Ingress, REST exposed externally via
cert-manager-issued TLS, raw gRPC kept cluster-internal per design.md §7a).
helmfile.yaml.gotmpl + releases.d/*.gotmpl wire strimzi-operator ->
kafka-cluster -> redis -> {queue-crd, management-service} via `needs:`.
Directory is releases.d, not helmfile.d as originally sketched in design.md
section 7b: Helmfile v1 treats a literal "helmfile.d" directory as a special
auto-discovery mode that conflicts with an explicit top-level helmfile.yaml.
Files use .gotmpl (required by Helmfile v1 for {{ }}-templated files) and
each declares its own environments: block, since nested helmfiles don't
inherit the parent's resolved values in this version.
Namespace is sqs throughout. environments/homelab.yaml carries no secrets.
Validated locally via helm lint/helm template (all 3 charts) and
`helmfile -e homelab build` (dependency ordering + value substitution) — no
cluster contact made. Live apply is a separate, explicitly-confirmed step.
Assembles tasks 1/6/7/8 into a runnable cmd/server binary: QueueServiceServer
handlers translating kafkamgmt.v1 proto to internal/core/queue's plain Go
types, a lazy per-queue Kafka consumer registry for ReceiveMessage, and a
Redis-scan-based queue discovery loop that starts a reaper goroutine per
queue (queue lifecycle isn't exposed over gRPC, so this is the server's only
signal). Promotes kmsvc-proto to a direct go.mod dependency.
Handler-level integration tests run against kfake+miniredis (same documented
tradeoff as tasks 5-7's envtest/testcontainers substitution), exercising
send->receive->delete through the real QueueServiceServer implementation.
Remove local proto/, buf.yaml, buf.gen.yaml, and generated
internal/api/v1/*.pb.go. The message-plane contract now lives in the
sibling repo kmsvc-proto (forgejo.riotpiao.homelab.com/rock/kmsvc-proto),
fetched via go get — no local buf/protoc plugin install needed.
Nothing in this repo imported internal/api/v1 yet, so this is a clean
swap with no call-site changes.
JWKS caching via lestrrat-go/jwx, signature/iss/aud/exp validation, and
a single gRPC interceptor that grpc-gateway's forwarded headers make
work identically for REST callers.
Per-queue ticker that scans vis_index for expired in-flight messages and
drives each through the atomic reap.lua check-and-act, routing maxed-out
messages to their queue's configured DLQ via the existing SendMessage
path. Safe to run from multiple replicas against the same queue.
Implements SendMessage (size cap, FIFO dedup, shard routing), ReceiveMessage
(long-poll loop across active+closing shards, FIFO per-group exclusivity
gating, redelivery hand-out), DeleteMessage (ack + low-watermark offset
advancement), and ChangeMessageVisibility against the Redis state layer
and a real Kafka producer/consumer.
Reconciles Queue CRs into Kafka topics + Redis shard-map/queue-meta state:
creates shard-0 on first reconcile, splits a shard's hash range into two
children once its split threshold is crossed, drains and closes a parent
shard once its consumer group has fully caught up and its retention
window has elapsed, and tears down every shard's topic + Redis state on
deletion. Includes the manager entrypoint (cmd/queue-operator) and RBAC.
Implements the kmsvc: key schema (inflight, pending/watermark keyed by
shard+partition, vis_index, dedup, fifo_lock, queue meta, shard map) plus
the atomic reap/ack Lua scripts used for safe multi-replica redelivery
and DLQ routing.
QueueSpec/QueueStatus model the shard-based (Kinesis-style hash-range
splitting) queue lifecycle object, with the controller-gen-generated
deepcopy and CRD YAML.
Adds queue_service.proto (SendMessage, SendMessageBatch, ReceiveMessage,
DeleteMessage, DeleteMessageBatch, ChangeMessageVisibility(Batch)) with
grpc-gateway REST annotations, plus the generated Go server/client and
gateway stubs.
Story Crater Bot