Deployed ClusterRole only had get/list/watch/update/patch on temporalworkers,
missing create/delete needed by reconcileTemporalWorker's cross-namespace
(sqs -> temporal) CreateOrUpdate call, and never granted apps/deployments at
all -- both required for the auto-provisioned TemporalWorker + backing
Deployment to reconcile successfully.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Resolves each shard topic's replica broker IDs (internal/kafka.Admin.
ReplicaBrokerIDs) to the topology.kubernetes.io/zone labels of the nodes
hosting those brokers (ZoneLocator), and writes the result into
ShardStatus.AvailabilityZones each reconcile. Uses mgr.GetAPIReader()
rather than the cached client for the Pod/Node lookups, since the cached
client would otherwise require cluster-wide list/watch RBAC on Pods just
to serve occasional point Gets.
main.go called Reconcile(ctx, req.Name) without req.Namespace, so the
Get against the namespaced Queue CRD always 404'd and was silently
swallowed as success -- no shard topics or Redis state were ever created.
Separately, ShardStatus.HashRangeStart/End were uint32, but controller-gen
maps that to OpenAPI format:int32, whose max (2147483647) is smaller than
FullHashRangeEnd (0xFFFFFFFF), so the apiserver rejected every status
update with the (misleadingly empty-looking) "must be of type integer with
format int32" error. Widened to int64, regenerated the CRD, and synced the
chart's bundled copy.
Go's GC doesn't respect cgroup memory limits on its own, risking OOMKill
under load; set GOMEMLIMIT to ~90% of each deployment's resources.limits.memory.
Also switch imagePullPolicy to Always so :latest tags aren't served stale
from node-local cache after a new push.
Local charts: kafka-cluster (Strimzi Kafka+KafkaNodePool CRs, 3-replica KRaft
topology, 5Gi memory cap, Longhorn storage, pod anti-affinity), queue-crd
(Queue CRD + queue-operator Deployment/RBAC), management-service
(Deployment/Service/ConfigMap/HPA/Ingress, REST exposed externally via
cert-manager-issued TLS, raw gRPC kept cluster-internal per design.md §7a).
helmfile.yaml.gotmpl + releases.d/*.gotmpl wire strimzi-operator ->
kafka-cluster -> redis -> {queue-crd, management-service} via `needs:`.
Directory is releases.d, not helmfile.d as originally sketched in design.md
section 7b: Helmfile v1 treats a literal "helmfile.d" directory as a special
auto-discovery mode that conflicts with an explicit top-level helmfile.yaml.
Files use .gotmpl (required by Helmfile v1 for {{ }}-templated files) and
each declares its own environments: block, since nested helmfiles don't
inherit the parent's resolved values in this version.
Namespace is sqs throughout. environments/homelab.yaml carries no secrets.
Validated locally via helm lint/helm template (all 3 charts) and
`helmfile -e homelab build` (dependency ordering + value substitution) — no
cluster contact made. Live apply is a separate, explicitly-confirmed step.
