homelab-frontend/docs/KNOWN-ISSUES.md

Known cluster issues

Pre-existing problems found while specifying this gateway. None are caused by this repo, and none block phases 0-6. Recorded so they are not rediscovered or mistaken for new breakage.

Verified 2026-08-19 against context admin@homelab-cluster.

---

1. TemporalWorker CRD is stale — queue-operator reconcile fails every ~17 min

Status: open, deliberately deferred. Affects task 7.3.

The live temporalworkers.kmsvc.io CRD and the one in ~/workplace/kmsvc-manage/config/crd/kmsvc.io_temporalworkers.yaml share exactly one field — namespace.

	spec properties
live CRD	activityTypes, concurrency, namespace, taskQueue, workflowTypes
repo CRD	affinity, image, imagePullPolicy, namespace, nodeSelector, replicas, resources, tolerations

The live schema has no image field, so the API server prunes image from the CR that queue-operator writes. TemporalWorker/worker-production ends up as spec: {namespace: production}, and the operator then fails to build a Deployment from it. The live CRD also lacks a status subresource, producing a second error.

Observed on a loop, most recently 21:25:39Z:

failed to create or update deployment ... error: "Deployment.apps \"worker-production\"
  is invalid: spec.template.spec.containers[0].image: Required value"
Reconciler error ... "update status failed: temporalworkers.kmsvc.io
  \"worker-production\" not found"

Impact is narrower than it looks. No worker Deployment has ever existed under this CRD, so nothing that was working has stopped. Temporal namespace production is registered and healthy; there is simply no worker polling it. The practical cost is log noise, not lost work. That is why this is deferred rather than treated as an incident.

Neither object is under GitOps. The CRD and the Queue/agent-worker-queue CR both carry only kubectl.kubernetes.io/last-applied-configuration — no argocd.argoproj.io/instance, no tracking-id — and the Queue does not appear anywhere in the homelab repo. They were hand-applied and predate GitOps coverage.

Fix, when it is worth doing:

1. Bring temporalworkers.kmsvc.io and the Queue CR into the homelab GitOps repo.
2. Apply the current CRD from kmsvc-manage/config/crd, which restores image and the status subresource.
3. Ensure the operator sets spec.image on the CR it creates.

Do not hand-apply the CRD as a one-off. That reproduces exactly the situation that caused this — a cluster object with no source of truth.

To silence the loop without fixing it: remove the temporal.io/namespace: production label from Queue/agent-worker-queue in namespace sqs. The operator returns early when the label is absent. Reversible by re-adding it.

---

2. kmsvc.riotpiao.com is unauthenticated

Status: open. Relevant to task 7.2.

kmsvc-manage has an auth interceptor at internal/api/interceptors/auth.go, but it is never wired: cmd/server/main.go constructs a bare grpc.NewServer() with no interceptor options. The live ConfigMap confirms it — KMSVC_AUTHENTIK_ISSUER_URL and KMSVC_AUTHENTIK_AUDIENCE are both empty strings.

Both the REST surface (8080) and the gRPC surface (9090) are open.

There is a second, subtler problem. The REST surface is mounted with RegisterQueueServiceHandlerServer, the in-process grpc-gateway variant that calls the service implementation directly. It bypasses gRPC interceptors entirely. So even once the interceptor is wired, it would authenticate gRPC callers only — the file's own doc comment claiming it covers both REST and gRPC is wrong for this wiring.

Consequence for this gateway: /sqs/* must own authentication itself. Do not assume the upstream will enforce anything.

---

3. kmsvc-redis-master.sqs:6379 has no authentication

ALLOW_EMPTY_PASSWORD=yes, TLS off, Bitnami chart with auth.enabled=false, no password secret in the namespace. Anything with network reach has full unauthenticated read/write.

A NetworkPolicy is the only control. Relevant to task 6.2.

---

4. macos-bluebubbles pod will never schedule

sms Argo Application is Synced/Degraded. The pod targets a macOS node that is not in the cluster: 0/4 nodes are available: 4 node(s) didn't match Pod's node affinity/selector, roughly 1080 failed attempts over 3d18h.

Not transient. Needs either that node or removal of the Application. Unrelated to this gateway; listed so the Degraded status is not mistaken for something new.

---

5. Documentation that does not match reality

• kmsvc-manage/TEMPORAL_INTEGRATION.md is aspirational. It documents apiVersion: temporal.kmsvc.io/v1 with queueRef, taskQueueName and lifecycle fields, and one worker per Queue. Reality is kmsvc.io/v1, none of those fields, and one worker per Temporal namespace. Do not source API documentation from it.
• Module paths disagree across repos: kmsvc-proto declares forgejo.riotpiao.homelab.com/..., while kmsvc-manage and kmsvc-sdk import forgejo.riotpiao.com/.... The .homelab.com domain is fully retired — every subdomain NXDOMAINs.
• kmsvc-cli README says the gRPC ingress uses TLS passthrough. It uses nginx.ingress.kubernetes.io/backend-protocol: GRPC, which terminates TLS at nginx. Functionally fine for clients; the wording is wrong.