homelab-frontend/docs/KNOWN-ISSUES.md

# Known cluster issues

Pre-existing problems found while specifying this gateway. None are caused by this
repo, and none block phases 0-6. Recorded so they are not rediscovered or mistaken
for new breakage.

Verified 2026-08-19 against context `admin@homelab-cluster`.

---

## 1. TemporalWorker CRD is stale — queue-operator reconcile fails every ~17 min

**Status:** open, deliberately deferred. Affects [task 7.3](../tasks/7.3-workflow-prefix.md).

The live `temporalworkers.kmsvc.io` CRD and the one in
`~/workplace/kmsvc-manage/config/crd/kmsvc.io_temporalworkers.yaml` share exactly one
field — `namespace`.

| | spec properties |
|---|---|
| live CRD | `activityTypes`, `concurrency`, `namespace`, `taskQueue`, `workflowTypes` |
| repo CRD | `affinity`, `image`, `imagePullPolicy`, `namespace`, `nodeSelector`, `replicas`, `resources`, `tolerations` |

The live schema has no `image` field, so the API server **prunes** `image` from the CR
that `queue-operator` writes. `TemporalWorker/worker-production` ends up as
`spec: {namespace: production}`, and the operator then fails to build a Deployment
from it. The live CRD also lacks a status subresource, producing a second error.

Observed on a loop, most recently 21:25:39Z:

```
failed to create or update deployment ... error: "Deployment.apps \"worker-production\"
  is invalid: spec.template.spec.containers[0].image: Required value"
Reconciler error ... "update status failed: temporalworkers.kmsvc.io
  \"worker-production\" not found"
```

**Impact is narrower than it looks.** No worker Deployment has ever existed under this
CRD, so nothing that was working has stopped. Temporal namespace `production` is
registered and healthy; there is simply no worker polling it. The practical cost is log
noise, not lost work. That is why this is deferred rather than treated as an incident.

**Neither object is under GitOps.** The CRD and the `Queue/agent-worker-queue` CR both
carry only `kubectl.kubernetes.io/last-applied-configuration` — no
`argocd.argoproj.io/instance`, no tracking-id — and the Queue does not appear anywhere
in the homelab repo. They were hand-applied and predate GitOps coverage.

**Fix, when it is worth doing:**

1. Bring `temporalworkers.kmsvc.io` and the Queue CR into the homelab GitOps repo.
2. Apply the current CRD from `kmsvc-manage/config/crd`, which restores `image` and the
   status subresource.
3. Ensure the operator sets `spec.image` on the CR it creates.

Do not hand-apply the CRD as a one-off. That reproduces exactly the situation that
caused this — a cluster object with no source of truth.

**To silence the loop without fixing it:** remove the `temporal.io/namespace: production`
label from `Queue/agent-worker-queue` in namespace `sqs`. The operator returns early when
the label is absent. Reversible by re-adding it.

---

## 2. `kmsvc.riotpiao.com` is unauthenticated

**Status:** open. Relevant to [task 7.2](../tasks/7.2-sqs-prefix.md).

`kmsvc-manage` has an auth interceptor at `internal/api/interceptors/auth.go`, but it is
never wired: `cmd/server/main.go` constructs a bare `grpc.NewServer()` with no
interceptor options. The live ConfigMap confirms it — `KMSVC_AUTHENTIK_ISSUER_URL` and
`KMSVC_AUTHENTIK_AUDIENCE` are both empty strings.

Both the REST surface (8080) and the gRPC surface (9090) are open.

There is a second, subtler problem. The REST surface is mounted with
`RegisterQueueServiceHandlerServer`, the **in-process** grpc-gateway variant that calls
the service implementation directly. It bypasses gRPC interceptors entirely. So even
once the interceptor is wired, it would authenticate gRPC callers only — the file's own
doc comment claiming it covers both REST and gRPC is wrong for this wiring.

Consequence for this gateway: `/sqs/*` must own authentication itself. Do not assume the
upstream will enforce anything.

---

## 3. `kmsvc-redis-master.sqs:6379` has no authentication

`ALLOW_EMPTY_PASSWORD=yes`, TLS off, Bitnami chart with `auth.enabled=false`, no password
secret in the namespace. Anything with network reach has full unauthenticated read/write.

A NetworkPolicy is the only control. Relevant to [task 6.2](../tasks/6.2-kubernetes-manifests.md).

---

## 4. `macos-bluebubbles` pod will never schedule

`sms` Argo Application is `Synced`/`Degraded`. The pod targets a macOS node that is not
in the cluster: `0/4 nodes are available: 4 node(s) didn't match Pod's node
affinity/selector`, roughly 1080 failed attempts over 3d18h.

Not transient. Needs either that node or removal of the Application. Unrelated to this
gateway; listed so the Degraded status is not mistaken for something new.

---

## 5. Documentation that does not match reality

- `kmsvc-manage/TEMPORAL_INTEGRATION.md` is aspirational. It documents
  `apiVersion: temporal.kmsvc.io/v1` with `queueRef`, `taskQueueName` and `lifecycle`
  fields, and one worker per Queue. Reality is `kmsvc.io/v1`, none of those fields, and
  one worker per Temporal *namespace*. Do not source API documentation from it.
- Module paths disagree across repos: `kmsvc-proto` declares
  `forgejo.riotpiao.homelab.com/...`, while `kmsvc-manage` and `kmsvc-sdk` import
  `forgejo.riotpiao.com/...`. The `.homelab.com` domain is fully retired — every
  subdomain NXDOMAINs.
- `kmsvc-cli` README says the gRPC ingress uses TLS passthrough. It uses
  `nginx.ingress.kubernetes.io/backend-protocol: GRPC`, which terminates TLS at nginx.
  Functionally fine for clients; the wording is wrong.