Story Crater BotandClaude Opus 5
058f11cf2b
Baseline for the Kong replacement on api.riotpiao.com. Brings the working tree under version control for the first time: gateway source, the task board that drives the agent runs, test fixtures, and K8s manifests. Anchor the gateway ignore rule to the repo root. Unanchored, "gateway" also matched the cmd/gateway/ source directory, so the program entrypoint was excluded from every commit. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
119 lines
5.4 KiB
Markdown
119 lines
5.4 KiB
Markdown
# Known cluster issues
|
|
|
|
Pre-existing problems found while specifying this gateway. None are caused by this
|
|
repo, and none block phases 0-6. Recorded so they are not rediscovered or mistaken
|
|
for new breakage.
|
|
|
|
Verified 2026-08-19 against context `admin@homelab-cluster`.
|
|
|
|
---
|
|
|
|
## 1. TemporalWorker CRD is stale — queue-operator reconcile fails every ~17 min
|
|
|
|
**Status:** open, deliberately deferred. Affects [task 7.3](../tasks/7.3-workflow-prefix.md).
|
|
|
|
The live `temporalworkers.kmsvc.io` CRD and the one in
|
|
`~/workplace/kmsvc-manage/config/crd/kmsvc.io_temporalworkers.yaml` share exactly one
|
|
field — `namespace`.
|
|
|
|
| | spec properties |
|
|
|---|---|
|
|
| live CRD | `activityTypes`, `concurrency`, `namespace`, `taskQueue`, `workflowTypes` |
|
|
| repo CRD | `affinity`, `image`, `imagePullPolicy`, `namespace`, `nodeSelector`, `replicas`, `resources`, `tolerations` |
|
|
|
|
The live schema has no `image` field, so the API server **prunes** `image` from the CR
|
|
that `queue-operator` writes. `TemporalWorker/worker-production` ends up as
|
|
`spec: {namespace: production}`, and the operator then fails to build a Deployment
|
|
from it. The live CRD also lacks a status subresource, producing a second error.
|
|
|
|
Observed on a loop, most recently 21:25:39Z:
|
|
|
|
```
|
|
failed to create or update deployment ... error: "Deployment.apps \"worker-production\"
|
|
is invalid: spec.template.spec.containers[0].image: Required value"
|
|
Reconciler error ... "update status failed: temporalworkers.kmsvc.io
|
|
\"worker-production\" not found"
|
|
```
|
|
|
|
**Impact is narrower than it looks.** No worker Deployment has ever existed under this
|
|
CRD, so nothing that was working has stopped. Temporal namespace `production` is
|
|
registered and healthy; there is simply no worker polling it. The practical cost is log
|
|
noise, not lost work. That is why this is deferred rather than treated as an incident.
|
|
|
|
**Neither object is under GitOps.** The CRD and the `Queue/agent-worker-queue` CR both
|
|
carry only `kubectl.kubernetes.io/last-applied-configuration` — no
|
|
`argocd.argoproj.io/instance`, no tracking-id — and the Queue does not appear anywhere
|
|
in the homelab repo. They were hand-applied and predate GitOps coverage.
|
|
|
|
**Fix, when it is worth doing:**
|
|
|
|
1. Bring `temporalworkers.kmsvc.io` and the Queue CR into the homelab GitOps repo.
|
|
2. Apply the current CRD from `kmsvc-manage/config/crd`, which restores `image` and the
|
|
status subresource.
|
|
3. Ensure the operator sets `spec.image` on the CR it creates.
|
|
|
|
Do not hand-apply the CRD as a one-off. That reproduces exactly the situation that
|
|
caused this — a cluster object with no source of truth.
|
|
|
|
**To silence the loop without fixing it:** remove the `temporal.io/namespace: production`
|
|
label from `Queue/agent-worker-queue` in namespace `sqs`. The operator returns early when
|
|
the label is absent. Reversible by re-adding it.
|
|
|
|
---
|
|
|
|
## 2. `kmsvc.riotpiao.com` is unauthenticated
|
|
|
|
**Status:** open. Relevant to [task 7.2](../tasks/7.2-sqs-prefix.md).
|
|
|
|
`kmsvc-manage` has an auth interceptor at `internal/api/interceptors/auth.go`, but it is
|
|
never wired: `cmd/server/main.go` constructs a bare `grpc.NewServer()` with no
|
|
interceptor options. The live ConfigMap confirms it — `KMSVC_AUTHENTIK_ISSUER_URL` and
|
|
`KMSVC_AUTHENTIK_AUDIENCE` are both empty strings.
|
|
|
|
Both the REST surface (8080) and the gRPC surface (9090) are open.
|
|
|
|
There is a second, subtler problem. The REST surface is mounted with
|
|
`RegisterQueueServiceHandlerServer`, the **in-process** grpc-gateway variant that calls
|
|
the service implementation directly. It bypasses gRPC interceptors entirely. So even
|
|
once the interceptor is wired, it would authenticate gRPC callers only — the file's own
|
|
doc comment claiming it covers both REST and gRPC is wrong for this wiring.
|
|
|
|
Consequence for this gateway: `/sqs/*` must own authentication itself. Do not assume the
|
|
upstream will enforce anything.
|
|
|
|
---
|
|
|
|
## 3. `kmsvc-redis-master.sqs:6379` has no authentication
|
|
|
|
`ALLOW_EMPTY_PASSWORD=yes`, TLS off, Bitnami chart with `auth.enabled=false`, no password
|
|
secret in the namespace. Anything with network reach has full unauthenticated read/write.
|
|
|
|
A NetworkPolicy is the only control. Relevant to [task 6.2](../tasks/6.2-kubernetes-manifests.md).
|
|
|
|
---
|
|
|
|
## 4. `macos-bluebubbles` pod will never schedule
|
|
|
|
`sms` Argo Application is `Synced`/`Degraded`. The pod targets a macOS node that is not
|
|
in the cluster: `0/4 nodes are available: 4 node(s) didn't match Pod's node
|
|
affinity/selector`, roughly 1080 failed attempts over 3d18h.
|
|
|
|
Not transient. Needs either that node or removal of the Application. Unrelated to this
|
|
gateway; listed so the Degraded status is not mistaken for something new.
|
|
|
|
---
|
|
|
|
## 5. Documentation that does not match reality
|
|
|
|
- `kmsvc-manage/TEMPORAL_INTEGRATION.md` is aspirational. It documents
|
|
`apiVersion: temporal.kmsvc.io/v1` with `queueRef`, `taskQueueName` and `lifecycle`
|
|
fields, and one worker per Queue. Reality is `kmsvc.io/v1`, none of those fields, and
|
|
one worker per Temporal *namespace*. Do not source API documentation from it.
|
|
- Module paths disagree across repos: `kmsvc-proto` declares
|
|
`forgejo.riotpiao.homelab.com/...`, while `kmsvc-manage` and `kmsvc-sdk` import
|
|
`forgejo.riotpiao.com/...`. The `.homelab.com` domain is fully retired — every
|
|
subdomain NXDOMAINs.
|
|
- `kmsvc-cli` README says the gRPC ingress uses TLS passthrough. It uses
|
|
`nginx.ingress.kubernetes.io/backend-protocol: GRPC`, which terminates TLS at nginx.
|
|
Functionally fine for clients; the wording is wrong.
|