chore: initial commit of Go API gateway

Baseline for the Kong replacement on api.riotpiao.com. Brings the working tree under version control for the first time: gateway source, the task board that drives the agent runs, test fixtures, and K8s manifests. Anchor the gateway ignore rule to the repo root. Unanchored, "gateway" also matched the cmd/gateway/ source directory, so the program entrypoint was excluded from every commit. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-19 20:54:34 -07:00
commit 058f11cf2b
109 changed files with 8992 additions and 0 deletions
@@ -0,0 +1,324 @@
+# API — LLM surfaces
+
+Two protocol dialects over the same models and the same slot controller.
+
+| Prefix | Dialect | Endpoint | Client |
+|---|---|---|---|
+| `/v1` | OpenAI-compatible | `POST /v1/chat/completions` | pi, OpenAI SDKs |
+| `/llm` | Anthropic Messages | `POST /llm/v1/messages` | riotpiao frontend (first-party) |
+
+Status marks below:
+**[LIVE]** verified against the running cluster on 2026-08-19.
+**[SPEC]** the contract this gateway must implement; not built yet.
+
+---
+
+## Models
+
+| `model` value | Upstream | Engine | Context | Notes |
+|---|---|---|---|---|
+| `reasoning` | `reasoning-predictor.llm-serving:80` | vLLM, DeepSeek-R1-Distill-Qwen-32B | 16384 | emits `reasoning_content`; 8 sequence slots total |
+| `ornith:35b` | `ornith-predictor.llm-serving:80` | Ollama | 131072 | reliable tool calling |
+| `qwen2.5:3b-instruct` | `ornith-predictor.llm-serving:80` | Ollama | 32768 | same pods as ornith |
+| `nomic-ai/nomic-embed-text-v2-moe` | `embeddings-predictor.llm-serving:80` | TEI | — | embeddings only |
+| `BAAI/bge-reranker-base` | `reranker-predictor.llm-serving:80` | TEI | — | rerank only |
+
+`reasoning` runs 2 replicas x `--max-num-seqs=4`. Those **8 slots are the scarcest resource in the cluster** and are shared across both dialects.
+
+---
+
+## Authentication [SPEC]
+
+Ships behind a flag, default off. The model API is unauthenticated today.
+
+```
+Authorization: Bearer <authentik-jwt>
+```
+
+### Decided — Bearer on both surfaces
+
+`Authorization: Bearer <jwt>` is the only accepted credential, on `/v1` and `/llm` alike. One auth path, consistent with G5, validated against Authentik via JWKS.
+
+**Known divergence from Anthropic:** the real Anthropic API authenticates with `x-api-key` and requires `anthropic-version: 2023-06-01`. A stock Anthropic SDK pointed at `/llm` will send `x-api-key` and get a 401.
+
+This is accepted, not overlooked. The `/llm` client is the first-party riotpiao frontend, which sends whatever we tell it to. If a real Anthropic SDK ever needs to reach this gateway, accepting `x-api-key` as a second credential source is an additive change — a small branch in one middleware, not a redesign.
+
+`anthropic-version` is accepted and ignored if present, and never required.
+
+The 401 for an `x-api-key`-only request must name the problem — say that Bearer is required — rather than returning a bare 401. The Kong retirement was caused by exactly this failure mode: a gateway that rejected the header clients actually send, without saying why.
+
+---
+
+## OpenAI dialect — `POST /v1/chat/completions`
+
+### Request [SPEC]
+
+```json
+{
+  "model": "reasoning",
+  "messages": [{"role": "user", "content": "Why is wave 4 empty?"}],
+  "max_tokens": 2000,
+  "temperature": 0.7,
+  "stream": false
+}
+```
+
+`model` is required and selects the upstream. The body is forwarded byte-identical — the gateway reads `model`, it does not rewrite it.
+
+### Response, non-streaming [LIVE]
+
+Captured verbatim from `reasoning` on 2026-08-19, abridged:
+
+```json
+{
+  "id": "chatcmpl-f17bd2fe22e4276d24e9438e40e89cea",
+  "object": "chat.completion",
+  "created": 1787172340,
+  "model": "reasoning",
+  "choices": [{
+    "index": 0,
+    "message": {
+      "role": "assistant",
+      "content": "\n\nTo find the current weather in Toronto...",
+      "reasoning_content": "Okay, so I need to figure out...",
+      "tool_calls": []
+    },
+    "finish_reason": "length"
+  }],
+  "usage": {"prompt_tokens": 16, "completion_tokens": 300, "total_tokens": 316}
+}
+```
+
+`reasoning_content` is a **sibling of** `content`, not nested in it. This is a vLLM extension produced by `--reasoning-parser=deepseek_r1`; it is not part of the OpenAI spec. Pass it through untouched.
+
+### The two engines disagree on the field name [LIVE]
+
+Verified 2026-08-19 by calling both:
+
+| Upstream | Engine | Reasoning field |
+|---|---|---|
+| `reasoning-predictor` | vLLM | `reasoning_content` |
+| `ornith-predictor` | Ollama | `reasoning` |
+
+Neither is in the OpenAI spec, so neither is wrong — they are two vendor extensions that
+happen to mean the same thing. The gateway must recognise **both** when mapping to the
+Anthropic `thinking` block, or `ornith:35b` responses will silently lose their reasoning
+on the `/llm` surface.
+
+Do not normalise them on the `/v1` surface. That surface passes bodies through
+untouched, and a client asking for `ornith:35b` should get exactly what Ollama sent.
+Normalisation belongs in the canonical request model (task 2.9), which is the layer that
+exists to absorb precisely this kind of upstream difference.
+
+### Response, streaming [SPEC]
+
+```
+data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"reasoning_content":"Okay"},"finish_reason":null}]}
+
+data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Wave"},"finish_reason":null}]}
+
+data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
+
+data: [DONE]
+```
+
+Data-only frames, no `event:` lines. Terminated by the literal `data: [DONE]`.
+
+### Legacy aliases [LIVE, being retired]
+
+`POST /v1/{reasoning,ornith,qwen}/chat/completions` force `model` to the corresponding value regardless of the body. They exist only because Kong could not dispatch on the body. Removed once callers migrate.
+
+### `GET /v1/models` [SPEC]
+
+```json
+{"object":"list","data":[{"id":"reasoning","object":"model","owned_by":"homelab","created":0}]}
+```
+
+Derived from the registry, never hardcoded.
+
+### Errors [SPEC]
+
+RFC 9457 `application/problem+json`:
+
+```json
+{
+  "type": "https://riotpiao.com/errors/unknown-model",
+  "title": "Unknown model",
+  "status": 400,
+  "detail": "\"gpt-4\" is not available",
+  "validModels": ["reasoning", "ornith:35b", "qwen2.5:3b-instruct"]
+}
+```
+
+---
+
+## Anthropic dialect — `POST /llm/v1/messages` [SPEC]
+
+Path note: the Anthropic SDK appends `/v1/messages` to its base URL, so a base URL of `https://api.riotpiao.com/llm` produces exactly this path.
+
+### Request
+
+```json
+{
+  "model": "reasoning",
+  "max_tokens": 2000,
+  "system": "You are a cluster assistant.",
+  "messages": [
+    {"role": "user", "content": "Why is wave 4 empty?"}
+  ],
+  "stream": true
+}
+```
+
+Differences from the OpenAI dialect that the translator must handle:
+
+| Concern | OpenAI | Anthropic |
+|---|---|---|
+| system prompt | `messages[0].role = "system"` | top-level `system` field |
+| `max_tokens` | optional | **required** |
+| content | string | string *or* block array |
+| roles | system/user/assistant/tool | user/assistant only |
+| stop | `stop` | `stop_sequences` |
+
+`max_tokens` being required is a real divergence — the gateway must either reject its absence with a clear error or apply a documented default. Pick one and state it; do not silently default.
+
+### Response, non-streaming
+
+```json
+{
+  "id": "msg_01ABC",
+  "type": "message",
+  "role": "assistant",
+  "model": "reasoning",
+  "content": [
+    {"type": "thinking", "thinking": "Waves are sort keys, not a sequence..."},
+    {"type": "text", "text": "Wave 4 is empty. Waves are sort keys..."}
+  ],
+  "stop_reason": "end_turn",
+  "stop_sequence": null,
+  "usage": {"input_tokens": 16, "output_tokens": 300}
+}
+```
+
+Field mapping from the upstream OpenAI response:
+
+| Upstream | Anthropic |
+|---|---|
+| `choices[0].message.reasoning_content` | `content[]` block `{"type":"thinking","thinking":...}` |
+| `choices[0].message.content` | `content[]` block `{"type":"text","text":...}` |
+| `finish_reason: "stop"` | `stop_reason: "end_turn"` |
+| `finish_reason: "length"` | `stop_reason: "max_tokens"` |
+| `usage.prompt_tokens` | `usage.input_tokens` |
+| `usage.completion_tokens` | `usage.output_tokens` |
+
+The thinking block precedes the text block.
+
+### Response, streaming
+
+Anthropic SSE uses **named events with content-block indices**, unlike OpenAI's flat frames. Verified event sequence:
+
+```
+event: message_start
+data: {"type":"message_start","message":{"id":"msg_01ABC","type":"message","role":"assistant","model":"reasoning","content":[],"usage":{"input_tokens":16,"output_tokens":0}}}
+
+event: content_block_start
+data: {"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":""}}
+
+event: content_block_delta
+data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"Waves are sort keys"}}
+
+event: content_block_stop
+data: {"type":"content_block_stop","index":0}
+
+event: content_block_start
+data: {"type":"content_block_start","index":1,"content_block":{"type":"text","text":""}}
+
+event: content_block_delta
+data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"Wave 4 is empty."}}
+
+event: content_block_stop
+data: {"type":"content_block_stop","index":1}
+
+event: message_delta
+data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":300}}
+
+event: message_stop
+data: {"type":"message_stop"}
+```
+
+Three details that are easy to get wrong:
+
+- In `message_delta`, `usage` is a **sibling of** `delta`, not inside it.
+- The delta field name matches the delta type: `thinking_delta` carries `.thinking`, `text_delta` carries `.text`.
+- Block index 0 is thinking, index 1 is text. **You only learn reasoning has ended when `content` first appears in an upstream chunk**, so the thinking block must be closed before the text block opens. If a response has no `reasoning_content` at all, the text block is index 0 and no thinking block is emitted.
+
+### Queue position — non-standard extension
+
+Anthropic's event set has no way to say "you are queued", because the stream implicitly begins after a slot is acquired. With only 8 slots, queueing is normal here.
+
+Emitted **before** `message_start`:
+
+```
+event: queue
+data: {"type":"queue","position":3}
+```
+
+This is deliberately outside the Anthropic spec. It is safe only because the client is first-party; a strict Anthropic client would ignore the unknown event and show nothing while queued.
+
+### Errors
+
+Anthropic error shape, **not** RFC 9457 — the same rejection renders differently depending on which surface received it:
+
+```json
+{"type":"error","error":{"type":"invalid_request_error","message":"Unknown model \"gpt-4\". Available: reasoning, ornith:35b, qwen2.5:3b-instruct"}}
+```
+
+| Condition | HTTP | `error.type` |
+|---|---|---|
+| unknown or missing model | 400 | `invalid_request_error` |
+| `max_tokens` absent (if required) | 400 | `invalid_request_error` |
+| malformed JSON | 400 | `invalid_request_error` |
+| unsupported feature requested | 400 | `invalid_request_error` |
+| not authenticated | 401 | `authentication_error` |
+| budget exhausted or queue full | 429 | `rate_limit_error` |
+| upstream failure | 502 | `api_error` |
+
+### Deliberately not implemented
+
+Each returns 400 naming the unsupported feature — never a silent partial implementation:
+
+tool use and `tool_result` turns, image content blocks, prompt-caching headers, the batch API, multi-block user content, `thinking.budget_tokens` configuration.
+
+The target client is the riotpiao frontend. Widening scope is a code change with a test, not an accident.
+
+---
+
+## Shared behaviour, both dialects
+
+**One slot controller, keyed by upstream.** A `/v1` request and a `/llm` request contend for the same 8 `reasoning` slots and the same queue, in arrival order. Per-dialect semaphores would each believe they were within budget while together exceeding the physical limit.
+
+**Streaming is unbuffered** and a client disconnect cancels the upstream immediately. An orphaned generation holds a slot until it completes on its own, which for a 32B model on a Volta GPU can run to minutes.
+
+**Timeouts** [LIVE]: chat routes are connect 10s / read 1h / write 1h. The hour is deliberate — a 32B model on this hardware routinely exceeds 60s. Any shorter application cap is enforced in gateway logic, never by shortening the proxy timeout.
+
+**Tool calling** [LIVE]: `reasoning` honours an explicit `tool_choice` but returns `tool_calls: []` under `tool_choice: "auto"` — it reasons about the tool in prose instead. `ornith:35b` returns `finish_reason: "tool_calls"` correctly under `auto`. This is a model property; the gateway does not compensate for it.
+
+---
+
+## Examples
+
+```bash
+# OpenAI dialect
+curl -s https://api.riotpiao.com/v1/chat/completions \
+  -H 'content-type: application/json' \
+  -d '{"model":"reasoning","messages":[{"role":"user","content":"Why is wave 4 empty?"}],"max_tokens":500}'
+
+# Anthropic dialect, streaming
+curl -N -s https://api.riotpiao.com/llm/v1/messages \
+  -H 'content-type: application/json' \
+  -d '{"model":"reasoning","max_tokens":500,"stream":true,
+       "messages":[{"role":"user","content":"Why is wave 4 empty?"}]}'
+
+# model list
+curl -s https://api.riotpiao.com/v1/models
+```
@@ -0,0 +1,263 @@
+# API — queue surface (`/sqs/*`)
+
+Fronts the Kafka Management Service (`kmsvc`) in namespace `sqs`. SQS-shaped
+message-plane API over Kafka.
+
+Status marks:
+**[LIVE]** verified against the running cluster and the committed proto on 2026-08-19.
+**[SPEC]** the contract this gateway must implement; not built yet.
+
+Source of truth for shapes:
+`~/workplace/kmsvc-proto/proto/kafkamgmt/v1/queue_service.proto`.
+
+---
+
+## The important finding: a REST surface already exists [LIVE]
+
+**Do not build gRPC-to-JSON transcoding.** `kmsvc-manage` already mounts grpc-gateway:
+
+```go
+mux := runtime.NewServeMux()
+kafkamgmtv1.RegisterQueueServiceHandlerServer(ctx, mux, svc)
+```
+
+The upstream serves plain REST/JSON on **:8080** and plain gRPC on **:9090**. Neither
+gRPC-Web nor server reflection is enabled.
+
+So `/sqs/*` is a **path-stripping reverse proxy plus authentication**, not a protocol
+translator. That makes it dramatically cheaper than the LLM surface.
+
+```
+api.riotpiao.com/sqs/v1/queues/{q}/messages
+        |  strip /sqs, authenticate
+        v
+management-service.sqs.svc.cluster.local:8080/v1/queues/{q}/messages
+```
+
+Upstream: Deployment `management-service`, 3 replicas, HPA 3-9, Service ClusterIP
+`10.98.3.138`, ports `8080` (http) and `9090` (grpc).
+
+---
+
+## Endpoints [LIVE — HTTP annotations from the proto]
+
+Six operations. All unary. No streaming, no subscribe.
+
+| Method | Path (after `/sqs` strip) | RPC |
+|---|---|---|
+| POST | `/v1/queues/{queue_name}/messages` | `SendMessage` |
+| POST | `/v1/queues/{queue_name}/messages:batch` | `SendMessageBatch` |
+| GET | `/v1/queues/{queue_name}/messages` | `ReceiveMessage` |
+| DELETE | `/v1/queues/{queue_name}/messages/{receipt_handle}` | `DeleteMessage` |
+| POST | `/v1/queues/{queue_name}/messages:batchDelete` | `DeleteMessageBatch` |
+| PATCH | `/v1/queues/{queue_name}/messages/{receipt_handle}` | `ChangeMessageVisibility` |
+
+---
+
+## Two wire-format traps [LIVE]
+
+Both follow from grpc-gateway defaults, and both will surprise anyone who reads only
+the proto.
+
+**1. `bytes` fields are base64 in JSON.** `SendMessageRequest.message_body` and
+`Message.body` are proto `bytes`. The JSONPB marshaler encodes them as base64 strings.
+Sending raw text will not do what you expect.
+
+**2. Field names are lowerCamelCase.** `cmd/server/main.go` calls bare
+`runtime.NewServeMux()` with no marshaler options, so `OrigName` is false. The wire uses
+`messageBody`, `receiptHandle`, `maxNumberOfMessages` — not the snake_case names in the
+proto.
+
+Document both prominently or every first-time caller loses an hour.
+
+---
+
+## Message shapes [LIVE — from the proto]
+
+### Send
+
+```
+POST /sqs/v1/queues/agent-worker-queue/messages
+{
+  "messageBody": "aGVsbG8gd29ybGQ=",          // base64 of "hello world"
+  "messageAttributes": {"values": {"k": "v"}},
+  "messageGroupId": "",                        // FIFO only
+  "messageDeduplicationId": "",                // FIFO only
+  "delaySeconds": 0                            // 0-900
+}
+-> {"messageId": "...", "sequenceNumber": ""}  // sequenceNumber FIFO only
+```
+
+### Receive — long poll
+
+```
+GET /sqs/v1/queues/agent-worker-queue/messages
+      ?maxNumberOfMessages=10      // <= 10
+      &waitTimeSeconds=20          // 0-20
+      &visibilityTimeoutSeconds=30 // optional override
+
+-> {"messages": [{
+     "messageId": "...",
+     "receiptHandle": "...",
+     "body": "aGVsbG8gd29ybGQ=",
+     "attributes": {"values": {}},
+     "receiveCount": 1,
+     "messageGroupId": "",
+     "enqueuedAt": "2026-08-19T16:29:07Z"
+   }]}
+```
+
+### Delete — the ack
+
+```
+DELETE /sqs/v1/queues/agent-worker-queue/messages/{receiptHandle}
+-> {}
+```
+
+### Change visibility
+
+```
+PATCH /sqs/v1/queues/agent-worker-queue/messages/{receiptHandle}
+{"visibilityTimeoutSeconds": 60}   // 0-43200
+-> {}
+```
+
+### Batch
+
+Both batch calls take `entries[]` with a caller-assigned `id`, and return partial
+success:
+
+```json
+{"successful": [{"id": "1", "messageId": "..."}],
+ "failed":     [{"id": "2", "error": "..."}]}
+```
+
+A batch call can return 200 with entries in `failed`. Callers must inspect the body,
+not just the status.
+
+### Limits [LIVE — from the SDK]
+
+`MaxMessageBodyBytes = 262144` (256 KiB), `MaxReceiveMessages = 10`,
+`MaxWaitTimeSeconds = 20`.
+
+---
+
+## Semantics
+
+At-least-once, SQS-style. Receive leases a message for the visibility timeout; the
+caller must `DeleteMessage` to acknowledge. An un-deleted message reappears after the
+timeout and `receiveCount` increments. After `maxReceiveCount` (default 5) it goes to
+the DLQ if one is configured.
+
+**Long-polling matters for the gateway.** `waitTimeSeconds` up to 20 means a `GET` can
+legitimately hold open for 20 seconds returning nothing. Read timeouts must exceed that
+comfortably, and a client disconnect must cancel upstream — the same requirement as the
+LLM surface, for the same reason.
+
+---
+
+## Error mapping [SPEC]
+
+The SDK maps gRPC codes to sentinel errors; grpc-gateway maps them to HTTP. Use this as
+the gateway's status contract:
+
+| gRPC code | HTTP | SDK sentinel |
+|---|---|---|
+| `NotFound` | 404 | `ErrQueueNotFound` |
+| `AlreadyExists` | 409 | `ErrAlreadyExists` |
+| `InvalidArgument` | 400 | `ErrInvalidArgument` |
+| `Unauthenticated` | 401 | `ErrUnauthenticated` |
+| `ResourceExhausted` | 429 | `ErrMessageTooLarge` |
+
+Upstream errors arrive in the grpc-gateway envelope
+`{"code": 5, "message": "Not Found", "details": []}`. Decide deliberately whether
+`/sqs/*` passes that through or re-renders it as RFC 9457 to match `/v1/*`.
+Recommendation: **pass through**, so the gateway does not become a second, subtly
+different error vocabulary for the same upstream.
+
+---
+
+## Queue lifecycle is NOT in this API [LIVE]
+
+There is no `CreateQueue`, `DeleteQueue`, or `ListQueues` RPC. The proto says so
+explicitly:
+
+```proto
+// Queue lifecycle (create/delete/configure) is managed via the Queue CRD,
+// not this service
+```
+
+Queues are Kubernetes resources — `queues.kmsvc.io/v1`, namespaced. `kmsvc-cli`'s
+`create-queue` and `delete-queue` talk to the Kubernetes API, not to kmsvc.
+
+**This is a hard boundary for the gateway.** Exposing queue creation over `/sqs/*` would
+require the gateway to hold Kubernetes write credentials, which violates **G2**. Do not
+add it. If declarative queue management ever needs a public surface, it belongs behind a
+separate component with its own RBAC — not in the public edge process.
+
+Queue spec fields, for reference when reading a queue's configuration:
+`fifoQueue`, `isDLQ`, `deadLetterTargetQueue`, `delaySeconds` (0-900),
+`maxReceiveCount` (default 5), `messageRetentionPeriodSeconds` (default 345600),
+`visibilityTimeoutSeconds` (default 30), `minShards`, `maxShards` (default 8),
+`partitionsPerShard` (default 6), `shardSplitThresholdBytesPerSec`,
+`shardSplitCooldownSeconds`.
+
+Kafka topics are named `kmsvc.{queue}.shard-{id}` and are created by `queue-operator`
+directly via the Kafka Admin API — there are no `KafkaTopic` CRs.
+
+Currently one queue exists: `agent-worker-queue` in namespace `sqs`, phase `Ready`,
+1 shard.
+
+---
+
+## Authentication [SPEC]
+
+`Authorization: Bearer <jwt>`, same as every other gateway surface.
+
+**The upstream enforces nothing.** `kmsvc`'s auth interceptor exists but is never wired,
+and the REST surface is mounted with the in-process grpc-gateway variant that bypasses
+gRPC interceptors regardless. Both `:8080` and `:9090` are currently open, and
+`kmsvc.riotpiao.com` is publicly routed.
+
+The gateway is therefore the only authentication boundary for this surface. See
+[KNOWN-ISSUES.md](KNOWN-ISSUES.md) §2.
+
+---
+
+## Out of scope
+
+- **Workflow start.** Nothing in kmsvc starts a Temporal workflow — no such RPC exists,
+  and grep for `ExecuteWorkflow`/`StartWorkflow` across `kmsvc-manage`, `kmsvc-sdk` and
+  `kmsvc-cli` returns nothing. A caller dials `temporal-frontend.temporal.svc:7233`
+  with a Temporal SDK directly. A `/workflow/*` surface is net-new code, not a proxy
+  route — see [task 7.3](../tasks/7.3-workflow-prefix.md) and KNOWN-ISSUES.md §1.
+- **DLQ operations.** `kmsvc-cli`'s `dlq peek` and `dlq redrive` are client-side
+  compositions of the six RPCs, not server operations. Redrive is a non-atomic
+  Receive-Send-Delete. If `/sqs/*` should offer redrive, that is new logic with real
+  failure modes, not a proxied call.
+- **Kafka direct access.** No external listener exists; the bootstrap
+  `kmsvc-kafka-bootstrap.sqs.svc.cluster.local:9092` is cluster-internal only. The
+  gateway proxies kmsvc, never Kafka.
+
+---
+
+## Examples
+
+```bash
+Q=agent-worker-queue
+
+# send (body must be base64)
+curl -s -X POST https://api.riotpiao.com/sqs/v1/queues/$Q/messages \
+  -H 'content-type: application/json' \
+  -d "{\"messageBody\":\"$(printf 'hello world' | base64)\"}"
+
+# receive, long poll 20s
+curl -s "https://api.riotpiao.com/sqs/v1/queues/$Q/messages?maxNumberOfMessages=10&waitTimeSeconds=20"
+
+# acknowledge
+curl -s -X DELETE https://api.riotpiao.com/sqs/v1/queues/$Q/messages/$RECEIPT
+
+# extend the lease
+curl -s -X PATCH https://api.riotpiao.com/sqs/v1/queues/$Q/messages/$RECEIPT \
+  -H 'content-type: application/json' -d '{"visibilityTimeoutSeconds":60}'
+```
@@ -0,0 +1,118 @@
+# Known cluster issues
+
+Pre-existing problems found while specifying this gateway. None are caused by this
+repo, and none block phases 0-6. Recorded so they are not rediscovered or mistaken
+for new breakage.
+
+Verified 2026-08-19 against context `admin@homelab-cluster`.
+
+---
+
+## 1. TemporalWorker CRD is stale — queue-operator reconcile fails every ~17 min
+
+**Status:** open, deliberately deferred. Affects [task 7.3](../tasks/7.3-workflow-prefix.md).
+
+The live `temporalworkers.kmsvc.io` CRD and the one in
+`~/workplace/kmsvc-manage/config/crd/kmsvc.io_temporalworkers.yaml` share exactly one
+field — `namespace`.
+
+| | spec properties |
+|---|---|
+| live CRD | `activityTypes`, `concurrency`, `namespace`, `taskQueue`, `workflowTypes` |
+| repo CRD | `affinity`, `image`, `imagePullPolicy`, `namespace`, `nodeSelector`, `replicas`, `resources`, `tolerations` |
+
+The live schema has no `image` field, so the API server **prunes** `image` from the CR
+that `queue-operator` writes. `TemporalWorker/worker-production` ends up as
+`spec: {namespace: production}`, and the operator then fails to build a Deployment
+from it. The live CRD also lacks a status subresource, producing a second error.
+
+Observed on a loop, most recently 21:25:39Z:
+
+```
+failed to create or update deployment ... error: "Deployment.apps \"worker-production\"
+  is invalid: spec.template.spec.containers[0].image: Required value"
+Reconciler error ... "update status failed: temporalworkers.kmsvc.io
+  \"worker-production\" not found"
+```
+
+**Impact is narrower than it looks.** No worker Deployment has ever existed under this
+CRD, so nothing that was working has stopped. Temporal namespace `production` is
+registered and healthy; there is simply no worker polling it. The practical cost is log
+noise, not lost work. That is why this is deferred rather than treated as an incident.
+
+**Neither object is under GitOps.** The CRD and the `Queue/agent-worker-queue` CR both
+carry only `kubectl.kubernetes.io/last-applied-configuration` — no
+`argocd.argoproj.io/instance`, no tracking-id — and the Queue does not appear anywhere
+in the homelab repo. They were hand-applied and predate GitOps coverage.
+
+**Fix, when it is worth doing:**
+
+1. Bring `temporalworkers.kmsvc.io` and the Queue CR into the homelab GitOps repo.
+2. Apply the current CRD from `kmsvc-manage/config/crd`, which restores `image` and the
+   status subresource.
+3. Ensure the operator sets `spec.image` on the CR it creates.
+
+Do not hand-apply the CRD as a one-off. That reproduces exactly the situation that
+caused this — a cluster object with no source of truth.
+
+**To silence the loop without fixing it:** remove the `temporal.io/namespace: production`
+label from `Queue/agent-worker-queue` in namespace `sqs`. The operator returns early when
+the label is absent. Reversible by re-adding it.
+
+---
+
+## 2. `kmsvc.riotpiao.com` is unauthenticated
+
+**Status:** open. Relevant to [task 7.2](../tasks/7.2-sqs-prefix.md).
+
+`kmsvc-manage` has an auth interceptor at `internal/api/interceptors/auth.go`, but it is
+never wired: `cmd/server/main.go` constructs a bare `grpc.NewServer()` with no
+interceptor options. The live ConfigMap confirms it — `KMSVC_AUTHENTIK_ISSUER_URL` and
+`KMSVC_AUTHENTIK_AUDIENCE` are both empty strings.
+
+Both the REST surface (8080) and the gRPC surface (9090) are open.
+
+There is a second, subtler problem. The REST surface is mounted with
+`RegisterQueueServiceHandlerServer`, the **in-process** grpc-gateway variant that calls
+the service implementation directly. It bypasses gRPC interceptors entirely. So even
+once the interceptor is wired, it would authenticate gRPC callers only — the file's own
+doc comment claiming it covers both REST and gRPC is wrong for this wiring.
+
+Consequence for this gateway: `/sqs/*` must own authentication itself. Do not assume the
+upstream will enforce anything.
+
+---
+
+## 3. `kmsvc-redis-master.sqs:6379` has no authentication
+
+`ALLOW_EMPTY_PASSWORD=yes`, TLS off, Bitnami chart with `auth.enabled=false`, no password
+secret in the namespace. Anything with network reach has full unauthenticated read/write.
+
+A NetworkPolicy is the only control. Relevant to [task 6.2](../tasks/6.2-kubernetes-manifests.md).
+
+---
+
+## 4. `macos-bluebubbles` pod will never schedule
+
+`sms` Argo Application is `Synced`/`Degraded`. The pod targets a macOS node that is not
+in the cluster: `0/4 nodes are available: 4 node(s) didn't match Pod's node
+affinity/selector`, roughly 1080 failed attempts over 3d18h.
+
+Not transient. Needs either that node or removal of the Application. Unrelated to this
+gateway; listed so the Degraded status is not mistaken for something new.
+
+---
+
+## 5. Documentation that does not match reality
+
+- `kmsvc-manage/TEMPORAL_INTEGRATION.md` is aspirational. It documents
+  `apiVersion: temporal.kmsvc.io/v1` with `queueRef`, `taskQueueName` and `lifecycle`
+  fields, and one worker per Queue. Reality is `kmsvc.io/v1`, none of those fields, and
+  one worker per Temporal *namespace*. Do not source API documentation from it.
+- Module paths disagree across repos: `kmsvc-proto` declares
+  `forgejo.riotpiao.homelab.com/...`, while `kmsvc-manage` and `kmsvc-sdk` import
+  `forgejo.riotpiao.com/...`. The `.homelab.com` domain is fully retired — every
+  subdomain NXDOMAINs.
+- `kmsvc-cli` README says the gRPC ingress uses TLS passthrough. It uses
+  `nginx.ingress.kubernetes.io/backend-protocol: GRPC`, which terminates TLS at nginx.
+  Functionally fine for clients; the wording is wrong.
@@ -0,0 +1,143 @@
+# Kong retirement — inventory and cutover
+
+Everything Kong does on `api.riotpiao.com` today, and where it goes. Inventory
+verified live against context `admin@homelab-cluster` on 2026-08-19.
+
+Source of the objects being retired: `~/workplace/homelab/k8s/apps/api/` and
+`k8s/argocd/apps/55-api-gateway.yaml`.
+
+## What is running now
+
+Kong OSS 3.4.1, Helm chart from `https://charts.konghq.com`, DB-less, namespace
+`api`, Argo Application `kong` at sync wave 7. Two replicas. Fronted by
+`ingress-nginx` via Ingress `api/api`, which catch-alls `/` on `api.riotpiao.com`
+to `kong-proxy:80`.
+
+Eleven ReplicaSets exist on the Kong Deployment, the newest minutes old — this
+config is being actively iterated, so re-verify the inventory immediately before
+cutover.
+
+## Routing table to port
+
+Seven `ingressClassName: kong` Ingresses. Six in `llm-serving`, one in `agent-pod`.
+
+| Method | Path | Upstream | Transform applied by Kong |
+|---|---|---|---|
+| GET | `/v1/models` | — | `request-termination`: static 200 JSON, upstream never contacted |
+| POST | `/v1/reasoning/chat/completions` | `reasoning-predictor:80` | force body `model=reasoning`, rewrite URI to `/v1/chat/completions` |
+| POST | `/v1/ornith/chat/completions` | `ornith-predictor:80` | force body `model=ornith:35b`, rewrite URI |
+| POST | `/v1/qwen/chat/completions` | `ornith-predictor:80` | force body `model=qwen2.5:3b-instruct`, rewrite URI |
+| POST | `/v1/embeddings` | `embeddings-predictor:80` | none — TEI already serves the canonical path |
+| POST | `/v1/rerank` | `reranker-predictor:80` | rewrite URI to `/rerank` (TEI does not serve `/v1/rerank`) |
+| GET/WS | `/console`, `/run`, `/sessions` | `agent-hub:9090` (`agent-pod` ns) | none, `strip-path: false` |
+
+Upstream model map, from the manifest comments and confirmed live:
+
+- `reasoning` → `reasoning-predictor` — vLLM, DeepSeek-R1-Distill-Qwen-32B, 2 replicas,
+  `--max-num-seqs=4`, `--max-model-len=16384`, `--reasoning-parser=deepseek_r1`,
+  `--enable-auto-tool-choice --tool-call-parser=hermes`
+- `ornith:35b` → `ornith-predictor` — Ollama, 2 replicas
+- `qwen2.5:3b-instruct` → `ornith-predictor` — same pods; both models stay resident via
+  `OLLAMA_MAX_LOADED_MODELS=2`, `OLLAMA_KEEP_ALIVE=-1`
+- `nomic-ai/nomic-embed-text-v2-moe` → `embeddings-predictor` — TEI
+- `BAAI/bge-reranker-base` → `reranker-predictor` — TEI
+
+### The path-per-model surface goes away
+
+The three chat paths exist only because Kong OSS cannot dispatch on the request
+body. The gateway serves a single `POST /v1/chat/completions` and selects the
+upstream from the body's `model` field.
+
+Keep the old paths as aliases during cutover so live clients do not break, then
+remove them once callers have migrated. pi is a live caller today.
+
+### `/v1/models` should not be ported verbatim
+
+Kong serves a hardcoded list via `request-termination`. The manifest already flags
+that it can drift from what the engines actually serve. Derive the response from
+the gateway's configured upstream map instead, so the list cannot disagree with
+what routing will accept.
+
+## Plugins being retired
+
+| Plugin | Scope | Replacement |
+|---|---|---|
+| `llm-rewrite-reasoning` / `-ornith` / `-qwen` | llm-serving | body-based dispatch in `internal/llm` |
+| `llm-rewrite-rerank` | llm-serving | per-upstream path rewrite in the route table |
+| `llm-models-list` | llm-serving | derived from the upstream map |
+| `prometheus` | **cluster-wide** | `internal/observability` — must expose bandwidth, latency, status codes, upstream health or observability regresses |
+
+No `rate-limiting` plugin exists anywhere in the cluster. REQUIREMENTS.md §4 Tier 2
+describes it as an existing layer; it is not built. Nothing to migrate — it is net
+new work, and it now belongs in the gateway rather than in Kong.
+
+## Auth: currently off, must land on
+
+`KongConsumer model-invoker` exists in namespace `api` and stays defined, but the
+`key-auth` plugin is commented out and every route has `model-key-auth` stripped
+from its `konghq.com/plugins` annotation.
+
+**The model API is unauthenticated right now.** Confirmed live 2026-08-19: a request
+to `/v1/reasoning/chat/completions` with no credentials returns 200.
+
+The reason is recorded in `model-auth.yaml` — Kong's `key-auth` accepts a raw
+`apikey:` header but rejects `Authorization: Bearer`, which blocks every
+OpenAI-compatible client. That is why `~/.pi/agent/models.json` carries a
+`customHeaders: {apikey: ...}` block.
+
+The gateway reads Bearer tokens directly and validates them against Authentik via
+JWKS. `AUTH-PLAN.md`'s pinned-RSA-key approach and its rotation runbook are not
+needed and should not be carried over.
+
+Ship auth behind a flag. Turning it on breaks every current caller until they hold
+a token — pi included.
+
+## Timeouts
+
+Kong today:
+
+| Route class | connect | read | write |
+|---|---|---|---|
+| chat | 10s | **1h** | 1h |
+| embeddings / rerank | 10s | 10m | 10m |
+
+nginx in front sets `proxy-read-timeout: 3600`, `proxy-send-timeout: 3600`,
+`proxy-buffering: off`, `proxy-body-size: 0`. Those stay — they are what makes token
+streaming work, and the gateway needs the same treatment from nginx.
+
+The 1-hour read timeout is deliberate: a 32B model on a Volta GPU routinely exceeds
+60s. Any shorter server-side cap must be enforced *in the gateway*, not by shortening
+the proxy timeout, or long legitimate generations get truncated mid-stream.
+
+## Cutover
+
+Reversible at every step. Kong keeps serving until the last step.
+
+1. Deploy the gateway alongside Kong, unexposed. Verify in-cluster against
+   `http://homelab-frontend.api.svc.cluster.local`.
+2. Compare gateway and Kong responses for every route in the table above, including
+   a streaming chat request and a client disconnect mid-stream.
+3. Repoint Ingress `api/api` from `kong-proxy:80` to the gateway Service. **This is
+   the cutover.** Reverting is a one-line change to the same Ingress.
+4. Soak. Watch gateway metrics and pi traffic.
+5. Delete the seven kong-class Ingresses and the six KongPlugin CRs.
+6. Remove the `kong` Application from `k8s/argocd/apps/55-api-gateway.yaml`; let Argo
+   prune the Helm release, the CRDs and namespace leftovers.
+
+Steps 1–4 are reversible in seconds. Step 5 onward is not — do not start it until the
+soak is clean.
+
+All of this flows through git and Argo. No `kubectl apply`, no `helm upgrade`.
+
+## Loose ends
+
+- `agent-pod/console` is publicly routed, unauthenticated, accepts free-form prompts
+  into a shell-capable container, and exposes a WebSocket. Migrating it behind the
+  gateway's auth is a security fix, not merely a port. Treat WebSocket upgrade as an
+  explicit requirement of the proxy layer.
+- Eight `*.example.com` hosts exist on istio-class Ingresses in `llm-serving`
+  (`{embeddings,ornith,reasoning,reranker}[-predictor]-llm-serving.example.com`).
+  KServe defaults, not public, not Kong's — out of scope here, but they exist and
+  should not be mistaken for gateway routes.
+- Ingress class split across the cluster is 7 kong / 17 nginx / 4 istio. Only the 7
+  kong ones are in scope.
@@ -0,0 +1,136 @@
+# ADR-0001 — Retire Kong OSS in favour of a Go API gateway
+
+Status: Accepted
+Date: 2026-08-19
+Deciders: rock
+
+## Context
+
+`api.riotpiao.com` is currently served by Kong OSS 3.4.1 (Helm, DB-less, namespace `api`,
+Argo wave 7), sitting behind ingress-nginx which owns TLS. Kong routes to the KServe
+model predictors in `llm-serving` via seven `ingressClassName: kong` Ingresses and six
+`KongPlugin` CRs.
+
+Three separate capabilities were attempted on Kong OSS. All three failed, and each
+failure is already documented in-repo by the person who hit it:
+
+**1. Body-based model dispatch is not expressible.**
+From `k8s/apps/api/llm-routes.yaml`:
+
+> a single `/v1/chat/completions` endpoint that dispatches on the body's `model` field is
+> not expressible in Kong OSS (`ai-proxy-advanced`, which does multi-target model routing,
+> is Enterprise-only).
+
+The workaround is a path-per-model surface (`/v1/reasoning/chat/completions`,
+`/v1/ornith/...`, `/v1/qwen/...`) with a `request-transformer` force-overwriting the body's
+`model` field. This is not OpenAI-standard, so every client needs bespoke configuration —
+visible today in `~/.pi/agent/models.json`, which carries three separate provider entries
+for what should be one endpoint.
+
+**2. OIDC is Enterprise-only.**
+`k8s/apps/api/AUTH-PLAN.md` routes around the missing `openid-connect` plugin using the
+built-in `jwt` plugin, which requires pinning Authentik's RSA public key onto a
+KongConsumer. That plan lists its own consequence:
+
+> Pinning `rsa_public_key`: Authentik key rotation would break it — document a rotation
+> runbook, or have the provision script re-export the cert PEM into the Kong credential on
+> each run.
+
+A rotation runbook is a standing operational liability accepted only because the gateway
+cannot fetch JWKS itself.
+
+**3. `key-auth` cannot read `Authorization: Bearer`.**
+From `k8s/apps/api/model-auth.yaml`:
+
+> a raw `apikey: <key>` header succeeds (200), the same request with only
+> `Authorization: Bearer <key>` fails (401). No OpenAI-SDK-compatible client (pi included)
+> sends a raw apikey header or lets you customize the header name, so every such client was
+> hard-blocked.
+
+Consequence: authentication on the model routes is **currently disabled**. Verified live
+2026-08-19 — `api.riotpiao.com/v1/reasoning/chat/completions` answers unauthenticated.
+
+Separately, the intended surface has grown beyond LLM routing. The target is a
+capability-per-subdomain API over cluster services — `sqs.riotpiao.com` for queue
+operations, `workflow.riotpiao.com` for Temporal, `cluster.riotpiao.com` for atlas — each
+needing request shaping, per-caller budgets and streaming semantics that are application
+concerns, not gateway-plugin concerns.
+
+## Decision
+
+Retire Kong OSS entirely. Replace it with a purpose-built Go service,
+`homelab-frontend`, which owns north-south routing, authentication, and request shaping
+for every public capability on `*.riotpiao.com`.
+
+ingress-nginx keeps the edge and TLS. It forwards to the gateway instead of `kong-proxy`.
+
+Authentication is Authentik OIDC, validated by fetching JWKS from
+`https://authentik.riotpiao.com` at runtime.
+
+## Options considered
+
+**A. Stay on Kong OSS, accept the workarounds.**
+Keeps a battle-tested proxy and its Prometheus plugin. But the path-per-model surface stays
+non-standard, the RSA pinning runbook stays, and auth stays off until someone writes a
+`request-transformer` shim to copy Bearer into an `apikey` header. Every new capability
+(`sqs`, `workflow`) inherits the same constraints.
+
+**B. Buy Kong Enterprise.**
+`ai-proxy-advanced` and `openid-connect` solve 1 and 2. Does not solve the genuinely
+application-level requirements at all — signed session cookies, per-session daily message
+budgets, a 6-of-8 GPU sequence-slot semaphore with a bounded queue, and
+disconnect-cancels-upstream are not gateway features in any tier. Cost for a homelab is not
+justifiable.
+
+**C. Go gateway, Kong retained for LLM paths only.**
+Gradual migration, lower risk. But it means running two gateways indefinitely, splitting the
+routing table across Kong CRDs and Go code, and keeping the Kong Helm release and its CRDs.
+The split is the thing most likely to drift.
+
+**D. Go gateway, Kong retired entirely.** — chosen
+One routing table, one auth implementation, one place to reason about timeouts. The logic
+being replaced is small: four `request-transformer` plugins that set a body field and
+rewrite a URI, one `request-termination` serving a static JSON model list, and one
+`prometheus` plugin. That is on the order of a hundred lines of Go, against roughly 480
+lines of YAML it retires.
+
+## Consequences
+
+### Gained
+
+- **Standard OpenAI surface.** One `POST /v1/chat/completions`, model selected from the
+  request body. Any OpenAI SDK works unmodified. The three pi provider entries collapse to
+  one.
+- **Working authentication.** Bearer tokens are read from the header, because it is our
+  code. JWKS is fetched and cached with automatic rotation handling, so the AUTH-PLAN.md
+  rotation runbook is deleted rather than written.
+- **Application-level policy becomes possible.** GPU slot semaphore, per-session budgets,
+  disconnect propagation and SSE handling live where the state is.
+- **One timeout story.** Kong currently sets `read-timeout: 3600000` (1 hour) on chat
+  routes, which silently defeats any shorter server-side cap. Retiring Kong removes the
+  conflicting layer.
+- **~480 lines of gateway YAML deleted**, plus the Kong CRDs, the Helm release, and its
+  `ServerSideApply` workaround for oversized CRD annotations.
+
+### Lost / assumed
+
+- **We now own proxy correctness.** Connection pooling, retries, timeout propagation,
+  streaming passthrough, header hygiene, graceful shutdown. `net/http/httputil.ReverseProxy`
+  covers most of it, but it is our bug surface now.
+- **Kong's Prometheus plugin goes away.** The gateway must expose equivalent metrics itself
+  (bandwidth, latency, status codes, upstream health) or observability regresses.
+- **Migration touches live traffic.** pi depends on `api.riotpiao.com` today. Cutover must
+  be reversible — see `docs/MIGRATION-kong.md`.
+- **`agent-pod/console` is a kong-class Ingress** exposing `/console` (WebSocket), `/run`
+  and `/sessions`. It must migrate too, and it is currently unauthenticated and publicly
+  routed while accepting free-form prompts into a shell-capable container. Putting it behind
+  the gateway's Authentik auth is a security improvement, not just a port.
+
+### Risks
+
+- Enabling Authentik auth will break any client currently relying on the unauthenticated
+  surface — including pi, until its `models.json` is updated. Auth must ship behind a flag
+  and be enabled deliberately.
+- Kong's `request-termination` for `/v1/models` returns a **static** list that can drift
+  from what the engines actually serve. Porting it verbatim ports the bug; the gateway
+  should derive the list from configured upstreams instead.