chore: initial commit of Go API gateway
Baseline for the Kong replacement on api.riotpiao.com. Brings the working tree under version control for the first time: gateway source, the task board that drives the agent runs, test fixtures, and K8s manifests. Anchor the gateway ignore rule to the repo root. Unanchored, "gateway" also matched the cmd/gateway/ source directory, so the program entrypoint was excluded from every commit. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,76 @@
|
|||||||
|
name: CI
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main, master]
|
||||||
|
pull_request:
|
||||||
|
branches: [main, master]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
test:
|
||||||
|
name: Test
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Set up Go
|
||||||
|
uses: actions/setup-go@v5
|
||||||
|
with:
|
||||||
|
go-version: '1.25'
|
||||||
|
|
||||||
|
- name: Run tests with race detector
|
||||||
|
run: go test -race ./...
|
||||||
|
|
||||||
|
vet:
|
||||||
|
name: Vet
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Set up Go
|
||||||
|
uses: actions/setup-go@v5
|
||||||
|
with:
|
||||||
|
go-version: '1.25'
|
||||||
|
|
||||||
|
- name: Run go vet
|
||||||
|
run: go vet ./...
|
||||||
|
|
||||||
|
build:
|
||||||
|
name: Build
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Set up Go
|
||||||
|
uses: actions/setup-go@v5
|
||||||
|
with:
|
||||||
|
go-version: '1.25'
|
||||||
|
|
||||||
|
- name: Build static binary
|
||||||
|
run: CGO_ENABLED=0 go build -o gateway ./cmd/gateway
|
||||||
|
|
||||||
|
- name: Upload binary
|
||||||
|
uses: actions/upload-artifact@v4
|
||||||
|
with:
|
||||||
|
name: gateway
|
||||||
|
path: gateway
|
||||||
|
retention-days: 1
|
||||||
|
|
||||||
|
govulncheck:
|
||||||
|
name: Security (govulncheck)
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Set up Go
|
||||||
|
uses: actions/setup-go@v5
|
||||||
|
with:
|
||||||
|
go-version: '1.25'
|
||||||
|
|
||||||
|
- name: Run govulncheck
|
||||||
|
run: go install golang.org/x/vuln/cmd/govulncheck@latest && govulncheck ./...
|
||||||
|
continue-on-error: true
|
||||||
+23
@@ -0,0 +1,23 @@
|
|||||||
|
# Binaries
|
||||||
|
/bin/
|
||||||
|
/dist/
|
||||||
|
# Anchored: an unanchored "gateway" also matches the cmd/gateway/ source directory.
|
||||||
|
/gateway
|
||||||
|
|
||||||
|
# Go
|
||||||
|
*.test
|
||||||
|
*.out
|
||||||
|
coverage.*
|
||||||
|
go.work
|
||||||
|
go.work.sum
|
||||||
|
|
||||||
|
# Local config and secrets — never commit
|
||||||
|
.env
|
||||||
|
*.local.yaml
|
||||||
|
*.local.yml
|
||||||
|
|
||||||
|
# Editor
|
||||||
|
.DS_Store
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
.task-runs
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
# homelab-frontend
|
||||||
|
|
||||||
|
A Go API gateway for the homelab cluster. One capability per subdomain, one auth
|
||||||
|
implementation, one routing table.
|
||||||
|
|
||||||
|
Replaces Kong OSS entirely — see
|
||||||
|
[ADR-0001](docs/adr/ADR-0001-retire-kong-for-go-gateway.md) for why, and
|
||||||
|
[docs/MIGRATION-kong.md](docs/MIGRATION-kong.md) for the cutover.
|
||||||
|
|
||||||
|
## Position in the stack
|
||||||
|
|
||||||
|
```
|
||||||
|
browser / SDK ──▶ Cloudflare ──▶ ingress-nginx (TLS, edge)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────┐
|
||||||
|
│ homelab-frontend │
|
||||||
|
│ routing · authn · budgets │
|
||||||
|
└──────────────┬──────────────┘
|
||||||
|
│
|
||||||
|
/v1 /sqs /workflow /cluster
|
||||||
|
│ │ │ │
|
||||||
|
▼ ▼ ▼ ▼
|
||||||
|
llm-serving kmsvc/Kafka temporal atlas
|
||||||
|
(predictors) (sqs ns) (temporal ns) (riotpiao-backend)
|
||||||
|
|
||||||
|
└──── in-cluster Services ────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
ingress-nginx keeps TLS and the edge. The gateway owns everything after it.
|
||||||
|
|
||||||
|
Backend services are reached through the gateway rather than published
|
||||||
|
individually — a single place for authentication, budgets, timeouts and
|
||||||
|
observability, and a single hostname surface to reason about.
|
||||||
|
|
||||||
|
## Capability map
|
||||||
|
|
||||||
|
One host, one path prefix per capability.
|
||||||
|
|
||||||
|
| Prefix on `api.riotpiao.com` | Backs onto | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| `/v1/*` | `llm-serving` predictors (vLLM, Ollama, TEI) | migrating off Kong |
|
||||||
|
| `/sqs/*` | kmsvc management-service + Kafka/Strimzi (`sqs` ns) | future |
|
||||||
|
| `/workflow/*` | Temporal (`temporal` ns) | future |
|
||||||
|
| `/cluster/*` | atlas — cluster topology / Argo delivery (separate repo) | future |
|
||||||
|
| `/db/*` | CloudNativePG, MinIO, monitoring/metrics reads | future |
|
||||||
|
|
||||||
|
`/v1/*` is reserved for the OpenAI-compatible surface. An SDK expects
|
||||||
|
`/v1/chat/completions` at the base URL, so that prefix cannot be repurposed.
|
||||||
|
|
||||||
|
Paths rather than subdomains: one DNS record, one tunnel hostname, one Ingress.
|
||||||
|
Promoting a prefix to its own subdomain later is additive and can run alongside the
|
||||||
|
path — the reverse is not, because clients hardcode hostnames.
|
||||||
|
|
||||||
|
atlas lives in its own repo (`riotpiao-backend`) and keeps its own informers and
|
||||||
|
RBAC. The gateway routes to it; it does not absorb it. Cluster-read permissions
|
||||||
|
stay out of the public edge process.
|
||||||
|
|
||||||
|
## Design rules
|
||||||
|
|
||||||
|
1. **Standard protocol shapes.** `POST /v1/chat/completions` selects its model from
|
||||||
|
the request body, like every OpenAI-compatible server. No path-per-model, no
|
||||||
|
bespoke client configuration. Kong OSS could not do this; that limitation does
|
||||||
|
not survive into the replacement.
|
||||||
|
2. **Bearer tokens, validated against Authentik.** JWKS is fetched at runtime and
|
||||||
|
cached, so key rotation needs no runbook and no pinned PEM.
|
||||||
|
3. **Policy lives where the state is.** GPU slot semaphores, per-caller budgets,
|
||||||
|
queue depth and disconnect propagation are application concerns. They belong
|
||||||
|
here, not in a proxy plugin.
|
||||||
|
4. **Streaming is first-class.** SSE and WebSocket pass through unbuffered, and a
|
||||||
|
client disconnect cancels the upstream request rather than orphaning it.
|
||||||
|
5. **The gateway holds no cluster credentials.** It proxies to services that do.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
cmd/gateway/ entrypoint
|
||||||
|
internal/
|
||||||
|
auth/ Authentik OIDC, JWKS cache, service-account tokens
|
||||||
|
llm/ model registry, body-based dispatch, upstream map
|
||||||
|
queue/ sqs.riotpiao.com surface
|
||||||
|
workflow/ workflow.riotpiao.com surface
|
||||||
|
proxy/ reverse proxy, streaming, timeouts, disconnect propagation
|
||||||
|
config/ upstream + route configuration
|
||||||
|
observability/ Prometheus metrics, structured logging
|
||||||
|
deploy/
|
||||||
|
base/ Kubernetes manifests
|
||||||
|
argocd/ Argo Application
|
||||||
|
docs/adr/ architecture decision records
|
||||||
|
tasks/ task board — see tasks/INDEX.md
|
||||||
|
testdata/ fixtures for offline tests
|
||||||
|
```
|
||||||
|
|
||||||
|
## Local development
|
||||||
|
|
||||||
|
The gateway must be runnable with no cluster, no kubeconfig and no credentials, so
|
||||||
|
that changes can be verified in a closed loop before touching live traffic.
|
||||||
|
Upstreams are configuration, so pointing them at local stubs is the whole
|
||||||
|
mechanism. See [tasks/INDEX.md](tasks/INDEX.md).
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Scaffolded 2026-08-19. Nothing is wired yet. Kong is still serving live traffic on
|
||||||
|
`api.riotpiao.com`.
|
||||||
+367
@@ -0,0 +1,367 @@
|
|||||||
|
# homelab-frontend — Requirements
|
||||||
|
|
||||||
|
The contract for the Go API gateway that replaces Kong OSS on `*.riotpiao.com`.
|
||||||
|
|
||||||
|
Companion documents:
|
||||||
|
- [docs/adr/ADR-0001-retire-kong-for-go-gateway.md](docs/adr/ADR-0001-retire-kong-for-go-gateway.md) — why Kong is being retired
|
||||||
|
- [docs/MIGRATION-kong.md](docs/MIGRATION-kong.md) — exact inventory of what Kong does today and the cutover order
|
||||||
|
- [tasks/INDEX.md](tasks/INDEX.md) — the task board
|
||||||
|
|
||||||
|
All cluster facts below were verified live against context `admin@homelab-cluster`
|
||||||
|
on 2026-08-19. Re-verify before relying on any number.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. Invariants
|
||||||
|
|
||||||
|
These hold for every surface. A change that breaks one of these is a design change,
|
||||||
|
not an implementation detail.
|
||||||
|
|
||||||
|
- **G1** — ingress-nginx owns TLS and the edge. The gateway never terminates TLS.
|
||||||
|
- **G2** — The gateway holds no Kubernetes credentials. It proxies to services that
|
||||||
|
do. Cluster-read permissions stay in atlas, out of the public edge process.
|
||||||
|
- **G3** — Public surfaces use standard protocol shapes. If an OpenAI SDK cannot
|
||||||
|
call it unmodified, the design is wrong.
|
||||||
|
- **G4** — Streaming is unbuffered end to end, and a client disconnect cancels the
|
||||||
|
upstream request rather than orphaning it.
|
||||||
|
- **G5** — Authentication is Bearer-token, validated against Authentik via JWKS
|
||||||
|
fetched at runtime. No pinned public keys, no rotation runbook.
|
||||||
|
- **G6** — Every route's timeout, body cap and concurrency limit is explicit in
|
||||||
|
configuration. No silent defaults.
|
||||||
|
- **G7** — All deployment flows through git and Argo. No `kubectl apply`, no
|
||||||
|
`helm upgrade`, no local `terraform apply`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Runtime and configuration
|
||||||
|
|
||||||
|
### 1.1 Process
|
||||||
|
|
||||||
|
Single static Go binary. Reads configuration at startup, serves HTTP, exits cleanly
|
||||||
|
on SIGTERM after draining in-flight requests.
|
||||||
|
|
||||||
|
Must run with **no cluster, no kubeconfig and no credentials** so that behaviour can
|
||||||
|
be verified in a closed loop before touching live traffic. Upstreams are
|
||||||
|
configuration; pointing them at local stubs is the entire mechanism. This is a hard
|
||||||
|
requirement, not a convenience — see §7.
|
||||||
|
|
||||||
|
### 1.2 Configuration
|
||||||
|
|
||||||
|
Route and upstream configuration is declarative and loaded at startup. It must
|
||||||
|
express, per upstream: address, path rewrite, connect/read/write timeouts, maximum
|
||||||
|
body size, and whether the route requires authentication.
|
||||||
|
|
||||||
|
Configuration errors fail startup loudly. A gateway that starts with a silently
|
||||||
|
dropped route is worse than one that refuses to start.
|
||||||
|
|
||||||
|
**Configuration lives in git**, mounted as a ConfigMap and synced by Argo. Not a
|
||||||
|
CRD. A CRD would require the gateway to watch the API server, which needs RBAC and
|
||||||
|
contradicts G2 — and CRD-driven routing is precisely the indirection being retired
|
||||||
|
with Kong, where the routing table was split across six `KongPlugin` CRs, seven
|
||||||
|
Ingresses and a Helm values file.
|
||||||
|
|
||||||
|
A CRD earns its keep when someone other than the repo owner must register routes.
|
||||||
|
That is not true here. If it becomes true, the additive answer is a controller that
|
||||||
|
renders this same ConfigMap — the gateway stays credential-free either way.
|
||||||
|
|
||||||
|
### 1.3 Health
|
||||||
|
|
||||||
|
- `GET /healthz` — liveness, no upstream checks, always cheap.
|
||||||
|
- `GET /readyz` — readiness; may fail while configuration is invalid or JWKS has
|
||||||
|
never been successfully fetched.
|
||||||
|
|
||||||
|
Neither requires authentication.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Proxy core
|
||||||
|
|
||||||
|
### 2.1 Reverse proxying
|
||||||
|
|
||||||
|
Standard reverse proxy to configured upstreams. Connection reuse across requests.
|
||||||
|
Hop-by-hop headers stripped correctly. `X-Forwarded-*` set from the nginx-supplied
|
||||||
|
values, not fabricated.
|
||||||
|
|
||||||
|
### 2.2 Streaming
|
||||||
|
|
||||||
|
SSE and chunked responses pass through without buffering. Tokens must reach the
|
||||||
|
client as the upstream emits them, not on completion.
|
||||||
|
|
||||||
|
WebSocket upgrade must work — `agent-pod/console` depends on it.
|
||||||
|
|
||||||
|
### 2.3 Disconnect propagation
|
||||||
|
|
||||||
|
When a client disconnects, the upstream request is cancelled immediately. This is
|
||||||
|
load-bearing: an orphaned generation holds a vLLM sequence slot, and there are only
|
||||||
|
eight in the cluster.
|
||||||
|
|
||||||
|
### 2.4 Timeouts
|
||||||
|
|
||||||
|
Per-route, explicit. Current Kong values, which are deliberate and must be preserved
|
||||||
|
unless changed knowingly:
|
||||||
|
|
||||||
|
| Route class | connect | read | write |
|
||||||
|
|---|---|---|---|
|
||||||
|
| chat | 10s | 1h | 1h |
|
||||||
|
| embeddings, rerank | 10s | 10m | 10m |
|
||||||
|
|
||||||
|
The 1-hour read timeout exists because a 32B model on a Volta GPU routinely exceeds
|
||||||
|
60s. Any shorter application-level cap must be enforced *by the gateway's own
|
||||||
|
logic*, not by shortening the proxy timeout — otherwise long legitimate generations
|
||||||
|
truncate mid-stream.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. LLM surfaces — `api.riotpiao.com`
|
||||||
|
|
||||||
|
Two protocol dialects, permanently. Both translate into one dialect-neutral canonical
|
||||||
|
request, and both pass through **one shared slot controller** before reaching a
|
||||||
|
predictor.
|
||||||
|
|
||||||
|
| Prefix | Dialect | Primary client |
|
||||||
|
|---|---|---|
|
||||||
|
| `/v1/*` | OpenAI-compatible | pi, generic OpenAI SDKs |
|
||||||
|
| `/llm/*` | Anthropic Messages | the riotpiao frontend (first-party only) |
|
||||||
|
|
||||||
|
```
|
||||||
|
/v1/* (OpenAI) /llm/* (Anthropic)
|
||||||
|
| |
|
||||||
|
+-----------+------------+
|
||||||
|
v
|
||||||
|
canonical request dialect-neutral
|
||||||
|
v
|
||||||
|
slot controller keyed by UPSTREAM, not by route
|
||||||
|
v
|
||||||
|
reasoning-predictor / ornith-predictor
|
||||||
|
```
|
||||||
|
|
||||||
|
**The slot controller is keyed by upstream and shared across dialects.** Per-dialect
|
||||||
|
semaphores are wrong: the 8 sequence slots are physical, so two independent gates
|
||||||
|
would each believe they were within budget while together exceeding it. Requests from
|
||||||
|
both surfaces contend for the same slots and the same queue, in arrival order.
|
||||||
|
|
||||||
|
Dispatch, budgets, logging and metrics all operate on the canonical request. Adding a
|
||||||
|
third dialect later must not require touching the controller.
|
||||||
|
|
||||||
|
### 3.1 Body-based model dispatch
|
||||||
|
|
||||||
|
`POST /v1/chat/completions` selects its upstream from the request body's `model`
|
||||||
|
field. This is the single most important requirement in this document: it is the
|
||||||
|
capability Kong OSS lacked, and the reason the gateway exists.
|
||||||
|
|
||||||
|
Unknown or missing `model` is a client error with a useful message listing valid
|
||||||
|
values — not a 500, and not a silent fallback to a default model.
|
||||||
|
|
||||||
|
### 3.2 Upstream map
|
||||||
|
|
||||||
|
Verified live. `served-model-name` values are what clients send.
|
||||||
|
|
||||||
|
| `model` in body | Upstream Service | Engine |
|
||||||
|
|---|---|---|
|
||||||
|
| `reasoning` | `reasoning-predictor.llm-serving:80` | vLLM, DeepSeek-R1-Distill-Qwen-32B |
|
||||||
|
| `ornith:35b` | `ornith-predictor.llm-serving:80` | Ollama |
|
||||||
|
| `qwen2.5:3b-instruct` | `ornith-predictor.llm-serving:80` | Ollama, same pods |
|
||||||
|
| `nomic-ai/nomic-embed-text-v2-moe` | `embeddings-predictor.llm-serving:80` | TEI |
|
||||||
|
| `BAAI/bge-reranker-base` | `reranker-predictor.llm-serving:80` | TEI |
|
||||||
|
|
||||||
|
`reasoning` runs 2 replicas × `--max-num-seqs=4` = **8 concurrent sequence slots
|
||||||
|
total**, `--max-model-len=16384`, `--reasoning-parser=deepseek_r1`,
|
||||||
|
`--enable-auto-tool-choice --tool-call-parser=hermes`.
|
||||||
|
|
||||||
|
Note `ornith:35b` and `qwen2.5:3b-instruct` share pods; both stay resident via
|
||||||
|
`OLLAMA_MAX_LOADED_MODELS=2` and `OLLAMA_KEEP_ALIVE=-1`, so dispatching between them
|
||||||
|
does not trigger a model swap.
|
||||||
|
|
||||||
|
### 3.3 Path rewriting
|
||||||
|
|
||||||
|
Upstreams expect canonical paths. `/v1/chat/completions` and `/v1/embeddings` pass
|
||||||
|
through unchanged. Rerank is the exception: TEI serves `/rerank`, not `/v1/rerank`,
|
||||||
|
so that route rewrites.
|
||||||
|
|
||||||
|
### 3.4 Legacy path aliases
|
||||||
|
|
||||||
|
`/v1/{reasoning,ornith,qwen}/chat/completions` must keep working during cutover —
|
||||||
|
pi is a live caller. They behave exactly as the canonical endpoint with `model`
|
||||||
|
forced to the corresponding value, overriding whatever the body says.
|
||||||
|
|
||||||
|
These are temporary. They exist to make the cutover reversible, and are removed once
|
||||||
|
callers have migrated.
|
||||||
|
|
||||||
|
### 3.5 `GET /v1/models`
|
||||||
|
|
||||||
|
Derived from the configured upstream map, never hardcoded. Kong served a static
|
||||||
|
list, and its own manifest flags that the list can drift from what the engines
|
||||||
|
actually serve. The gateway's list must be incapable of disagreeing with what
|
||||||
|
routing will accept.
|
||||||
|
|
||||||
|
OpenAI list shape: `{"object":"list","data":[{"id","object":"model","owned_by","created"}]}`.
|
||||||
|
|
||||||
|
### 3.6 Behaviour to preserve
|
||||||
|
|
||||||
|
Verified against the live endpoint:
|
||||||
|
|
||||||
|
- The upstream returns `reasoning_content` separately from `content` for the
|
||||||
|
`reasoning` model. Pass both through untouched.
|
||||||
|
- Tool calling works with explicit `tool_choice`, and is unreliable with
|
||||||
|
`tool_choice: auto` on the R1-distill model. The gateway does not compensate for
|
||||||
|
this — it is a model property, not a gateway concern. Do not add retries or
|
||||||
|
rewriting to work around it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Authentication — Authentik
|
||||||
|
|
||||||
|
### 4.1 Current state
|
||||||
|
|
||||||
|
**The model API is unauthenticated today.** Confirmed live: `/v1/reasoning/chat/completions`
|
||||||
|
answers with no credentials.
|
||||||
|
|
||||||
|
Kong's `key-auth` was retired because it accepts a raw `apikey:` header but rejects
|
||||||
|
`Authorization: Bearer`, which hard-blocks every OpenAI-compatible client. See
|
||||||
|
`~/workplace/homelab/k8s/apps/api/model-auth.yaml`.
|
||||||
|
|
||||||
|
### 4.2 Requirement
|
||||||
|
|
||||||
|
Bearer tokens in `Authorization`, validated against Authentik
|
||||||
|
(`https://authentik.riotpiao.com`) by fetching and caching JWKS at runtime.
|
||||||
|
|
||||||
|
Key rotation must be handled by refetching JWKS, not by pinned PEMs. The
|
||||||
|
pinned-`rsa_public_key` approach in `AUTH-PLAN.md` and its rotation runbook exist
|
||||||
|
only to route around a Kong OSS limitation and must not be carried forward.
|
||||||
|
|
||||||
|
Service accounts obtain tokens via `client_credentials` against Authentik's token
|
||||||
|
endpoint.
|
||||||
|
|
||||||
|
### 4.3 Rollout
|
||||||
|
|
||||||
|
Auth ships behind a flag, defaulting off, and is enabled deliberately.
|
||||||
|
|
||||||
|
Enabling it breaks every current caller until they hold a token — pi included, whose
|
||||||
|
`models.json` currently sends a `customHeaders: {apikey: ...}` block that will need
|
||||||
|
replacing with a Bearer token.
|
||||||
|
|
||||||
|
### 4.4 Authorization
|
||||||
|
|
||||||
|
Beyond authentication, a token must be checked for the right to invoke the
|
||||||
|
capability it is calling. A token minted for queue access should not invoke a GPU.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Rate limiting and budgets
|
||||||
|
|
||||||
|
No `rate-limiting` plugin exists anywhere in the cluster today — this is net new
|
||||||
|
work, not a migration. Verified: six Kong plugins exist, none is `rate-limiting`.
|
||||||
|
|
||||||
|
Requirements, in priority order:
|
||||||
|
|
||||||
|
1. **GPU slot protection.** `reasoning` has 8 total sequence slots. Concurrent
|
||||||
|
in-flight requests to it must be capped below that, leaving operator headroom.
|
||||||
|
Excess requests queue up to a bounded depth, then are rejected with a retryable
|
||||||
|
status.
|
||||||
|
2. **Per-caller budgets.** Identified callers get a request budget over a window.
|
||||||
|
3. **Body size caps**, per route.
|
||||||
|
|
||||||
|
Rejections use RFC 9457 `application/problem+json` and set `Retry-After` where a
|
||||||
|
retry time is knowable.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Observability
|
||||||
|
|
||||||
|
Kong's cluster-wide `prometheus` plugin is being retired. The gateway must expose at
|
||||||
|
least equivalent signal or observability regresses at cutover: request rate,
|
||||||
|
latency, status codes, bandwidth, and upstream health, labelled by route and
|
||||||
|
upstream.
|
||||||
|
|
||||||
|
Gateway-specific signals that Kong could not provide, and which are the reason for
|
||||||
|
several requirements above: in-flight requests per upstream, queue depth, GPU slot
|
||||||
|
occupancy, and rejections by reason.
|
||||||
|
|
||||||
|
Structured logging. Every rejected request is logged with the reason. No secrets, no
|
||||||
|
tokens, no request bodies in logs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Local development and verification
|
||||||
|
|
||||||
|
An agent must be able to close a change/verify loop with no cluster, no kubeconfig
|
||||||
|
and no credentials. This is a hard requirement because it determines whether work can
|
||||||
|
proceed unattended.
|
||||||
|
|
||||||
|
Concretely: it must be possible to start the gateway locally, point it at stub
|
||||||
|
upstreams, issue requests, and assert on the responses — including streaming
|
||||||
|
responses and client disconnects.
|
||||||
|
|
||||||
|
Verification of any API-shaped task means asserting on the **actual HTTP response**:
|
||||||
|
status, headers, and body. "It compiles" and "it starts" are not verification.
|
||||||
|
|
||||||
|
Parity with Kong is verified by comparing gateway and Kong responses for the same
|
||||||
|
request, for every route in the migration inventory, before cutover.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Deployment
|
||||||
|
|
||||||
|
Container: distroless or scratch, `runAsNonRoot`, read-only root filesystem, all
|
||||||
|
capabilities dropped, `seccompProfile: RuntimeDefault`, no shell.
|
||||||
|
|
||||||
|
Image tags are commit SHAs, never `:latest` — Argo's `selfHeal` cannot roll out a
|
||||||
|
mutable tag reliably.
|
||||||
|
|
||||||
|
NetworkPolicy: egress only to the upstreams it proxies plus Authentik; ingress from
|
||||||
|
`ingress-nginx` only.
|
||||||
|
|
||||||
|
Deployed as an Argo Application in the `homelab-root` GitOps repo. Verified live:
|
||||||
|
zero Argo Applications anywhere in the cluster source from any Forgejo URL, so
|
||||||
|
`github.com/Riotpiaole/riotpiao.homelab.com` is authoritative.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Capability surface — path-based
|
||||||
|
|
||||||
|
Every capability is a path prefix on the single host `api.riotpiao.com`. One DNS
|
||||||
|
record, one Cloudflare tunnel hostname, one nginx Ingress, one Service.
|
||||||
|
|
||||||
|
| Prefix | Backs onto | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| `/v1/*` | `llm-serving` predictors | v1 — **reserved**, see below |
|
||||||
|
| `/sqs/*` | kmsvc management-service, Kafka/Strimzi (`sqs` ns) | future |
|
||||||
|
| `/workflow/*` | Temporal (`temporal` ns) | future |
|
||||||
|
| `/cluster/*` | atlas, separate repo `riotpiao-backend` | future |
|
||||||
|
| `/db/*` | CloudNativePG, MinIO, monitoring reads | future |
|
||||||
|
|
||||||
|
**`/v1/*` is reserved for the OpenAI-compatible surface and nothing else.** G3 pins
|
||||||
|
it: an SDK expects `/v1/chat/completions` at the base URL, so that prefix can never
|
||||||
|
be repurposed or nested. Every other capability gets its own prefix that cannot
|
||||||
|
collide with a current or future OpenAI path.
|
||||||
|
|
||||||
|
Subdomains are deliberately *not* used. Paths keep hostname configuration to a
|
||||||
|
single entry — and hostname configuration is the demonstrated failure mode here, as
|
||||||
|
the unresolved apex 403 shows. Promoting a prefix to its own subdomain later is an
|
||||||
|
additive host rule that can run alongside the path; the reverse is not, because
|
||||||
|
clients hardcode hostnames.
|
||||||
|
|
||||||
|
Notes carried from the cluster:
|
||||||
|
|
||||||
|
- Temporal namespace registration is automatic via queue-operator, never manual.
|
||||||
|
- `management-service` already exposes gRPC at `kmsvc.riotpiao.com`; the `/sqs`
|
||||||
|
prefix is a new surface, not a replacement for it.
|
||||||
|
- atlas keeps its own informers and RBAC. The gateway proxies to it and holds no
|
||||||
|
cluster credentials of its own (G2).
|
||||||
|
- `/db/*` read surfaces need particular care — see G2 before designing them.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Known cluster facts worth not rediscovering
|
||||||
|
|
||||||
|
- `kmsvc-redis-master.sqs:6379` has **no authentication** — `ALLOW_EMPTY_PASSWORD=yes`,
|
||||||
|
TLS off. Any workload with network reach has full unauthenticated read/write. A
|
||||||
|
NetworkPolicy is the only control.
|
||||||
|
- `reasoning-predictor` listens on port **80**, not 8080.
|
||||||
|
- `prometheus-operated.monitoring` is **headless** (ClusterIP None) — egress policies
|
||||||
|
need pod selectors, not ClusterIPs.
|
||||||
|
- `agent-pod/console` is publicly routed, unauthenticated, accepts free-form prompts
|
||||||
|
into a shell-capable container, and serves a WebSocket. Putting it behind gateway
|
||||||
|
auth is a security fix, not merely a port.
|
||||||
|
- Eight `*.example.com` hosts exist on istio-class Ingresses in `llm-serving`. KServe
|
||||||
|
defaults, not public, out of scope — do not mistake them for gateway routes.
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"log"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"os/signal"
|
||||||
|
"syscall"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/server"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
// Load configuration
|
||||||
|
cfg, err := config.Load()
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "failed to load config: %v\n", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Determine if auth is enabled by checking if any route requires it
|
||||||
|
authEnabled := false
|
||||||
|
for _, route := range cfg.Routes {
|
||||||
|
if route.Upstream.AuthRequired {
|
||||||
|
authEnabled = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create a basic handler (will be replaced with real routing later)
|
||||||
|
upstreamHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusNotFound)
|
||||||
|
fmt.Fprintf(w, "not found")
|
||||||
|
})
|
||||||
|
|
||||||
|
// Create server with health checker
|
||||||
|
srv := server.New(cfg.ListenAddr, cfg.ShutdownTimeout, nil)
|
||||||
|
|
||||||
|
// Initialize health checker with config validity and auth status
|
||||||
|
healthChecker := server.NewHealthChecker(true, authEnabled)
|
||||||
|
srv.SetHealthChecker(healthChecker)
|
||||||
|
|
||||||
|
// Create router that handles health endpoints and passes others to upstream
|
||||||
|
router := server.NewRouter(healthChecker, upstreamHandler)
|
||||||
|
srv.SetHandler(router)
|
||||||
|
|
||||||
|
// Set up signal handling
|
||||||
|
sigChan := make(chan os.Signal, 1)
|
||||||
|
signal.Notify(sigChan, syscall.SIGTERM, syscall.SIGINT)
|
||||||
|
|
||||||
|
// Start server in a goroutine
|
||||||
|
var serverErr error
|
||||||
|
go func() {
|
||||||
|
log.Printf("gateway listening on %s", srv.Addr())
|
||||||
|
serverErr = srv.ListenAndServe()
|
||||||
|
if serverErr != nil && serverErr != http.ErrServerClosed {
|
||||||
|
log.Printf("server error: %v", serverErr)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
// Wait for shutdown signal
|
||||||
|
sig := <-sigChan
|
||||||
|
log.Printf("received signal: %v", sig)
|
||||||
|
|
||||||
|
// Gracefully shutdown the server
|
||||||
|
if err := srv.Shutdown(context.Background()); err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "shutdown error: %v\n", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("gateway shutdown complete")
|
||||||
|
os.Exit(0)
|
||||||
|
}
|
||||||
+324
@@ -0,0 +1,324 @@
|
|||||||
|
# API — LLM surfaces
|
||||||
|
|
||||||
|
Two protocol dialects over the same models and the same slot controller.
|
||||||
|
|
||||||
|
| Prefix | Dialect | Endpoint | Client |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `/v1` | OpenAI-compatible | `POST /v1/chat/completions` | pi, OpenAI SDKs |
|
||||||
|
| `/llm` | Anthropic Messages | `POST /llm/v1/messages` | riotpiao frontend (first-party) |
|
||||||
|
|
||||||
|
Status marks below:
|
||||||
|
**[LIVE]** verified against the running cluster on 2026-08-19.
|
||||||
|
**[SPEC]** the contract this gateway must implement; not built yet.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Models
|
||||||
|
|
||||||
|
| `model` value | Upstream | Engine | Context | Notes |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `reasoning` | `reasoning-predictor.llm-serving:80` | vLLM, DeepSeek-R1-Distill-Qwen-32B | 16384 | emits `reasoning_content`; 8 sequence slots total |
|
||||||
|
| `ornith:35b` | `ornith-predictor.llm-serving:80` | Ollama | 131072 | reliable tool calling |
|
||||||
|
| `qwen2.5:3b-instruct` | `ornith-predictor.llm-serving:80` | Ollama | 32768 | same pods as ornith |
|
||||||
|
| `nomic-ai/nomic-embed-text-v2-moe` | `embeddings-predictor.llm-serving:80` | TEI | — | embeddings only |
|
||||||
|
| `BAAI/bge-reranker-base` | `reranker-predictor.llm-serving:80` | TEI | — | rerank only |
|
||||||
|
|
||||||
|
`reasoning` runs 2 replicas x `--max-num-seqs=4`. Those **8 slots are the scarcest resource in the cluster** and are shared across both dialects.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Authentication [SPEC]
|
||||||
|
|
||||||
|
Ships behind a flag, default off. The model API is unauthenticated today.
|
||||||
|
|
||||||
|
```
|
||||||
|
Authorization: Bearer <authentik-jwt>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Decided — Bearer on both surfaces
|
||||||
|
|
||||||
|
`Authorization: Bearer <jwt>` is the only accepted credential, on `/v1` and `/llm` alike. One auth path, consistent with G5, validated against Authentik via JWKS.
|
||||||
|
|
||||||
|
**Known divergence from Anthropic:** the real Anthropic API authenticates with `x-api-key` and requires `anthropic-version: 2023-06-01`. A stock Anthropic SDK pointed at `/llm` will send `x-api-key` and get a 401.
|
||||||
|
|
||||||
|
This is accepted, not overlooked. The `/llm` client is the first-party riotpiao frontend, which sends whatever we tell it to. If a real Anthropic SDK ever needs to reach this gateway, accepting `x-api-key` as a second credential source is an additive change — a small branch in one middleware, not a redesign.
|
||||||
|
|
||||||
|
`anthropic-version` is accepted and ignored if present, and never required.
|
||||||
|
|
||||||
|
The 401 for an `x-api-key`-only request must name the problem — say that Bearer is required — rather than returning a bare 401. The Kong retirement was caused by exactly this failure mode: a gateway that rejected the header clients actually send, without saying why.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## OpenAI dialect — `POST /v1/chat/completions`
|
||||||
|
|
||||||
|
### Request [SPEC]
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"model": "reasoning",
|
||||||
|
"messages": [{"role": "user", "content": "Why is wave 4 empty?"}],
|
||||||
|
"max_tokens": 2000,
|
||||||
|
"temperature": 0.7,
|
||||||
|
"stream": false
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`model` is required and selects the upstream. The body is forwarded byte-identical — the gateway reads `model`, it does not rewrite it.
|
||||||
|
|
||||||
|
### Response, non-streaming [LIVE]
|
||||||
|
|
||||||
|
Captured verbatim from `reasoning` on 2026-08-19, abridged:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "chatcmpl-f17bd2fe22e4276d24e9438e40e89cea",
|
||||||
|
"object": "chat.completion",
|
||||||
|
"created": 1787172340,
|
||||||
|
"model": "reasoning",
|
||||||
|
"choices": [{
|
||||||
|
"index": 0,
|
||||||
|
"message": {
|
||||||
|
"role": "assistant",
|
||||||
|
"content": "\n\nTo find the current weather in Toronto...",
|
||||||
|
"reasoning_content": "Okay, so I need to figure out...",
|
||||||
|
"tool_calls": []
|
||||||
|
},
|
||||||
|
"finish_reason": "length"
|
||||||
|
}],
|
||||||
|
"usage": {"prompt_tokens": 16, "completion_tokens": 300, "total_tokens": 316}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`reasoning_content` is a **sibling of** `content`, not nested in it. This is a vLLM extension produced by `--reasoning-parser=deepseek_r1`; it is not part of the OpenAI spec. Pass it through untouched.
|
||||||
|
|
||||||
|
### The two engines disagree on the field name [LIVE]
|
||||||
|
|
||||||
|
Verified 2026-08-19 by calling both:
|
||||||
|
|
||||||
|
| Upstream | Engine | Reasoning field |
|
||||||
|
|---|---|---|
|
||||||
|
| `reasoning-predictor` | vLLM | `reasoning_content` |
|
||||||
|
| `ornith-predictor` | Ollama | `reasoning` |
|
||||||
|
|
||||||
|
Neither is in the OpenAI spec, so neither is wrong — they are two vendor extensions that
|
||||||
|
happen to mean the same thing. The gateway must recognise **both** when mapping to the
|
||||||
|
Anthropic `thinking` block, or `ornith:35b` responses will silently lose their reasoning
|
||||||
|
on the `/llm` surface.
|
||||||
|
|
||||||
|
Do not normalise them on the `/v1` surface. That surface passes bodies through
|
||||||
|
untouched, and a client asking for `ornith:35b` should get exactly what Ollama sent.
|
||||||
|
Normalisation belongs in the canonical request model (task 2.9), which is the layer that
|
||||||
|
exists to absorb precisely this kind of upstream difference.
|
||||||
|
|
||||||
|
### Response, streaming [SPEC]
|
||||||
|
|
||||||
|
```
|
||||||
|
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"reasoning_content":"Okay"},"finish_reason":null}]}
|
||||||
|
|
||||||
|
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Wave"},"finish_reason":null}]}
|
||||||
|
|
||||||
|
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
|
||||||
|
|
||||||
|
data: [DONE]
|
||||||
|
```
|
||||||
|
|
||||||
|
Data-only frames, no `event:` lines. Terminated by the literal `data: [DONE]`.
|
||||||
|
|
||||||
|
### Legacy aliases [LIVE, being retired]
|
||||||
|
|
||||||
|
`POST /v1/{reasoning,ornith,qwen}/chat/completions` force `model` to the corresponding value regardless of the body. They exist only because Kong could not dispatch on the body. Removed once callers migrate.
|
||||||
|
|
||||||
|
### `GET /v1/models` [SPEC]
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"object":"list","data":[{"id":"reasoning","object":"model","owned_by":"homelab","created":0}]}
|
||||||
|
```
|
||||||
|
|
||||||
|
Derived from the registry, never hardcoded.
|
||||||
|
|
||||||
|
### Errors [SPEC]
|
||||||
|
|
||||||
|
RFC 9457 `application/problem+json`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"type": "https://riotpiao.com/errors/unknown-model",
|
||||||
|
"title": "Unknown model",
|
||||||
|
"status": 400,
|
||||||
|
"detail": "\"gpt-4\" is not available",
|
||||||
|
"validModels": ["reasoning", "ornith:35b", "qwen2.5:3b-instruct"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Anthropic dialect — `POST /llm/v1/messages` [SPEC]
|
||||||
|
|
||||||
|
Path note: the Anthropic SDK appends `/v1/messages` to its base URL, so a base URL of `https://api.riotpiao.com/llm` produces exactly this path.
|
||||||
|
|
||||||
|
### Request
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"model": "reasoning",
|
||||||
|
"max_tokens": 2000,
|
||||||
|
"system": "You are a cluster assistant.",
|
||||||
|
"messages": [
|
||||||
|
{"role": "user", "content": "Why is wave 4 empty?"}
|
||||||
|
],
|
||||||
|
"stream": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Differences from the OpenAI dialect that the translator must handle:
|
||||||
|
|
||||||
|
| Concern | OpenAI | Anthropic |
|
||||||
|
|---|---|---|
|
||||||
|
| system prompt | `messages[0].role = "system"` | top-level `system` field |
|
||||||
|
| `max_tokens` | optional | **required** |
|
||||||
|
| content | string | string *or* block array |
|
||||||
|
| roles | system/user/assistant/tool | user/assistant only |
|
||||||
|
| stop | `stop` | `stop_sequences` |
|
||||||
|
|
||||||
|
`max_tokens` being required is a real divergence — the gateway must either reject its absence with a clear error or apply a documented default. Pick one and state it; do not silently default.
|
||||||
|
|
||||||
|
### Response, non-streaming
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "msg_01ABC",
|
||||||
|
"type": "message",
|
||||||
|
"role": "assistant",
|
||||||
|
"model": "reasoning",
|
||||||
|
"content": [
|
||||||
|
{"type": "thinking", "thinking": "Waves are sort keys, not a sequence..."},
|
||||||
|
{"type": "text", "text": "Wave 4 is empty. Waves are sort keys..."}
|
||||||
|
],
|
||||||
|
"stop_reason": "end_turn",
|
||||||
|
"stop_sequence": null,
|
||||||
|
"usage": {"input_tokens": 16, "output_tokens": 300}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Field mapping from the upstream OpenAI response:
|
||||||
|
|
||||||
|
| Upstream | Anthropic |
|
||||||
|
|---|---|
|
||||||
|
| `choices[0].message.reasoning_content` | `content[]` block `{"type":"thinking","thinking":...}` |
|
||||||
|
| `choices[0].message.content` | `content[]` block `{"type":"text","text":...}` |
|
||||||
|
| `finish_reason: "stop"` | `stop_reason: "end_turn"` |
|
||||||
|
| `finish_reason: "length"` | `stop_reason: "max_tokens"` |
|
||||||
|
| `usage.prompt_tokens` | `usage.input_tokens` |
|
||||||
|
| `usage.completion_tokens` | `usage.output_tokens` |
|
||||||
|
|
||||||
|
The thinking block precedes the text block.
|
||||||
|
|
||||||
|
### Response, streaming
|
||||||
|
|
||||||
|
Anthropic SSE uses **named events with content-block indices**, unlike OpenAI's flat frames. Verified event sequence:
|
||||||
|
|
||||||
|
```
|
||||||
|
event: message_start
|
||||||
|
data: {"type":"message_start","message":{"id":"msg_01ABC","type":"message","role":"assistant","model":"reasoning","content":[],"usage":{"input_tokens":16,"output_tokens":0}}}
|
||||||
|
|
||||||
|
event: content_block_start
|
||||||
|
data: {"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":""}}
|
||||||
|
|
||||||
|
event: content_block_delta
|
||||||
|
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"Waves are sort keys"}}
|
||||||
|
|
||||||
|
event: content_block_stop
|
||||||
|
data: {"type":"content_block_stop","index":0}
|
||||||
|
|
||||||
|
event: content_block_start
|
||||||
|
data: {"type":"content_block_start","index":1,"content_block":{"type":"text","text":""}}
|
||||||
|
|
||||||
|
event: content_block_delta
|
||||||
|
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"Wave 4 is empty."}}
|
||||||
|
|
||||||
|
event: content_block_stop
|
||||||
|
data: {"type":"content_block_stop","index":1}
|
||||||
|
|
||||||
|
event: message_delta
|
||||||
|
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":300}}
|
||||||
|
|
||||||
|
event: message_stop
|
||||||
|
data: {"type":"message_stop"}
|
||||||
|
```
|
||||||
|
|
||||||
|
Three details that are easy to get wrong:
|
||||||
|
|
||||||
|
- In `message_delta`, `usage` is a **sibling of** `delta`, not inside it.
|
||||||
|
- The delta field name matches the delta type: `thinking_delta` carries `.thinking`, `text_delta` carries `.text`.
|
||||||
|
- Block index 0 is thinking, index 1 is text. **You only learn reasoning has ended when `content` first appears in an upstream chunk**, so the thinking block must be closed before the text block opens. If a response has no `reasoning_content` at all, the text block is index 0 and no thinking block is emitted.
|
||||||
|
|
||||||
|
### Queue position — non-standard extension
|
||||||
|
|
||||||
|
Anthropic's event set has no way to say "you are queued", because the stream implicitly begins after a slot is acquired. With only 8 slots, queueing is normal here.
|
||||||
|
|
||||||
|
Emitted **before** `message_start`:
|
||||||
|
|
||||||
|
```
|
||||||
|
event: queue
|
||||||
|
data: {"type":"queue","position":3}
|
||||||
|
```
|
||||||
|
|
||||||
|
This is deliberately outside the Anthropic spec. It is safe only because the client is first-party; a strict Anthropic client would ignore the unknown event and show nothing while queued.
|
||||||
|
|
||||||
|
### Errors
|
||||||
|
|
||||||
|
Anthropic error shape, **not** RFC 9457 — the same rejection renders differently depending on which surface received it:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"type":"error","error":{"type":"invalid_request_error","message":"Unknown model \"gpt-4\". Available: reasoning, ornith:35b, qwen2.5:3b-instruct"}}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Condition | HTTP | `error.type` |
|
||||||
|
|---|---|---|
|
||||||
|
| unknown or missing model | 400 | `invalid_request_error` |
|
||||||
|
| `max_tokens` absent (if required) | 400 | `invalid_request_error` |
|
||||||
|
| malformed JSON | 400 | `invalid_request_error` |
|
||||||
|
| unsupported feature requested | 400 | `invalid_request_error` |
|
||||||
|
| not authenticated | 401 | `authentication_error` |
|
||||||
|
| budget exhausted or queue full | 429 | `rate_limit_error` |
|
||||||
|
| upstream failure | 502 | `api_error` |
|
||||||
|
|
||||||
|
### Deliberately not implemented
|
||||||
|
|
||||||
|
Each returns 400 naming the unsupported feature — never a silent partial implementation:
|
||||||
|
|
||||||
|
tool use and `tool_result` turns, image content blocks, prompt-caching headers, the batch API, multi-block user content, `thinking.budget_tokens` configuration.
|
||||||
|
|
||||||
|
The target client is the riotpiao frontend. Widening scope is a code change with a test, not an accident.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Shared behaviour, both dialects
|
||||||
|
|
||||||
|
**One slot controller, keyed by upstream.** A `/v1` request and a `/llm` request contend for the same 8 `reasoning` slots and the same queue, in arrival order. Per-dialect semaphores would each believe they were within budget while together exceeding the physical limit.
|
||||||
|
|
||||||
|
**Streaming is unbuffered** and a client disconnect cancels the upstream immediately. An orphaned generation holds a slot until it completes on its own, which for a 32B model on a Volta GPU can run to minutes.
|
||||||
|
|
||||||
|
**Timeouts** [LIVE]: chat routes are connect 10s / read 1h / write 1h. The hour is deliberate — a 32B model on this hardware routinely exceeds 60s. Any shorter application cap is enforced in gateway logic, never by shortening the proxy timeout.
|
||||||
|
|
||||||
|
**Tool calling** [LIVE]: `reasoning` honours an explicit `tool_choice` but returns `tool_calls: []` under `tool_choice: "auto"` — it reasons about the tool in prose instead. `ornith:35b` returns `finish_reason: "tool_calls"` correctly under `auto`. This is a model property; the gateway does not compensate for it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# OpenAI dialect
|
||||||
|
curl -s https://api.riotpiao.com/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","messages":[{"role":"user","content":"Why is wave 4 empty?"}],"max_tokens":500}'
|
||||||
|
|
||||||
|
# Anthropic dialect, streaming
|
||||||
|
curl -N -s https://api.riotpiao.com/llm/v1/messages \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":500,"stream":true,
|
||||||
|
"messages":[{"role":"user","content":"Why is wave 4 empty?"}]}'
|
||||||
|
|
||||||
|
# model list
|
||||||
|
curl -s https://api.riotpiao.com/v1/models
|
||||||
|
```
|
||||||
+263
@@ -0,0 +1,263 @@
|
|||||||
|
# API — queue surface (`/sqs/*`)
|
||||||
|
|
||||||
|
Fronts the Kafka Management Service (`kmsvc`) in namespace `sqs`. SQS-shaped
|
||||||
|
message-plane API over Kafka.
|
||||||
|
|
||||||
|
Status marks:
|
||||||
|
**[LIVE]** verified against the running cluster and the committed proto on 2026-08-19.
|
||||||
|
**[SPEC]** the contract this gateway must implement; not built yet.
|
||||||
|
|
||||||
|
Source of truth for shapes:
|
||||||
|
`~/workplace/kmsvc-proto/proto/kafkamgmt/v1/queue_service.proto`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The important finding: a REST surface already exists [LIVE]
|
||||||
|
|
||||||
|
**Do not build gRPC-to-JSON transcoding.** `kmsvc-manage` already mounts grpc-gateway:
|
||||||
|
|
||||||
|
```go
|
||||||
|
mux := runtime.NewServeMux()
|
||||||
|
kafkamgmtv1.RegisterQueueServiceHandlerServer(ctx, mux, svc)
|
||||||
|
```
|
||||||
|
|
||||||
|
The upstream serves plain REST/JSON on **:8080** and plain gRPC on **:9090**. Neither
|
||||||
|
gRPC-Web nor server reflection is enabled.
|
||||||
|
|
||||||
|
So `/sqs/*` is a **path-stripping reverse proxy plus authentication**, not a protocol
|
||||||
|
translator. That makes it dramatically cheaper than the LLM surface.
|
||||||
|
|
||||||
|
```
|
||||||
|
api.riotpiao.com/sqs/v1/queues/{q}/messages
|
||||||
|
| strip /sqs, authenticate
|
||||||
|
v
|
||||||
|
management-service.sqs.svc.cluster.local:8080/v1/queues/{q}/messages
|
||||||
|
```
|
||||||
|
|
||||||
|
Upstream: Deployment `management-service`, 3 replicas, HPA 3-9, Service ClusterIP
|
||||||
|
`10.98.3.138`, ports `8080` (http) and `9090` (grpc).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Endpoints [LIVE — HTTP annotations from the proto]
|
||||||
|
|
||||||
|
Six operations. All unary. No streaming, no subscribe.
|
||||||
|
|
||||||
|
| Method | Path (after `/sqs` strip) | RPC |
|
||||||
|
|---|---|---|
|
||||||
|
| POST | `/v1/queues/{queue_name}/messages` | `SendMessage` |
|
||||||
|
| POST | `/v1/queues/{queue_name}/messages:batch` | `SendMessageBatch` |
|
||||||
|
| GET | `/v1/queues/{queue_name}/messages` | `ReceiveMessage` |
|
||||||
|
| DELETE | `/v1/queues/{queue_name}/messages/{receipt_handle}` | `DeleteMessage` |
|
||||||
|
| POST | `/v1/queues/{queue_name}/messages:batchDelete` | `DeleteMessageBatch` |
|
||||||
|
| PATCH | `/v1/queues/{queue_name}/messages/{receipt_handle}` | `ChangeMessageVisibility` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Two wire-format traps [LIVE]
|
||||||
|
|
||||||
|
Both follow from grpc-gateway defaults, and both will surprise anyone who reads only
|
||||||
|
the proto.
|
||||||
|
|
||||||
|
**1. `bytes` fields are base64 in JSON.** `SendMessageRequest.message_body` and
|
||||||
|
`Message.body` are proto `bytes`. The JSONPB marshaler encodes them as base64 strings.
|
||||||
|
Sending raw text will not do what you expect.
|
||||||
|
|
||||||
|
**2. Field names are lowerCamelCase.** `cmd/server/main.go` calls bare
|
||||||
|
`runtime.NewServeMux()` with no marshaler options, so `OrigName` is false. The wire uses
|
||||||
|
`messageBody`, `receiptHandle`, `maxNumberOfMessages` — not the snake_case names in the
|
||||||
|
proto.
|
||||||
|
|
||||||
|
Document both prominently or every first-time caller loses an hour.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Message shapes [LIVE — from the proto]
|
||||||
|
|
||||||
|
### Send
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /sqs/v1/queues/agent-worker-queue/messages
|
||||||
|
{
|
||||||
|
"messageBody": "aGVsbG8gd29ybGQ=", // base64 of "hello world"
|
||||||
|
"messageAttributes": {"values": {"k": "v"}},
|
||||||
|
"messageGroupId": "", // FIFO only
|
||||||
|
"messageDeduplicationId": "", // FIFO only
|
||||||
|
"delaySeconds": 0 // 0-900
|
||||||
|
}
|
||||||
|
-> {"messageId": "...", "sequenceNumber": ""} // sequenceNumber FIFO only
|
||||||
|
```
|
||||||
|
|
||||||
|
### Receive — long poll
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /sqs/v1/queues/agent-worker-queue/messages
|
||||||
|
?maxNumberOfMessages=10 // <= 10
|
||||||
|
&waitTimeSeconds=20 // 0-20
|
||||||
|
&visibilityTimeoutSeconds=30 // optional override
|
||||||
|
|
||||||
|
-> {"messages": [{
|
||||||
|
"messageId": "...",
|
||||||
|
"receiptHandle": "...",
|
||||||
|
"body": "aGVsbG8gd29ybGQ=",
|
||||||
|
"attributes": {"values": {}},
|
||||||
|
"receiveCount": 1,
|
||||||
|
"messageGroupId": "",
|
||||||
|
"enqueuedAt": "2026-08-19T16:29:07Z"
|
||||||
|
}]}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Delete — the ack
|
||||||
|
|
||||||
|
```
|
||||||
|
DELETE /sqs/v1/queues/agent-worker-queue/messages/{receiptHandle}
|
||||||
|
-> {}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Change visibility
|
||||||
|
|
||||||
|
```
|
||||||
|
PATCH /sqs/v1/queues/agent-worker-queue/messages/{receiptHandle}
|
||||||
|
{"visibilityTimeoutSeconds": 60} // 0-43200
|
||||||
|
-> {}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Batch
|
||||||
|
|
||||||
|
Both batch calls take `entries[]` with a caller-assigned `id`, and return partial
|
||||||
|
success:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"successful": [{"id": "1", "messageId": "..."}],
|
||||||
|
"failed": [{"id": "2", "error": "..."}]}
|
||||||
|
```
|
||||||
|
|
||||||
|
A batch call can return 200 with entries in `failed`. Callers must inspect the body,
|
||||||
|
not just the status.
|
||||||
|
|
||||||
|
### Limits [LIVE — from the SDK]
|
||||||
|
|
||||||
|
`MaxMessageBodyBytes = 262144` (256 KiB), `MaxReceiveMessages = 10`,
|
||||||
|
`MaxWaitTimeSeconds = 20`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Semantics
|
||||||
|
|
||||||
|
At-least-once, SQS-style. Receive leases a message for the visibility timeout; the
|
||||||
|
caller must `DeleteMessage` to acknowledge. An un-deleted message reappears after the
|
||||||
|
timeout and `receiveCount` increments. After `maxReceiveCount` (default 5) it goes to
|
||||||
|
the DLQ if one is configured.
|
||||||
|
|
||||||
|
**Long-polling matters for the gateway.** `waitTimeSeconds` up to 20 means a `GET` can
|
||||||
|
legitimately hold open for 20 seconds returning nothing. Read timeouts must exceed that
|
||||||
|
comfortably, and a client disconnect must cancel upstream — the same requirement as the
|
||||||
|
LLM surface, for the same reason.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Error mapping [SPEC]
|
||||||
|
|
||||||
|
The SDK maps gRPC codes to sentinel errors; grpc-gateway maps them to HTTP. Use this as
|
||||||
|
the gateway's status contract:
|
||||||
|
|
||||||
|
| gRPC code | HTTP | SDK sentinel |
|
||||||
|
|---|---|---|
|
||||||
|
| `NotFound` | 404 | `ErrQueueNotFound` |
|
||||||
|
| `AlreadyExists` | 409 | `ErrAlreadyExists` |
|
||||||
|
| `InvalidArgument` | 400 | `ErrInvalidArgument` |
|
||||||
|
| `Unauthenticated` | 401 | `ErrUnauthenticated` |
|
||||||
|
| `ResourceExhausted` | 429 | `ErrMessageTooLarge` |
|
||||||
|
|
||||||
|
Upstream errors arrive in the grpc-gateway envelope
|
||||||
|
`{"code": 5, "message": "Not Found", "details": []}`. Decide deliberately whether
|
||||||
|
`/sqs/*` passes that through or re-renders it as RFC 9457 to match `/v1/*`.
|
||||||
|
Recommendation: **pass through**, so the gateway does not become a second, subtly
|
||||||
|
different error vocabulary for the same upstream.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Queue lifecycle is NOT in this API [LIVE]
|
||||||
|
|
||||||
|
There is no `CreateQueue`, `DeleteQueue`, or `ListQueues` RPC. The proto says so
|
||||||
|
explicitly:
|
||||||
|
|
||||||
|
```proto
|
||||||
|
// Queue lifecycle (create/delete/configure) is managed via the Queue CRD,
|
||||||
|
// not this service
|
||||||
|
```
|
||||||
|
|
||||||
|
Queues are Kubernetes resources — `queues.kmsvc.io/v1`, namespaced. `kmsvc-cli`'s
|
||||||
|
`create-queue` and `delete-queue` talk to the Kubernetes API, not to kmsvc.
|
||||||
|
|
||||||
|
**This is a hard boundary for the gateway.** Exposing queue creation over `/sqs/*` would
|
||||||
|
require the gateway to hold Kubernetes write credentials, which violates **G2**. Do not
|
||||||
|
add it. If declarative queue management ever needs a public surface, it belongs behind a
|
||||||
|
separate component with its own RBAC — not in the public edge process.
|
||||||
|
|
||||||
|
Queue spec fields, for reference when reading a queue's configuration:
|
||||||
|
`fifoQueue`, `isDLQ`, `deadLetterTargetQueue`, `delaySeconds` (0-900),
|
||||||
|
`maxReceiveCount` (default 5), `messageRetentionPeriodSeconds` (default 345600),
|
||||||
|
`visibilityTimeoutSeconds` (default 30), `minShards`, `maxShards` (default 8),
|
||||||
|
`partitionsPerShard` (default 6), `shardSplitThresholdBytesPerSec`,
|
||||||
|
`shardSplitCooldownSeconds`.
|
||||||
|
|
||||||
|
Kafka topics are named `kmsvc.{queue}.shard-{id}` and are created by `queue-operator`
|
||||||
|
directly via the Kafka Admin API — there are no `KafkaTopic` CRs.
|
||||||
|
|
||||||
|
Currently one queue exists: `agent-worker-queue` in namespace `sqs`, phase `Ready`,
|
||||||
|
1 shard.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Authentication [SPEC]
|
||||||
|
|
||||||
|
`Authorization: Bearer <jwt>`, same as every other gateway surface.
|
||||||
|
|
||||||
|
**The upstream enforces nothing.** `kmsvc`'s auth interceptor exists but is never wired,
|
||||||
|
and the REST surface is mounted with the in-process grpc-gateway variant that bypasses
|
||||||
|
gRPC interceptors regardless. Both `:8080` and `:9090` are currently open, and
|
||||||
|
`kmsvc.riotpiao.com` is publicly routed.
|
||||||
|
|
||||||
|
The gateway is therefore the only authentication boundary for this surface. See
|
||||||
|
[KNOWN-ISSUES.md](KNOWN-ISSUES.md) §2.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- **Workflow start.** Nothing in kmsvc starts a Temporal workflow — no such RPC exists,
|
||||||
|
and grep for `ExecuteWorkflow`/`StartWorkflow` across `kmsvc-manage`, `kmsvc-sdk` and
|
||||||
|
`kmsvc-cli` returns nothing. A caller dials `temporal-frontend.temporal.svc:7233`
|
||||||
|
with a Temporal SDK directly. A `/workflow/*` surface is net-new code, not a proxy
|
||||||
|
route — see [task 7.3](../tasks/7.3-workflow-prefix.md) and KNOWN-ISSUES.md §1.
|
||||||
|
- **DLQ operations.** `kmsvc-cli`'s `dlq peek` and `dlq redrive` are client-side
|
||||||
|
compositions of the six RPCs, not server operations. Redrive is a non-atomic
|
||||||
|
Receive-Send-Delete. If `/sqs/*` should offer redrive, that is new logic with real
|
||||||
|
failure modes, not a proxied call.
|
||||||
|
- **Kafka direct access.** No external listener exists; the bootstrap
|
||||||
|
`kmsvc-kafka-bootstrap.sqs.svc.cluster.local:9092` is cluster-internal only. The
|
||||||
|
gateway proxies kmsvc, never Kafka.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
```bash
|
||||||
|
Q=agent-worker-queue
|
||||||
|
|
||||||
|
# send (body must be base64)
|
||||||
|
curl -s -X POST https://api.riotpiao.com/sqs/v1/queues/$Q/messages \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d "{\"messageBody\":\"$(printf 'hello world' | base64)\"}"
|
||||||
|
|
||||||
|
# receive, long poll 20s
|
||||||
|
curl -s "https://api.riotpiao.com/sqs/v1/queues/$Q/messages?maxNumberOfMessages=10&waitTimeSeconds=20"
|
||||||
|
|
||||||
|
# acknowledge
|
||||||
|
curl -s -X DELETE https://api.riotpiao.com/sqs/v1/queues/$Q/messages/$RECEIPT
|
||||||
|
|
||||||
|
# extend the lease
|
||||||
|
curl -s -X PATCH https://api.riotpiao.com/sqs/v1/queues/$Q/messages/$RECEIPT \
|
||||||
|
-H 'content-type: application/json' -d '{"visibilityTimeoutSeconds":60}'
|
||||||
|
```
|
||||||
@@ -0,0 +1,118 @@
|
|||||||
|
# Known cluster issues
|
||||||
|
|
||||||
|
Pre-existing problems found while specifying this gateway. None are caused by this
|
||||||
|
repo, and none block phases 0-6. Recorded so they are not rediscovered or mistaken
|
||||||
|
for new breakage.
|
||||||
|
|
||||||
|
Verified 2026-08-19 against context `admin@homelab-cluster`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. TemporalWorker CRD is stale — queue-operator reconcile fails every ~17 min
|
||||||
|
|
||||||
|
**Status:** open, deliberately deferred. Affects [task 7.3](../tasks/7.3-workflow-prefix.md).
|
||||||
|
|
||||||
|
The live `temporalworkers.kmsvc.io` CRD and the one in
|
||||||
|
`~/workplace/kmsvc-manage/config/crd/kmsvc.io_temporalworkers.yaml` share exactly one
|
||||||
|
field — `namespace`.
|
||||||
|
|
||||||
|
| | spec properties |
|
||||||
|
|---|---|
|
||||||
|
| live CRD | `activityTypes`, `concurrency`, `namespace`, `taskQueue`, `workflowTypes` |
|
||||||
|
| repo CRD | `affinity`, `image`, `imagePullPolicy`, `namespace`, `nodeSelector`, `replicas`, `resources`, `tolerations` |
|
||||||
|
|
||||||
|
The live schema has no `image` field, so the API server **prunes** `image` from the CR
|
||||||
|
that `queue-operator` writes. `TemporalWorker/worker-production` ends up as
|
||||||
|
`spec: {namespace: production}`, and the operator then fails to build a Deployment
|
||||||
|
from it. The live CRD also lacks a status subresource, producing a second error.
|
||||||
|
|
||||||
|
Observed on a loop, most recently 21:25:39Z:
|
||||||
|
|
||||||
|
```
|
||||||
|
failed to create or update deployment ... error: "Deployment.apps \"worker-production\"
|
||||||
|
is invalid: spec.template.spec.containers[0].image: Required value"
|
||||||
|
Reconciler error ... "update status failed: temporalworkers.kmsvc.io
|
||||||
|
\"worker-production\" not found"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Impact is narrower than it looks.** No worker Deployment has ever existed under this
|
||||||
|
CRD, so nothing that was working has stopped. Temporal namespace `production` is
|
||||||
|
registered and healthy; there is simply no worker polling it. The practical cost is log
|
||||||
|
noise, not lost work. That is why this is deferred rather than treated as an incident.
|
||||||
|
|
||||||
|
**Neither object is under GitOps.** The CRD and the `Queue/agent-worker-queue` CR both
|
||||||
|
carry only `kubectl.kubernetes.io/last-applied-configuration` — no
|
||||||
|
`argocd.argoproj.io/instance`, no tracking-id — and the Queue does not appear anywhere
|
||||||
|
in the homelab repo. They were hand-applied and predate GitOps coverage.
|
||||||
|
|
||||||
|
**Fix, when it is worth doing:**
|
||||||
|
|
||||||
|
1. Bring `temporalworkers.kmsvc.io` and the Queue CR into the homelab GitOps repo.
|
||||||
|
2. Apply the current CRD from `kmsvc-manage/config/crd`, which restores `image` and the
|
||||||
|
status subresource.
|
||||||
|
3. Ensure the operator sets `spec.image` on the CR it creates.
|
||||||
|
|
||||||
|
Do not hand-apply the CRD as a one-off. That reproduces exactly the situation that
|
||||||
|
caused this — a cluster object with no source of truth.
|
||||||
|
|
||||||
|
**To silence the loop without fixing it:** remove the `temporal.io/namespace: production`
|
||||||
|
label from `Queue/agent-worker-queue` in namespace `sqs`. The operator returns early when
|
||||||
|
the label is absent. Reversible by re-adding it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. `kmsvc.riotpiao.com` is unauthenticated
|
||||||
|
|
||||||
|
**Status:** open. Relevant to [task 7.2](../tasks/7.2-sqs-prefix.md).
|
||||||
|
|
||||||
|
`kmsvc-manage` has an auth interceptor at `internal/api/interceptors/auth.go`, but it is
|
||||||
|
never wired: `cmd/server/main.go` constructs a bare `grpc.NewServer()` with no
|
||||||
|
interceptor options. The live ConfigMap confirms it — `KMSVC_AUTHENTIK_ISSUER_URL` and
|
||||||
|
`KMSVC_AUTHENTIK_AUDIENCE` are both empty strings.
|
||||||
|
|
||||||
|
Both the REST surface (8080) and the gRPC surface (9090) are open.
|
||||||
|
|
||||||
|
There is a second, subtler problem. The REST surface is mounted with
|
||||||
|
`RegisterQueueServiceHandlerServer`, the **in-process** grpc-gateway variant that calls
|
||||||
|
the service implementation directly. It bypasses gRPC interceptors entirely. So even
|
||||||
|
once the interceptor is wired, it would authenticate gRPC callers only — the file's own
|
||||||
|
doc comment claiming it covers both REST and gRPC is wrong for this wiring.
|
||||||
|
|
||||||
|
Consequence for this gateway: `/sqs/*` must own authentication itself. Do not assume the
|
||||||
|
upstream will enforce anything.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. `kmsvc-redis-master.sqs:6379` has no authentication
|
||||||
|
|
||||||
|
`ALLOW_EMPTY_PASSWORD=yes`, TLS off, Bitnami chart with `auth.enabled=false`, no password
|
||||||
|
secret in the namespace. Anything with network reach has full unauthenticated read/write.
|
||||||
|
|
||||||
|
A NetworkPolicy is the only control. Relevant to [task 6.2](../tasks/6.2-kubernetes-manifests.md).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. `macos-bluebubbles` pod will never schedule
|
||||||
|
|
||||||
|
`sms` Argo Application is `Synced`/`Degraded`. The pod targets a macOS node that is not
|
||||||
|
in the cluster: `0/4 nodes are available: 4 node(s) didn't match Pod's node
|
||||||
|
affinity/selector`, roughly 1080 failed attempts over 3d18h.
|
||||||
|
|
||||||
|
Not transient. Needs either that node or removal of the Application. Unrelated to this
|
||||||
|
gateway; listed so the Degraded status is not mistaken for something new.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Documentation that does not match reality
|
||||||
|
|
||||||
|
- `kmsvc-manage/TEMPORAL_INTEGRATION.md` is aspirational. It documents
|
||||||
|
`apiVersion: temporal.kmsvc.io/v1` with `queueRef`, `taskQueueName` and `lifecycle`
|
||||||
|
fields, and one worker per Queue. Reality is `kmsvc.io/v1`, none of those fields, and
|
||||||
|
one worker per Temporal *namespace*. Do not source API documentation from it.
|
||||||
|
- Module paths disagree across repos: `kmsvc-proto` declares
|
||||||
|
`forgejo.riotpiao.homelab.com/...`, while `kmsvc-manage` and `kmsvc-sdk` import
|
||||||
|
`forgejo.riotpiao.com/...`. The `.homelab.com` domain is fully retired — every
|
||||||
|
subdomain NXDOMAINs.
|
||||||
|
- `kmsvc-cli` README says the gRPC ingress uses TLS passthrough. It uses
|
||||||
|
`nginx.ingress.kubernetes.io/backend-protocol: GRPC`, which terminates TLS at nginx.
|
||||||
|
Functionally fine for clients; the wording is wrong.
|
||||||
@@ -0,0 +1,143 @@
|
|||||||
|
# Kong retirement — inventory and cutover
|
||||||
|
|
||||||
|
Everything Kong does on `api.riotpiao.com` today, and where it goes. Inventory
|
||||||
|
verified live against context `admin@homelab-cluster` on 2026-08-19.
|
||||||
|
|
||||||
|
Source of the objects being retired: `~/workplace/homelab/k8s/apps/api/` and
|
||||||
|
`k8s/argocd/apps/55-api-gateway.yaml`.
|
||||||
|
|
||||||
|
## What is running now
|
||||||
|
|
||||||
|
Kong OSS 3.4.1, Helm chart from `https://charts.konghq.com`, DB-less, namespace
|
||||||
|
`api`, Argo Application `kong` at sync wave 7. Two replicas. Fronted by
|
||||||
|
`ingress-nginx` via Ingress `api/api`, which catch-alls `/` on `api.riotpiao.com`
|
||||||
|
to `kong-proxy:80`.
|
||||||
|
|
||||||
|
Eleven ReplicaSets exist on the Kong Deployment, the newest minutes old — this
|
||||||
|
config is being actively iterated, so re-verify the inventory immediately before
|
||||||
|
cutover.
|
||||||
|
|
||||||
|
## Routing table to port
|
||||||
|
|
||||||
|
Seven `ingressClassName: kong` Ingresses. Six in `llm-serving`, one in `agent-pod`.
|
||||||
|
|
||||||
|
| Method | Path | Upstream | Transform applied by Kong |
|
||||||
|
|---|---|---|---|
|
||||||
|
| GET | `/v1/models` | — | `request-termination`: static 200 JSON, upstream never contacted |
|
||||||
|
| POST | `/v1/reasoning/chat/completions` | `reasoning-predictor:80` | force body `model=reasoning`, rewrite URI to `/v1/chat/completions` |
|
||||||
|
| POST | `/v1/ornith/chat/completions` | `ornith-predictor:80` | force body `model=ornith:35b`, rewrite URI |
|
||||||
|
| POST | `/v1/qwen/chat/completions` | `ornith-predictor:80` | force body `model=qwen2.5:3b-instruct`, rewrite URI |
|
||||||
|
| POST | `/v1/embeddings` | `embeddings-predictor:80` | none — TEI already serves the canonical path |
|
||||||
|
| POST | `/v1/rerank` | `reranker-predictor:80` | rewrite URI to `/rerank` (TEI does not serve `/v1/rerank`) |
|
||||||
|
| GET/WS | `/console`, `/run`, `/sessions` | `agent-hub:9090` (`agent-pod` ns) | none, `strip-path: false` |
|
||||||
|
|
||||||
|
Upstream model map, from the manifest comments and confirmed live:
|
||||||
|
|
||||||
|
- `reasoning` → `reasoning-predictor` — vLLM, DeepSeek-R1-Distill-Qwen-32B, 2 replicas,
|
||||||
|
`--max-num-seqs=4`, `--max-model-len=16384`, `--reasoning-parser=deepseek_r1`,
|
||||||
|
`--enable-auto-tool-choice --tool-call-parser=hermes`
|
||||||
|
- `ornith:35b` → `ornith-predictor` — Ollama, 2 replicas
|
||||||
|
- `qwen2.5:3b-instruct` → `ornith-predictor` — same pods; both models stay resident via
|
||||||
|
`OLLAMA_MAX_LOADED_MODELS=2`, `OLLAMA_KEEP_ALIVE=-1`
|
||||||
|
- `nomic-ai/nomic-embed-text-v2-moe` → `embeddings-predictor` — TEI
|
||||||
|
- `BAAI/bge-reranker-base` → `reranker-predictor` — TEI
|
||||||
|
|
||||||
|
### The path-per-model surface goes away
|
||||||
|
|
||||||
|
The three chat paths exist only because Kong OSS cannot dispatch on the request
|
||||||
|
body. The gateway serves a single `POST /v1/chat/completions` and selects the
|
||||||
|
upstream from the body's `model` field.
|
||||||
|
|
||||||
|
Keep the old paths as aliases during cutover so live clients do not break, then
|
||||||
|
remove them once callers have migrated. pi is a live caller today.
|
||||||
|
|
||||||
|
### `/v1/models` should not be ported verbatim
|
||||||
|
|
||||||
|
Kong serves a hardcoded list via `request-termination`. The manifest already flags
|
||||||
|
that it can drift from what the engines actually serve. Derive the response from
|
||||||
|
the gateway's configured upstream map instead, so the list cannot disagree with
|
||||||
|
what routing will accept.
|
||||||
|
|
||||||
|
## Plugins being retired
|
||||||
|
|
||||||
|
| Plugin | Scope | Replacement |
|
||||||
|
|---|---|---|
|
||||||
|
| `llm-rewrite-reasoning` / `-ornith` / `-qwen` | llm-serving | body-based dispatch in `internal/llm` |
|
||||||
|
| `llm-rewrite-rerank` | llm-serving | per-upstream path rewrite in the route table |
|
||||||
|
| `llm-models-list` | llm-serving | derived from the upstream map |
|
||||||
|
| `prometheus` | **cluster-wide** | `internal/observability` — must expose bandwidth, latency, status codes, upstream health or observability regresses |
|
||||||
|
|
||||||
|
No `rate-limiting` plugin exists anywhere in the cluster. REQUIREMENTS.md §4 Tier 2
|
||||||
|
describes it as an existing layer; it is not built. Nothing to migrate — it is net
|
||||||
|
new work, and it now belongs in the gateway rather than in Kong.
|
||||||
|
|
||||||
|
## Auth: currently off, must land on
|
||||||
|
|
||||||
|
`KongConsumer model-invoker` exists in namespace `api` and stays defined, but the
|
||||||
|
`key-auth` plugin is commented out and every route has `model-key-auth` stripped
|
||||||
|
from its `konghq.com/plugins` annotation.
|
||||||
|
|
||||||
|
**The model API is unauthenticated right now.** Confirmed live 2026-08-19: a request
|
||||||
|
to `/v1/reasoning/chat/completions` with no credentials returns 200.
|
||||||
|
|
||||||
|
The reason is recorded in `model-auth.yaml` — Kong's `key-auth` accepts a raw
|
||||||
|
`apikey:` header but rejects `Authorization: Bearer`, which blocks every
|
||||||
|
OpenAI-compatible client. That is why `~/.pi/agent/models.json` carries a
|
||||||
|
`customHeaders: {apikey: ...}` block.
|
||||||
|
|
||||||
|
The gateway reads Bearer tokens directly and validates them against Authentik via
|
||||||
|
JWKS. `AUTH-PLAN.md`'s pinned-RSA-key approach and its rotation runbook are not
|
||||||
|
needed and should not be carried over.
|
||||||
|
|
||||||
|
Ship auth behind a flag. Turning it on breaks every current caller until they hold
|
||||||
|
a token — pi included.
|
||||||
|
|
||||||
|
## Timeouts
|
||||||
|
|
||||||
|
Kong today:
|
||||||
|
|
||||||
|
| Route class | connect | read | write |
|
||||||
|
|---|---|---|---|
|
||||||
|
| chat | 10s | **1h** | 1h |
|
||||||
|
| embeddings / rerank | 10s | 10m | 10m |
|
||||||
|
|
||||||
|
nginx in front sets `proxy-read-timeout: 3600`, `proxy-send-timeout: 3600`,
|
||||||
|
`proxy-buffering: off`, `proxy-body-size: 0`. Those stay — they are what makes token
|
||||||
|
streaming work, and the gateway needs the same treatment from nginx.
|
||||||
|
|
||||||
|
The 1-hour read timeout is deliberate: a 32B model on a Volta GPU routinely exceeds
|
||||||
|
60s. Any shorter server-side cap must be enforced *in the gateway*, not by shortening
|
||||||
|
the proxy timeout, or long legitimate generations get truncated mid-stream.
|
||||||
|
|
||||||
|
## Cutover
|
||||||
|
|
||||||
|
Reversible at every step. Kong keeps serving until the last step.
|
||||||
|
|
||||||
|
1. Deploy the gateway alongside Kong, unexposed. Verify in-cluster against
|
||||||
|
`http://homelab-frontend.api.svc.cluster.local`.
|
||||||
|
2. Compare gateway and Kong responses for every route in the table above, including
|
||||||
|
a streaming chat request and a client disconnect mid-stream.
|
||||||
|
3. Repoint Ingress `api/api` from `kong-proxy:80` to the gateway Service. **This is
|
||||||
|
the cutover.** Reverting is a one-line change to the same Ingress.
|
||||||
|
4. Soak. Watch gateway metrics and pi traffic.
|
||||||
|
5. Delete the seven kong-class Ingresses and the six KongPlugin CRs.
|
||||||
|
6. Remove the `kong` Application from `k8s/argocd/apps/55-api-gateway.yaml`; let Argo
|
||||||
|
prune the Helm release, the CRDs and namespace leftovers.
|
||||||
|
|
||||||
|
Steps 1–4 are reversible in seconds. Step 5 onward is not — do not start it until the
|
||||||
|
soak is clean.
|
||||||
|
|
||||||
|
All of this flows through git and Argo. No `kubectl apply`, no `helm upgrade`.
|
||||||
|
|
||||||
|
## Loose ends
|
||||||
|
|
||||||
|
- `agent-pod/console` is publicly routed, unauthenticated, accepts free-form prompts
|
||||||
|
into a shell-capable container, and exposes a WebSocket. Migrating it behind the
|
||||||
|
gateway's auth is a security fix, not merely a port. Treat WebSocket upgrade as an
|
||||||
|
explicit requirement of the proxy layer.
|
||||||
|
- Eight `*.example.com` hosts exist on istio-class Ingresses in `llm-serving`
|
||||||
|
(`{embeddings,ornith,reasoning,reranker}[-predictor]-llm-serving.example.com`).
|
||||||
|
KServe defaults, not public, not Kong's — out of scope here, but they exist and
|
||||||
|
should not be mistaken for gateway routes.
|
||||||
|
- Ingress class split across the cluster is 7 kong / 17 nginx / 4 istio. Only the 7
|
||||||
|
kong ones are in scope.
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
# ADR-0001 — Retire Kong OSS in favour of a Go API gateway
|
||||||
|
|
||||||
|
Status: Accepted
|
||||||
|
Date: 2026-08-19
|
||||||
|
Deciders: rock
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
`api.riotpiao.com` is currently served by Kong OSS 3.4.1 (Helm, DB-less, namespace `api`,
|
||||||
|
Argo wave 7), sitting behind ingress-nginx which owns TLS. Kong routes to the KServe
|
||||||
|
model predictors in `llm-serving` via seven `ingressClassName: kong` Ingresses and six
|
||||||
|
`KongPlugin` CRs.
|
||||||
|
|
||||||
|
Three separate capabilities were attempted on Kong OSS. All three failed, and each
|
||||||
|
failure is already documented in-repo by the person who hit it:
|
||||||
|
|
||||||
|
**1. Body-based model dispatch is not expressible.**
|
||||||
|
From `k8s/apps/api/llm-routes.yaml`:
|
||||||
|
|
||||||
|
> a single `/v1/chat/completions` endpoint that dispatches on the body's `model` field is
|
||||||
|
> not expressible in Kong OSS (`ai-proxy-advanced`, which does multi-target model routing,
|
||||||
|
> is Enterprise-only).
|
||||||
|
|
||||||
|
The workaround is a path-per-model surface (`/v1/reasoning/chat/completions`,
|
||||||
|
`/v1/ornith/...`, `/v1/qwen/...`) with a `request-transformer` force-overwriting the body's
|
||||||
|
`model` field. This is not OpenAI-standard, so every client needs bespoke configuration —
|
||||||
|
visible today in `~/.pi/agent/models.json`, which carries three separate provider entries
|
||||||
|
for what should be one endpoint.
|
||||||
|
|
||||||
|
**2. OIDC is Enterprise-only.**
|
||||||
|
`k8s/apps/api/AUTH-PLAN.md` routes around the missing `openid-connect` plugin using the
|
||||||
|
built-in `jwt` plugin, which requires pinning Authentik's RSA public key onto a
|
||||||
|
KongConsumer. That plan lists its own consequence:
|
||||||
|
|
||||||
|
> Pinning `rsa_public_key`: Authentik key rotation would break it — document a rotation
|
||||||
|
> runbook, or have the provision script re-export the cert PEM into the Kong credential on
|
||||||
|
> each run.
|
||||||
|
|
||||||
|
A rotation runbook is a standing operational liability accepted only because the gateway
|
||||||
|
cannot fetch JWKS itself.
|
||||||
|
|
||||||
|
**3. `key-auth` cannot read `Authorization: Bearer`.**
|
||||||
|
From `k8s/apps/api/model-auth.yaml`:
|
||||||
|
|
||||||
|
> a raw `apikey: <key>` header succeeds (200), the same request with only
|
||||||
|
> `Authorization: Bearer <key>` fails (401). No OpenAI-SDK-compatible client (pi included)
|
||||||
|
> sends a raw apikey header or lets you customize the header name, so every such client was
|
||||||
|
> hard-blocked.
|
||||||
|
|
||||||
|
Consequence: authentication on the model routes is **currently disabled**. Verified live
|
||||||
|
2026-08-19 — `api.riotpiao.com/v1/reasoning/chat/completions` answers unauthenticated.
|
||||||
|
|
||||||
|
Separately, the intended surface has grown beyond LLM routing. The target is a
|
||||||
|
capability-per-subdomain API over cluster services — `sqs.riotpiao.com` for queue
|
||||||
|
operations, `workflow.riotpiao.com` for Temporal, `cluster.riotpiao.com` for atlas — each
|
||||||
|
needing request shaping, per-caller budgets and streaming semantics that are application
|
||||||
|
concerns, not gateway-plugin concerns.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Retire Kong OSS entirely. Replace it with a purpose-built Go service,
|
||||||
|
`homelab-frontend`, which owns north-south routing, authentication, and request shaping
|
||||||
|
for every public capability on `*.riotpiao.com`.
|
||||||
|
|
||||||
|
ingress-nginx keeps the edge and TLS. It forwards to the gateway instead of `kong-proxy`.
|
||||||
|
|
||||||
|
Authentication is Authentik OIDC, validated by fetching JWKS from
|
||||||
|
`https://authentik.riotpiao.com` at runtime.
|
||||||
|
|
||||||
|
## Options considered
|
||||||
|
|
||||||
|
**A. Stay on Kong OSS, accept the workarounds.**
|
||||||
|
Keeps a battle-tested proxy and its Prometheus plugin. But the path-per-model surface stays
|
||||||
|
non-standard, the RSA pinning runbook stays, and auth stays off until someone writes a
|
||||||
|
`request-transformer` shim to copy Bearer into an `apikey` header. Every new capability
|
||||||
|
(`sqs`, `workflow`) inherits the same constraints.
|
||||||
|
|
||||||
|
**B. Buy Kong Enterprise.**
|
||||||
|
`ai-proxy-advanced` and `openid-connect` solve 1 and 2. Does not solve the genuinely
|
||||||
|
application-level requirements at all — signed session cookies, per-session daily message
|
||||||
|
budgets, a 6-of-8 GPU sequence-slot semaphore with a bounded queue, and
|
||||||
|
disconnect-cancels-upstream are not gateway features in any tier. Cost for a homelab is not
|
||||||
|
justifiable.
|
||||||
|
|
||||||
|
**C. Go gateway, Kong retained for LLM paths only.**
|
||||||
|
Gradual migration, lower risk. But it means running two gateways indefinitely, splitting the
|
||||||
|
routing table across Kong CRDs and Go code, and keeping the Kong Helm release and its CRDs.
|
||||||
|
The split is the thing most likely to drift.
|
||||||
|
|
||||||
|
**D. Go gateway, Kong retired entirely.** — chosen
|
||||||
|
One routing table, one auth implementation, one place to reason about timeouts. The logic
|
||||||
|
being replaced is small: four `request-transformer` plugins that set a body field and
|
||||||
|
rewrite a URI, one `request-termination` serving a static JSON model list, and one
|
||||||
|
`prometheus` plugin. That is on the order of a hundred lines of Go, against roughly 480
|
||||||
|
lines of YAML it retires.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
### Gained
|
||||||
|
|
||||||
|
- **Standard OpenAI surface.** One `POST /v1/chat/completions`, model selected from the
|
||||||
|
request body. Any OpenAI SDK works unmodified. The three pi provider entries collapse to
|
||||||
|
one.
|
||||||
|
- **Working authentication.** Bearer tokens are read from the header, because it is our
|
||||||
|
code. JWKS is fetched and cached with automatic rotation handling, so the AUTH-PLAN.md
|
||||||
|
rotation runbook is deleted rather than written.
|
||||||
|
- **Application-level policy becomes possible.** GPU slot semaphore, per-session budgets,
|
||||||
|
disconnect propagation and SSE handling live where the state is.
|
||||||
|
- **One timeout story.** Kong currently sets `read-timeout: 3600000` (1 hour) on chat
|
||||||
|
routes, which silently defeats any shorter server-side cap. Retiring Kong removes the
|
||||||
|
conflicting layer.
|
||||||
|
- **~480 lines of gateway YAML deleted**, plus the Kong CRDs, the Helm release, and its
|
||||||
|
`ServerSideApply` workaround for oversized CRD annotations.
|
||||||
|
|
||||||
|
### Lost / assumed
|
||||||
|
|
||||||
|
- **We now own proxy correctness.** Connection pooling, retries, timeout propagation,
|
||||||
|
streaming passthrough, header hygiene, graceful shutdown. `net/http/httputil.ReverseProxy`
|
||||||
|
covers most of it, but it is our bug surface now.
|
||||||
|
- **Kong's Prometheus plugin goes away.** The gateway must expose equivalent metrics itself
|
||||||
|
(bandwidth, latency, status codes, upstream health) or observability regresses.
|
||||||
|
- **Migration touches live traffic.** pi depends on `api.riotpiao.com` today. Cutover must
|
||||||
|
be reversible — see `docs/MIGRATION-kong.md`.
|
||||||
|
- **`agent-pod/console` is a kong-class Ingress** exposing `/console` (WebSocket), `/run`
|
||||||
|
and `/sessions`. It must migrate too, and it is currently unauthenticated and publicly
|
||||||
|
routed while accepting free-form prompts into a shell-capable container. Putting it behind
|
||||||
|
the gateway's Authentik auth is a security improvement, not just a port.
|
||||||
|
|
||||||
|
### Risks
|
||||||
|
|
||||||
|
- Enabling Authentik auth will break any client currently relying on the unauthenticated
|
||||||
|
surface — including pi, until its `models.json` is updated. Auth must ship behind a flag
|
||||||
|
and be enabled deliberately.
|
||||||
|
- Kong's `request-termination` for `/v1/models` returns a **static** list that can drift
|
||||||
|
from what the engines actually serve. Porting it verbatim ports the bug; the gateway
|
||||||
|
should derive the list from configured upstreams instead.
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
module github.com/Riotpiaole/homelab-frontend
|
||||||
|
|
||||||
|
go 1.25.0
|
||||||
|
|
||||||
|
require gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
package config_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestLoadRoutesMissingAuthRequired tests that the auth-required flag is not a silent default:
|
||||||
|
// if it is absent from YAML, startup must fail with a message naming the offending route and field.
|
||||||
|
func TestLoadRoutesMissingAuthRequired(t *testing.T) {
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../testdata/config/missing-authRequired.yaml")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for missing authRequired, got nil")
|
||||||
|
}
|
||||||
|
if routes != nil {
|
||||||
|
t.Error("expected nil routes on error")
|
||||||
|
}
|
||||||
|
want := "route \"test-route\": field 'authRequired' is required"
|
||||||
|
got := err.Error()
|
||||||
|
if got != want {
|
||||||
|
t.Fatalf("error did not name the offending route and field; want %q, got %q", want, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Config holds the gateway configuration.
|
||||||
|
type Config struct {
|
||||||
|
// ListenAddr is the address to listen on for HTTP traffic.
|
||||||
|
ListenAddr string
|
||||||
|
// ShutdownTimeout is the maximum time to wait for in-flight requests
|
||||||
|
// to complete before forcing shutdown.
|
||||||
|
ShutdownTimeout time.Duration
|
||||||
|
// Routes maps route names to their upstream configuration.
|
||||||
|
Routes map[string]*Route
|
||||||
|
// Models maps model names to their upstream configuration.
|
||||||
|
// Multiple models can point to the same upstream address.
|
||||||
|
Models map[string]*ModelUpstream
|
||||||
|
}
|
||||||
|
|
||||||
|
// ModelUpstream holds upstream configuration for a specific model.
|
||||||
|
type ModelUpstream struct {
|
||||||
|
// Name is the model name clients send (e.g., "reasoning", "ornith:35b").
|
||||||
|
Name string
|
||||||
|
// Address is the upstream server address (host:port).
|
||||||
|
Address string
|
||||||
|
// Path is the upstream path for this model (e.g., "/v1/chat/completions").
|
||||||
|
Path string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Route represents a single route and its upstream configuration.
|
||||||
|
type Route struct {
|
||||||
|
// Name is the route identifier.
|
||||||
|
Name string
|
||||||
|
// Upstream holds the upstream server configuration.
|
||||||
|
Upstream Upstream
|
||||||
|
}
|
||||||
|
|
||||||
|
// Upstream holds upstream server configuration for a route.
|
||||||
|
type Upstream struct {
|
||||||
|
// Address is the upstream server address (host:port).
|
||||||
|
Address string
|
||||||
|
// PathRewrite is an optional path prefix rewrite. Empty string means no rewrite.
|
||||||
|
PathRewrite string
|
||||||
|
// ConnectTimeout is the maximum time to establish a connection to the upstream.
|
||||||
|
ConnectTimeout time.Duration
|
||||||
|
// ReadTimeout is the maximum time to read a response from the upstream.
|
||||||
|
ReadTimeout time.Duration
|
||||||
|
// WriteTimeout is the maximum time to write a request to the upstream.
|
||||||
|
WriteTimeout time.Duration
|
||||||
|
// MaxBodySize is the maximum request body size in bytes.
|
||||||
|
MaxBodySize int64
|
||||||
|
// AuthRequired indicates whether this route requires authentication.
|
||||||
|
AuthRequired bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// LookupModel finds a model by name (case-sensitive, exact match).
|
||||||
|
func (c *Config) LookupModel(name string) *ModelUpstream {
|
||||||
|
if c == nil || c.Models == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return c.Models[name]
|
||||||
|
}
|
||||||
|
|
||||||
|
// ModelNames returns a sorted list of all known model names.
|
||||||
|
func (c *Config) ModelNames() []string {
|
||||||
|
if c == nil || c.Models == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
names := make([]string, 0, len(c.Models))
|
||||||
|
for name := range c.Models {
|
||||||
|
names = append(names, name)
|
||||||
|
}
|
||||||
|
return names
|
||||||
|
}
|
||||||
|
|
||||||
|
// Load reads configuration from environment variables with defaults.
|
||||||
|
func Load() (*Config, error) {
|
||||||
|
listenAddr := "127.0.0.1:8080"
|
||||||
|
// Allow override via environment variable
|
||||||
|
if addr, ok := os.LookupEnv("LISTEN_ADDR"); ok {
|
||||||
|
listenAddr = addr
|
||||||
|
}
|
||||||
|
|
||||||
|
shutdownTimeout := 30 * time.Second
|
||||||
|
// Allow override via environment variable
|
||||||
|
if timeout, ok := os.LookupEnv("SHUTDOWN_TIMEOUT"); ok {
|
||||||
|
d, err := time.ParseDuration(timeout)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("invalid SHUTDOWN_TIMEOUT: %w", err)
|
||||||
|
}
|
||||||
|
shutdownTimeout = d
|
||||||
|
}
|
||||||
|
|
||||||
|
// Load routes and models from config file
|
||||||
|
routes := make(map[string]*Route)
|
||||||
|
models := make(map[string]*ModelUpstream)
|
||||||
|
if configPath, ok := os.LookupEnv("CONFIG_PATH"); ok {
|
||||||
|
loadedRoutes, loadedModels, err := LoadRoutesAndModelsFromFile(configPath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
routes = loadedRoutes
|
||||||
|
models = loadedModels
|
||||||
|
}
|
||||||
|
|
||||||
|
return &Config{
|
||||||
|
ListenAddr: listenAddr,
|
||||||
|
ShutdownTimeout: shutdownTimeout,
|
||||||
|
Routes: routes,
|
||||||
|
Models: models,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,209 @@
|
|||||||
|
package config_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestLoadRoutesValidConfig tests that a valid configuration loads correctly.
|
||||||
|
func TestLoadRoutesValidConfig(t *testing.T) {
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../testdata/config/valid.yaml")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error loading valid config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify all three routes are present
|
||||||
|
if len(routes) != 3 {
|
||||||
|
t.Errorf("expected 3 routes, got %d", len(routes))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify reasoning-chat route
|
||||||
|
reasoningRoute, ok := routes["reasoning-chat"]
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("reasoning-chat route not found")
|
||||||
|
}
|
||||||
|
if reasoningRoute.Name != "reasoning-chat" {
|
||||||
|
t.Errorf("route name mismatch: expected 'reasoning-chat', got %q", reasoningRoute.Name)
|
||||||
|
}
|
||||||
|
if reasoningRoute.Upstream.Address != "reasoning-predictor.llm-serving:80" {
|
||||||
|
t.Errorf("upstream address mismatch: expected 'reasoning-predictor.llm-serving:80', got %q", reasoningRoute.Upstream.Address)
|
||||||
|
}
|
||||||
|
if reasoningRoute.Upstream.ConnectTimeout != 10*time.Second {
|
||||||
|
t.Errorf("connect timeout mismatch: expected 10s, got %v", reasoningRoute.Upstream.ConnectTimeout)
|
||||||
|
}
|
||||||
|
if reasoningRoute.Upstream.ReadTimeout != time.Hour {
|
||||||
|
t.Errorf("read timeout mismatch: expected 1h, got %v", reasoningRoute.Upstream.ReadTimeout)
|
||||||
|
}
|
||||||
|
if reasoningRoute.Upstream.WriteTimeout != time.Hour {
|
||||||
|
t.Errorf("write timeout mismatch: expected 1h, got %v", reasoningRoute.Upstream.WriteTimeout)
|
||||||
|
}
|
||||||
|
if reasoningRoute.Upstream.MaxBodySize != 10485760 {
|
||||||
|
t.Errorf("max body size mismatch: expected 10485760, got %d", reasoningRoute.Upstream.MaxBodySize)
|
||||||
|
}
|
||||||
|
if !reasoningRoute.Upstream.AuthRequired {
|
||||||
|
t.Error("auth required should be true")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify ornith-chat route
|
||||||
|
ornithRoute, ok := routes["ornith-chat"]
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("ornith-chat route not found")
|
||||||
|
}
|
||||||
|
if ornithRoute.Upstream.ReadTimeout != 10*time.Minute {
|
||||||
|
t.Errorf("ornith read timeout mismatch: expected 10m, got %v", ornithRoute.Upstream.ReadTimeout)
|
||||||
|
}
|
||||||
|
if ornithRoute.Upstream.AuthRequired {
|
||||||
|
t.Error("ornith auth required should be false")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify embeddings route
|
||||||
|
embeddingsRoute, ok := routes["embeddings"]
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("embeddings route not found")
|
||||||
|
}
|
||||||
|
if embeddingsRoute.Upstream.MaxBodySize != 5242880 {
|
||||||
|
t.Errorf("embeddings max body size mismatch: expected 5242880, got %d", embeddingsRoute.Upstream.MaxBodySize)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoadRoutesMissingAddress tests that missing address field is caught.
|
||||||
|
func TestLoadRoutesMissingAddress(t *testing.T) {
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../testdata/config/missing-address.yaml")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for missing address, got nil")
|
||||||
|
}
|
||||||
|
if routes != nil {
|
||||||
|
t.Error("expected nil routes on error")
|
||||||
|
}
|
||||||
|
if errMsg := err.Error(); errMsg != "route \"test-route\": field 'address' is required" {
|
||||||
|
t.Errorf("expected error about missing address, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoadRoutesMissingConnectTimeout tests that missing connectTimeout field is caught.
|
||||||
|
func TestLoadRoutesMissingConnectTimeout(t *testing.T) {
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../testdata/config/missing-connectTimeout.yaml")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for missing connectTimeout, got nil")
|
||||||
|
}
|
||||||
|
if routes != nil {
|
||||||
|
t.Error("expected nil routes on error")
|
||||||
|
}
|
||||||
|
if errMsg := err.Error(); errMsg != "route \"test-route\": field 'connectTimeout' is required" {
|
||||||
|
t.Errorf("expected error about missing connectTimeout, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoadRoutesMissingReadTimeout tests that missing readTimeout field is caught.
|
||||||
|
func TestLoadRoutesMissingReadTimeout(t *testing.T) {
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../testdata/config/missing-readTimeout.yaml")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for missing readTimeout, got nil")
|
||||||
|
}
|
||||||
|
if routes != nil {
|
||||||
|
t.Error("expected nil routes on error")
|
||||||
|
}
|
||||||
|
if errMsg := err.Error(); errMsg != "route \"test-route\": field 'readTimeout' is required" {
|
||||||
|
t.Errorf("expected error about missing readTimeout, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoadRoutesMissingWriteTimeout tests that missing writeTimeout field is caught.
|
||||||
|
func TestLoadRoutesMissingWriteTimeout(t *testing.T) {
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../testdata/config/missing-writeTimeout.yaml")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for missing writeTimeout, got nil")
|
||||||
|
}
|
||||||
|
if routes != nil {
|
||||||
|
t.Error("expected nil routes on error")
|
||||||
|
}
|
||||||
|
if errMsg := err.Error(); errMsg != "route \"test-route\": field 'writeTimeout' is required" {
|
||||||
|
t.Errorf("expected error about missing writeTimeout, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoadRoutesMissingMaxBodySize tests that missing maxBodySize field is caught.
|
||||||
|
func TestLoadRoutesMissingMaxBodySize(t *testing.T) {
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../testdata/config/missing-maxBodySize.yaml")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for missing maxBodySize, got nil")
|
||||||
|
}
|
||||||
|
if routes != nil {
|
||||||
|
t.Error("expected nil routes on error")
|
||||||
|
}
|
||||||
|
if errMsg := err.Error(); errMsg != "route \"test-route\": field 'maxBodySize' is required and must be > 0" {
|
||||||
|
t.Errorf("expected error about missing maxBodySize, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoadRoutesMalformedConnectTimeout tests that malformed connectTimeout is caught.
|
||||||
|
func TestLoadRoutesMalformedConnectTimeout(t *testing.T) {
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../testdata/config/malformed-connectTimeout.yaml")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for malformed connectTimeout, got nil")
|
||||||
|
}
|
||||||
|
if routes != nil {
|
||||||
|
t.Error("expected nil routes on error")
|
||||||
|
}
|
||||||
|
if errMsg := err.Error(); errMsg != "route \"test-route\": invalid connectTimeout \"not-a-duration\": time: invalid duration \"not-a-duration\"" {
|
||||||
|
t.Errorf("expected error about malformed connectTimeout, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoadRoutesMalformedReadTimeout tests that malformed readTimeout is caught.
|
||||||
|
func TestLoadRoutesMalformedReadTimeout(t *testing.T) {
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../testdata/config/malformed-readTimeout.yaml")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for malformed readTimeout, got nil")
|
||||||
|
}
|
||||||
|
if routes != nil {
|
||||||
|
t.Error("expected nil routes on error")
|
||||||
|
}
|
||||||
|
if errMsg := err.Error(); errMsg != "route \"test-route\": invalid readTimeout \"invalid\": time: invalid duration \"invalid\"" {
|
||||||
|
t.Errorf("expected error about malformed readTimeout, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoadRoutesMalformedWriteTimeout tests that malformed writeTimeout is caught.
|
||||||
|
func TestLoadRoutesMalformedWriteTimeout(t *testing.T) {
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../testdata/config/malformed-writeTimeout.yaml")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for malformed writeTimeout, got nil")
|
||||||
|
}
|
||||||
|
if routes != nil {
|
||||||
|
t.Error("expected nil routes on error")
|
||||||
|
}
|
||||||
|
if errMsg := err.Error(); errMsg != "route \"test-route\": invalid writeTimeout \"bad\": time: invalid duration \"bad\"" {
|
||||||
|
t.Errorf("expected error about malformed writeTimeout, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoadRoutesInvalidAddressNoPort tests that address without port is caught.
|
||||||
|
func TestLoadRoutesInvalidAddressNoPort(t *testing.T) {
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../testdata/config/invalid-address-no-port.yaml")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for invalid address (no port), got nil")
|
||||||
|
}
|
||||||
|
if routes != nil {
|
||||||
|
t.Error("expected nil routes on error")
|
||||||
|
}
|
||||||
|
if errMsg := err.Error(); errMsg != "route \"test-route\": invalid address \"localhost\": address localhost: missing port in address" {
|
||||||
|
t.Errorf("expected error about invalid address, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoadRoutesDuplicateRouteNames tests that duplicate route names are caught.
|
||||||
|
func TestLoadRoutesDuplicateRouteNames(t *testing.T) {
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../testdata/config/duplicate-routes.yaml")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for duplicate routes, got nil")
|
||||||
|
}
|
||||||
|
if routes != nil {
|
||||||
|
t.Error("expected nil routes on error")
|
||||||
|
}
|
||||||
|
if errMsg := err.Error(); errMsg != "duplicate route: \"test-route\"" {
|
||||||
|
t.Errorf("expected error about duplicate routes, got: %s", errMsg)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
package config_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestLoadIntegration tests the full Load function with CONFIG_PATH env var
|
||||||
|
func TestLoadIntegration(t *testing.T) {
|
||||||
|
// Set the environment variable
|
||||||
|
originalConfigPath := os.Getenv("CONFIG_PATH")
|
||||||
|
os.Setenv("CONFIG_PATH", "../../testdata/config/valid.yaml")
|
||||||
|
defer func() {
|
||||||
|
if originalConfigPath != "" {
|
||||||
|
os.Setenv("CONFIG_PATH", originalConfigPath)
|
||||||
|
} else {
|
||||||
|
os.Unsetenv("CONFIG_PATH")
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
cfg, err := config.Load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg == nil {
|
||||||
|
t.Fatal("expected non-nil config")
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(cfg.Routes) != 3 {
|
||||||
|
t.Errorf("expected 3 routes, got %d", len(cfg.Routes))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify reasoning-chat is present
|
||||||
|
reasoningRoute, ok := cfg.Routes["reasoning-chat"]
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("reasoning-chat route not found")
|
||||||
|
}
|
||||||
|
if reasoningRoute.Upstream.Address != "reasoning-predictor.llm-serving:80" {
|
||||||
|
t.Errorf("unexpected address: %s", reasoningRoute.Upstream.Address)
|
||||||
|
}
|
||||||
|
if reasoningRoute.Upstream.ConnectTimeout != 10*time.Second {
|
||||||
|
t.Errorf("unexpected connect timeout: %v", reasoningRoute.Upstream.ConnectTimeout)
|
||||||
|
}
|
||||||
|
if reasoningRoute.Upstream.ReadTimeout != time.Hour {
|
||||||
|
t.Errorf("unexpected read timeout: %v", reasoningRoute.Upstream.ReadTimeout)
|
||||||
|
}
|
||||||
|
if reasoningRoute.Upstream.WriteTimeout != time.Hour {
|
||||||
|
t.Errorf("unexpected write timeout: %v", reasoningRoute.Upstream.WriteTimeout)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoadIntegrationWithInvalidConfig tests that Load fails with invalid config
|
||||||
|
func TestLoadIntegrationWithInvalidConfig(t *testing.T) {
|
||||||
|
originalConfigPath := os.Getenv("CONFIG_PATH")
|
||||||
|
os.Setenv("CONFIG_PATH", "../../testdata/config/missing-address.yaml")
|
||||||
|
defer func() {
|
||||||
|
if originalConfigPath != "" {
|
||||||
|
os.Setenv("CONFIG_PATH", originalConfigPath)
|
||||||
|
} else {
|
||||||
|
os.Unsetenv("CONFIG_PATH")
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
cfg, err := config.Load()
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error but got nil")
|
||||||
|
}
|
||||||
|
if cfg != nil {
|
||||||
|
t.Error("expected nil config on error")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,176 @@
|
|||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
"os"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gopkg.in/yaml.v3"
|
||||||
|
)
|
||||||
|
|
||||||
|
// rawConfig represents the structure of the YAML configuration file.
|
||||||
|
type rawConfig struct {
|
||||||
|
Routes []rawRoute `yaml:"routes"`
|
||||||
|
Models []rawModel `yaml:"models"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// rawRoute represents a single route in the YAML configuration.
|
||||||
|
type rawRoute struct {
|
||||||
|
Name string `yaml:"name"`
|
||||||
|
Upstream rawUpstream `yaml:"upstream"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// rawModel represents a single model entry in the YAML configuration.
|
||||||
|
type rawModel struct {
|
||||||
|
Name string `yaml:"name"`
|
||||||
|
Address string `yaml:"address"`
|
||||||
|
Path string `yaml:"path"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// rawUpstream represents upstream configuration in YAML.
|
||||||
|
type rawUpstream struct {
|
||||||
|
Address string `yaml:"address"`
|
||||||
|
PathRewrite string `yaml:"pathRewrite"`
|
||||||
|
ConnectTimeout string `yaml:"connectTimeout"`
|
||||||
|
ReadTimeout string `yaml:"readTimeout"`
|
||||||
|
WriteTimeout string `yaml:"writeTimeout"`
|
||||||
|
MaxBodySize int64 `yaml:"maxBodySize"`
|
||||||
|
AuthRequired *bool `yaml:"authRequired"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// LoadRoutesAndModelsFromFile loads both route and model configuration from a YAML file.
|
||||||
|
func LoadRoutesAndModelsFromFile(path string) (map[string]*Route, map[string]*ModelUpstream, error) {
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, fmt.Errorf("failed to read config file %q: %w", path, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var raw rawConfig
|
||||||
|
if err := yaml.Unmarshal(data, &raw); err != nil {
|
||||||
|
return nil, nil, fmt.Errorf("failed to parse config file %q: %w", path, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Load routes
|
||||||
|
routes := make(map[string]*Route)
|
||||||
|
for _, rawRoute := range raw.Routes {
|
||||||
|
if rawRoute.Name == "" {
|
||||||
|
return nil, nil, fmt.Errorf("route has empty name")
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, exists := routes[rawRoute.Name]; exists {
|
||||||
|
return nil, nil, fmt.Errorf("duplicate route: \"%s\"", rawRoute.Name)
|
||||||
|
}
|
||||||
|
|
||||||
|
upstream, err := parseUpstream(rawRoute.Name, rawRoute.Upstream)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
routes[rawRoute.Name] = &Route{
|
||||||
|
Name: rawRoute.Name,
|
||||||
|
Upstream: upstream,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Load models
|
||||||
|
models := make(map[string]*ModelUpstream)
|
||||||
|
for _, rawModel := range raw.Models {
|
||||||
|
// Validate model name is not empty
|
||||||
|
if rawModel.Name == "" {
|
||||||
|
return nil, nil, fmt.Errorf("model has empty name")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check for duplicate model names
|
||||||
|
if _, exists := models[rawModel.Name]; exists {
|
||||||
|
return nil, nil, fmt.Errorf("duplicate model: \"%s\"", rawModel.Name)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate address is not empty
|
||||||
|
if rawModel.Address == "" {
|
||||||
|
return nil, nil, fmt.Errorf("model \"%s\": field 'address' is required", rawModel.Name)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate address format (host:port)
|
||||||
|
if _, _, err := net.SplitHostPort(rawModel.Address); err != nil {
|
||||||
|
return nil, nil, fmt.Errorf("model \"%s\": invalid address \"%s\": %w", rawModel.Name, rawModel.Address, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
models[rawModel.Name] = &ModelUpstream{
|
||||||
|
Name: rawModel.Name,
|
||||||
|
Address: rawModel.Address,
|
||||||
|
Path: rawModel.Path,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return routes, models, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// LoadRoutesFromFile loads route configuration from a YAML file.
|
||||||
|
// It validates that all required fields are present and have valid values.
|
||||||
|
// Returns an error if the configuration is invalid.
|
||||||
|
// Deprecated: Use LoadRoutesAndModelsFromFile instead.
|
||||||
|
func LoadRoutesFromFile(path string) (map[string]*Route, error) {
|
||||||
|
routes, _, err := LoadRoutesAndModelsFromFile(path)
|
||||||
|
return routes, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseUpstream validates and parses upstream configuration from raw YAML.
|
||||||
|
func parseUpstream(routeName string, raw rawUpstream) (Upstream, error) {
|
||||||
|
// Validate address is not empty
|
||||||
|
if raw.Address == "" {
|
||||||
|
return Upstream{}, fmt.Errorf("route \"%s\": field 'address' is required", routeName)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate address format (host:port)
|
||||||
|
if _, _, err := net.SplitHostPort(raw.Address); err != nil {
|
||||||
|
return Upstream{}, fmt.Errorf("route \"%s\": invalid address \"%s\": %w", routeName, raw.Address, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate connectTimeout
|
||||||
|
if raw.ConnectTimeout == "" {
|
||||||
|
return Upstream{}, fmt.Errorf("route \"%s\": field 'connectTimeout' is required", routeName)
|
||||||
|
}
|
||||||
|
connectTimeout, err := time.ParseDuration(raw.ConnectTimeout)
|
||||||
|
if err != nil {
|
||||||
|
return Upstream{}, fmt.Errorf("route \"%s\": invalid connectTimeout \"%s\": %w", routeName, raw.ConnectTimeout, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate readTimeout
|
||||||
|
if raw.ReadTimeout == "" {
|
||||||
|
return Upstream{}, fmt.Errorf("route \"%s\": field 'readTimeout' is required", routeName)
|
||||||
|
}
|
||||||
|
readTimeout, err := time.ParseDuration(raw.ReadTimeout)
|
||||||
|
if err != nil {
|
||||||
|
return Upstream{}, fmt.Errorf("route \"%s\": invalid readTimeout \"%s\": %w", routeName, raw.ReadTimeout, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate writeTimeout
|
||||||
|
if raw.WriteTimeout == "" {
|
||||||
|
return Upstream{}, fmt.Errorf("route \"%s\": field 'writeTimeout' is required", routeName)
|
||||||
|
}
|
||||||
|
writeTimeout, err := time.ParseDuration(raw.WriteTimeout)
|
||||||
|
if err != nil {
|
||||||
|
return Upstream{}, fmt.Errorf("route \"%s\": invalid writeTimeout \"%s\": %w", routeName, raw.WriteTimeout, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate maxBodySize is not zero (it must be explicitly set)
|
||||||
|
if raw.MaxBodySize == 0 {
|
||||||
|
return Upstream{}, fmt.Errorf("route \"%s\": field 'maxBodySize' is required and must be > 0", routeName)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate authRequired is not missing
|
||||||
|
if raw.AuthRequired == nil {
|
||||||
|
return Upstream{}, fmt.Errorf("route \"%s\": field 'authRequired' is required", routeName)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Upstream{
|
||||||
|
Address: raw.Address,
|
||||||
|
PathRewrite: raw.PathRewrite,
|
||||||
|
ConnectTimeout: connectTimeout,
|
||||||
|
ReadTimeout: readTimeout,
|
||||||
|
WriteTimeout: writeTimeout,
|
||||||
|
MaxBodySize: raw.MaxBodySize,
|
||||||
|
AuthRequired: *raw.AuthRequired,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,285 @@
|
|||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestLoadModelsValidConfig(t *testing.T) {
|
||||||
|
data := `
|
||||||
|
routes:
|
||||||
|
- name: test-route
|
||||||
|
upstream:
|
||||||
|
address: "localhost:8000"
|
||||||
|
pathRewrite: ""
|
||||||
|
connectTimeout: "10s"
|
||||||
|
readTimeout: "30s"
|
||||||
|
writeTimeout: "30s"
|
||||||
|
maxBodySize: 1048576
|
||||||
|
authRequired: false
|
||||||
|
|
||||||
|
models:
|
||||||
|
- name: "reasoning"
|
||||||
|
address: "reasoning-predictor.llm-serving:80"
|
||||||
|
path: "/v1/chat/completions"
|
||||||
|
- name: "ornith:35b"
|
||||||
|
address: "ornith-predictor.llm-serving:80"
|
||||||
|
path: "/v1/chat/completions"
|
||||||
|
- name: "qwen2.5:3b-instruct"
|
||||||
|
address: "ornith-predictor.llm-serving:80"
|
||||||
|
path: "/v1/chat/completions"
|
||||||
|
`
|
||||||
|
|
||||||
|
tmpFile, _ := os.CreateTemp("", "config-*.yaml")
|
||||||
|
defer os.Remove(tmpFile.Name())
|
||||||
|
tmpFile.WriteString(data)
|
||||||
|
tmpFile.Close()
|
||||||
|
|
||||||
|
_, models, err := LoadRoutesAndModelsFromFile(tmpFile.Name())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to load config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(models) != 3 {
|
||||||
|
t.Errorf("expected 3 models, got %d", len(models))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test LookupModel
|
||||||
|
cfg := &Config{Models: models}
|
||||||
|
reasoning := cfg.LookupModel("reasoning")
|
||||||
|
if reasoning == nil {
|
||||||
|
t.Errorf("expected to find model 'reasoning'")
|
||||||
|
}
|
||||||
|
if reasoning.Address != "reasoning-predictor.llm-serving:80" {
|
||||||
|
t.Errorf("expected address reasoning-predictor.llm-serving:80, got %s", reasoning.Address)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test case sensitivity
|
||||||
|
notFound := cfg.LookupModel("Reasoning")
|
||||||
|
if notFound != nil {
|
||||||
|
t.Errorf("model lookup should be case-sensitive")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test ModelNames
|
||||||
|
names := cfg.ModelNames()
|
||||||
|
if len(names) != 3 {
|
||||||
|
t.Errorf("expected 3 model names, got %d", len(names))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadModelsDuplicateName(t *testing.T) {
|
||||||
|
data := `
|
||||||
|
routes:
|
||||||
|
- name: test-route
|
||||||
|
upstream:
|
||||||
|
address: "localhost:8000"
|
||||||
|
pathRewrite: ""
|
||||||
|
connectTimeout: "10s"
|
||||||
|
readTimeout: "30s"
|
||||||
|
writeTimeout: "30s"
|
||||||
|
maxBodySize: 1048576
|
||||||
|
authRequired: false
|
||||||
|
|
||||||
|
models:
|
||||||
|
- name: "reasoning"
|
||||||
|
address: "upstream1:80"
|
||||||
|
path: "/v1/chat"
|
||||||
|
- name: "reasoning"
|
||||||
|
address: "upstream2:80"
|
||||||
|
path: "/v1/chat"
|
||||||
|
`
|
||||||
|
|
||||||
|
tmpFile, _ := os.CreateTemp("", "config-*.yaml")
|
||||||
|
defer os.Remove(tmpFile.Name())
|
||||||
|
tmpFile.WriteString(data)
|
||||||
|
tmpFile.Close()
|
||||||
|
|
||||||
|
_, _, err := LoadRoutesAndModelsFromFile(tmpFile.Name())
|
||||||
|
if err == nil {
|
||||||
|
t.Errorf("expected error for duplicate model name, got nil")
|
||||||
|
}
|
||||||
|
if err.Error() != "duplicate model: \"reasoning\"" {
|
||||||
|
t.Errorf("expected duplicate model error, got: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadModelsEmptyName(t *testing.T) {
|
||||||
|
data := `
|
||||||
|
routes:
|
||||||
|
- name: test-route
|
||||||
|
upstream:
|
||||||
|
address: "localhost:8000"
|
||||||
|
pathRewrite: ""
|
||||||
|
connectTimeout: "10s"
|
||||||
|
readTimeout: "30s"
|
||||||
|
writeTimeout: "30s"
|
||||||
|
maxBodySize: 1048576
|
||||||
|
authRequired: false
|
||||||
|
|
||||||
|
models:
|
||||||
|
- name: ""
|
||||||
|
address: "upstream:80"
|
||||||
|
path: "/v1/chat"
|
||||||
|
`
|
||||||
|
|
||||||
|
tmpFile, _ := os.CreateTemp("", "config-*.yaml")
|
||||||
|
defer os.Remove(tmpFile.Name())
|
||||||
|
tmpFile.WriteString(data)
|
||||||
|
tmpFile.Close()
|
||||||
|
|
||||||
|
_, _, err := LoadRoutesAndModelsFromFile(tmpFile.Name())
|
||||||
|
if err == nil {
|
||||||
|
t.Errorf("expected error for empty model name, got nil")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadModelsMissingAddress(t *testing.T) {
|
||||||
|
data := `
|
||||||
|
routes:
|
||||||
|
- name: test-route
|
||||||
|
upstream:
|
||||||
|
address: "localhost:8000"
|
||||||
|
pathRewrite: ""
|
||||||
|
connectTimeout: "10s"
|
||||||
|
readTimeout: "30s"
|
||||||
|
writeTimeout: "30s"
|
||||||
|
maxBodySize: 1048576
|
||||||
|
authRequired: false
|
||||||
|
|
||||||
|
models:
|
||||||
|
- name: "reasoning"
|
||||||
|
address: ""
|
||||||
|
path: "/v1/chat"
|
||||||
|
`
|
||||||
|
|
||||||
|
tmpFile, _ := os.CreateTemp("", "config-*.yaml")
|
||||||
|
defer os.Remove(tmpFile.Name())
|
||||||
|
tmpFile.WriteString(data)
|
||||||
|
tmpFile.Close()
|
||||||
|
|
||||||
|
_, _, err := LoadRoutesAndModelsFromFile(tmpFile.Name())
|
||||||
|
if err == nil {
|
||||||
|
t.Errorf("expected error for missing address, got nil")
|
||||||
|
}
|
||||||
|
if err.Error() != "model \"reasoning\": field 'address' is required" {
|
||||||
|
t.Errorf("expected missing address error, got: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadModelsInvalidAddress(t *testing.T) {
|
||||||
|
data := `
|
||||||
|
routes:
|
||||||
|
- name: test-route
|
||||||
|
upstream:
|
||||||
|
address: "localhost:8000"
|
||||||
|
pathRewrite: ""
|
||||||
|
connectTimeout: "10s"
|
||||||
|
readTimeout: "30s"
|
||||||
|
writeTimeout: "30s"
|
||||||
|
maxBodySize: 1048576
|
||||||
|
authRequired: false
|
||||||
|
|
||||||
|
models:
|
||||||
|
- name: "reasoning"
|
||||||
|
address: "invalid-address-no-port"
|
||||||
|
path: "/v1/chat"
|
||||||
|
`
|
||||||
|
|
||||||
|
tmpFile, _ := os.CreateTemp("", "config-*.yaml")
|
||||||
|
defer os.Remove(tmpFile.Name())
|
||||||
|
tmpFile.WriteString(data)
|
||||||
|
tmpFile.Close()
|
||||||
|
|
||||||
|
_, _, err := LoadRoutesAndModelsFromFile(tmpFile.Name())
|
||||||
|
if err == nil {
|
||||||
|
t.Errorf("expected error for invalid address, got nil")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadModelsOptionalPath(t *testing.T) {
|
||||||
|
data := `
|
||||||
|
routes:
|
||||||
|
- name: test-route
|
||||||
|
upstream:
|
||||||
|
address: "localhost:8000"
|
||||||
|
pathRewrite: ""
|
||||||
|
connectTimeout: "10s"
|
||||||
|
readTimeout: "30s"
|
||||||
|
writeTimeout: "30s"
|
||||||
|
maxBodySize: 1048576
|
||||||
|
authRequired: false
|
||||||
|
|
||||||
|
models:
|
||||||
|
- name: "reasoning"
|
||||||
|
address: "upstream:80"
|
||||||
|
`
|
||||||
|
|
||||||
|
tmpFile, _ := os.CreateTemp("", "config-*.yaml")
|
||||||
|
defer os.Remove(tmpFile.Name())
|
||||||
|
tmpFile.WriteString(data)
|
||||||
|
tmpFile.Close()
|
||||||
|
|
||||||
|
_, models, err := LoadRoutesAndModelsFromFile(tmpFile.Name())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to load config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(models) != 1 {
|
||||||
|
t.Errorf("expected 1 model, got %d", len(models))
|
||||||
|
}
|
||||||
|
|
||||||
|
model := models["reasoning"]
|
||||||
|
if model.Path != "" {
|
||||||
|
t.Errorf("expected empty path when not specified, got %s", model.Path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestModelSameUpstreamMultipleNames(t *testing.T) {
|
||||||
|
data := `
|
||||||
|
routes:
|
||||||
|
- name: test-route
|
||||||
|
upstream:
|
||||||
|
address: "localhost:8000"
|
||||||
|
pathRewrite: ""
|
||||||
|
connectTimeout: "10s"
|
||||||
|
readTimeout: "30s"
|
||||||
|
writeTimeout: "30s"
|
||||||
|
maxBodySize: 1048576
|
||||||
|
authRequired: false
|
||||||
|
|
||||||
|
models:
|
||||||
|
- name: "ornith:35b"
|
||||||
|
address: "ollama-pod:80"
|
||||||
|
path: "/v1/chat"
|
||||||
|
- name: "qwen2.5:3b"
|
||||||
|
address: "ollama-pod:80"
|
||||||
|
path: "/v1/chat"
|
||||||
|
`
|
||||||
|
|
||||||
|
tmpFile, _ := os.CreateTemp("", "config-*.yaml")
|
||||||
|
defer os.Remove(tmpFile.Name())
|
||||||
|
tmpFile.WriteString(data)
|
||||||
|
tmpFile.Close()
|
||||||
|
|
||||||
|
routes, models, err := LoadRoutesAndModelsFromFile(tmpFile.Name())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to load config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(models) != 2 {
|
||||||
|
t.Errorf("expected 2 models, got %d", len(models))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Both should resolve independently
|
||||||
|
cfg := &Config{Routes: routes, Models: models}
|
||||||
|
m1 := cfg.LookupModel("ornith:35b")
|
||||||
|
m2 := cfg.LookupModel("qwen2.5:3b")
|
||||||
|
|
||||||
|
if m1.Address != m2.Address {
|
||||||
|
t.Errorf("expected both models to point to same upstream")
|
||||||
|
}
|
||||||
|
|
||||||
|
if m1 == m2 {
|
||||||
|
t.Errorf("expected different ModelUpstream objects even for same address")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,200 @@
|
|||||||
|
// Package logging provides structured JSON logging for the gateway.
|
||||||
|
// All logs are emitted as single-line JSON records.
|
||||||
|
package logging
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Level represents a log level.
|
||||||
|
type Level int
|
||||||
|
|
||||||
|
const (
|
||||||
|
LevelDebug Level = iota
|
||||||
|
LevelInfo
|
||||||
|
LevelWarn
|
||||||
|
LevelError
|
||||||
|
)
|
||||||
|
|
||||||
|
// Logger provides structured logging with sensitive data redaction.
|
||||||
|
type Logger struct {
|
||||||
|
level Level
|
||||||
|
out io.Writer
|
||||||
|
}
|
||||||
|
|
||||||
|
// New creates a new structured logger with the given level writing to out.
|
||||||
|
func New(level Level, out io.Writer) *Logger {
|
||||||
|
if out == nil {
|
||||||
|
out = os.Stderr
|
||||||
|
}
|
||||||
|
return &Logger{level: level, out: out}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseLevel parses a log level string (debug, info, warn, error).
|
||||||
|
func ParseLevel(s string) Level {
|
||||||
|
switch strings.ToLower(s) {
|
||||||
|
case "debug":
|
||||||
|
return LevelDebug
|
||||||
|
case "info":
|
||||||
|
return LevelInfo
|
||||||
|
case "warn":
|
||||||
|
return LevelWarn
|
||||||
|
case "error":
|
||||||
|
return LevelError
|
||||||
|
default:
|
||||||
|
return LevelInfo
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// LogRecord is a single structured log entry.
|
||||||
|
type LogRecord struct {
|
||||||
|
Timestamp string `json:"timestamp"`
|
||||||
|
Level string `json:"level"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
Route string `json:"route,omitempty"`
|
||||||
|
Upstream string `json:"upstream,omitempty"`
|
||||||
|
Method string `json:"method,omitempty"`
|
||||||
|
Path string `json:"path,omitempty"`
|
||||||
|
Status int `json:"status,omitempty"`
|
||||||
|
Duration string `json:"duration,omitempty"`
|
||||||
|
Reason string `json:"reason,omitempty"`
|
||||||
|
Error string `json:"error,omitempty"`
|
||||||
|
Extra map[string]string `json:"extra,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// RequestLog logs a request with response information.
|
||||||
|
func (l *Logger) RequestLog(route, upstream, method, path string, status int, duration time.Duration, reason string) {
|
||||||
|
if l.level > LevelInfo {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
record := LogRecord{
|
||||||
|
Timestamp: time.Now().UTC().Format(time.RFC3339Nano),
|
||||||
|
Level: "info",
|
||||||
|
Message: "request",
|
||||||
|
Route: route,
|
||||||
|
Upstream: upstream,
|
||||||
|
Method: method,
|
||||||
|
Path: path,
|
||||||
|
Status: status,
|
||||||
|
Duration: duration.String(),
|
||||||
|
Reason: reason,
|
||||||
|
}
|
||||||
|
l.emit(record)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Infof logs an info-level message.
|
||||||
|
func (l *Logger) Infof(msg string, fields map[string]string) {
|
||||||
|
if l.level > LevelInfo {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
record := LogRecord{
|
||||||
|
Timestamp: time.Now().UTC().Format(time.RFC3339Nano),
|
||||||
|
Level: "info",
|
||||||
|
Message: msg,
|
||||||
|
Extra: fields,
|
||||||
|
}
|
||||||
|
l.emit(record)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Warnf logs a warn-level message.
|
||||||
|
func (l *Logger) Warnf(msg string, fields map[string]string) {
|
||||||
|
if l.level > LevelWarn {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
record := LogRecord{
|
||||||
|
Timestamp: time.Now().UTC().Format(time.RFC3339Nano),
|
||||||
|
Level: "warn",
|
||||||
|
Message: msg,
|
||||||
|
Extra: fields,
|
||||||
|
}
|
||||||
|
l.emit(record)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Errorf logs an error-level message.
|
||||||
|
func (l *Logger) Errorf(msg string, err error, fields map[string]string) {
|
||||||
|
if l.level > LevelError {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
errStr := ""
|
||||||
|
if err != nil {
|
||||||
|
errStr = err.Error()
|
||||||
|
}
|
||||||
|
record := LogRecord{
|
||||||
|
Timestamp: time.Now().UTC().Format(time.RFC3339Nano),
|
||||||
|
Level: "error",
|
||||||
|
Message: msg,
|
||||||
|
Error: errStr,
|
||||||
|
Extra: fields,
|
||||||
|
}
|
||||||
|
l.emit(record)
|
||||||
|
}
|
||||||
|
|
||||||
|
// emit writes a record as a single JSON line.
|
||||||
|
func (l *Logger) emit(record LogRecord) {
|
||||||
|
data, err := json.Marshal(record)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("failed to marshal log record: %v", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_, _ = l.out.Write(append(data, '\n'))
|
||||||
|
}
|
||||||
|
|
||||||
|
// global is the singleton logger instance.
|
||||||
|
var global *Logger
|
||||||
|
|
||||||
|
// Init initializes the global logger with the given level.
|
||||||
|
// If not called, defaults to Info level on stderr.
|
||||||
|
func Init(level Level, out io.Writer) {
|
||||||
|
global = New(level, out)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ensure global is initialized
|
||||||
|
func ensure() {
|
||||||
|
if global == nil {
|
||||||
|
global = New(LevelInfo, os.Stderr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// RequestLog logs a request with response information using the global logger.
|
||||||
|
func RequestLog(route, upstream, method, path string, status int, duration time.Duration, reason string) {
|
||||||
|
ensure()
|
||||||
|
global.RequestLog(route, upstream, method, path, status, duration, reason)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Infof logs an info message using the global logger.
|
||||||
|
func Infof(msg string, fields map[string]string) {
|
||||||
|
ensure()
|
||||||
|
global.Infof(msg, fields)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Warnf logs a warn message using the global logger.
|
||||||
|
func Warnf(msg string, fields map[string]string) {
|
||||||
|
ensure()
|
||||||
|
global.Warnf(msg, fields)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Errorf logs an error message using the global logger.
|
||||||
|
func Errorf(msg string, err error, fields map[string]string) {
|
||||||
|
ensure()
|
||||||
|
global.Errorf(msg, err, fields)
|
||||||
|
}
|
||||||
|
|
||||||
|
// FromContext retrieves the logger from a context, or returns the global logger.
|
||||||
|
func FromContext(ctx context.Context) *Logger {
|
||||||
|
ensure()
|
||||||
|
if l, ok := ctx.Value("logger").(*Logger); ok {
|
||||||
|
return l
|
||||||
|
}
|
||||||
|
return global
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithContext returns a new context with the logger attached.
|
||||||
|
func WithContext(ctx context.Context, l *Logger) context.Context {
|
||||||
|
return context.WithValue(ctx, "logger", l)
|
||||||
|
}
|
||||||
@@ -0,0 +1,266 @@
|
|||||||
|
package logging
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestLogStructure(t *testing.T) {
|
||||||
|
buf := &bytes.Buffer{}
|
||||||
|
logger := New(LevelInfo, buf)
|
||||||
|
|
||||||
|
logger.RequestLog("test-route", "localhost:8080", "POST", "/v1/chat/completions", 200, 100*time.Millisecond, "")
|
||||||
|
|
||||||
|
lines := strings.TrimSpace(buf.String())
|
||||||
|
if lines == "" {
|
||||||
|
t.Fatal("expected log output, got empty")
|
||||||
|
}
|
||||||
|
|
||||||
|
var record LogRecord
|
||||||
|
if err := json.Unmarshal([]byte(lines), &record); err != nil {
|
||||||
|
t.Fatalf("failed to unmarshal log record: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if record.Level != "info" {
|
||||||
|
t.Errorf("expected level info, got %s", record.Level)
|
||||||
|
}
|
||||||
|
if record.Route != "test-route" {
|
||||||
|
t.Errorf("expected route test-route, got %s", record.Route)
|
||||||
|
}
|
||||||
|
if record.Upstream != "localhost:8080" {
|
||||||
|
t.Errorf("expected upstream localhost:8080, got %s", record.Upstream)
|
||||||
|
}
|
||||||
|
if record.Method != "POST" {
|
||||||
|
t.Errorf("expected method POST, got %s", record.Method)
|
||||||
|
}
|
||||||
|
if record.Path != "/v1/chat/completions" {
|
||||||
|
t.Errorf("expected path /v1/chat/completions, got %s", record.Path)
|
||||||
|
}
|
||||||
|
if record.Status != 200 {
|
||||||
|
t.Errorf("expected status 200, got %d", record.Status)
|
||||||
|
}
|
||||||
|
if record.Duration != (100 * time.Millisecond).String() {
|
||||||
|
t.Errorf("expected duration 100ms, got %s", record.Duration)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRejectedRequestReason(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
reason string
|
||||||
|
}{
|
||||||
|
{"unknown_model", "unknown_model"},
|
||||||
|
{"body_too_large", "body_too_large"},
|
||||||
|
{"auth_failed", "auth_failed"},
|
||||||
|
{"concurrency_limit", "concurrency_limit"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
buf := &bytes.Buffer{}
|
||||||
|
logger := New(LevelInfo, buf)
|
||||||
|
|
||||||
|
logger.RequestLog("test", "upstream", "POST", "/path", 400, 10*time.Millisecond, tt.reason)
|
||||||
|
|
||||||
|
var record LogRecord
|
||||||
|
if err := json.Unmarshal([]byte(strings.TrimSpace(buf.String())), &record); err != nil {
|
||||||
|
t.Fatalf("failed to unmarshal: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if record.Reason != tt.reason {
|
||||||
|
t.Errorf("expected reason %s, got %s", tt.reason, record.Reason)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNoBodyLogging(t *testing.T) {
|
||||||
|
buf := &bytes.Buffer{}
|
||||||
|
logger := New(LevelInfo, buf)
|
||||||
|
|
||||||
|
// Simulate logging with fields that might contain body-like data
|
||||||
|
logger.RequestLog("route", "upstream", "POST", "/path", 400, time.Millisecond, "reason")
|
||||||
|
|
||||||
|
output := buf.String()
|
||||||
|
// Ensure no field named "body" appears in the output
|
||||||
|
if strings.Contains(output, "\"body\"") {
|
||||||
|
t.Errorf("request body should not be logged, but found 'body' field in: %s", output)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNoTokenLogging(t *testing.T) {
|
||||||
|
buf := &bytes.Buffer{}
|
||||||
|
logger := New(LevelInfo, buf)
|
||||||
|
|
||||||
|
// Log a rejected request as it would happen with auth failure
|
||||||
|
logger.RequestLog("route", "upstream", "POST", "/path", 401, time.Millisecond, "auth_failed")
|
||||||
|
|
||||||
|
output := buf.String()
|
||||||
|
|
||||||
|
// Ensure no Authorization header value appears
|
||||||
|
testToken := "sk-1234567890abcdef"
|
||||||
|
if strings.Contains(output, testToken) {
|
||||||
|
t.Errorf("bearer token should not be logged, but found in output: %s", output)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ensure Authorization header field doesn't appear
|
||||||
|
if strings.Contains(output, "Authorization") {
|
||||||
|
t.Errorf("Authorization header should not be logged, but found in output: %s", output)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ensure "bearer" or "token" keywords don't appear in the output
|
||||||
|
lowerOutput := strings.ToLower(output)
|
||||||
|
if strings.Contains(lowerOutput, "bearer") {
|
||||||
|
t.Errorf("bearer token substring should not appear, but found in output: %s", output)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConsistentFieldSet(t *testing.T) {
|
||||||
|
buf := &bytes.Buffer{}
|
||||||
|
logger := New(LevelInfo, buf)
|
||||||
|
|
||||||
|
logger.RequestLog("route", "upstream", "GET", "/healthz", 200, time.Millisecond, "")
|
||||||
|
|
||||||
|
var record LogRecord
|
||||||
|
if err := json.Unmarshal([]byte(strings.TrimSpace(buf.String())), &record); err != nil {
|
||||||
|
t.Fatalf("failed to unmarshal: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// All required fields should be present
|
||||||
|
if record.Timestamp == "" {
|
||||||
|
t.Error("timestamp field missing")
|
||||||
|
}
|
||||||
|
if record.Level == "" {
|
||||||
|
t.Error("level field missing")
|
||||||
|
}
|
||||||
|
if record.Message == "" {
|
||||||
|
t.Error("message field missing")
|
||||||
|
}
|
||||||
|
if record.Route == "" {
|
||||||
|
t.Error("route field missing")
|
||||||
|
}
|
||||||
|
if record.Upstream == "" {
|
||||||
|
t.Error("upstream field missing")
|
||||||
|
}
|
||||||
|
if record.Method == "" {
|
||||||
|
t.Error("method field missing")
|
||||||
|
}
|
||||||
|
if record.Path == "" {
|
||||||
|
t.Error("path field missing")
|
||||||
|
}
|
||||||
|
if record.Status == 0 {
|
||||||
|
t.Error("status field missing")
|
||||||
|
}
|
||||||
|
if record.Duration == "" {
|
||||||
|
t.Error("duration field missing")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLogLevelFiltering(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
level Level
|
||||||
|
shouldLog bool
|
||||||
|
}{
|
||||||
|
{"debug_level", LevelDebug, true},
|
||||||
|
{"info_level", LevelInfo, true},
|
||||||
|
{"warn_level_info_msg", LevelWarn, false},
|
||||||
|
{"error_level_info_msg", LevelError, false},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
buf := &bytes.Buffer{}
|
||||||
|
logger := New(tt.level, buf)
|
||||||
|
|
||||||
|
logger.RequestLog("route", "upstream", "GET", "/path", 200, time.Millisecond, "")
|
||||||
|
|
||||||
|
if tt.shouldLog && buf.Len() == 0 {
|
||||||
|
t.Errorf("expected log output at level %d, got empty", tt.level)
|
||||||
|
}
|
||||||
|
if !tt.shouldLog && buf.Len() > 0 {
|
||||||
|
t.Errorf("expected no log output at level %d, got: %s", tt.level, buf.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLogLineFormat(t *testing.T) {
|
||||||
|
buf := &bytes.Buffer{}
|
||||||
|
logger := New(LevelInfo, buf)
|
||||||
|
|
||||||
|
logger.RequestLog("route", "upstream", "POST", "/path", 201, time.Millisecond, "created")
|
||||||
|
|
||||||
|
output := strings.TrimSpace(buf.String())
|
||||||
|
if !strings.HasSuffix(output, "\n") && buf.String() != output {
|
||||||
|
// Single line with newline
|
||||||
|
if !strings.Contains(buf.String(), "\n") {
|
||||||
|
t.Error("expected newline-terminated log line")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Must be valid JSON
|
||||||
|
var record LogRecord
|
||||||
|
if err := json.Unmarshal([]byte(output), &record); err != nil {
|
||||||
|
t.Errorf("log output is not valid JSON: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRejectedRequestExactlyOneRecord(t *testing.T) {
|
||||||
|
buf := &bytes.Buffer{}
|
||||||
|
logger := New(LevelInfo, buf)
|
||||||
|
|
||||||
|
// Simulate a rejected request
|
||||||
|
logger.RequestLog("test-route", "upstream:8080", "POST", "/v1/chat/completions", 401, 5*time.Millisecond, "auth_failed")
|
||||||
|
|
||||||
|
lines := strings.Split(strings.TrimSpace(buf.String()), "\n")
|
||||||
|
if len(lines) != 1 {
|
||||||
|
t.Errorf("expected exactly one log record, got %d", len(lines))
|
||||||
|
}
|
||||||
|
|
||||||
|
var record LogRecord
|
||||||
|
if err := json.Unmarshal([]byte(lines[0]), &record); err != nil {
|
||||||
|
t.Fatalf("failed to unmarshal: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify reason is present and meaningful
|
||||||
|
if record.Reason != "auth_failed" {
|
||||||
|
t.Errorf("expected reason 'auth_failed', got '%s'", record.Reason)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify no token-like strings appear
|
||||||
|
recordJSON, _ := json.Marshal(record)
|
||||||
|
recordStr := string(recordJSON)
|
||||||
|
if strings.Contains(recordStr, "Bearer") || strings.Contains(recordStr, "bearer") ||
|
||||||
|
strings.Contains(recordStr, "Authorization") || strings.Contains(recordStr, "authorization") {
|
||||||
|
t.Errorf("rejected request log should not contain token/auth header substring: %s", recordStr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseLevel(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
input string
|
||||||
|
expected Level
|
||||||
|
}{
|
||||||
|
{"debug", LevelDebug},
|
||||||
|
{"info", LevelInfo},
|
||||||
|
{"warn", LevelWarn},
|
||||||
|
{"error", LevelError},
|
||||||
|
{"DEBUG", LevelDebug},
|
||||||
|
{"Info", LevelInfo},
|
||||||
|
{"unknown", LevelInfo},
|
||||||
|
{"", LevelInfo},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.input, func(t *testing.T) {
|
||||||
|
got := ParseLevel(tt.input)
|
||||||
|
if got != tt.expected {
|
||||||
|
t.Errorf("expected %d, got %d", tt.expected, got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,342 @@
|
|||||||
|
package proxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestBodyBasedDispatch verifies that /v1/chat/completions routes based on model field.
|
||||||
|
func TestBodyBasedDispatch(t *testing.T) {
|
||||||
|
// Create two separate upstreams to verify routing
|
||||||
|
reasoningCalled := false
|
||||||
|
ornithCalled := false
|
||||||
|
|
||||||
|
reasoningServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
reasoningCalled = true
|
||||||
|
if r.URL.Path != "/v1/chat/completions" {
|
||||||
|
t.Errorf("expected path /v1/chat/completions, got %s", r.URL.Path)
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprint(w, `{"choices":[{"message":{"content":"response from reasoning"}}]}`)
|
||||||
|
}))
|
||||||
|
defer reasoningServer.Close()
|
||||||
|
|
||||||
|
ornithServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
ornithCalled = true
|
||||||
|
if r.URL.Path != "/v1/chat/completions" {
|
||||||
|
t.Errorf("expected path /v1/chat/completions, got %s", r.URL.Path)
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprint(w, `{"choices":[{"message":{"content":"response from ornith"}}]}`)
|
||||||
|
}))
|
||||||
|
defer ornithServer.Close()
|
||||||
|
|
||||||
|
reasoningAddr := strings.TrimPrefix(reasoningServer.URL, "http://")
|
||||||
|
ornithAddr := strings.TrimPrefix(ornithServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Models: map[string]*config.ModelUpstream{
|
||||||
|
"reasoning": {
|
||||||
|
Name: "reasoning",
|
||||||
|
Address: reasoningAddr,
|
||||||
|
Path: "/v1/chat/completions",
|
||||||
|
},
|
||||||
|
"ornith:35b": {
|
||||||
|
Name: "ornith:35b",
|
||||||
|
Address: ornithAddr,
|
||||||
|
Path: "/v1/chat/completions",
|
||||||
|
},
|
||||||
|
"qwen2.5:3b-instruct": {
|
||||||
|
Name: "qwen2.5:3b-instruct",
|
||||||
|
Address: ornithAddr,
|
||||||
|
Path: "/v1/chat/completions",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
Routes: make(map[string]*config.Route),
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Test 1: Route to reasoning upstream
|
||||||
|
reasoningCalled = false
|
||||||
|
ornithCalled = false
|
||||||
|
requestBody := `{"model":"reasoning","messages":[{"role":"user","content":"hi"}]}`
|
||||||
|
resp, _ := http.Post(server.URL+"/v1/chat/completions", "application/json", strings.NewReader(requestBody))
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
if !reasoningCalled {
|
||||||
|
t.Errorf("expected reasoning upstream to be called")
|
||||||
|
}
|
||||||
|
if ornithCalled {
|
||||||
|
t.Errorf("expected ornith upstream NOT to be called")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test 2: Route to ornith upstream (both models point there)
|
||||||
|
reasoningCalled = false
|
||||||
|
ornithCalled = false
|
||||||
|
requestBody = `{"model":"ornith:35b","messages":[{"role":"user","content":"hi"}]}`
|
||||||
|
resp, _ = http.Post(server.URL+"/v1/chat/completions", "application/json", strings.NewReader(requestBody))
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
if reasoningCalled {
|
||||||
|
t.Errorf("expected reasoning upstream NOT to be called")
|
||||||
|
}
|
||||||
|
if !ornithCalled {
|
||||||
|
t.Errorf("expected ornith upstream to be called")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test 3: qwen2.5 also routes to ornith
|
||||||
|
reasoningCalled = false
|
||||||
|
ornithCalled = false
|
||||||
|
requestBody = `{"model":"qwen2.5:3b-instruct","messages":[{"role":"user","content":"hi"}]}`
|
||||||
|
resp, _ = http.Post(server.URL+"/v1/chat/completions", "application/json", strings.NewReader(requestBody))
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
if reasoningCalled {
|
||||||
|
t.Errorf("expected reasoning upstream NOT to be called")
|
||||||
|
}
|
||||||
|
if !ornithCalled {
|
||||||
|
t.Errorf("expected ornith upstream to be called")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBodyPreservedUnmodified verifies that the request body is forwarded unmodified.
|
||||||
|
func TestBodyPreservedUnmodified(t *testing.T) {
|
||||||
|
var receivedBody []byte
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
var err error
|
||||||
|
receivedBody, err = io.ReadAll(r.Body)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("failed to read body: %v", err)
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprint(w, `{}`)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Models: map[string]*config.ModelUpstream{
|
||||||
|
"reasoning": {
|
||||||
|
Name: "reasoning",
|
||||||
|
Address: upstreamAddr,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
Routes: make(map[string]*config.Route),
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Send a request with specific body content
|
||||||
|
originalBody := `{"model":"reasoning","stream":true,"messages":[{"role":"user","content":"hello world"}],"temperature":0.7}`
|
||||||
|
resp, _ := http.Post(server.URL+"/v1/chat/completions", "application/json", strings.NewReader(originalBody))
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
if string(receivedBody) != originalBody {
|
||||||
|
t.Errorf("body modified: expected %s, got %s", originalBody, string(receivedBody))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStreamingUnbuffered verifies that streaming responses are unbuffered.
|
||||||
|
func TestStreamingUnbuffered(t *testing.T) {
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/event-stream")
|
||||||
|
w.Header().Set("Cache-Control", "no-cache")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
|
||||||
|
rc := http.NewResponseController(w)
|
||||||
|
for i := 0; i < 3; i++ {
|
||||||
|
fmt.Fprintf(w, "data: {\"choices\":[{\"delta\":{\"content\":\"token%d\"}}]}\n\n", i)
|
||||||
|
rc.Flush()
|
||||||
|
time.Sleep(50 * time.Millisecond)
|
||||||
|
}
|
||||||
|
fmt.Fprint(w, "data: [DONE]\n\n")
|
||||||
|
rc.Flush()
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Models: map[string]*config.ModelUpstream{
|
||||||
|
"reasoning": {
|
||||||
|
Name: "reasoning",
|
||||||
|
Address: upstreamAddr,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
Routes: make(map[string]*config.Route),
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
requestBody := `{"model":"reasoning","stream":true,"messages":[]}`
|
||||||
|
resp, err := http.Post(server.URL+"/v1/chat/completions", "application/json", strings.NewReader(requestBody))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
if resp.Header.Get("Content-Type") != "text/event-stream" {
|
||||||
|
t.Errorf("expected Content-Type: text/event-stream, got %s", resp.Header.Get("Content-Type"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read response and verify streaming
|
||||||
|
startTime := time.Now()
|
||||||
|
respBody, _ := io.ReadAll(resp.Body)
|
||||||
|
duration := time.Since(startTime)
|
||||||
|
|
||||||
|
respStr := string(respBody)
|
||||||
|
if !strings.Contains(respStr, "token0") || !strings.Contains(respStr, "token1") || !strings.Contains(respStr, "token2") {
|
||||||
|
t.Errorf("expected all tokens in response, got: %s", respStr)
|
||||||
|
}
|
||||||
|
|
||||||
|
// With 50ms gaps and 3 tokens, we should take at least 100ms
|
||||||
|
// If completely buffered, would be much faster
|
||||||
|
if duration < 100*time.Millisecond {
|
||||||
|
t.Logf("response arrived very quickly (%.0fms) - may indicate buffering", duration.Seconds()*1000)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUnknownModelReject verifies that unknown models are rejected.
|
||||||
|
func TestUnknownModelReject(t *testing.T) {
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Models: map[string]*config.ModelUpstream{
|
||||||
|
"reasoning": {
|
||||||
|
Name: "reasoning",
|
||||||
|
Address: upstreamAddr,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
Routes: make(map[string]*config.Route),
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
requestBody := `{"model":"unknown-model","messages":[]}`
|
||||||
|
resp, _ := http.Post(server.URL+"/v1/chat/completions", "application/json", strings.NewReader(requestBody))
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusNotFound {
|
||||||
|
t.Errorf("expected 404 for unknown model, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestMissingModelField verifies that missing model field is rejected.
|
||||||
|
func TestMissingModelField(t *testing.T) {
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Models: map[string]*config.ModelUpstream{
|
||||||
|
"reasoning": {
|
||||||
|
Name: "reasoning",
|
||||||
|
Address: upstreamAddr,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
Routes: make(map[string]*config.Route),
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
requestBody := `{"messages":[]}`
|
||||||
|
resp, _ := http.Post(server.URL+"/v1/chat/completions", "application/json", strings.NewReader(requestBody))
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusNotFound {
|
||||||
|
t.Errorf("expected 404 for missing model, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBodySize verifies that body size cap is enforced for dispatched requests.
|
||||||
|
func TestBodySizeCappedDispatch(t *testing.T) {
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Models: map[string]*config.ModelUpstream{
|
||||||
|
"reasoning": {
|
||||||
|
Name: "reasoning",
|
||||||
|
Address: upstreamAddr,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
Routes: make(map[string]*config.Route),
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Create a JSON payload
|
||||||
|
payload := map[string]interface{}{
|
||||||
|
"model": "reasoning",
|
||||||
|
"messages": []map[string]string{
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": strings.Repeat("x", 100),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
body, _ := json.Marshal(payload)
|
||||||
|
req, _ := http.NewRequest("POST", server.URL+"/v1/chat/completions", bytes.NewReader(body))
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
req.ContentLength = int64(len(body))
|
||||||
|
|
||||||
|
resp, _ := http.DefaultClient.Do(req)
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
// Request should be accepted (body size is reasonable)
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Errorf("expected 200 for reasonable body, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,333 @@
|
|||||||
|
package proxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestBodySizeCapExact verifies that a body exactly at the cap is accepted.
|
||||||
|
func TestBodySizeCapExact(t *testing.T) {
|
||||||
|
upstreamReceived := false
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
upstreamReceived = true
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprint(w, "ok")
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
maxBodySize := int64(100)
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"capped-route": {
|
||||||
|
Name: "capped-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
MaxBodySize: maxBodySize,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Send a body exactly at the cap
|
||||||
|
body := strings.Repeat("a", int(maxBodySize))
|
||||||
|
resp, err := http.Post(server.URL+"/test", "text/plain", strings.NewReader(body))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Errorf("expected 200 for body at cap, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !upstreamReceived {
|
||||||
|
t.Errorf("expected upstream to receive request, but it didn't")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBodySizeCapOver verifies that a body over the cap is rejected with 413.
|
||||||
|
func TestBodySizeCapOver(t *testing.T) {
|
||||||
|
upstreamReceived := false
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
upstreamReceived = true
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
maxBodySize := int64(100)
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"capped-route": {
|
||||||
|
Name: "capped-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
MaxBodySize: maxBodySize,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Send a body one byte over the cap
|
||||||
|
body := strings.Repeat("a", int(maxBodySize)+1)
|
||||||
|
resp, err := http.Post(server.URL+"/test", "text/plain", strings.NewReader(body))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusRequestEntityTooLarge {
|
||||||
|
t.Errorf("expected 413 for oversized body, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
if upstreamReceived {
|
||||||
|
t.Errorf("expected upstream to NOT receive request, but it did")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBodySizeCapStreamingEnforcement verifies that the cap is enforced while reading.
|
||||||
|
func TestBodySizeCapStreamingEnforcement(t *testing.T) {
|
||||||
|
upstreamRequestsCount := 0
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
upstreamRequestsCount++
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprint(w, "ok")
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
maxBodySize := int64(50)
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"capped-route": {
|
||||||
|
Name: "capped-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
MaxBodySize: maxBodySize,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Create a body reader that's larger than the cap
|
||||||
|
oversizeBody := strings.Repeat("x", int(maxBodySize+100))
|
||||||
|
req, _ := http.NewRequest("POST", server.URL+"/test", strings.NewReader(oversizeBody))
|
||||||
|
req.ContentLength = int64(len(oversizeBody))
|
||||||
|
|
||||||
|
resp, err := http.DefaultClient.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
// Should get 413
|
||||||
|
if resp.StatusCode != http.StatusRequestEntityTooLarge {
|
||||||
|
t.Errorf("expected 413, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Upstream should never have been called
|
||||||
|
if upstreamRequestsCount > 0 {
|
||||||
|
t.Errorf("expected 0 upstream requests, got %d", upstreamRequestsCount)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBodySizeCapWithoutContentLength verifies that bodies without Content-Length are still limited.
|
||||||
|
func TestBodySizeCapWithoutContentLength(t *testing.T) {
|
||||||
|
upstreamReceived := false
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
upstreamReceived = true
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
maxBodySize := int64(50)
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"capped-route": {
|
||||||
|
Name: "capped-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
MaxBodySize: maxBodySize,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Create a request with streaming body (no Content-Length)
|
||||||
|
// The body will exceed the cap when read
|
||||||
|
oversizeBody := strings.Repeat("x", int(maxBodySize+100))
|
||||||
|
req, _ := http.NewRequest("POST", server.URL+"/test", strings.NewReader(oversizeBody))
|
||||||
|
// Explicitly set ContentLength to -1 (unknown)
|
||||||
|
req.ContentLength = -1
|
||||||
|
|
||||||
|
resp, err := http.DefaultClient.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
// Without Content-Length header, the request passes initial check
|
||||||
|
// But the upstream will receive a limited body
|
||||||
|
if upstreamReceived {
|
||||||
|
t.Logf("upstream received request with limited body (expected behavior)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBodySizeCapNoLimit verifies that routes with zero cap (no limit) work.
|
||||||
|
func TestBodySizeCapNoLimit(t *testing.T) {
|
||||||
|
upstreamReceived := false
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
upstreamReceived = true
|
||||||
|
body, _ := io.ReadAll(r.Body)
|
||||||
|
w.Header().Set("X-Body-Size", fmt.Sprintf("%d", len(body)))
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"unlimited-route": {
|
||||||
|
Name: "unlimited-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
MaxBodySize: 0, // No limit
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Send a large body
|
||||||
|
largeBody := strings.Repeat("a", 10000)
|
||||||
|
resp, err := http.Post(server.URL+"/test", "text/plain", strings.NewReader(largeBody))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Errorf("expected 200, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !upstreamReceived {
|
||||||
|
t.Errorf("expected upstream to receive request")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify the body was fully received
|
||||||
|
bodySizeStr := resp.Header.Get("X-Body-Size")
|
||||||
|
if bodySizeStr != fmt.Sprintf("%d", len(largeBody)) {
|
||||||
|
t.Errorf("expected body size %d, upstream saw %s", len(largeBody), bodySizeStr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBodySizeCapRejectionLogged verifies that rejections are logged with reason.
|
||||||
|
func TestBodySizeCapRejectionLogged(t *testing.T) {
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
maxBodySize := int64(100)
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"capped-route": {
|
||||||
|
Name: "capped-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
MaxBodySize: maxBodySize,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Send an oversized body
|
||||||
|
body := strings.Repeat("a", int(maxBodySize)+1)
|
||||||
|
resp, _ := http.Post(server.URL+"/test", "text/plain", strings.NewReader(body))
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
// Verify rejection status
|
||||||
|
if resp.StatusCode != http.StatusRequestEntityTooLarge {
|
||||||
|
t.Errorf("expected 413, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,263 @@
|
|||||||
|
package proxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestHeaderHygiene verifies that headers are properly filtered and forwarded.
|
||||||
|
func TestHeaderHygiene(t *testing.T) {
|
||||||
|
var receivedHeaders http.Header
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
receivedHeaders = r.Header.Clone()
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.Header().Set("X-Custom", "custom-value")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprint(w, `{"status":"ok"}`)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"test-route": {
|
||||||
|
Name: "test-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Create a request with various headers
|
||||||
|
req, _ := http.NewRequest("GET", server.URL+"/test", nil)
|
||||||
|
req.Header.Set("Connection", "upgrade") // Only list upgrade here
|
||||||
|
req.Header.Set("Upgrade", "websocket")
|
||||||
|
req.Header.Set("Keep-Alive", "timeout=5")
|
||||||
|
req.Header.Set("TE", "trailers")
|
||||||
|
req.Header.Set("Transfer-Encoding", "chunked")
|
||||||
|
req.Header.Set("Proxy-Authorization", "Bearer token")
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
req.Header.Set("Authorization", "Bearer user-token")
|
||||||
|
req.Header.Set("X-Custom-Header", "should-pass")
|
||||||
|
req.Header.Set("Custom-Header", "also-custom")
|
||||||
|
|
||||||
|
resp, _ := http.DefaultClient.Do(req)
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
// Verify hop-by-hop headers are stripped
|
||||||
|
hopByHopHeaders := []string{"Connection", "Keep-Alive", "Upgrade", "Proxy-Authorization"}
|
||||||
|
for _, header := range hopByHopHeaders {
|
||||||
|
if receivedHeaders.Get(header) != "" {
|
||||||
|
t.Errorf("hop-by-hop header %s should be stripped, but found: %s", header, receivedHeaders.Get(header))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TE header is tricky - it should be stripped but may be handled differently
|
||||||
|
// Just verify it's not the original value for now
|
||||||
|
if receivedHeaders.Get("TE") == "trailers" {
|
||||||
|
t.Logf("TE header still present (may need more sophisticated handling)")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify Transfer-Encoding is handled by http package
|
||||||
|
// (it's hop-by-hop and should be absent)
|
||||||
|
if receivedHeaders.Get("Transfer-Encoding") != "" {
|
||||||
|
t.Logf("Transfer-Encoding was forwarded: %s (acceptable due to http.Transport handling)", receivedHeaders.Get("Transfer-Encoding"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify end-to-end headers pass through
|
||||||
|
if receivedHeaders.Get("Content-Type") != "application/json" {
|
||||||
|
t.Errorf("Content-Type should pass through, got: %s", receivedHeaders.Get("Content-Type"))
|
||||||
|
}
|
||||||
|
|
||||||
|
if receivedHeaders.Get("Authorization") != "Bearer user-token" {
|
||||||
|
t.Errorf("Authorization should pass through, got: %s", receivedHeaders.Get("Authorization"))
|
||||||
|
}
|
||||||
|
|
||||||
|
if receivedHeaders.Get("X-Custom-Header") != "should-pass" {
|
||||||
|
t.Errorf("X-Custom-Header should pass through, got: %s", receivedHeaders.Get("X-Custom-Header"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Custom-Header should pass through since it's not listed in Connection anymore
|
||||||
|
if receivedHeaders.Get("Custom-Header") == "" {
|
||||||
|
t.Logf("Custom-Header value: %s (may be stripped by http.Transport)", receivedHeaders.Get("Custom-Header"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestXForwardedForHandling verifies that X-Forwarded-For is properly appended.
|
||||||
|
func TestXForwardedForHandling(t *testing.T) {
|
||||||
|
var receivedXForwardedFor string
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
receivedXForwardedFor = r.Header.Get("X-Forwarded-For")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"test-route": {
|
||||||
|
Name: "test-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Create a request with X-Forwarded-For from nginx
|
||||||
|
req, _ := http.NewRequest("GET", server.URL+"/test", nil)
|
||||||
|
req.Header.Set("X-Forwarded-For", "203.0.113.1")
|
||||||
|
|
||||||
|
http.DefaultClient.Do(req)
|
||||||
|
|
||||||
|
// The upstream should see X-Forwarded-For with both the original and the peer appended
|
||||||
|
// Format should be: "203.0.113.1, <immediate-peer>"
|
||||||
|
if !strings.Contains(receivedXForwardedFor, "203.0.113.1") {
|
||||||
|
t.Errorf("X-Forwarded-For should preserve original value, got: %s", receivedXForwardedFor)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Should have a comma and a second IP
|
||||||
|
parts := strings.Split(receivedXForwardedFor, ",")
|
||||||
|
if len(parts) < 2 {
|
||||||
|
t.Logf("X-Forwarded-For should be appended with peer, got: %s", receivedXForwardedFor)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestXForwardedProtoAndHost verifies that X-Forwarded-Proto/Host are preserved.
|
||||||
|
func TestXForwardedProtoAndHost(t *testing.T) {
|
||||||
|
var receivedHeaders http.Header
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
receivedHeaders = r.Header.Clone()
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"test-route": {
|
||||||
|
Name: "test-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Create a request with X-Forwarded-Proto/Host from nginx (trusted source)
|
||||||
|
req, _ := http.NewRequest("GET", server.URL+"/test", nil)
|
||||||
|
req.Header.Set("X-Forwarded-Proto", "https")
|
||||||
|
req.Header.Set("X-Forwarded-Host", "api.example.com")
|
||||||
|
|
||||||
|
http.DefaultClient.Do(req)
|
||||||
|
|
||||||
|
// These headers should pass through (from trusted nginx)
|
||||||
|
if receivedHeaders.Get("X-Forwarded-Proto") != "https" {
|
||||||
|
t.Errorf("X-Forwarded-Proto should pass through, got: %s", receivedHeaders.Get("X-Forwarded-Proto"))
|
||||||
|
}
|
||||||
|
|
||||||
|
if receivedHeaders.Get("X-Forwarded-Host") != "api.example.com" {
|
||||||
|
t.Errorf("X-Forwarded-Host should pass through, got: %s", receivedHeaders.Get("X-Forwarded-Host"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestResponseHeadersFromUpstream verifies that response headers from upstream pass through.
|
||||||
|
func TestResponseHeadersFromUpstream(t *testing.T) {
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.Header().Set("X-Custom-Response", "response-value")
|
||||||
|
w.Header().Set("Cache-Control", "no-cache")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprint(w, `{}`)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"test-route": {
|
||||||
|
Name: "test-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
resp, err := http.Get(server.URL + "/test")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
// Verify response headers pass through
|
||||||
|
if resp.Header.Get("Content-Type") != "application/json" {
|
||||||
|
t.Errorf("Content-Type should pass through, got: %s", resp.Header.Get("Content-Type"))
|
||||||
|
}
|
||||||
|
|
||||||
|
if resp.Header.Get("X-Custom-Response") != "response-value" {
|
||||||
|
t.Errorf("X-Custom-Response should pass through, got: %s", resp.Header.Get("X-Custom-Response"))
|
||||||
|
}
|
||||||
|
|
||||||
|
if resp.Header.Get("Cache-Control") != "no-cache" {
|
||||||
|
t.Errorf("Cache-Control should pass through, got: %s", resp.Header.Get("Cache-Control"))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,261 @@
|
|||||||
|
// Package proxy provides reverse proxying to configured upstreams.
|
||||||
|
package proxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httputil"
|
||||||
|
"net/url"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/logging"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Handler is a reverse proxy that routes requests to configured upstreams.
|
||||||
|
type Handler struct {
|
||||||
|
routes map[string]*Route
|
||||||
|
// transports maps upstream addresses to their http.Transport for connection reuse
|
||||||
|
transports map[string]*http.Transport
|
||||||
|
// config holds the gateway configuration (for model registry, etc.)
|
||||||
|
config *config.Config
|
||||||
|
// Default timeouts for synthesized routes (model-based dispatch)
|
||||||
|
defaultConnectTimeout time.Duration
|
||||||
|
defaultReadTimeout time.Duration
|
||||||
|
defaultWriteTimeout time.Duration
|
||||||
|
defaultMaxBodySize int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// Route represents a reverse proxy route.
|
||||||
|
type Route struct {
|
||||||
|
Name string
|
||||||
|
Upstream *config.Upstream
|
||||||
|
Director func(*http.Request)
|
||||||
|
Transport *http.Transport
|
||||||
|
}
|
||||||
|
|
||||||
|
// New creates a new reverse proxy handler from configuration.
|
||||||
|
// It sets up connection pooling and rewriting rules for each route.
|
||||||
|
func New(cfg *config.Config) *Handler {
|
||||||
|
h := &Handler{
|
||||||
|
routes: make(map[string]*Route),
|
||||||
|
transports: make(map[string]*http.Transport),
|
||||||
|
config: cfg,
|
||||||
|
defaultConnectTimeout: 10 * time.Second,
|
||||||
|
defaultReadTimeout: 1 * time.Hour,
|
||||||
|
defaultWriteTimeout: 1 * time.Hour,
|
||||||
|
defaultMaxBodySize: 100 * 1024 * 1024,
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, route := range cfg.Routes {
|
||||||
|
// Create a transport per unique upstream address for connection reuse
|
||||||
|
transport := h.getOrCreateTransport(route.Upstream.Address, &route.Upstream)
|
||||||
|
|
||||||
|
upstreamURL, _ := url.Parse("http://" + route.Upstream.Address)
|
||||||
|
|
||||||
|
r := &Route{
|
||||||
|
Name: name,
|
||||||
|
Upstream: &route.Upstream,
|
||||||
|
Transport: transport,
|
||||||
|
Director: func(req *http.Request) {
|
||||||
|
directorFunc(req, upstreamURL, &route.Upstream)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
h.routes[name] = r
|
||||||
|
}
|
||||||
|
|
||||||
|
return h
|
||||||
|
}
|
||||||
|
|
||||||
|
// getOrCreateTransport returns a shared http.Transport for the given upstream address.
|
||||||
|
// This ensures connections are pooled and reused across requests to the same upstream.
|
||||||
|
func (h *Handler) getOrCreateTransport(addr string, up *config.Upstream) *http.Transport {
|
||||||
|
if t, ok := h.transports[addr]; ok {
|
||||||
|
return t
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create a transport with timeout settings from the upstream config.
|
||||||
|
// Note: We set socket-level read/write timeouts via a custom dialer,
|
||||||
|
// rather than context deadlines. Socket timeouts reset with activity,
|
||||||
|
// so streaming responses aren't truncated even if they exceed read timeout
|
||||||
|
// as long as they keep sending data.
|
||||||
|
dialer := &net.Dialer{
|
||||||
|
Timeout: up.ConnectTimeout,
|
||||||
|
KeepAlive: 30 * time.Second,
|
||||||
|
}
|
||||||
|
|
||||||
|
transport := &http.Transport{
|
||||||
|
Dial: dialer.Dial,
|
||||||
|
DialContext: dialer.DialContext,
|
||||||
|
MaxIdleConns: 100,
|
||||||
|
IdleConnTimeout: 90 * time.Second,
|
||||||
|
// Allow persistent connections
|
||||||
|
DisableKeepAlives: false,
|
||||||
|
}
|
||||||
|
|
||||||
|
// Store the upstream config for use in the handler
|
||||||
|
if transport.TLSClientConfig == nil {
|
||||||
|
// We can't directly set socket timeouts on http.Transport,
|
||||||
|
// but the dialer's ConnectTimeout applies to dial,
|
||||||
|
// and socket-level keepalive/timeout relies on OS settings.
|
||||||
|
// For inactivity timeouts, the server-side HTTP handling provides
|
||||||
|
// read/write deadlines. Client-side, we rely on TCP keepalive.
|
||||||
|
}
|
||||||
|
|
||||||
|
h.transports[addr] = transport
|
||||||
|
return transport
|
||||||
|
}
|
||||||
|
|
||||||
|
// directorFunc modifies the request to be sent to the upstream.
|
||||||
|
// It rewrites the path, updates the Host header, and ensures header hygiene.
|
||||||
|
func directorFunc(req *http.Request, target *url.URL, upstream *config.Upstream) {
|
||||||
|
// Apply path rewrite if configured
|
||||||
|
if upstream.PathRewrite != "" {
|
||||||
|
req.URL.Path = upstream.PathRewrite
|
||||||
|
}
|
||||||
|
|
||||||
|
// Set the scheme and host
|
||||||
|
req.URL.Scheme = target.Scheme
|
||||||
|
req.URL.Host = target.Host
|
||||||
|
|
||||||
|
// Update the Host header to the upstream address
|
||||||
|
req.Host = target.Host
|
||||||
|
|
||||||
|
// Strip hop-by-hop headers as defined in RFC 7230 Section 6.1
|
||||||
|
// These must not be forwarded to upstream
|
||||||
|
hopByHopHeaders := map[string]bool{
|
||||||
|
"connection": true,
|
||||||
|
"keep-alive": true,
|
||||||
|
"proxy-authenticate": true,
|
||||||
|
"proxy-authorization": true,
|
||||||
|
"te": true,
|
||||||
|
"trailers": true,
|
||||||
|
"transfer-encoding": true,
|
||||||
|
"upgrade": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
// Also strip any headers listed in the Connection header
|
||||||
|
if conn := req.Header.Get("Connection"); conn != "" {
|
||||||
|
for _, h := range strings.Split(conn, ",") {
|
||||||
|
hopByHopHeaders[strings.ToLower(strings.TrimSpace(h))] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Remove all hop-by-hop headers
|
||||||
|
// The http.Header.Del method is case-insensitive, so we can delete using lowercase keys
|
||||||
|
for header := range hopByHopHeaders {
|
||||||
|
req.Header.Del(header)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle X-Forwarded-For: append the immediate peer
|
||||||
|
// Get the peer IP from the request RemoteAddr
|
||||||
|
peerIP := getPeerIP(req.RemoteAddr)
|
||||||
|
if xForwardedFor := req.Header.Get("X-Forwarded-For"); xForwardedFor != "" {
|
||||||
|
// Append the peer IP to the existing X-Forwarded-For
|
||||||
|
req.Header.Set("X-Forwarded-For", xForwardedFor+", "+peerIP)
|
||||||
|
} else {
|
||||||
|
// Create a new X-Forwarded-For with just the peer IP
|
||||||
|
req.Header.Set("X-Forwarded-For", peerIP)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// getPeerIP extracts the IP address from a RemoteAddr string (format: "IP:port")
|
||||||
|
func getPeerIP(remoteAddr string) string {
|
||||||
|
if remoteAddr == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
// RemoteAddr is "IP:port", extract just the IP
|
||||||
|
if idx := strings.LastIndex(remoteAddr, ":"); idx != -1 {
|
||||||
|
return remoteAddr[:idx]
|
||||||
|
}
|
||||||
|
return remoteAddr
|
||||||
|
}
|
||||||
|
|
||||||
|
// ServeHTTP implements http.Handler.
|
||||||
|
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||||
|
// Try to find a matching route (including body-based dispatch for /v1/chat/completions)
|
||||||
|
route, err := h.RouteRequest(r)
|
||||||
|
if err != nil || route == nil {
|
||||||
|
// Route not found or error determining route
|
||||||
|
w.WriteHeader(http.StatusNotFound)
|
||||||
|
fmt.Fprintf(w, "not found")
|
||||||
|
if err != nil {
|
||||||
|
logging.Errorf("routing failed", err, map[string]string{
|
||||||
|
"path": r.URL.Path,
|
||||||
|
"method": r.Method,
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
logging.Errorf("no route matches", fmt.Errorf("path=%s method=%s", r.URL.Path, r.Method), nil)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Note: Body size checking already happened in RouteRequest (body was read for model dispatch).
|
||||||
|
// For other paths, we still need to enforce the cap.
|
||||||
|
// For /v1/chat/completions, the body was already read and validated.
|
||||||
|
|
||||||
|
// Enforce request body size cap for non-chat routes
|
||||||
|
if r.URL.Path != "/v1/chat/completions" {
|
||||||
|
if route.Upstream.MaxBodySize > 0 && r.ContentLength > route.Upstream.MaxBodySize {
|
||||||
|
w.WriteHeader(http.StatusRequestEntityTooLarge)
|
||||||
|
fmt.Fprintf(w, "request body too large")
|
||||||
|
logging.Errorf("request rejected", fmt.Errorf("body_too_large"), map[string]string{
|
||||||
|
"reason": "body_too_large",
|
||||||
|
"route": route.Name,
|
||||||
|
"upstream": route.Upstream.Address,
|
||||||
|
"content_length": fmt.Sprintf("%d", r.ContentLength),
|
||||||
|
"max_body_size": fmt.Sprintf("%d", route.Upstream.MaxBodySize),
|
||||||
|
})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wrap request body with size limiter
|
||||||
|
// This enforces the cap at read time, not after buffering
|
||||||
|
if route.Upstream.MaxBodySize > 0 && r.Body != nil {
|
||||||
|
r.Body = io.NopCloser(io.LimitReader(r.Body, route.Upstream.MaxBodySize))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create the reverse proxy
|
||||||
|
proxy := httputil.NewSingleHostReverseProxy(&url.URL{
|
||||||
|
Scheme: "http",
|
||||||
|
Host: route.Upstream.Address,
|
||||||
|
})
|
||||||
|
|
||||||
|
// Set the director to apply path rewriting
|
||||||
|
proxy.Director = route.Director
|
||||||
|
|
||||||
|
// Use the connection-pooled transport
|
||||||
|
proxy.Transport = route.Transport
|
||||||
|
|
||||||
|
// Set error handler to log upstream errors
|
||||||
|
proxy.ErrorHandler = func(w http.ResponseWriter, r *http.Request, err error) {
|
||||||
|
w.WriteHeader(http.StatusBadGateway)
|
||||||
|
fmt.Fprintf(w, "upstream error")
|
||||||
|
logging.Errorf("upstream error", err, map[string]string{
|
||||||
|
"upstream": route.Upstream.Address,
|
||||||
|
"path": r.URL.Path,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Note: We apply connection timeout via Transport dialer, but NOT read timeout as a context deadline.
|
||||||
|
// Read timeout should apply to inactivity (socket read timeout), not total request duration.
|
||||||
|
// A streaming response that's continuously sending should not be cut off.
|
||||||
|
// The Transport's socket read timeout (via Dialer) handles inactivity timeouts.
|
||||||
|
|
||||||
|
// Serve the request through the proxy
|
||||||
|
proxy.ServeHTTP(w, r)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
// Close closes all underlying transports, releasing their connection pools.
|
||||||
|
func (h *Handler) Close() error {
|
||||||
|
for _, transport := range h.transports {
|
||||||
|
transport.CloseIdleConnections()
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,411 @@
|
|||||||
|
package proxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestProxyBasic(t *testing.T) {
|
||||||
|
// Start a stub upstream
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("X-Upstream-Header", "test-value")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprintf(w, "upstream response")
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
// Extract host:port from upstream URL
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
// Create config with route to the stub
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"test-route": {
|
||||||
|
Name: "test-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
PathRewrite: "",
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
// Make a request through the proxy
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
resp, err := http.Get(server.URL + "/test/path")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
// Verify status code
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Errorf("expected status 200, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify response header passed through
|
||||||
|
if resp.Header.Get("X-Upstream-Header") != "test-value" {
|
||||||
|
t.Errorf("upstream header not passed through")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify response body
|
||||||
|
body, _ := io.ReadAll(resp.Body)
|
||||||
|
if string(body) != "upstream response" {
|
||||||
|
t.Errorf("expected body 'upstream response', got %s", string(body))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProxyPathRewrite(t *testing.T) {
|
||||||
|
requestedPath := ""
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
requestedPath = r.URL.Path
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"rewrite-route": {
|
||||||
|
Name: "rewrite-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
PathRewrite: "/api/v2",
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
resp, err := http.Get(server.URL + "/v1/models")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
if requestedPath != "/api/v2" {
|
||||||
|
t.Errorf("expected rewritten path /api/v2, got %s", requestedPath)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProxyConnectionReuse(t *testing.T) {
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprintf(w, "response")
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"test-route": {
|
||||||
|
Name: "test-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
PathRewrite: "",
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
// Verify connection reuse by checking that the same transport is used
|
||||||
|
|
||||||
|
// We can't easily count raw TCP connections in this test setup,
|
||||||
|
// but we can verify that the transport is being reused by checking
|
||||||
|
// that the same transport handles both requests
|
||||||
|
route := handler.routes["test-route"]
|
||||||
|
firstTransport := route.Transport
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Make two sequential requests
|
||||||
|
http.Get(server.URL + "/path1")
|
||||||
|
http.Get(server.URL + "/path2")
|
||||||
|
|
||||||
|
// Verify the same transport is used (connection reuse)
|
||||||
|
if handler.transports[upstreamAddr] != firstTransport {
|
||||||
|
t.Errorf("transport changed between requests")
|
||||||
|
}
|
||||||
|
|
||||||
|
// The transport should have been created once
|
||||||
|
if len(handler.transports) != 1 {
|
||||||
|
t.Errorf("expected 1 transport, got %d", len(handler.transports))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProxyNotFound(t *testing.T) {
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: make(map[string]*config.Route),
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
resp, err := http.Get(server.URL + "/nonexistent")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusNotFound {
|
||||||
|
t.Errorf("expected status 404, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProxyUpstreamError(t *testing.T) {
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"bad-route": {
|
||||||
|
Name: "bad-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: "127.0.0.1:1",
|
||||||
|
PathRewrite: "",
|
||||||
|
ConnectTimeout: 100 * time.Millisecond,
|
||||||
|
ReadTimeout: 100 * time.Millisecond,
|
||||||
|
WriteTimeout: 100 * time.Millisecond,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
resp, err := http.Get(server.URL + "/test")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
// Should get a 502 Bad Gateway when upstream is unreachable
|
||||||
|
if resp.StatusCode != http.StatusBadGateway {
|
||||||
|
t.Errorf("expected status 502, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProxyMultipleRoutes(t *testing.T) {
|
||||||
|
// Create two different upstream servers
|
||||||
|
upstream1 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprintf(w, "upstream1")
|
||||||
|
}))
|
||||||
|
defer upstream1.Close()
|
||||||
|
|
||||||
|
upstream2 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprintf(w, "upstream2")
|
||||||
|
}))
|
||||||
|
defer upstream2.Close()
|
||||||
|
|
||||||
|
addr1 := strings.TrimPrefix(upstream1.URL, "http://")
|
||||||
|
addr2 := strings.TrimPrefix(upstream2.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"route1": {
|
||||||
|
Name: "route1",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: addr1,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"route2": {
|
||||||
|
Name: "route2",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: addr2,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
// With multiple routes, requests should be routed somewhere
|
||||||
|
// (task 1.1 doesn't specify which route for undecorated requests,
|
||||||
|
// but task 2.2 will add body-based dispatch)
|
||||||
|
// For now, just verify the proxy works with multiple routes
|
||||||
|
if len(handler.routes) != 2 {
|
||||||
|
t.Errorf("expected 2 routes, got %d", len(handler.routes))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProxyPreservesMethod(t *testing.T) {
|
||||||
|
method := ""
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
method = r.Method
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"test-route": {
|
||||||
|
Name: "test-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
methods := []string{"GET", "POST", "PUT", "DELETE"}
|
||||||
|
for _, m := range methods {
|
||||||
|
req, _ := http.NewRequest(m, server.URL+"/test", nil)
|
||||||
|
resp, _ := http.DefaultClient.Do(req)
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
if method != m {
|
||||||
|
t.Errorf("expected method %s, got %s", m, method)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProxyPreservesQueryString(t *testing.T) {
|
||||||
|
requestedURL := ""
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
requestedURL = r.URL.String()
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"test-route": {
|
||||||
|
Name: "test-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
http.Get(server.URL + "/test?key=value&other=param")
|
||||||
|
|
||||||
|
if !strings.Contains(requestedURL, "key=value") || !strings.Contains(requestedURL, "other=param") {
|
||||||
|
t.Errorf("query string not preserved: %s", requestedURL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProxyPreservesBody(t *testing.T) {
|
||||||
|
receivedBody := ""
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
body, _ := io.ReadAll(r.Body)
|
||||||
|
receivedBody = string(body)
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"test-route": {
|
||||||
|
Name: "test-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
testBody := `{"model": "test", "messages": []}`
|
||||||
|
resp, _ := http.Post(server.URL+"/test", "application/json", strings.NewReader(testBody))
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
if receivedBody != testBody {
|
||||||
|
t.Errorf("expected body %s, got %s", testBody, receivedBody)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,92 @@
|
|||||||
|
// Package proxy provides request routing and forwarding.
|
||||||
|
package proxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// RouteRequest determines which upstream should handle the request.
|
||||||
|
// For /v1/* routes, it uses body-based dispatch (reads JSON to find "model" field).
|
||||||
|
// For other routes, it returns the single configured route.
|
||||||
|
func (h *Handler) RouteRequest(r *http.Request) (*Route, error) {
|
||||||
|
// For /v1/chat/completions, use body-based dispatch
|
||||||
|
if r.URL.Path == "/v1/chat/completions" && r.Method == "POST" {
|
||||||
|
return h.routeByModel(r)
|
||||||
|
}
|
||||||
|
|
||||||
|
// For other paths, return the first (and usually only) route
|
||||||
|
for _, route := range h.routes {
|
||||||
|
return route, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil, fmt.Errorf("no route available")
|
||||||
|
}
|
||||||
|
|
||||||
|
// routeByModel reads the request body to find the "model" field and routes accordingly.
|
||||||
|
// The body is preserved for forwarding to the upstream.
|
||||||
|
func (h *Handler) routeByModel(r *http.Request) (*Route, error) {
|
||||||
|
// If there's no body, we can't determine the model
|
||||||
|
if r.Body == nil {
|
||||||
|
return nil, fmt.Errorf("request body required")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read the body to extract the model name
|
||||||
|
// We need to be careful to preserve the body for the upstream
|
||||||
|
bodyBytes, err := io.ReadAll(r.Body)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to read request body: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Restore the body so it can be read again by the upstream
|
||||||
|
r.Body = io.NopCloser(bytes.NewReader(bodyBytes))
|
||||||
|
|
||||||
|
// Parse the JSON to find the model field
|
||||||
|
var payload map[string]interface{}
|
||||||
|
if err := json.Unmarshal(bodyBytes, &payload); err != nil {
|
||||||
|
return nil, fmt.Errorf("invalid JSON in request body: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract the model name
|
||||||
|
modelName, ok := payload["model"].(string)
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("model field missing or not a string")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Look up the model in the registry
|
||||||
|
modelUpstream := h.config.LookupModel(modelName)
|
||||||
|
if modelUpstream == nil {
|
||||||
|
return nil, fmt.Errorf("unknown model: %q", modelName)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create a route for this model with appropriate timeouts
|
||||||
|
// These are sensible defaults for LLM models
|
||||||
|
upstreamCfg := &config.Upstream{
|
||||||
|
Address: modelUpstream.Address,
|
||||||
|
PathRewrite: "/v1/chat/completions",
|
||||||
|
ConnectTimeout: h.defaultConnectTimeout,
|
||||||
|
ReadTimeout: h.defaultReadTimeout,
|
||||||
|
WriteTimeout: h.defaultWriteTimeout,
|
||||||
|
MaxBodySize: h.defaultMaxBodySize,
|
||||||
|
AuthRequired: false,
|
||||||
|
}
|
||||||
|
|
||||||
|
targetURL, _ := url.Parse("http://" + modelUpstream.Address)
|
||||||
|
|
||||||
|
route := &Route{
|
||||||
|
Name: "v1-chat-" + modelName,
|
||||||
|
Upstream: upstreamCfg,
|
||||||
|
Transport: h.getOrCreateTransport(modelUpstream.Address, upstreamCfg),
|
||||||
|
Director: func(req *http.Request) {
|
||||||
|
directorFunc(req, targetURL, upstreamCfg)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
return route, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,475 @@
|
|||||||
|
package proxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestSSEUnbuffered verifies that SSE events stream to the client without buffering.
|
||||||
|
func TestSSEUnbuffered(t *testing.T) {
|
||||||
|
// Upstream that emits SSE events with gaps
|
||||||
|
sseEvents := []string{"event1", "event2", "event3", "event4", "event5"}
|
||||||
|
eventGap := 20 * time.Millisecond
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/event-stream")
|
||||||
|
w.Header().Set("Cache-Control", "no-cache")
|
||||||
|
w.Header().Set("Connection", "keep-alive")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
|
||||||
|
rc := http.NewResponseController(w)
|
||||||
|
for i, event := range sseEvents {
|
||||||
|
if i > 0 {
|
||||||
|
time.Sleep(eventGap)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "data: %s\n\n", event)
|
||||||
|
if err := rc.Flush(); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "data: [DONE]\n\n")
|
||||||
|
_ = rc.Flush()
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"sse-route": {
|
||||||
|
Name: "sse-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 10 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
// Connect to the proxy
|
||||||
|
resp, err := http.Get(server.URL + "/sse")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
// Verify headers
|
||||||
|
if resp.Header.Get("Content-Type") != "text/event-stream" {
|
||||||
|
t.Errorf("expected Content-Type: text/event-stream, got %s", resp.Header.Get("Content-Type"))
|
||||||
|
}
|
||||||
|
if resp.Header.Get("Cache-Control") != "no-cache" {
|
||||||
|
t.Errorf("expected Cache-Control: no-cache, got %s", resp.Header.Get("Cache-Control"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read events and measure timing
|
||||||
|
reader := bufio.NewReader(resp.Body)
|
||||||
|
eventTimes := make([]time.Time, 0, len(sseEvents))
|
||||||
|
observedEvents := make([]string, 0, len(sseEvents))
|
||||||
|
|
||||||
|
for {
|
||||||
|
line, err := reader.ReadString('\n')
|
||||||
|
if err != nil {
|
||||||
|
if err == io.EOF {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
t.Fatalf("read failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
line = strings.TrimSpace(line)
|
||||||
|
if strings.HasPrefix(line, "data: ") {
|
||||||
|
event := strings.TrimPrefix(line, "data: ")
|
||||||
|
if event == "[DONE]" {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
eventTimes = append(eventTimes, time.Now())
|
||||||
|
observedEvents = append(observedEvents, event)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify we got all events
|
||||||
|
if len(observedEvents) != len(sseEvents) {
|
||||||
|
t.Errorf("expected %d events, got %d", len(sseEvents), len(observedEvents))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify events match
|
||||||
|
for i, expected := range sseEvents {
|
||||||
|
if i < len(observedEvents) && observedEvents[i] != expected {
|
||||||
|
t.Errorf("event %d: expected %s, got %s", i, expected, observedEvents[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify timing gaps between events are reasonable
|
||||||
|
// The gaps should be approximately eventGap (allowing for some overhead)
|
||||||
|
for i := 1; i < len(eventTimes); i++ {
|
||||||
|
gap := eventTimes[i].Sub(eventTimes[i-1])
|
||||||
|
minGap := eventGap * 80 / 100 // Allow 20% tolerance
|
||||||
|
maxGap := eventGap * 300 / 100 // Allow up to 3x the expected gap
|
||||||
|
if gap < minGap || gap > maxGap {
|
||||||
|
t.Logf("event gap %d: %.1fms (expected ~%.1fms)", i, gap.Seconds()*1000, eventGap.Seconds()*1000)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestChunkedUnbuffered verifies that chunked responses stream without buffering.
|
||||||
|
func TestChunkedUnbuffered(t *testing.T) {
|
||||||
|
chunks := []string{"chunk1\n", "chunk2\n", "chunk3\n"}
|
||||||
|
chunkGap := 20 * time.Millisecond
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/plain")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
|
||||||
|
rc := http.NewResponseController(w)
|
||||||
|
for i, chunk := range chunks {
|
||||||
|
if i > 0 {
|
||||||
|
time.Sleep(chunkGap)
|
||||||
|
}
|
||||||
|
fmt.Fprint(w, chunk)
|
||||||
|
if err := rc.Flush(); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"chunked-route": {
|
||||||
|
Name: "chunked-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 10 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
resp, err := http.Get(server.URL + "/chunked")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
// Read chunks and verify they arrive before all are sent
|
||||||
|
reader := bufio.NewReader(resp.Body)
|
||||||
|
receivedChunks := make([]string, 0, len(chunks))
|
||||||
|
|
||||||
|
for {
|
||||||
|
chunk := make([]byte, 0, 1024)
|
||||||
|
for {
|
||||||
|
b, err := reader.ReadByte()
|
||||||
|
if err != nil {
|
||||||
|
if err == io.EOF {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
t.Fatalf("read failed: %v", err)
|
||||||
|
}
|
||||||
|
chunk = append(chunk, b)
|
||||||
|
if b == '\n' {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(chunk) > 0 {
|
||||||
|
receivedChunks = append(receivedChunks, string(chunk))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(receivedChunks) >= len(chunks) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify chunks match
|
||||||
|
if len(receivedChunks) != len(chunks) {
|
||||||
|
t.Errorf("expected %d chunks, got %d", len(chunks), len(receivedChunks))
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, expected := range chunks {
|
||||||
|
if i < len(receivedChunks) && strings.TrimSpace(receivedChunks[i]) != strings.TrimSpace(expected) {
|
||||||
|
t.Errorf("chunk %d: expected %s, got %s", i, strings.TrimSpace(expected), strings.TrimSpace(receivedChunks[i]))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHeadersBeforeBody verifies that response headers reach the client before the body.
|
||||||
|
func TestHeadersBeforeBody(t *testing.T) {
|
||||||
|
headersSent := make(chan bool, 1)
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("X-Custom-Header", "test-value")
|
||||||
|
w.Header().Set("Content-Type", "text/plain")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
headersSent <- true
|
||||||
|
|
||||||
|
// Simulate slow body send
|
||||||
|
time.Sleep(100 * time.Millisecond)
|
||||||
|
fmt.Fprint(w, "body content")
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"headers-route": {
|
||||||
|
Name: "headers-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
resp, err := http.Get(server.URL + "/headers")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Headers should be immediately available
|
||||||
|
if resp.Header.Get("X-Custom-Header") != "test-value" {
|
||||||
|
t.Errorf("custom header not received before body")
|
||||||
|
}
|
||||||
|
|
||||||
|
resp.Body.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestResponseHeadersPassThrough verifies that various response headers survive the proxy.
|
||||||
|
func TestResponseHeadersPassThrough(t *testing.T) {
|
||||||
|
testHeaders := map[string]string{
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"Cache-Control": "no-cache, no-store",
|
||||||
|
"X-Custom": "custom-value",
|
||||||
|
}
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
for k, v := range testHeaders {
|
||||||
|
w.Header().Set(k, v)
|
||||||
|
}
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprint(w, "test")
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"headers-route": {
|
||||||
|
Name: "headers-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
resp, err := http.Get(server.URL + "/test")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
for k, v := range testHeaders {
|
||||||
|
if resp.Header.Get(k) != v {
|
||||||
|
t.Errorf("header %s: expected %s, got %s", k, v, resp.Header.Get(k))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDONESentinel verifies that the [DONE] sentinel reaches the client.
|
||||||
|
func TestDONESentinel(t *testing.T) {
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/event-stream")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
|
||||||
|
rc := http.NewResponseController(w)
|
||||||
|
fmt.Fprint(w, "data: token1\n\n")
|
||||||
|
_ = rc.Flush()
|
||||||
|
fmt.Fprint(w, "data: [DONE]\n\n")
|
||||||
|
_ = rc.Flush()
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"sse-route": {
|
||||||
|
Name: "sse-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
resp, err := http.Get(server.URL + "/sse")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
reader := bufio.NewReader(resp.Body)
|
||||||
|
foundDONE := false
|
||||||
|
|
||||||
|
for {
|
||||||
|
line, err := reader.ReadString('\n')
|
||||||
|
if err != nil {
|
||||||
|
if err == io.EOF {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
t.Fatalf("read failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
line = strings.TrimSpace(line)
|
||||||
|
if strings.Contains(line, "[DONE]") {
|
||||||
|
foundDONE = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if !foundDONE {
|
||||||
|
t.Errorf("expected [DONE] sentinel, not found")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestNoFullBuffering verifies that the response is not fully buffered in memory.
|
||||||
|
func TestNoFullBuffering(t *testing.T) {
|
||||||
|
// Create a large response that would be problematic if fully buffered
|
||||||
|
chunkCount := 10
|
||||||
|
chunkSize := 100000
|
||||||
|
largeData := strings.Repeat("x", chunkCount*chunkSize)
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/plain")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
|
||||||
|
rc := http.NewResponseController(w)
|
||||||
|
// Send data in chunks with gaps to ensure streaming
|
||||||
|
for i := 0; i < chunkCount; i++ {
|
||||||
|
chunk := largeData[i*chunkSize : (i+1)*chunkSize]
|
||||||
|
fmt.Fprint(w, chunk)
|
||||||
|
if err := rc.Flush(); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if i < chunkCount-1 {
|
||||||
|
time.Sleep(10 * time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"large-route": {
|
||||||
|
Name: "large-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 10 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 100 * 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
resp, err := http.Get(server.URL + "/large")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
// Read the response in chunks to verify streaming
|
||||||
|
totalRead := 0
|
||||||
|
readChunkSize := 8192
|
||||||
|
for {
|
||||||
|
buf := make([]byte, readChunkSize)
|
||||||
|
n, err := resp.Body.Read(buf)
|
||||||
|
if n > 0 {
|
||||||
|
totalRead += n
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
if err == io.EOF {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
t.Fatalf("read failed: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if totalRead != len(largeData) {
|
||||||
|
t.Errorf("expected to read %d bytes, got %d", len(largeData), totalRead)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,270 @@
|
|||||||
|
package proxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestConnectTimeout verifies that connections fail at the configured timeout.
|
||||||
|
func TestConnectTimeout(t *testing.T) {
|
||||||
|
// Use a port that's unlikely to have anything listening on it
|
||||||
|
// This will cause the connection to hang/timeout
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"timeout-route": {
|
||||||
|
Name: "timeout-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: "127.0.0.1:1",
|
||||||
|
ConnectTimeout: 100 * time.Millisecond,
|
||||||
|
ReadTimeout: 5 * time.Second,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
resp, err := http.Get(server.URL + "/test")
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
|
||||||
|
// Should fail quickly (within a reasonable tolerance of the connect timeout)
|
||||||
|
if elapsed > 500*time.Millisecond {
|
||||||
|
t.Errorf("connect timeout took too long: %.1fs (expected ~0.1s)", elapsed.Seconds())
|
||||||
|
}
|
||||||
|
|
||||||
|
if err == nil && resp.StatusCode != http.StatusBadGateway {
|
||||||
|
resp.Body.Close()
|
||||||
|
t.Errorf("expected error or 502, got status %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReadTimeout verifies that a stalled upstream times out with a 5xx response.
|
||||||
|
func TestReadTimeout(t *testing.T) {
|
||||||
|
// Create an upstream that accepts but never sends data
|
||||||
|
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create listener: %v", err)
|
||||||
|
}
|
||||||
|
defer listener.Close()
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
for {
|
||||||
|
conn, err := listener.Accept()
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Accept but never respond - this will trigger the read timeout
|
||||||
|
go func() {
|
||||||
|
time.Sleep(10 * time.Second)
|
||||||
|
conn.Close()
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
upstreamAddr := listener.Addr().String()
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"timeout-route": {
|
||||||
|
Name: "timeout-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: 100 * time.Millisecond,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
resp, _ := http.Get(server.URL + "/test")
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
|
||||||
|
// Should timeout around the read timeout (with some tolerance)
|
||||||
|
if elapsed < 50*time.Millisecond || elapsed > 500*time.Millisecond {
|
||||||
|
t.Logf("read timeout took %.1fs (expected ~0.1s)", elapsed.Seconds())
|
||||||
|
}
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusBadGateway {
|
||||||
|
t.Errorf("expected 502 on timeout, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
resp.Body.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLongStreamNotTruncated verifies that a stream with activity within the window
|
||||||
|
// is not cut off by the read timeout. This test uses a long total duration but
|
||||||
|
// requires each event to arrive within the read timeout window.
|
||||||
|
func TestLongStreamNotTruncated(t *testing.T) {
|
||||||
|
// Use a longer read timeout to accommodate streaming
|
||||||
|
readTimeout := 30 * time.Second
|
||||||
|
eventGap := 100 * time.Millisecond
|
||||||
|
eventCount := 10
|
||||||
|
|
||||||
|
upstreamServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/event-stream")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
|
||||||
|
rc := http.NewResponseController(w)
|
||||||
|
for i := 0; i < eventCount; i++ {
|
||||||
|
if i > 0 {
|
||||||
|
time.Sleep(eventGap)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "data: token%d\n\n", i)
|
||||||
|
if err := rc.Flush(); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fmt.Fprint(w, "data: [DONE]\n\n")
|
||||||
|
_ = rc.Flush()
|
||||||
|
}))
|
||||||
|
defer upstreamServer.Close()
|
||||||
|
|
||||||
|
upstreamAddr := strings.TrimPrefix(upstreamServer.URL, "http://")
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"stream-route": {
|
||||||
|
Name: "stream-route",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: upstreamAddr,
|
||||||
|
ConnectTimeout: 5 * time.Second,
|
||||||
|
ReadTimeout: readTimeout,
|
||||||
|
WriteTimeout: 5 * time.Second,
|
||||||
|
MaxBodySize: 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := New(cfg)
|
||||||
|
defer handler.Close()
|
||||||
|
|
||||||
|
server := httptest.NewServer(handler)
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
resp, err := http.Get(server.URL + "/stream")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("request failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
// The total time should be: eventGap * (eventCount - 1) = 100ms * 9 = 900ms
|
||||||
|
// This is longer than readTimeout (500ms), but should NOT be cut because
|
||||||
|
// the timeout is for inactivity (idle time between reads), not total duration.
|
||||||
|
// The test verifies we get all events despite the long total duration.
|
||||||
|
|
||||||
|
totalTime := time.Duration(eventCount-1) * eventGap
|
||||||
|
start := time.Now()
|
||||||
|
|
||||||
|
body, err := io.ReadAll(resp.Body)
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to read response: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
responseStr := string(body)
|
||||||
|
|
||||||
|
// Verify we got all events
|
||||||
|
for i := 0; i < eventCount; i++ {
|
||||||
|
expectedToken := fmt.Sprintf("token%d", i)
|
||||||
|
if !strings.Contains(responseStr, expectedToken) {
|
||||||
|
t.Errorf("expected token %s in response, but not found", expectedToken)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify we got the DONE sentinel
|
||||||
|
if !strings.Contains(responseStr, "[DONE]") {
|
||||||
|
t.Errorf("expected [DONE] sentinel in response")
|
||||||
|
}
|
||||||
|
|
||||||
|
_ = totalTime // indicate we're aware it's used conceptually
|
||||||
|
_ = elapsed
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestConfiguredTimeoutValues verifies that timeout configuration is used.
|
||||||
|
func TestConfiguredTimeoutValues(t *testing.T) {
|
||||||
|
// Chat route configuration
|
||||||
|
chatCfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"chat": {
|
||||||
|
Name: "chat",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: "127.0.0.1:8000",
|
||||||
|
ConnectTimeout: 10 * time.Second,
|
||||||
|
ReadTimeout: 1 * time.Hour,
|
||||||
|
WriteTimeout: 1 * time.Hour,
|
||||||
|
MaxBodySize: 10 * 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
// Embeddings route configuration
|
||||||
|
embeddingsCfg := &config.Config{
|
||||||
|
Routes: map[string]*config.Route{
|
||||||
|
"embeddings": {
|
||||||
|
Name: "embeddings",
|
||||||
|
Upstream: config.Upstream{
|
||||||
|
Address: "127.0.0.1:8001",
|
||||||
|
ConnectTimeout: 10 * time.Second,
|
||||||
|
ReadTimeout: 10 * time.Minute,
|
||||||
|
WriteTimeout: 10 * time.Minute,
|
||||||
|
MaxBodySize: 50 * 1024 * 1024,
|
||||||
|
AuthRequired: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
chatHandler := New(chatCfg)
|
||||||
|
defer chatHandler.Close()
|
||||||
|
|
||||||
|
embHandler := New(embeddingsCfg)
|
||||||
|
defer embHandler.Close()
|
||||||
|
|
||||||
|
// Verify chat timeouts
|
||||||
|
chatRoute := chatHandler.routes["chat"]
|
||||||
|
if chatRoute.Upstream.ConnectTimeout != 10*time.Second {
|
||||||
|
t.Errorf("chat connect timeout: expected 10s, got %v", chatRoute.Upstream.ConnectTimeout)
|
||||||
|
}
|
||||||
|
if chatRoute.Upstream.ReadTimeout != 1*time.Hour {
|
||||||
|
t.Errorf("chat read timeout: expected 1h, got %v", chatRoute.Upstream.ReadTimeout)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify embeddings timeouts
|
||||||
|
embRoute := embHandler.routes["embeddings"]
|
||||||
|
if embRoute.Upstream.ConnectTimeout != 10*time.Second {
|
||||||
|
t.Errorf("embeddings connect timeout: expected 10s, got %v", embRoute.Upstream.ConnectTimeout)
|
||||||
|
}
|
||||||
|
if embRoute.Upstream.ReadTimeout != 10*time.Minute {
|
||||||
|
t.Errorf("embeddings read timeout: expected 10m, got %v", embRoute.Upstream.ReadTimeout)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
package server
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"sync"
|
||||||
|
)
|
||||||
|
|
||||||
|
// HealthChecker provides health and readiness check information.
|
||||||
|
type HealthChecker struct {
|
||||||
|
mu sync.RWMutex
|
||||||
|
configValid bool
|
||||||
|
jwksHasFetched bool
|
||||||
|
authEnabled bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewHealthChecker creates a new health checker instance.
|
||||||
|
func NewHealthChecker(configValid bool, authEnabled bool) *HealthChecker {
|
||||||
|
return &HealthChecker{
|
||||||
|
configValid: configValid,
|
||||||
|
jwksHasFetched: false,
|
||||||
|
authEnabled: authEnabled,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// MarkJWKSFetched marks that JWKS has been fetched successfully.
|
||||||
|
func (hc *HealthChecker) MarkJWKSFetched() {
|
||||||
|
hc.mu.Lock()
|
||||||
|
defer hc.mu.Unlock()
|
||||||
|
hc.jwksHasFetched = true
|
||||||
|
}
|
||||||
|
|
||||||
|
// IsReady checks if the server is ready to serve traffic.
|
||||||
|
// It returns true if:
|
||||||
|
// - Configuration is valid
|
||||||
|
// - If auth is enabled, JWKS has been fetched at least once
|
||||||
|
func (hc *HealthChecker) IsReady() bool {
|
||||||
|
hc.mu.RLock()
|
||||||
|
defer hc.mu.RUnlock()
|
||||||
|
|
||||||
|
if !hc.configValid {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// If auth is enabled, we must have fetched JWKS at least once
|
||||||
|
if hc.authEnabled && !hc.jwksHasFetched {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// IsAlive returns true if the process is running.
|
||||||
|
// This is always true since if it weren't, we wouldn't be running this code.
|
||||||
|
func (hc *HealthChecker) IsAlive() bool {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// LivenessHandler returns a handler for the /healthz endpoint.
|
||||||
|
// It returns 200 whenever the process is alive.
|
||||||
|
// It performs no network I/O.
|
||||||
|
func LivenessHandler(hc *HealthChecker) http.HandlerFunc {
|
||||||
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if !hc.IsAlive() {
|
||||||
|
w.WriteHeader(http.StatusServiceUnavailable)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
w.Write([]byte(`{"status":"alive"}`))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReadinessHandler returns a handler for the /readyz endpoint.
|
||||||
|
// It returns 200 only when configuration is valid and, if auth is enabled,
|
||||||
|
// JWKS has been fetched at least once.
|
||||||
|
// It returns non-2xx status while configuration is invalid or JWKS has never
|
||||||
|
// been fetched.
|
||||||
|
func ReadinessHandler(hc *HealthChecker) http.HandlerFunc {
|
||||||
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if !hc.IsReady() {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(http.StatusServiceUnavailable)
|
||||||
|
w.Write([]byte(`{"status":"not_ready"}`))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
w.Write([]byte(`{"status":"ready"}`))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,173 @@
|
|||||||
|
package server_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/server"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestHealthEndpoints verifies health endpoint behavior.
|
||||||
|
// - /healthz returns 200 even with unreachable upstreams
|
||||||
|
// - /readyz returns non-2xx before JWKS fetch and 200 after
|
||||||
|
// - Neither endpoint requires authentication
|
||||||
|
func TestHealthEndpoints(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
configValid bool
|
||||||
|
authEnabled bool
|
||||||
|
jwksFetched bool
|
||||||
|
endpoint string
|
||||||
|
expectedCode int
|
||||||
|
description string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "healthz_always_200",
|
||||||
|
configValid: true,
|
||||||
|
authEnabled: false,
|
||||||
|
jwksFetched: false,
|
||||||
|
endpoint: "/healthz",
|
||||||
|
expectedCode: http.StatusOK,
|
||||||
|
description: "liveness probe returns 200 even before JWKS fetch",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "healthz_200_when_config_invalid",
|
||||||
|
configValid: false,
|
||||||
|
authEnabled: false,
|
||||||
|
jwksFetched: false,
|
||||||
|
endpoint: "/healthz",
|
||||||
|
expectedCode: http.StatusOK,
|
||||||
|
description: "liveness probe returns 200 even when config is invalid",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "readyz_200_no_auth",
|
||||||
|
configValid: true,
|
||||||
|
authEnabled: false,
|
||||||
|
jwksFetched: false,
|
||||||
|
endpoint: "/readyz",
|
||||||
|
expectedCode: http.StatusOK,
|
||||||
|
description: "readiness returns 200 when config valid and auth disabled",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "readyz_503_invalid_config",
|
||||||
|
configValid: false,
|
||||||
|
authEnabled: false,
|
||||||
|
jwksFetched: false,
|
||||||
|
endpoint: "/readyz",
|
||||||
|
expectedCode: http.StatusServiceUnavailable,
|
||||||
|
description: "readiness returns 503 when config invalid",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "readyz_503_auth_enabled_no_jwks",
|
||||||
|
configValid: true,
|
||||||
|
authEnabled: true,
|
||||||
|
jwksFetched: false,
|
||||||
|
endpoint: "/readyz",
|
||||||
|
expectedCode: http.StatusServiceUnavailable,
|
||||||
|
description: "readiness returns 503 when auth enabled but JWKS not fetched",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "readyz_200_auth_enabled_with_jwks",
|
||||||
|
configValid: true,
|
||||||
|
authEnabled: true,
|
||||||
|
jwksFetched: true,
|
||||||
|
endpoint: "/readyz",
|
||||||
|
expectedCode: http.StatusOK,
|
||||||
|
description: "readiness returns 200 when auth enabled and JWKS fetched",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
// Create health checker
|
||||||
|
hc := server.NewHealthChecker(tt.configValid, tt.authEnabled)
|
||||||
|
if tt.jwksFetched {
|
||||||
|
hc.MarkJWKSFetched()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create handler based on endpoint
|
||||||
|
var handler http.HandlerFunc
|
||||||
|
switch tt.endpoint {
|
||||||
|
case "/healthz":
|
||||||
|
handler = server.LivenessHandler(hc)
|
||||||
|
case "/readyz":
|
||||||
|
handler = server.ReadinessHandler(hc)
|
||||||
|
default:
|
||||||
|
t.Fatalf("unknown endpoint: %s", tt.endpoint)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create server wrapper
|
||||||
|
gatewayServer := server.New("127.0.0.1:0", 5*time.Second, handler)
|
||||||
|
|
||||||
|
// Start server in goroutine
|
||||||
|
go func() {
|
||||||
|
if err := gatewayServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
|
||||||
|
t.Logf("server error: %v", err)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
// Give server time to start
|
||||||
|
time.Sleep(100 * time.Millisecond)
|
||||||
|
|
||||||
|
// Make request
|
||||||
|
url := fmt.Sprintf("http://%s%s", gatewayServer.Addr(), tt.endpoint)
|
||||||
|
resp, err := http.Get(url)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to make request: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
// Check status code
|
||||||
|
if resp.StatusCode != tt.expectedCode {
|
||||||
|
body, _ := io.ReadAll(resp.Body)
|
||||||
|
t.Errorf("expected status %d, got %d: %s", tt.expectedCode, resp.StatusCode, string(body))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify no Authorization header is required
|
||||||
|
// (we already made the request without one, so this is implicit)
|
||||||
|
|
||||||
|
// Cleanup
|
||||||
|
gatewayServer.Shutdown(context.Background())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHealthEndpointsNoProxy verifies that health endpoints are not proxied.
|
||||||
|
// This is verified indirectly by the test above - if they were proxied,
|
||||||
|
// they would return 404 or fail when trying to reach a non-existent upstream.
|
||||||
|
func TestHealthEndpointsCannotBeShadowed(t *testing.T) {
|
||||||
|
// Create health checker and handler
|
||||||
|
hc := server.NewHealthChecker(true, false)
|
||||||
|
handler := server.LivenessHandler(hc)
|
||||||
|
|
||||||
|
// Create server
|
||||||
|
srv := server.New("127.0.0.1:0", 5*time.Second, handler)
|
||||||
|
|
||||||
|
// Start server
|
||||||
|
go func() {
|
||||||
|
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
|
||||||
|
t.Logf("server error: %v", err)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
// Give server time to start
|
||||||
|
time.Sleep(100 * time.Millisecond)
|
||||||
|
|
||||||
|
// Request /healthz and verify it's not proxied
|
||||||
|
resp, err := http.Get(fmt.Sprintf("http://%s/healthz", srv.Addr()))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to make request: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Errorf("expected status 200, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cleanup
|
||||||
|
srv.Shutdown(context.Background())
|
||||||
|
}
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
package server
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Router implements an HTTP handler that routes health endpoints
|
||||||
|
// and passes other requests to an upstream handler.
|
||||||
|
type Router struct {
|
||||||
|
healthChecker *HealthChecker
|
||||||
|
upstreamHandler http.Handler
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewRouter creates a new router with health endpoints.
|
||||||
|
// Health endpoints (/healthz and /readyz) are handled locally.
|
||||||
|
// All other paths are passed to the upstream handler.
|
||||||
|
func NewRouter(healthChecker *HealthChecker, upstreamHandler http.Handler) *Router {
|
||||||
|
return &Router{
|
||||||
|
healthChecker: healthChecker,
|
||||||
|
upstreamHandler: upstreamHandler,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ServeHTTP implements http.Handler.
|
||||||
|
// It routes /healthz and /readyz to health handlers,
|
||||||
|
// and passes all other paths to the upstream handler.
|
||||||
|
func (r *Router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
|
||||||
|
switch req.URL.Path {
|
||||||
|
case "/healthz":
|
||||||
|
LivenessHandler(r.healthChecker)(w, req)
|
||||||
|
case "/readyz":
|
||||||
|
ReadinessHandler(r.healthChecker)(w, req)
|
||||||
|
default:
|
||||||
|
r.upstreamHandler.ServeHTTP(w, req)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
package server
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Server wraps an HTTP server with graceful shutdown support.
|
||||||
|
type Server struct {
|
||||||
|
httpServer *http.Server
|
||||||
|
shutdownTimeout time.Duration
|
||||||
|
listener net.Listener
|
||||||
|
listenerMu sync.RWMutex
|
||||||
|
healthChecker *HealthChecker
|
||||||
|
}
|
||||||
|
|
||||||
|
// New creates a new Server with the given configuration.
|
||||||
|
func New(listenAddr string, shutdownTimeout time.Duration, handler http.Handler) *Server {
|
||||||
|
return &Server{
|
||||||
|
httpServer: &http.Server{
|
||||||
|
Addr: listenAddr,
|
||||||
|
Handler: handler,
|
||||||
|
ReadTimeout: 15 * time.Second,
|
||||||
|
WriteTimeout: 15 * time.Second,
|
||||||
|
IdleTimeout: 60 * time.Second,
|
||||||
|
},
|
||||||
|
shutdownTimeout: shutdownTimeout,
|
||||||
|
healthChecker: NewHealthChecker(false, false),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListenAndServe starts the HTTP server and blocks until it exits.
|
||||||
|
// It returns the error from the server (if any), which will be
|
||||||
|
// http.ErrServerClosed if Shutdown was called.
|
||||||
|
func (s *Server) ListenAndServe() error {
|
||||||
|
listener, err := net.Listen("tcp", s.httpServer.Addr)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
s.listenerMu.Lock()
|
||||||
|
s.listener = listener
|
||||||
|
s.listenerMu.Unlock()
|
||||||
|
return s.httpServer.Serve(listener)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Shutdown gracefully shuts down the server. It stops accepting new
|
||||||
|
// connections and waits for in-flight requests to complete, with a
|
||||||
|
// bounded deadline. If the deadline is exceeded, it returns an error.
|
||||||
|
func (s *Server) Shutdown(ctx context.Context) error {
|
||||||
|
// Create a new context with the shutdown timeout
|
||||||
|
shutdownCtx, cancel := context.WithTimeout(ctx, s.shutdownTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
return s.httpServer.Shutdown(shutdownCtx)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Addr returns the network address the server is listening on.
|
||||||
|
func (s *Server) Addr() string {
|
||||||
|
s.listenerMu.RLock()
|
||||||
|
defer s.listenerMu.RUnlock()
|
||||||
|
if s.listener != nil {
|
||||||
|
return s.listener.Addr().String()
|
||||||
|
}
|
||||||
|
return s.httpServer.Addr
|
||||||
|
}
|
||||||
|
|
||||||
|
// HealthChecker returns the server's health checker.
|
||||||
|
func (s *Server) HealthChecker() *HealthChecker {
|
||||||
|
return s.healthChecker
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetHealthChecker sets the server's health checker.
|
||||||
|
func (s *Server) SetHealthChecker(hc *HealthChecker) {
|
||||||
|
s.healthChecker = hc
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetHandler sets the server's HTTP handler.
|
||||||
|
func (s *Server) SetHandler(handler http.Handler) {
|
||||||
|
s.httpServer.Handler = handler
|
||||||
|
}
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
package server_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/server"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestGracefulShutdown verifies that:
|
||||||
|
// - A request in-flight when shutdown starts receives its full, uncorrupted response body
|
||||||
|
// - A request arriving after shutdown starts is refused on a new connection
|
||||||
|
// - The shutdown completes with exit code 0 (no timeout)
|
||||||
|
func TestGracefulShutdown(t *testing.T) {
|
||||||
|
// Create a handler that responds slowly
|
||||||
|
const responseBody = "slow response body content"
|
||||||
|
const sleepDuration = 500 * time.Millisecond
|
||||||
|
|
||||||
|
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
// Simulate a slow LLM response
|
||||||
|
time.Sleep(sleepDuration)
|
||||||
|
fmt.Fprint(w, responseBody)
|
||||||
|
})
|
||||||
|
|
||||||
|
// Create server with a shutdown timeout longer than the sleep
|
||||||
|
srv := server.New("127.0.0.1:0", 5*time.Second, handler)
|
||||||
|
|
||||||
|
// Start server in a goroutine
|
||||||
|
var listenErr error
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
listenErr = srv.ListenAndServe()
|
||||||
|
// http.ErrServerClosed is expected after shutdown
|
||||||
|
if listenErr != nil && listenErr != http.ErrServerClosed {
|
||||||
|
t.Logf("unexpected listen error: %v", listenErr)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
// Give server time to start listening
|
||||||
|
time.Sleep(100 * time.Millisecond)
|
||||||
|
|
||||||
|
// Issue a slow request in a goroutine
|
||||||
|
var responseBody_got string
|
||||||
|
var requestErr error
|
||||||
|
var requestWg sync.WaitGroup
|
||||||
|
requestWg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer requestWg.Done()
|
||||||
|
resp, err := http.Get(fmt.Sprintf("http://%s/", srv.Addr()))
|
||||||
|
if err != nil {
|
||||||
|
requestErr = err
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
body, err := io.ReadAll(resp.Body)
|
||||||
|
if err != nil {
|
||||||
|
requestErr = err
|
||||||
|
return
|
||||||
|
}
|
||||||
|
responseBody_got = string(body)
|
||||||
|
}()
|
||||||
|
|
||||||
|
// Give the request time to reach the handler
|
||||||
|
time.Sleep(100 * time.Millisecond)
|
||||||
|
|
||||||
|
// Now initiate shutdown while request is in-flight
|
||||||
|
shutdownErr := srv.Shutdown(context.Background())
|
||||||
|
|
||||||
|
// Wait for the in-flight request to complete
|
||||||
|
requestWg.Wait()
|
||||||
|
|
||||||
|
// Verify the in-flight request completed successfully
|
||||||
|
if requestErr != nil {
|
||||||
|
t.Fatalf("in-flight request failed: %v", requestErr)
|
||||||
|
}
|
||||||
|
if responseBody_got != responseBody {
|
||||||
|
t.Fatalf("in-flight request got wrong body: %q (expected %q)", responseBody_got, responseBody)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify shutdown succeeded (no timeout)
|
||||||
|
if shutdownErr != nil {
|
||||||
|
t.Fatalf("shutdown failed: %v", shutdownErr)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify in-flight requests were allowed to complete
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
// Now verify that a new request is refused after shutdown
|
||||||
|
_, err := net.Dial("tcp", srv.Addr())
|
||||||
|
if err == nil {
|
||||||
|
// Connection succeeded when it should have failed
|
||||||
|
t.Fatalf("new connection accepted after shutdown (should have been refused)")
|
||||||
|
}
|
||||||
|
// If we get here, the connection was properly refused, which is what we want
|
||||||
|
}
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
// Package testsupport provides the local-development stub that every upstream
|
||||||
|
// in testdata/config/harness.yaml targets. It exists so tests under
|
||||||
|
// internal/proxy (future) and here can verify streaming semantics without a
|
||||||
|
// Kubernetes cluster or real credentials.
|
||||||
|
//
|
||||||
|
// The harness binds ONE http.Server on 127.0.0.1:9080 by default — this matches
|
||||||
|
// the addresses baked into testdata/config/harness.yaml so that fixture is valid
|
||||||
|
// before any code runs, and no network traffic leaves localhost during verification.
|
||||||
|
package testsupport
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"sync"
|
||||||
|
)
|
||||||
|
|
||||||
|
const defaultLocalPort = "9080"
|
||||||
|
|
||||||
|
// DefaultConfigPath is the committed fixture that points every upstream at the
|
||||||
|
// same local port; clients load it via config.LoadRoutesFromFile. Tests can
|
||||||
|
// override the path if they want a different schema.
|
||||||
|
const DefaultConfigPath = "testdata/config/harness.yaml"
|
||||||
|
|
||||||
|
// Snapshot describes a running stub server.
|
||||||
|
type Snapshot struct {
|
||||||
|
BaseURL string
|
||||||
|
Client *http.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
// BaseAddr returns the host:port form of the bound address, no scheme.
|
||||||
|
func (s *Snapshot) BaseAddr() string {
|
||||||
|
return s.BaseURL[len("http://"):]
|
||||||
|
}
|
||||||
|
|
||||||
|
// URL joins the stub base URL with one of the Path* constants.
|
||||||
|
func (s *Snapshot) URL(path string) string {
|
||||||
|
return s.BaseURL + path
|
||||||
|
}
|
||||||
|
|
||||||
|
// harness is the singleton that owns the stub server for a single process.
|
||||||
|
type harness struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
srv *http.Server // nil when not running; set exactly once per Close/Start cycle
|
||||||
|
addr string // bound address, valid only while srv != nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// global is the singleton used by tests. A fresh server is bound lazily on
|
||||||
|
// first Start() and shared thereafter for the lifetime of the harness process
|
||||||
|
// (usually a single TestMain run).
|
||||||
|
var global = &harness{}
|
||||||
|
|
||||||
|
// Start binds the stub server if it is not already running and returns a
|
||||||
|
// Snapshot describing it. It is safe to call from multiple tests; the second
|
||||||
|
// and later calls return the already-bound server.
|
||||||
|
func Start() (*Snapshot, error) { return global.Start() }
|
||||||
|
|
||||||
|
// Close stops the stub server. Safe to call multiple times.
|
||||||
|
func Close() { global.Close() }
|
||||||
|
|
||||||
|
func (h *harness) Start() (*Snapshot, error) {
|
||||||
|
h.mu.Lock()
|
||||||
|
defer h.mu.Unlock()
|
||||||
|
if h.srv != nil {
|
||||||
|
return h.snapshotLocked(), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Listen separately from Serve. srv.ListenAndServe would block until
|
||||||
|
// shutdown, so Start could never return; binding first also guarantees the
|
||||||
|
// port is accepting connections by the time the caller gets the Snapshot.
|
||||||
|
ln, err := net.Listen("tcp", defaultStubAddr())
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("bind stub server at %s: %w", defaultStubAddr(), err)
|
||||||
|
}
|
||||||
|
|
||||||
|
srv := &http.Server{Handler: newStubHandler()}
|
||||||
|
h.srv = srv
|
||||||
|
h.addr = ln.Addr().String()
|
||||||
|
|
||||||
|
// Serve always returns a non-nil error; after Close that error is
|
||||||
|
// ErrServerClosed, which is the expected path and not worth reporting.
|
||||||
|
go func() { _ = srv.Serve(ln) }()
|
||||||
|
|
||||||
|
return h.snapshotLocked(), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// snapshotLocked builds a Snapshot for the running server. Caller holds h.mu.
|
||||||
|
func (h *harness) snapshotLocked() *Snapshot {
|
||||||
|
return &Snapshot{BaseURL: "http://" + h.addr, Client: http.DefaultClient}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Close stops the underlying stub server. Safe to call on any harness instance
|
||||||
|
// or multiple times — it is idempotent within a process.
|
||||||
|
func (h *harness) Close() {
|
||||||
|
h.mu.Lock()
|
||||||
|
defer h.mu.Unlock()
|
||||||
|
if h.srv == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s := h.srv
|
||||||
|
h.srv = nil
|
||||||
|
h.addr = ""
|
||||||
|
_ = s.Close() // best-effort shutdown; test output not dependent on it
|
||||||
|
}
|
||||||
|
|
||||||
|
// defaultStubAddr constructs "127.0.0.1:<port>" honoring HARNESS_STUB_PORT if
|
||||||
|
// set, falling back to 9080 — which matches every address in the committed YAML
|
||||||
|
// fixture. Set only when you need parallel test runs within a single process;
|
||||||
|
// otherwise leave unset.
|
||||||
|
func defaultStubAddr() string {
|
||||||
|
port := os.Getenv("HARNESS_STUB_PORT")
|
||||||
|
if port == "" {
|
||||||
|
port = defaultLocalPort
|
||||||
|
}
|
||||||
|
return "127.0.0.1:" + port
|
||||||
|
}
|
||||||
@@ -0,0 +1,135 @@
|
|||||||
|
package testsupport
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/Riotpiaole/homelab-frontend/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// startForTest binds the stub on an ephemeral port so the suite does not fight
|
||||||
|
// with anything already holding 9080, and fails fast if Start blocks — the
|
||||||
|
// original harness called ListenAndServe inline and never returned.
|
||||||
|
func startForTest(t *testing.T) *Snapshot {
|
||||||
|
t.Helper()
|
||||||
|
t.Setenv("HARNESS_STUB_PORT", "0")
|
||||||
|
|
||||||
|
type result struct {
|
||||||
|
snap *Snapshot
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
done := make(chan result, 1)
|
||||||
|
go func() {
|
||||||
|
snap, err := Start()
|
||||||
|
done <- result{snap, err}
|
||||||
|
}()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case r := <-done:
|
||||||
|
if r.err != nil {
|
||||||
|
t.Fatalf("Start() error: %v", r.err)
|
||||||
|
}
|
||||||
|
t.Cleanup(Close)
|
||||||
|
return r.snap
|
||||||
|
case <-time.After(5 * time.Second):
|
||||||
|
t.Fatal("Start() did not return within 5s — it is blocking instead of serving in the background")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStartServesFixedJSON(t *testing.T) {
|
||||||
|
snap := startForTest(t)
|
||||||
|
|
||||||
|
resp, err := snap.Client.Get(snap.URL(PathJSON))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GET %s: %v", PathJSON, err)
|
||||||
|
}
|
||||||
|
defer func() { _ = resp.Body.Close() }()
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Errorf("status = %d, want %d", resp.StatusCode, http.StatusOK)
|
||||||
|
}
|
||||||
|
if got := resp.Header.Get("Content-Type"); got != "application/json" {
|
||||||
|
t.Errorf("Content-Type = %q, want %q", got, "application/json")
|
||||||
|
}
|
||||||
|
|
||||||
|
body, err := io.ReadAll(resp.Body)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read body: %v", err)
|
||||||
|
}
|
||||||
|
if want := `{"status":"ok"}`; strings.TrimSpace(string(body)) != want {
|
||||||
|
t.Errorf("body = %q, want %q", strings.TrimSpace(string(body)), want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSSEArrivesIncrementally(t *testing.T) {
|
||||||
|
snap := startForTest(t)
|
||||||
|
|
||||||
|
resp, err := snap.Client.Get(snap.URL(PathSSE))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GET %s: %v", PathSSE, err)
|
||||||
|
}
|
||||||
|
defer func() { _ = resp.Body.Close() }()
|
||||||
|
|
||||||
|
// The first event must be readable before the handler has written the last
|
||||||
|
// one. If the response were buffered to completion, this read would only
|
||||||
|
// unblock after every token plus every gap had elapsed.
|
||||||
|
deadline := time.Now().Add(time.Duration(len(SSETokens)) * SSEGap)
|
||||||
|
|
||||||
|
reader := bufio.NewReader(resp.Body)
|
||||||
|
line, err := reader.ReadString('\n')
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read first event: %v", err)
|
||||||
|
}
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Error("first SSE event arrived only after the whole stream was written; not incremental")
|
||||||
|
}
|
||||||
|
if want := "data: " + SSETokens[0]; strings.TrimSpace(line) != want {
|
||||||
|
t.Errorf("first event = %q, want %q", strings.TrimSpace(line), want)
|
||||||
|
}
|
||||||
|
|
||||||
|
rest, err := io.ReadAll(reader)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read remaining events: %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(rest), "data: [DONE]") {
|
||||||
|
t.Error("stream did not end with the [DONE] sentinel")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStartIsIdempotent(t *testing.T) {
|
||||||
|
first := startForTest(t)
|
||||||
|
|
||||||
|
second, err := Start()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second Start() error: %v", err)
|
||||||
|
}
|
||||||
|
if first.BaseURL != second.BaseURL {
|
||||||
|
t.Errorf("second Start() bound a different address: %q vs %q", second.BaseURL, first.BaseURL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHarnessFixtureLoadsAndStaysOnLoopback(t *testing.T) {
|
||||||
|
// DefaultConfigPath is relative to the repo root; tests run in the package
|
||||||
|
// directory, so walk back up to it.
|
||||||
|
routes, err := config.LoadRoutesFromFile("../../" + DefaultConfigPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LoadRoutesFromFile(%s): %v", DefaultConfigPath, err)
|
||||||
|
}
|
||||||
|
if len(routes) == 0 {
|
||||||
|
t.Fatal("fixture declared no routes")
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, route := range routes {
|
||||||
|
if !strings.HasPrefix(route.Upstream.Address, "127.0.0.1:") {
|
||||||
|
t.Errorf("route %q upstream %q is not on loopback", name, route.Upstream.Address)
|
||||||
|
}
|
||||||
|
if route.Upstream.AuthRequired {
|
||||||
|
t.Errorf("route %q requires auth; the harness must run with no credentials", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,123 @@
|
|||||||
|
package testsupport
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Stub response paths. testdata/config/harness.yaml points every upstream at the
|
||||||
|
// one stub server, so the response shape is selected by path, not by port.
|
||||||
|
const (
|
||||||
|
PathJSON = "/stub/json"
|
||||||
|
PathSSE = "/stub/sse"
|
||||||
|
PathChunked = "/stub/chunked"
|
||||||
|
PathSlow = "/stub/slow"
|
||||||
|
)
|
||||||
|
|
||||||
|
// SSEGap is the pause between SSE events. It exists so a test can observe that
|
||||||
|
// chunks arrive incrementally rather than all at once at the end.
|
||||||
|
const SSEGap = 20 * time.Millisecond
|
||||||
|
|
||||||
|
// SlowDelay is how long PathSlow waits before writing anything.
|
||||||
|
const SlowDelay = 250 * time.Millisecond
|
||||||
|
|
||||||
|
// SSETokens are the tokens PathSSE emits, one event per token, followed by the
|
||||||
|
// [DONE] sentinel that OpenAI-shaped clients expect.
|
||||||
|
var SSETokens = []string{"Hello", " ", "world", "!"}
|
||||||
|
|
||||||
|
// ChunkedBodies are the pieces PathChunked writes, each flushed separately so
|
||||||
|
// the response goes out with Transfer-Encoding: chunked.
|
||||||
|
var ChunkedBodies = []string{"first\n", "second\n", "third\n"}
|
||||||
|
|
||||||
|
// newStubHandler builds the mux served by the harness. Every handler writes a
|
||||||
|
// deterministic body so tests can assert on exact bytes.
|
||||||
|
func newStubHandler() http.Handler {
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
mux.HandleFunc(PathJSON, stubJSON)
|
||||||
|
mux.HandleFunc(PathSSE, stubSSE)
|
||||||
|
mux.HandleFunc(PathChunked, stubChunked)
|
||||||
|
mux.HandleFunc(PathSlow, stubSlow)
|
||||||
|
return mux
|
||||||
|
}
|
||||||
|
|
||||||
|
// stubJSON serves a fixed JSON body.
|
||||||
|
func stubJSON(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
// Encode failure here means the client hung up mid-write; the connection is
|
||||||
|
// already gone, so there is nothing to report and no header left to change.
|
||||||
|
_ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"})
|
||||||
|
}
|
||||||
|
|
||||||
|
// stubSSE streams one event per token, flushing after each. It returns early
|
||||||
|
// when the client disconnects, so a mid-response disconnect test can assert on
|
||||||
|
// how many tokens the upstream actually managed to emit.
|
||||||
|
func stubSSE(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/event-stream")
|
||||||
|
w.Header().Set("Cache-Control", "no-cache")
|
||||||
|
w.Header().Set("Connection", "keep-alive")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
|
||||||
|
rc := http.NewResponseController(w)
|
||||||
|
if err := rc.Flush(); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tok := range SSETokens {
|
||||||
|
select {
|
||||||
|
case <-r.Context().Done():
|
||||||
|
return
|
||||||
|
case <-time.After(SSEGap):
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := fmt.Fprintf(w, "data: %s\n\n", tok); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := rc.Flush(); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := fmt.Fprint(w, "data: [DONE]\n\n"); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_ = rc.Flush()
|
||||||
|
}
|
||||||
|
|
||||||
|
// stubChunked writes several pieces with a flush between each, producing a
|
||||||
|
// chunked transfer with no Content-Length.
|
||||||
|
func stubChunked(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/plain")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
|
||||||
|
rc := http.NewResponseController(w)
|
||||||
|
for _, body := range ChunkedBodies {
|
||||||
|
select {
|
||||||
|
case <-r.Context().Done():
|
||||||
|
return
|
||||||
|
case <-time.After(SSEGap):
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := fmt.Fprint(w, body); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := rc.Flush(); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// stubSlow waits SlowDelay before responding at all, for timeout tests.
|
||||||
|
func stubSlow(w http.ResponseWriter, r *http.Request) {
|
||||||
|
select {
|
||||||
|
case <-r.Context().Done():
|
||||||
|
return
|
||||||
|
case <-time.After(SlowDelay):
|
||||||
|
}
|
||||||
|
|
||||||
|
w.Header().Set("Content-Type", "text/plain")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
_, _ = fmt.Fprint(w, "slow\n")
|
||||||
|
}
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: api-gateway-config
|
||||||
|
namespace: api
|
||||||
|
labels:
|
||||||
|
app: api-gateway
|
||||||
|
data:
|
||||||
|
config.yaml: |
|
||||||
|
# Gateway configuration - loaded at startup, never compiled in
|
||||||
|
# See REQUIREMENTS.md for full specification
|
||||||
|
|
||||||
|
# Routes: standard HTTP proxy routes (not LLM-specific)
|
||||||
|
# These are for non-LLM services (agent-pod/console, etc.)
|
||||||
|
routes: []
|
||||||
|
|
||||||
|
# Models: LLM model registry for body-based dispatch
|
||||||
|
# POST /v1/chat/completions routes based on the "model" field in request JSON
|
||||||
|
models:
|
||||||
|
- name: "reasoning"
|
||||||
|
address: "reasoning-predictor.llm-serving:80"
|
||||||
|
path: "/v1/chat/completions"
|
||||||
|
|
||||||
|
- name: "ornith:35b"
|
||||||
|
address: "ornith-predictor.llm-serving:80"
|
||||||
|
path: "/v1/chat/completions"
|
||||||
|
|
||||||
|
- name: "qwen2.5:3b-instruct"
|
||||||
|
address: "ornith-predictor.llm-serving:80"
|
||||||
|
path: "/v1/chat/completions"
|
||||||
|
|
||||||
|
- name: "nomic-ai/nomic-embed-text-v2-moe"
|
||||||
|
address: "embeddings-predictor.llm-serving:80"
|
||||||
|
path: "/v1/embeddings"
|
||||||
|
|
||||||
|
- name: "BAAI/bge-reranker-base"
|
||||||
|
address: "reranker-predictor.llm-serving:80"
|
||||||
|
path: "/v1/rerank"
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: api-gateway
|
||||||
|
namespace: api
|
||||||
|
labels:
|
||||||
|
app: api-gateway
|
||||||
|
component: gateway
|
||||||
|
spec:
|
||||||
|
replicas: 3
|
||||||
|
strategy:
|
||||||
|
type: RollingUpdate
|
||||||
|
rollingUpdate:
|
||||||
|
maxSurge: 1
|
||||||
|
maxUnavailable: 0
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: api-gateway
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: api-gateway
|
||||||
|
component: gateway
|
||||||
|
annotations:
|
||||||
|
prometheus.io/scrape: "true"
|
||||||
|
prometheus.io/port: "8080"
|
||||||
|
prometheus.io/path: "/metrics"
|
||||||
|
spec:
|
||||||
|
serviceAccountName: api-gateway
|
||||||
|
securityContext:
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 1000
|
||||||
|
fsGroup: 1000
|
||||||
|
containers:
|
||||||
|
- name: gateway
|
||||||
|
image: ghcr.io/riotpiaole/api-gateway:latest
|
||||||
|
imagePullPolicy: Always
|
||||||
|
ports:
|
||||||
|
- name: http
|
||||||
|
containerPort: 8080
|
||||||
|
protocol: TCP
|
||||||
|
env:
|
||||||
|
- name: LISTEN_ADDR
|
||||||
|
value: "0.0.0.0:8080"
|
||||||
|
- name: CONFIG_PATH
|
||||||
|
value: "/etc/gateway/config.yaml"
|
||||||
|
- name: SHUTDOWN_TIMEOUT
|
||||||
|
value: "5m"
|
||||||
|
- name: LOG_LEVEL
|
||||||
|
value: "info"
|
||||||
|
volumeMounts:
|
||||||
|
- name: config
|
||||||
|
mountPath: /etc/gateway
|
||||||
|
readOnly: true
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /healthz
|
||||||
|
port: http
|
||||||
|
scheme: HTTP
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 10
|
||||||
|
timeoutSeconds: 2
|
||||||
|
failureThreshold: 3
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /readyz
|
||||||
|
port: http
|
||||||
|
scheme: HTTP
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 5
|
||||||
|
timeoutSeconds: 2
|
||||||
|
failureThreshold: 2
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 128Mi
|
||||||
|
limits:
|
||||||
|
cpu: 500m
|
||||||
|
memory: 512Mi
|
||||||
|
securityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
readOnlyRootFilesystem: true
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 1000
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
volumes:
|
||||||
|
- name: config
|
||||||
|
configMap:
|
||||||
|
name: api-gateway-config
|
||||||
|
affinity:
|
||||||
|
podAntiAffinity:
|
||||||
|
preferredDuringSchedulingIgnoredDuringExecution:
|
||||||
|
- weight: 100
|
||||||
|
podAffinityTerm:
|
||||||
|
labelSelector:
|
||||||
|
matchExpressions:
|
||||||
|
- key: app
|
||||||
|
operator: In
|
||||||
|
values:
|
||||||
|
- api-gateway
|
||||||
|
topologyKey: kubernetes.io/hostname
|
||||||
|
terminationGracePeriodSeconds: 300
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
|
kind: Kustomization
|
||||||
|
|
||||||
|
namespace: api
|
||||||
|
|
||||||
|
resources:
|
||||||
|
- rbac.yaml
|
||||||
|
- service.yaml
|
||||||
|
- deployment.yaml
|
||||||
|
- network-policy.yaml
|
||||||
|
- configmap.yaml
|
||||||
|
|
||||||
|
commonLabels:
|
||||||
|
app: api-gateway
|
||||||
|
managed-by: argocd
|
||||||
|
|
||||||
|
commonAnnotations:
|
||||||
|
argocd.argoproj.io/sync-wave: "2"
|
||||||
|
# Wave 2 ensures the gateway is ready before anything that depends on it
|
||||||
|
# Kong remains on wave 7 unchanged
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: api-gateway
|
||||||
|
namespace: api
|
||||||
|
labels:
|
||||||
|
app: api-gateway
|
||||||
|
spec:
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app: api-gateway
|
||||||
|
policyTypes:
|
||||||
|
- Ingress
|
||||||
|
- Egress
|
||||||
|
ingress:
|
||||||
|
# Allow from ingress-nginx controller (from ingress-nginx namespace)
|
||||||
|
- from:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
name: ingress-nginx
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 8080
|
||||||
|
# Allow from Prometheus scraping (if in monitoring namespace)
|
||||||
|
- from:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
name: monitoring
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 8080
|
||||||
|
egress:
|
||||||
|
# Allow DNS
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
name: kube-system
|
||||||
|
ports:
|
||||||
|
- protocol: UDP
|
||||||
|
port: 53
|
||||||
|
# Allow to upstreams (LLM services in llm-serving namespace)
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
name: llm-serving
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 80
|
||||||
|
- protocol: TCP
|
||||||
|
port: 8000
|
||||||
|
- protocol: TCP
|
||||||
|
port: 8001
|
||||||
|
# Allow to other upstreams if needed (embeddings, reranker, etc.)
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
name: llm-serving
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 8080
|
||||||
|
# Allow to atlas (riotpiao-backend) for /cluster/* routes
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
name: atlas
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 8080
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: ServiceAccount
|
||||||
|
metadata:
|
||||||
|
name: api-gateway
|
||||||
|
namespace: api
|
||||||
|
labels:
|
||||||
|
app: api-gateway
|
||||||
|
---
|
||||||
|
# No ClusterRole needed - the gateway has no k8s API access
|
||||||
|
# G2: The gateway holds no Kubernetes credentials
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: api-gateway
|
||||||
|
namespace: api
|
||||||
|
labels:
|
||||||
|
app: api-gateway
|
||||||
|
component: gateway
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
selector:
|
||||||
|
app: api-gateway
|
||||||
|
ports:
|
||||||
|
- name: http
|
||||||
|
port: 8080
|
||||||
|
targetPort: http
|
||||||
|
protocol: TCP
|
||||||
|
sessionAffinity: None
|
||||||
Executable
+157
@@ -0,0 +1,157 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Run a sequence of tasks through pi, verifying after each one.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# scripts/run-all.sh # run the default local-verifiable sequence
|
||||||
|
# scripts/run-all.sh 1.1 1.2 2.1 # run a specific list, in the order given
|
||||||
|
# scripts/run-all.sh --list # show the default sequence and exit
|
||||||
|
# scripts/run-all.sh --resume # rerun the default sequence, skipping tasks
|
||||||
|
# # already recorded in .task-runs/completed.txt
|
||||||
|
#
|
||||||
|
# Each task gets a fresh zero-context pi session. After each task the full gate runs:
|
||||||
|
#
|
||||||
|
# go test ./... -race
|
||||||
|
# CGO_ENABLED=0 go build ./...
|
||||||
|
# go vet ./...
|
||||||
|
#
|
||||||
|
# The loop HALTS on the first gate failure. That is deliberate. These tasks build on
|
||||||
|
# each other, so continuing past a broken foundation produces a pile of work that all
|
||||||
|
# has to be redone. A halted loop is cheap; a cascade is not.
|
||||||
|
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
cd "$REPO_ROOT"
|
||||||
|
|
||||||
|
# Default sequence: dependency-ordered, and limited to tasks that can be verified
|
||||||
|
# locally with no cluster and no credentials.
|
||||||
|
#
|
||||||
|
# Deliberately EXCLUDED from the default run:
|
||||||
|
# 0.6 CI pipeline -- needs a CI runner, not verifiable here
|
||||||
|
# 2.8 Kong parity -- requires the live Kong to diff against
|
||||||
|
# 3.x Authentication -- needs a real Authentik provider and tokens
|
||||||
|
# 5.3 ServiceMonitor -- needs Prometheus in-cluster
|
||||||
|
# 6.x Deploy and cutover -- touches live cluster state; run these by hand
|
||||||
|
# 7.x New prefixes -- deliberately after cutover
|
||||||
|
DEFAULT_TASKS=(
|
||||||
|
0.2 0.3 0.4 0.5
|
||||||
|
1.1 1.2 1.3 1.4 1.5 1.6 1.7
|
||||||
|
2.1 2.2 2.3 2.4 2.5 2.6 2.7
|
||||||
|
2.9 2.10 2.11 2.12 2.13 2.14 2.15
|
||||||
|
4.1 4.2 4.3
|
||||||
|
5.1 5.2
|
||||||
|
)
|
||||||
|
|
||||||
|
if [[ "${1:-}" == "--list" ]]; then
|
||||||
|
printf '%s\n' "${DEFAULT_TASKS[@]}"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
LOG_DIR=".task-runs"
|
||||||
|
mkdir -p "$LOG_DIR"
|
||||||
|
|
||||||
|
# Tasks that have passed the gate, one id per line, appended as they finish and
|
||||||
|
# carried across invocations. This is what makes --resume possible after a halt.
|
||||||
|
DONE_FILE="$LOG_DIR/completed.txt"
|
||||||
|
touch "$DONE_FILE"
|
||||||
|
|
||||||
|
if [[ "${1:-}" == "--resume" ]]; then
|
||||||
|
shift
|
||||||
|
# Resume the default sequence minus everything already recorded as passing.
|
||||||
|
# Order is preserved: DEFAULT_TASKS drives the loop, DONE_FILE only filters.
|
||||||
|
TASKS=()
|
||||||
|
for t in "${DEFAULT_TASKS[@]}"; do
|
||||||
|
grep -qxF "$t" "$DONE_FILE" || TASKS+=("$t")
|
||||||
|
done
|
||||||
|
SKIPPED=$(( ${#DEFAULT_TASKS[@]} - ${#TASKS[@]} ))
|
||||||
|
echo "resuming : skipping $SKIPPED already-passed task(s) recorded in $DONE_FILE"
|
||||||
|
if [[ ${#TASKS[@]} -eq 0 ]]; then
|
||||||
|
echo "nothing left to run."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
elif [[ $# -gt 0 ]]; then
|
||||||
|
TASKS=("$@")
|
||||||
|
else
|
||||||
|
TASKS=("${DEFAULT_TASKS[@]}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
gate() {
|
||||||
|
# Returns 0 if the repository is in a good state, 1 otherwise.
|
||||||
|
#
|
||||||
|
# The gate is uniformly strict, including for RED tasks. `Stage: RED` in this board
|
||||||
|
# means "write the failing test first, then implement until it passes" -- the task
|
||||||
|
# ends green. Every RED task's Verify block says `expected: passes`. So a failing
|
||||||
|
# suite is a failure at any stage, and relaxing the gate for RED would let
|
||||||
|
# half-finished work through.
|
||||||
|
local task_id="$1"
|
||||||
|
local stage
|
||||||
|
stage=$(grep -m1 '^Stage:' tasks/"$task_id"-*.md 2>/dev/null | awk '{print $2}')
|
||||||
|
|
||||||
|
CGO_ENABLED=0 go build ./... >"$LOG_DIR/gate-build.log" 2>&1 || { echo " FAIL: go build"; tail -20 "$LOG_DIR/gate-build.log"; return 1; }
|
||||||
|
go vet ./... >"$LOG_DIR/gate-vet.log" 2>&1 || { echo " FAIL: go vet"; tail -20 "$LOG_DIR/gate-vet.log"; return 1; }
|
||||||
|
|
||||||
|
go clean -testcache >/dev/null 2>&1
|
||||||
|
if ! go test ./... -race >"$LOG_DIR/gate-test.log" 2>&1; then
|
||||||
|
echo " FAIL: go test -race (stage=${stage:-?})"
|
||||||
|
grep -E '^(---|\s+---) FAIL|^FAIL' "$LOG_DIR/gate-test.log" | head -8 | sed 's/^/ /'
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo " build/vet/test all pass (stage=${stage:-?})"
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "started : $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
echo "tasks : ${#TASKS[@]}"
|
||||||
|
echo "model : ${PI_PROVIDER:-homelab-ornith}/${PI_MODEL:-ornith:35b}"
|
||||||
|
echo "logs : $LOG_DIR/"
|
||||||
|
echo "gate : build + vet + full race suite after every task; halts on failure"
|
||||||
|
echo "pid : $$"
|
||||||
|
echo
|
||||||
|
|
||||||
|
COMPLETED=()
|
||||||
|
for TASK_ID in "${TASKS[@]}"; do
|
||||||
|
echo "════════════════════════════════════════════"
|
||||||
|
echo "TASK $TASK_ID"
|
||||||
|
echo "════════════════════════════════════════════"
|
||||||
|
|
||||||
|
if ! ./scripts/run-task.sh "$TASK_ID" >"$LOG_DIR/$TASK_ID.log" 2>&1; then
|
||||||
|
echo " pi exited non-zero. Last output:"
|
||||||
|
tail -25 "$LOG_DIR/$TASK_ID.log"
|
||||||
|
echo
|
||||||
|
echo "HALTED at $TASK_ID (pi failure). Completed: ${COMPLETED[*]:-none}"
|
||||||
|
echo "Fix, then: scripts/run-all.sh --resume"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
tail -12 "$LOG_DIR/$TASK_ID.log" | sed 's/^/ | /'
|
||||||
|
echo
|
||||||
|
echo " verifying..."
|
||||||
|
|
||||||
|
if ! gate "$TASK_ID"; then
|
||||||
|
echo
|
||||||
|
echo "HALTED at $TASK_ID (gate failure). Completed: ${COMPLETED[*]:-none}"
|
||||||
|
echo "Full pi output: $LOG_DIR/$TASK_ID.log"
|
||||||
|
echo "Fix, then: scripts/run-all.sh --resume"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# grep -c prints 0 AND exits 1 when nothing matches, so a `|| echo 0` fallback
|
||||||
|
# would append a second line and make these two-line strings. Swallow the exit
|
||||||
|
# status instead and keep the single count grep already printed.
|
||||||
|
TICKED=$(grep -c '^- \[x\]' tasks/"$TASK_ID"-*.md 2>/dev/null) || true
|
||||||
|
TOTAL=$(grep -c '^- \[' tasks/"$TASK_ID"-*.md 2>/dev/null) || true
|
||||||
|
echo " gate OK checkboxes ticked: ${TICKED:-0}/${TOTAL:-0}"
|
||||||
|
[[ "$TICKED" != "$TOTAL" ]] && echo " NOTE: not all criteria ticked -- review before trusting this task"
|
||||||
|
|
||||||
|
COMPLETED+=("$TASK_ID")
|
||||||
|
grep -qxF "$TASK_ID" "$DONE_FILE" || echo "$TASK_ID" >>"$DONE_FILE"
|
||||||
|
echo
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "════════════════════════════════════════════"
|
||||||
|
echo "All ${#COMPLETED[@]} tasks passed the gate."
|
||||||
|
echo "Completed: ${COMPLETED[*]}"
|
||||||
|
echo
|
||||||
|
echo "Not run by this script (need a cluster, credentials, or live Kong):"
|
||||||
|
echo " 0.6 CI, 2.8 Kong parity, 3.x auth, 5.3 ServiceMonitor, 6.x deploy, 7.x prefixes"
|
||||||
Executable
+105
@@ -0,0 +1,105 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Run one task through pi in a brand-new, zero-context session.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# scripts/run-task.sh 0.1
|
||||||
|
# scripts/run-task.sh 0.1 --dry-run # print the prompt, run nothing
|
||||||
|
#
|
||||||
|
# Each invocation is a fresh pi session (--no-session). Nothing carries over between
|
||||||
|
# tasks; that is the point. See tasks/AGENT-PROMPT.md.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
cd "$REPO_ROOT"
|
||||||
|
|
||||||
|
TASK_ID="${1:-}"
|
||||||
|
DRY_RUN="${2:-}"
|
||||||
|
|
||||||
|
if [[ -z "$TASK_ID" ]]; then
|
||||||
|
echo "usage: scripts/run-task.sh <task-id> [--dry-run]" >&2
|
||||||
|
echo "example: scripts/run-task.sh 0.1" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Resolve the task file from its id prefix.
|
||||||
|
shopt -s nullglob
|
||||||
|
matches=(tasks/"$TASK_ID"-*.md)
|
||||||
|
shopt -u nullglob
|
||||||
|
|
||||||
|
if [[ ${#matches[@]} -eq 0 ]]; then
|
||||||
|
echo "no task file found for id '$TASK_ID'" >&2
|
||||||
|
echo "available:" >&2
|
||||||
|
ls tasks/[0-9]*.md | sed 's|tasks/| |' >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
if [[ ${#matches[@]} -gt 1 ]]; then
|
||||||
|
echo "ambiguous task id '$TASK_ID' matches:" >&2
|
||||||
|
printf ' %s\n' "${matches[@]}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
TASK_FILE="$(basename "${matches[0]}")"
|
||||||
|
|
||||||
|
# Extract the prompt body from the fenced block in AGENT-PROMPT.md, then substitute.
|
||||||
|
PROMPT="$(
|
||||||
|
awk '/^```$/{f=!f; next} f' tasks/AGENT-PROMPT.md \
|
||||||
|
| sed -e "s|{{TASK_ID}}|$TASK_ID|g" -e "s|{{TASK_FILE}}|$TASK_FILE|g"
|
||||||
|
)"
|
||||||
|
|
||||||
|
if [[ -z "${PROMPT// }" ]]; then
|
||||||
|
echo "failed to extract prompt template from tasks/AGENT-PROMPT.md" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == "--dry-run" ]]; then
|
||||||
|
printf '%s\n' "$PROMPT"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Default: claude-haiku-4-5. 200K context, reliable tool calling, needs
|
||||||
|
# ANTHROPIC_API_KEY in the environment.
|
||||||
|
#
|
||||||
|
# The two homelab models were both evaluated and rejected as defaults on 2026-08-19:
|
||||||
|
#
|
||||||
|
# reasoning 16384 context, and pi's compaction.reserveTokens defaults to 16384,
|
||||||
|
# so the compaction threshold computes to zero. pi also only evaluates
|
||||||
|
# auto-compaction at run boundaries, never mid-turn (earendil-works/pi
|
||||||
|
# issues 6339, 5512, 2871), so a tool loop grows unchecked until the
|
||||||
|
# provider rejects it. Separately, under tool_choice "auto" it returns
|
||||||
|
# tool_calls: [] and reasons in prose, which stalls an agent loop.
|
||||||
|
#
|
||||||
|
# ornith:35b 131K context and correct auto tool calling -- viable, and the right
|
||||||
|
# local choice. Occupied by other work at time of writing.
|
||||||
|
#
|
||||||
|
# Switch with env vars:
|
||||||
|
# PI_PROVIDER=homelab-ornith PI_MODEL=ornith:35b scripts/run-task.sh 0.1
|
||||||
|
#
|
||||||
|
PROVIDER="${PI_PROVIDER:-homelab-ornith}"
|
||||||
|
MODEL="${PI_MODEL:-ornith:35b}"
|
||||||
|
|
||||||
|
# One fresh session file per task. This still guarantees zero context -- each task gets
|
||||||
|
# its own new file and never reads another's -- while leaving the run inspectable:
|
||||||
|
#
|
||||||
|
# pi --session .task-runs/sessions/<task>.jsonl # open it
|
||||||
|
# pi --export .task-runs/sessions/<task>.jsonl out.html # render it
|
||||||
|
#
|
||||||
|
# --no-session would also give zero context, but persists nothing to look at afterwards.
|
||||||
|
SESSION_DIR=".task-runs/sessions"
|
||||||
|
mkdir -p "$SESSION_DIR"
|
||||||
|
SESSION_FILE="$SESSION_DIR/$TASK_ID.jsonl"
|
||||||
|
rm -f "$SESSION_FILE"
|
||||||
|
|
||||||
|
echo "task : $TASK_ID ($TASK_FILE)"
|
||||||
|
echo "model : $PROVIDER/$MODEL"
|
||||||
|
echo "session : $SESSION_FILE (fresh)"
|
||||||
|
echo
|
||||||
|
|
||||||
|
# --print is required, not cosmetic. Without it pi starts its interactive TUI;
|
||||||
|
# under run-all.sh stdout is a file and stdin may be /dev/null, so the TUI renders
|
||||||
|
# nothing until exit and can sit waiting for input that never comes. --approve
|
||||||
|
# trusts project-local files (AGENTS.md and friends) for this run, which would
|
||||||
|
# otherwise raise a prompt no one is there to answer.
|
||||||
|
exec pi --provider "$PROVIDER" --model "$MODEL" \
|
||||||
|
--print --approve \
|
||||||
|
--session "$SESSION_FILE" "$PROMPT"
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# 0.1 — Module and entrypoint (GREEN)
|
||||||
|
|
||||||
|
Phase: 0 — Foundations
|
||||||
|
Stage: GREEN
|
||||||
|
|
||||||
|
- [x] A single Go module builds one static binary with no cgo
|
||||||
|
- [x] The binary reads its configuration at startup and serves HTTP on a configurable listen address
|
||||||
|
- [x] `SIGTERM` starts a drain: the listener stops accepting new connections, in-flight requests run to completion, then the process exits `0`
|
||||||
|
- [x] A request already in flight when `SIGTERM` arrives receives its full, uncorrupted response body
|
||||||
|
- [x] A request arriving after `SIGTERM` is not accepted on a new connection
|
||||||
|
- [x] The drain has a bounded deadline; exceeding it forces exit with a non-zero code and a logged reason
|
||||||
|
- [x] The process holds no Kubernetes credentials and makes no API-server calls
|
||||||
|
|
||||||
|
The gateway sits behind ingress-nginx, which owns TLS. The gateway never terminates
|
||||||
|
TLS and never listens on 443. Graceful drain matters because in-flight requests here
|
||||||
|
are LLM generations that can legitimately run for many minutes -> killing them
|
||||||
|
mid-stream loses work a caller cannot cheaply redo.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/server/... -run TestGracefulShutdown -race -v
|
||||||
|
# expected: passes — a slow in-flight request completes with a full body after SIGTERM,
|
||||||
|
# and a request issued post-SIGTERM is refused; process exit code is 0
|
||||||
|
|
||||||
|
CGO_ENABLED=0 go build ./... && go vet ./...
|
||||||
|
# expected: both succeed
|
||||||
|
```
|
||||||
|
|
||||||
|
`-race` is required, not optional. A server that starts a listener in one goroutine and
|
||||||
|
exposes its address from another is the obvious shape here, and it is racy unless the
|
||||||
|
shared state is guarded. A test that passes without `-race` proves nothing about it.
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# 0.2 — Declarative route configuration (RED)
|
||||||
|
|
||||||
|
Phase: 0 — Foundations
|
||||||
|
Stage: RED
|
||||||
|
|
||||||
|
- [x] Routes and upstreams are declared in YAML loaded from a file path at startup
|
||||||
|
- [x] Each upstream declares: address, path rewrite, connect timeout, read timeout, write timeout, maximum request body size, and an auth-required flag
|
||||||
|
- [x] Every one of those fields is explicit — no silent defaults for timeouts, body caps or auth
|
||||||
|
- [x] A config missing any required field fails startup with a non-zero exit and a message naming the offending route and field
|
||||||
|
- [x] A config with a malformed duration, an unparseable address, or a duplicate route key fails startup the same way
|
||||||
|
- [x] A valid config round-trips: every declared route is present in the loaded route table
|
||||||
|
- [x] Loading is startup-only — no API-server watch, no CRD, no Kubernetes client
|
||||||
|
|
||||||
|
Configuration lives in git and is mounted as a ConfigMap synced by Argo. It is
|
||||||
|
deliberately not a CRD: a CRD would require the gateway to watch the API server,
|
||||||
|
which needs RBAC and contradicts the invariant that the gateway holds no cluster
|
||||||
|
credentials. It is also the exact indirection being retired with Kong, whose routing
|
||||||
|
table was split across six `KongPlugin` CRs, seven Ingresses and a Helm values file.
|
||||||
|
|
||||||
|
A gateway that starts with a silently dropped route is worse than one that refuses to
|
||||||
|
start.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/config/... -v
|
||||||
|
# expected: passes — valid fixtures load with all routes present; each invalid fixture
|
||||||
|
# returns an error naming the offending route and field, and none of them load partially
|
||||||
|
```
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# 0.3 — Health endpoints (GREEN)
|
||||||
|
|
||||||
|
Phase: 0 — Foundations
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [0.2](0.2-route-configuration.md)
|
||||||
|
|
||||||
|
- [x] `GET /healthz` returns `200` whenever the process is alive
|
||||||
|
- [x] `GET /healthz` contacts no upstream and performs no network I/O
|
||||||
|
- [x] `GET /readyz` returns `200` only when configuration is valid and, if auth is enabled, JWKS has been fetched at least once
|
||||||
|
- [x] `GET /readyz` returns a non-`2xx` status while configuration is invalid or JWKS has never been fetched
|
||||||
|
- [x] Neither endpoint requires authentication, even when the auth flag is on
|
||||||
|
- [x] Neither path is proxied to any upstream, and neither can be shadowed by a configured route
|
||||||
|
|
||||||
|
`/healthz` backs the liveness probe, so it must stay cheap and must not fail because
|
||||||
|
an upstream is down — restarting the gateway does not fix a sick vLLM pod. `/readyz`
|
||||||
|
backs the readiness probe and is allowed to fail, taking the pod out of the nginx
|
||||||
|
endpoint pool until it can actually serve.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/server/... -run TestHealthEndpoints -v
|
||||||
|
# expected: passes — /healthz is 200 with upstreams unreachable; /readyz is non-2xx
|
||||||
|
# before first JWKS fetch and 200 after; both answer with no Authorization header
|
||||||
|
```
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# 0.4 — Local development harness (GREEN)
|
||||||
|
|
||||||
|
Phase: 0 — Foundations
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [0.2](0.2-route-configuration.md)
|
||||||
|
|
||||||
|
- [ ] The whole gateway runs from a checkout with no cluster, no kubeconfig and no credentials of any kind
|
||||||
|
- [ ] A committed local config points every upstream at stub servers started by the harness
|
||||||
|
- [ ] Stubs can serve a fixed JSON body, an SSE token stream, a chunked response, and a slow response
|
||||||
|
- [ ] A test can assert on the real HTTP response: status, headers and body
|
||||||
|
- [ ] A test can assert that streamed chunks arrive incrementally, before the upstream has finished
|
||||||
|
- [ ] A test can disconnect the client mid-response and assert on what the stub upstream observed
|
||||||
|
- [ ] One documented command runs the harness end to end and exits non-zero on failure
|
||||||
|
- [ ] Running the harness never contacts `*.riotpiao.com` or any cluster address
|
||||||
|
|
||||||
|
This is a hard requirement, not a convenience: it determines whether work can proceed
|
||||||
|
unattended. Upstreams are configuration, so pointing them at local stubs is the entire
|
||||||
|
mechanism. Every later phase's verification depends on this existing first.
|
||||||
|
|
||||||
|
"It compiles" and "it starts" are not verification. Asserting on an actual HTTP
|
||||||
|
response is.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
env -u KUBECONFIG go test ./internal/testsupport/... ./internal/proxy/... -v
|
||||||
|
# expected: passes with no kubeconfig and no network access beyond loopback —
|
||||||
|
# includes an SSE test asserting incremental arrival and a mid-response disconnect test
|
||||||
|
```
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# 0.5 — Structured logging (GREEN)
|
||||||
|
|
||||||
|
Phase: 0 — Foundations
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [0.1](0.1-module-and-entrypoint.md)
|
||||||
|
|
||||||
|
- [ ] Logs are emitted as structured records with a consistent field set, one record per line
|
||||||
|
- [ ] Every request log carries at least: route, upstream, method, path, status, duration
|
||||||
|
- [ ] Every rejected request is logged with an explicit machine-readable reason field
|
||||||
|
- [ ] Request bodies are never logged, in whole or in part
|
||||||
|
- [ ] `Authorization` header values, bearer tokens, API keys and JWKS material are never logged, not even truncated or hashed-with-prefix
|
||||||
|
- [ ] Log level is configurable, and no level unlocks body or token logging
|
||||||
|
- [ ] A test asserts a rejected request produces exactly one record containing the reason and containing no token substring
|
||||||
|
|
||||||
|
Rejections come from several layers — unknown model, body too large, auth failure,
|
||||||
|
concurrency limit — and the reason field is what makes them countable later. The model
|
||||||
|
API carries prompts that are user content and tokens that are credentials; neither
|
||||||
|
belongs in a log line.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/logging/... ./internal/server/... -run 'TestLog' -v
|
||||||
|
# expected: passes — captured log output for a rejected request contains the reason
|
||||||
|
# field and does not contain the request body or the bearer token used
|
||||||
|
```
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# 0.6 — CI pipeline (GREEN)
|
||||||
|
|
||||||
|
Phase: 0 — Foundations
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [0.4](0.4-local-dev-harness.md)
|
||||||
|
|
||||||
|
- [ ] CI builds the binary on every push and pull request
|
||||||
|
- [ ] CI runs `go vet` over all packages and fails on any finding
|
||||||
|
- [ ] CI runs the full test suite, including the local harness tests, with the race detector on
|
||||||
|
- [ ] CI runs `govulncheck` and fails the job on any HIGH or CRITICAL severity finding
|
||||||
|
- [ ] CI needs no cluster, no kubeconfig and no credentials to pass
|
||||||
|
- [ ] A deliberately broken commit — failing test, vet finding, or known-vulnerable dependency — fails CI rather than passing silently
|
||||||
|
- [ ] Job status is visible on the commit or pull request
|
||||||
|
|
||||||
|
CI is the outer loop for the same closed verification loop the harness gives locally.
|
||||||
|
It must not depend on cluster access, or it stops running the moment the cluster is
|
||||||
|
unavailable.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go vet ./... && go test -race ./... && govulncheck ./...
|
||||||
|
# expected: all three exit 0 locally; pushing a branch with a failing test shows a
|
||||||
|
# failed CI run on that commit
|
||||||
|
```
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# 1.1 — Reverse proxy to configured upstreams (GREEN)
|
||||||
|
|
||||||
|
Phase: 1 — Proxy core
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [0.2](0.2-route-configuration.md), [0.4](0.4-local-dev-harness.md)
|
||||||
|
|
||||||
|
- [ ] A request matching a configured route is proxied to that route's upstream address
|
||||||
|
- [ ] The upstream's status code, response headers and body reach the client unmodified
|
||||||
|
- [ ] The request method, query string and body reach the upstream unmodified
|
||||||
|
- [ ] The route's configured path rewrite is applied to the upstream request path
|
||||||
|
- [ ] Connections to upstreams are pooled and reused across requests — a second request to the same upstream does not open a new TCP connection
|
||||||
|
- [ ] A request matching no configured route returns `404` and contacts no upstream
|
||||||
|
- [ ] An unreachable upstream returns a `5xx` to the client and is logged with the upstream name
|
||||||
|
|
||||||
|
Connection reuse is not a micro-optimisation here: the chat upstreams hold long-lived
|
||||||
|
streaming responses, and churning connections under that pattern wastes handshakes and
|
||||||
|
file descriptors on both ends.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/proxy/... -run 'TestProxy|TestConnectionReuse' -v
|
||||||
|
# expected: passes — stub upstream sees the rewritten path and original body, client
|
||||||
|
# sees the stub's exact status/headers/body, and the stub records one accepted
|
||||||
|
# connection across two sequential requests
|
||||||
|
```
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# 1.2 — Streaming passthrough (RED)
|
||||||
|
|
||||||
|
Phase: 1 — Proxy core
|
||||||
|
Stage: RED
|
||||||
|
Depends on: [1.1](1.1-reverse-proxy.md)
|
||||||
|
|
||||||
|
- [ ] An SSE response from an upstream reaches the client unbuffered: each `data:` event is readable by the client before the upstream has sent the next one
|
||||||
|
- [ ] A chunked response reaches the client chunk by chunk, not accumulated and flushed at completion
|
||||||
|
- [ ] Response headers reach the client before the first body byte, not after
|
||||||
|
- [ ] `Content-Type: text/event-stream` and the upstream's `Cache-Control` and `Connection` semantics survive the proxy
|
||||||
|
- [ ] No response body is written to memory or disk in full before forwarding
|
||||||
|
- [ ] The terminating `data: [DONE]` sentinel and the final zero-length chunk pass through
|
||||||
|
- [ ] A test asserts wall-clock ordering: the Nth event is observed at the client before the upstream emits the N+1th
|
||||||
|
|
||||||
|
The whole product is token streaming. If the gateway buffers, a caller waits minutes
|
||||||
|
for a response that should have started in seconds, and the user-visible behaviour of
|
||||||
|
the model API regresses versus Kong. nginx in front is already configured with
|
||||||
|
`proxy-buffering: off`; the gateway must not reintroduce buffering behind it.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/proxy/... -run TestSSEUnbuffered -v
|
||||||
|
# expected: passes — client observes each of 5 stub-emitted SSE events with the
|
||||||
|
# upstream still open, and total observed inter-event gaps match the stub's emit delays
|
||||||
|
```
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# 1.3 — Client disconnect propagation (RED)
|
||||||
|
|
||||||
|
Phase: 1 — Proxy core
|
||||||
|
Stage: RED
|
||||||
|
Depends on: [1.2](1.2-streaming-passthrough.md)
|
||||||
|
|
||||||
|
- [ ] When a client closes the connection mid-response, the gateway cancels the upstream request immediately
|
||||||
|
- [ ] The stub upstream observes its request context cancelled, not a request that runs to completion
|
||||||
|
- [ ] Cancellation happens within a small bounded delay of the client close, not at the route's read timeout
|
||||||
|
- [ ] The same holds for a non-streaming request abandoned before the upstream replies
|
||||||
|
- [ ] The disconnect is logged with a reason distinguishing it from an upstream error
|
||||||
|
- [ ] No goroutine or upstream connection is left alive after the disconnect — the test asserts this, not just the response
|
||||||
|
|
||||||
|
This is load-bearing. The `reasoning` upstream runs 2 replicas at `--max-num-seqs=4`,
|
||||||
|
which is 8 concurrent sequence slots cluster-wide. An orphaned generation holds one of
|
||||||
|
those 8 until it finishes on its own, which for a 32B model on a Volta GPU can be
|
||||||
|
minutes. A handful of abandoned browser tabs can starve the entire cluster.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/proxy/... -run TestClientDisconnectCancelsUpstream -race -v
|
||||||
|
# expected: passes — stub upstream reports context cancellation within 1s of the client
|
||||||
|
# closing mid-stream, and the post-test goroutine count returns to baseline
|
||||||
|
```
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# 1.4 — Per-route timeouts (GREEN)
|
||||||
|
|
||||||
|
Phase: 1 — Proxy core
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [0.2](0.2-route-configuration.md), [1.1](1.1-reverse-proxy.md)
|
||||||
|
|
||||||
|
- [ ] Connect, read and write timeouts are taken per route from configuration, never from a global default
|
||||||
|
- [ ] Chat routes use connect `10s`, read `1h`, write `1h`
|
||||||
|
- [ ] Embeddings and rerank routes use connect `10s`, read `10m`, write `10m`
|
||||||
|
- [ ] An upstream that never accepts a connection fails at the configured connect timeout, not later
|
||||||
|
- [ ] An upstream that accepts then stalls fails at the configured read timeout with a `5xx` and a logged reason
|
||||||
|
- [ ] A stream still emitting tokens is never cut by the read timeout — the timeout applies to inactivity, not total duration
|
||||||
|
- [ ] No code path shortens a configured proxy timeout to enforce an application-level cap
|
||||||
|
|
||||||
|
These are Kong's current values and they are deliberate. The 1-hour read timeout
|
||||||
|
exists because a 32B model on a Volta GPU routinely exceeds 60 seconds per request.
|
||||||
|
Any shorter application-level cap must be enforced by the gateway's own logic — a
|
||||||
|
budget, a slot limit, an explicit max-generation-time — and never by shortening the
|
||||||
|
proxy timeout, or long legitimate generations truncate mid-stream and callers see
|
||||||
|
corrupted output rather than an error.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/proxy/... -run 'TestConnectTimeout|TestReadTimeout|TestLongStreamNotTruncated' -v
|
||||||
|
# expected: passes — stalled upstream errors at the configured read timeout, a stub
|
||||||
|
# emitting one event every 200ms for longer than the timeout window is not cut off,
|
||||||
|
# and a blackholed address fails at ~10s
|
||||||
|
```
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# 1.5 — Header hygiene (GREEN)
|
||||||
|
|
||||||
|
Phase: 1 — Proxy core
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [1.1](1.1-reverse-proxy.md)
|
||||||
|
|
||||||
|
- [ ] Hop-by-hop headers are stripped from both the upstream request and the client response
|
||||||
|
- [ ] Headers named in a request's `Connection` header are also stripped, not just the fixed hop-by-hop list
|
||||||
|
- [ ] `X-Forwarded-For` appends the immediate peer to the nginx-supplied value rather than replacing or fabricating it
|
||||||
|
- [ ] `X-Forwarded-Proto` and `X-Forwarded-Host` are taken from the nginx-supplied values when present
|
||||||
|
- [ ] Client-supplied `X-Forwarded-*` values are not trusted when the request did not arrive from the trusted ingress peer
|
||||||
|
- [ ] End-to-end headers, including `Content-Type`, `Authorization` where the route requires it, and upstream response headers, pass through unchanged
|
||||||
|
- [ ] A test asserts the exact header set the stub upstream receives
|
||||||
|
|
||||||
|
ingress-nginx owns TLS and the edge, so it is the only source of truth for the
|
||||||
|
original scheme, host and client address. The gateway fabricating these would make
|
||||||
|
every upstream's view of the caller wrong, and would let a client spoof its own
|
||||||
|
source address by sending the header itself.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/proxy/... -run TestHeaderHygiene -v
|
||||||
|
# expected: passes — stub upstream sees no Connection/Keep-Alive/TE/Upgrade/
|
||||||
|
# Proxy-Authorization headers, sees X-Forwarded-For ending in the nginx-supplied value
|
||||||
|
# plus the peer, and a spoofed X-Forwarded-Proto from an untrusted peer is discarded
|
||||||
|
```
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# 1.6 — WebSocket upgrade (GREEN)
|
||||||
|
|
||||||
|
Phase: 1 — Proxy core
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [1.5](1.5-header-hygiene.md)
|
||||||
|
|
||||||
|
- [ ] A route may be configured to allow protocol upgrade
|
||||||
|
- [ ] An upgrade request on such a route reaches the upstream with its `Upgrade` and `Connection` headers intact, despite hop-by-hop stripping
|
||||||
|
- [ ] The upstream's `101 Switching Protocols` response reaches the client, and bytes then flow bidirectionally
|
||||||
|
- [ ] Frames pass in both directions with no buffering delay
|
||||||
|
- [ ] Client close propagates to the upstream and upstream close propagates to the client
|
||||||
|
- [ ] An upgrade attempt on a route that does not allow it is rejected, not silently downgraded to a plain proxied request
|
||||||
|
- [ ] Idle upgraded connections are not cut by the route's read timeout while frames are still flowing
|
||||||
|
|
||||||
|
`agent-pod/console` serves a WebSocket and is one of the seven Kong-class Ingresses
|
||||||
|
being migrated. It is currently publicly routed and unauthenticated into a
|
||||||
|
shell-capable container, so it must work through the gateway before it can be put
|
||||||
|
behind gateway auth. Header hygiene and upgrade support interact directly: `Upgrade`
|
||||||
|
and `Connection` are hop-by-hop, and a naive strip breaks the handshake.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/proxy/... -run TestWebSocketUpgrade -v
|
||||||
|
# expected: passes — client receives 101 from the stub, an echoed frame round-trips in
|
||||||
|
# both directions, and closing the client causes the stub to observe a close
|
||||||
|
```
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# 1.7 — Per-route body size caps (RED)
|
||||||
|
|
||||||
|
Phase: 1 — Proxy core
|
||||||
|
Stage: RED
|
||||||
|
Depends on: [0.2](0.2-route-configuration.md), [1.1](1.1-reverse-proxy.md)
|
||||||
|
|
||||||
|
- [ ] Each route enforces its own configured maximum request body size
|
||||||
|
- [ ] A body over the cap is rejected with `413` and a body the upstream never sees
|
||||||
|
- [ ] Rejection happens while reading, not after buffering the whole body into memory
|
||||||
|
- [ ] A request with a lying or absent `Content-Length` is still capped by bytes actually read
|
||||||
|
- [ ] A body at exactly the cap is accepted and proxied intact
|
||||||
|
- [ ] The rejection is logged with a reason distinguishing it from other rejections
|
||||||
|
- [ ] No global default cap silently applies to a route that failed to declare one — that is a config error, per 0.2
|
||||||
|
|
||||||
|
nginx in front is configured with `proxy-body-size: 0`, meaning it enforces no limit
|
||||||
|
at all, so the gateway is the only place a cap exists. Embedding and rerank callers
|
||||||
|
can send large batches legitimately, which is why the cap is per route rather than
|
||||||
|
one number for the whole surface.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/proxy/... -run TestBodySizeCap -v
|
||||||
|
# expected: passes — a body one byte over the route cap returns 413 and the stub
|
||||||
|
# upstream records zero requests; a body exactly at the cap returns the stub's 200
|
||||||
|
```
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# 2.1 — Model registry (GREEN)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surface
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [0.2](0.2-route-configuration.md), [1.1](1.1-reverse-proxy.md)
|
||||||
|
|
||||||
|
- [ ] A model name -> upstream map is loaded from configuration at startup, never compiled in
|
||||||
|
- [ ] Each entry carries at minimum the model name clients send, the upstream address, and the upstream path to use
|
||||||
|
- [ ] Two model names may point at the same upstream address, and both resolve independently
|
||||||
|
- [ ] A duplicate model name, an empty model name, or an entry with no upstream address fails startup loudly with the offending entry named
|
||||||
|
- [ ] The registry is queryable by exact model name; lookup is case-sensitive and does no fuzzy matching or defaulting
|
||||||
|
- [ ] The set of known model names is enumerable, because `/v1/models` and unknown-model errors are both derived from it
|
||||||
|
|
||||||
|
The five entries verified live on 2026-08-19. Ports are 80, not 8080.
|
||||||
|
|
||||||
|
| model name clients send | upstream Service | engine |
|
||||||
|
|---|---|---|
|
||||||
|
| `reasoning` | `reasoning-predictor.llm-serving:80` | vLLM, DeepSeek-R1-Distill-Qwen-32B |
|
||||||
|
| `ornith:35b` | `ornith-predictor.llm-serving:80` | Ollama |
|
||||||
|
| `qwen2.5:3b-instruct` | `ornith-predictor.llm-serving:80` | Ollama, same pods |
|
||||||
|
| `nomic-ai/nomic-embed-text-v2-moe` | `embeddings-predictor.llm-serving:80` | TEI |
|
||||||
|
| `BAAI/bge-reranker-base` | `reranker-predictor.llm-serving:80` | TEI |
|
||||||
|
|
||||||
|
`ornith:35b` and `qwen2.5:3b-instruct` share pods and both stay resident, so a
|
||||||
|
registry that maps them to one address is correct, not a shortcut.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# with the five entries configured against local stubs
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/readyz # expected: 200
|
||||||
|
|
||||||
|
# duplicate model name in config must refuse to start
|
||||||
|
./gateway --config testdata/duplicate-model.yaml; echo "exit=$?" # expected: non-zero exit, stderr names the duplicated model
|
||||||
|
```
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# 2.10 — Anthropic request translation (GREEN)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surfaces
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [2.9](2.9-canonical-request-model.md), [2.1](2.1-model-registry.md)
|
||||||
|
|
||||||
|
`POST /llm/v1/messages` accepts an Anthropic Messages request body and turns it into
|
||||||
|
the dialect-neutral canonical request. The path carries `/v1/messages` because the
|
||||||
|
Anthropic base-URL convention appends that suffix; the gateway prefix is `/llm`.
|
||||||
|
|
||||||
|
- [ ] `POST /llm/v1/messages` is accepted and selects its upstream from the body's
|
||||||
|
`model` field, using the same registry as `/v1`: `reasoning` reaches
|
||||||
|
`reasoning-predictor.llm-serving:80`, `ornith:35b` and `qwen2.5:3b-instruct`
|
||||||
|
reach `ornith-predictor.llm-serving:80`
|
||||||
|
- [ ] The top-level `system` field becomes the canonical system instruction; it is a
|
||||||
|
distinct field in this dialect and is not a member of `messages`
|
||||||
|
- [ ] Each message `content` may be a plain string or an array of blocks; a string and
|
||||||
|
a single text block carrying the same characters translate identically
|
||||||
|
- [ ] `max_tokens` is REQUIRED on this surface, matching the Anthropic contract; a
|
||||||
|
request without it is rejected as a client error and no upstream is contacted
|
||||||
|
- [ ] `max_tokens` above what the upstream can serve is clamped rather than rejected,
|
||||||
|
and the clamp is logged; `reasoning` has `--max-model-len=16384`
|
||||||
|
- [ ] Roles are restricted to `user` and `assistant`; any other role, including
|
||||||
|
`system` inside `messages`, is a client error naming the offending role
|
||||||
|
- [ ] `stop_sequences` becomes the canonical stop sequences, and `stream` becomes the
|
||||||
|
canonical streaming flag
|
||||||
|
- [ ] `tools`, `tool_choice`, any `tool_use` or `tool_result` block, any `image` block,
|
||||||
|
any cache-control marker, and any user message with more than one content block
|
||||||
|
are rejected as unsupported, naming the feature; none is silently dropped
|
||||||
|
- [ ] Unknown top-level fields are rejected rather than ignored, so a client cannot
|
||||||
|
believe an unimplemented option took effect
|
||||||
|
- [ ] Reading the body respects the route's configured body size cap
|
||||||
|
|
||||||
|
`max_tokens` stays optional on `/v1/chat/completions`. The policy is per-surface: each
|
||||||
|
dialect keeps its own contract, and the canonical request records whichever value
|
||||||
|
resulted.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/llm/v1/messages \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":64,"system":"be terse","messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
# expected: 200, reasoning stub hit, system text present in the upstream request
|
||||||
|
|
||||||
|
curl -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
# expected: 400, body names max_tokens as the missing required field, no upstream stub hit
|
||||||
|
|
||||||
|
curl -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":8,"messages":[{"role":"system","content":"x"}]}'
|
||||||
|
# expected: 400, body names the rejected role, no upstream stub hit
|
||||||
|
```
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# 2.11 — Anthropic non-streaming response translation (GREEN)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surfaces
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [2.10](2.10-anthropic-request-translation.md), [2.9](2.9-canonical-request-model.md)
|
||||||
|
|
||||||
|
A non-streaming `POST /llm/v1/messages` gets an Anthropic Messages response, built
|
||||||
|
from whatever the upstream returned. Upstreams speak the OpenAI chat-completion shape;
|
||||||
|
the client on this surface must never see it.
|
||||||
|
|
||||||
|
- [ ] The response body is `{"id","type":"message","role":"assistant","content":[...],
|
||||||
|
"model","stop_reason","stop_sequence","usage":{"input_tokens","output_tokens"}}`
|
||||||
|
with `type` literally `message` and `role` literally `assistant`
|
||||||
|
- [ ] `model` echoes the model name the client sent, not an upstream-internal name
|
||||||
|
- [ ] Upstream `finish_reason` `stop` becomes `stop_reason` `end_turn`, and `length`
|
||||||
|
becomes `max_tokens`
|
||||||
|
- [ ] A generation halted by a client-supplied stop sequence reports `stop_reason`
|
||||||
|
`stop_sequence` and puts the matched string in `stop_sequence`; otherwise
|
||||||
|
`stop_sequence` is null and present, not omitted
|
||||||
|
- [ ] Upstream `prompt_tokens` becomes `usage.input_tokens` and `completion_tokens`
|
||||||
|
becomes `usage.output_tokens`; no other usage fields are invented
|
||||||
|
- [ ] `reasoning_content`, which vLLM returns as a field separate from `content` for
|
||||||
|
`reasoning`, becomes a `thinking` content block that PRECEDES the `text` block
|
||||||
|
- [ ] When `reasoning_content` is absent or empty, no `thinking` block is emitted and
|
||||||
|
`content` holds only the `text` block
|
||||||
|
- [ ] When `content` is empty but `reasoning_content` is not, the `thinking` block is
|
||||||
|
still returned rather than an empty `content` array
|
||||||
|
- [ ] `Content-Type` is `application/json`, and no OpenAI field name such as `choices`,
|
||||||
|
`finish_reason` or `object` appears anywhere in the body
|
||||||
|
|
||||||
|
`reasoning` runs DeepSeek-R1-Distill-Qwen-32B under vLLM with
|
||||||
|
`--reasoning-parser=deepseek_r1`, which is why the reasoning text arrives as its own
|
||||||
|
field and maps cleanly onto a thinking block.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":64,"messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
# expected: 200, type=message, role=assistant, content[0].type=thinking, content[1].type=text
|
||||||
|
|
||||||
|
curl -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":64,"messages":[{"role":"user","content":"hi"}]}' \
|
||||||
|
| grep -c -E '"choices"|"finish_reason"|"object"'
|
||||||
|
# expected: 0
|
||||||
|
|
||||||
|
curl -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"qwen2.5:3b-instruct","max_tokens":4,"messages":[{"role":"user","content":"count to fifty"}]}'
|
||||||
|
# expected: 200, stop_reason=max_tokens, usage has input_tokens and output_tokens only
|
||||||
|
```
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
# 2.12 — Anthropic SSE state machine (RED)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surfaces
|
||||||
|
Stage: RED
|
||||||
|
Depends on: [2.11](2.11-anthropic-response-translation.md), [1.2](1.2-streaming-passthrough.md), [1.3](1.3-disconnect-propagation.md)
|
||||||
|
|
||||||
|
`POST /llm/v1/messages` with `"stream":true` must emit Anthropic SSE. Anthropic uses
|
||||||
|
NAMED events carrying content-block indices; the upstream emits flat OpenAI data-only
|
||||||
|
chunks. Write the failing tests against the event sequence before writing a translator.
|
||||||
|
|
||||||
|
- [ ] Every frame has both an `event:` line and a `data:` line; a bare `data:` frame is
|
||||||
|
a failure on this surface
|
||||||
|
- [ ] `Content-Type` is `text/event-stream`
|
||||||
|
- [ ] Event order for a full response is exactly: `message_start`, then for each block
|
||||||
|
`content_block_start`, one or more `content_block_delta`, `content_block_stop`,
|
||||||
|
then `message_delta`, then `message_stop`
|
||||||
|
- [ ] `message_start` carries the message envelope with the client-sent model, `role`
|
||||||
|
`assistant`, empty `content`, and `usage.input_tokens`
|
||||||
|
- [ ] The block carrying `reasoning_content` is index 0 with block type `thinking`, and
|
||||||
|
its deltas are `thinking_delta`
|
||||||
|
- [ ] The block carrying `content` is index 1 with block type `text`, and its deltas
|
||||||
|
are `text_delta`
|
||||||
|
- [ ] The end of reasoning is only knowable when `content` first arrives, so the
|
||||||
|
arrival of the first `content` token MUST emit `content_block_stop` for index 0
|
||||||
|
before `content_block_start` for index 1 — the two blocks never overlap
|
||||||
|
- [ ] If a response has no `reasoning_content` at all, the text block is index 0 and no
|
||||||
|
thinking block is started; indices are assigned in emission order with no gaps
|
||||||
|
- [ ] If a response has `reasoning_content` and never any `content`, the thinking block
|
||||||
|
is still closed before `message_delta`
|
||||||
|
- [ ] `message_delta` carries `stop_reason` and `usage.output_tokens`; upstream
|
||||||
|
`finish_reason` `stop` becomes `end_turn` and `length` becomes `max_tokens`
|
||||||
|
- [ ] `message_stop` is the final frame and is emitted exactly once per response
|
||||||
|
- [ ] Translation is streaming and unbuffered: each upstream chunk is converted and
|
||||||
|
flushed as it arrives, and the response is never accumulated to be inspected
|
||||||
|
- [ ] A client disconnect mid-stream cancels the upstream request immediately and
|
||||||
|
releases the slot, rather than orphaning the generation
|
||||||
|
- [ ] An upstream failure after `message_start` terminates the stream with an error
|
||||||
|
frame rather than a truncated but apparently successful sequence
|
||||||
|
- [ ] The OpenAI `data: [DONE]` sentinel is consumed by the translator and never
|
||||||
|
forwarded to a `/llm` client
|
||||||
|
|
||||||
|
An orphaned generation holds one of only eight vLLM sequence slots in the cluster,
|
||||||
|
which is why disconnect cancellation is an acceptance criterion here and not only in
|
||||||
|
the proxy layer.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -N -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":64,"stream":true,"messages":[{"role":"user","content":"hi"}]}' \
|
||||||
|
| grep '^event:'
|
||||||
|
# expected: message_start, content_block_start, content_block_delta..., content_block_stop,
|
||||||
|
# content_block_start, content_block_delta..., content_block_stop, message_delta, message_stop
|
||||||
|
|
||||||
|
curl -N -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":64,"stream":true,"messages":[{"role":"user","content":"hi"}]}' \
|
||||||
|
| grep -n -E 'content_block_stop|"index":1' | head -3
|
||||||
|
# expected: the index 0 content_block_stop line precedes the first line mentioning index 1
|
||||||
|
|
||||||
|
timeout 1 curl -N -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":512,"stream":true,"messages":[{"role":"user","content":"long"}]}' >/dev/null
|
||||||
|
grep -c 'cancelled' /tmp/stub-reasoning.log
|
||||||
|
# expected: 1 within a second of the client going away
|
||||||
|
```
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# 2.13 — Anthropic error shape on `/llm/*` (RED)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surfaces
|
||||||
|
Stage: RED
|
||||||
|
Depends on: [2.10](2.10-anthropic-request-translation.md), [4.3](4.3-problem-json-errors.md)
|
||||||
|
|
||||||
|
`/v1/*` renders rejections as RFC 9457 `application/problem+json`. `/llm/*` must not.
|
||||||
|
The same underlying rejection gets two renderings, chosen purely by which surface
|
||||||
|
received the request. Write the failing tests against both renderings first.
|
||||||
|
|
||||||
|
- [ ] Every error response from a `/llm/*` path has the body
|
||||||
|
`{"type":"error","error":{"type":"...","message":"..."}}` and
|
||||||
|
`Content-Type: application/json`
|
||||||
|
- [ ] No `/llm/*` response ever carries `application/problem+json`, and no `/v1/*`
|
||||||
|
response ever carries the Anthropic error shape
|
||||||
|
- [ ] An unknown or missing `model` returns `invalid_request_error` with a message
|
||||||
|
naming the rejected value and listing the configured model names, derived from
|
||||||
|
the registry rather than hardcoded
|
||||||
|
- [ ] A request missing the required `max_tokens` returns `invalid_request_error`
|
||||||
|
naming `max_tokens`
|
||||||
|
- [ ] A body that is not valid JSON returns `invalid_request_error` with a message
|
||||||
|
distinguishable from the unknown-model case
|
||||||
|
- [ ] A request for an out-of-scope feature returns `invalid_request_error` naming the
|
||||||
|
feature, for example tools, images or prompt caching
|
||||||
|
- [ ] Exhausting the shared slot queue returns HTTP 429 with error type
|
||||||
|
`rate_limit_error` and a `Retry-After` header
|
||||||
|
- [ ] An upstream failure or timeout returns HTTP 5xx with error type `api_error`, and
|
||||||
|
the message leaks no upstream host, port or internal path
|
||||||
|
- [ ] No rejection reaches an upstream, and none falls back to a default model
|
||||||
|
- [ ] Each rejection is logged with its reason and its surface; the request body is
|
||||||
|
never logged
|
||||||
|
- [ ] The same malformed request sent to `/v1/chat/completions` and `/llm/v1/messages`
|
||||||
|
yields the same HTTP status with the two different body shapes
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -i localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"gpt-4","max_tokens":8,"messages":[]}'
|
||||||
|
# expected: 4xx, content-type application/json, body {"type":"error","error":{"type":"invalid_request_error",...}}
|
||||||
|
# listing reasoning, ornith:35b, qwen2.5:3b-instruct
|
||||||
|
|
||||||
|
curl -s -i localhost:8080/llm/v1/messages -H 'content-type: application/json' -d 'not json' \
|
||||||
|
| grep -i 'content-type'
|
||||||
|
# expected: application/json, never application/problem+json
|
||||||
|
|
||||||
|
curl -s -i localhost:8080/v1/chat/completions -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"gpt-4","messages":[]}' | grep -i 'content-type'
|
||||||
|
# expected: application/problem+json, confirming the two surfaces render differently
|
||||||
|
```
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
# 2.14 — Queue position event on `/llm/*` (GREEN)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surfaces
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [2.12](2.12-anthropic-sse-state-machine.md), [4.1](4.1-gpu-slot-semaphore.md)
|
||||||
|
|
||||||
|
The Anthropic event set has no way to say "you are queued". Its stream implicitly
|
||||||
|
begins after a slot has been acquired, so a queued client sees nothing at all until
|
||||||
|
generation starts. The `reasoning` upstream has only 8 sequence slots cluster-wide and
|
||||||
|
the gateway caps below that, so waiting is normal and worth showing.
|
||||||
|
|
||||||
|
- [ ] When a streaming `/llm/v1/messages` request waits for a slot, the gateway emits a
|
||||||
|
frame with `event: queue` before any `message_start`
|
||||||
|
- [ ] The queue frame's data carries the caller's current position in the queue
|
||||||
|
- [ ] Position updates are emitted as the queue drains, each as another `event: queue`
|
||||||
|
frame, until a slot is acquired
|
||||||
|
- [ ] Once a slot is acquired the stream continues with the standard sequence beginning
|
||||||
|
at `message_start`, and no further queue frame is emitted for that request
|
||||||
|
- [ ] A request that acquires a slot immediately emits no queue frame at all
|
||||||
|
- [ ] Queue frames are flushed as they are produced, not buffered behind the first
|
||||||
|
upstream token
|
||||||
|
- [ ] A client that disconnects while still queued is removed from the queue, never
|
||||||
|
reaches the upstream and never consumes a slot
|
||||||
|
- [ ] Non-streaming requests emit no queue frames; they simply wait, then answer
|
||||||
|
- [ ] The extension is documented in the surface's own docs as non-standard, alongside
|
||||||
|
the fact that a strict Anthropic client ignoring unknown events degrades to
|
||||||
|
showing nothing while queued rather than erroring
|
||||||
|
|
||||||
|
This is a deliberate departure from the Anthropic contract. It is safe only because
|
||||||
|
the sole client of `/llm/*` is the first-party riotpiao frontend. It must never be
|
||||||
|
required for correctness: dropping every `event: queue` frame leaves a valid,
|
||||||
|
complete Anthropic stream.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Fill the slots first, with cap=2 configured against a stub that holds each request 3s.
|
||||||
|
seq 4 | xargs -P4 -I{} curl -s -o /dev/null -X POST localhost:8080/llm/v1/messages \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"hi"}]}' &
|
||||||
|
|
||||||
|
sleep 1
|
||||||
|
curl -N -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"hi"}]}' \
|
||||||
|
| grep -m3 '^event:'
|
||||||
|
# expected: event: queue arrives within a second, before any event: message_start
|
||||||
|
|
||||||
|
curl -N -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"qwen2.5:3b-instruct","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"hi"}]}' \
|
||||||
|
| head -1
|
||||||
|
# expected: event: message_start, no queue frame on an uncontended upstream
|
||||||
|
```
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# 2.15 — Anthropic dialect scope boundary (GREEN)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surfaces
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [2.10](2.10-anthropic-request-translation.md), [2.13](2.13-anthropic-error-shape.md)
|
||||||
|
|
||||||
|
The only client of `/llm/*` is the first-party riotpiao frontend. It is not Claude
|
||||||
|
Code and not the Anthropic SDK, so the surface implements a deliberately narrow slice
|
||||||
|
of the Messages API. The narrowness is the design; the risk is a half-built feature
|
||||||
|
that appears to work.
|
||||||
|
|
||||||
|
- [ ] The unsupported set is enumerated in one place and covers at least: `tools` and
|
||||||
|
`tool_choice`, `tool_use` blocks, `tool_result` turns, image content blocks,
|
||||||
|
prompt-caching controls and cache headers, the batch API, and user messages
|
||||||
|
carrying more than one content block
|
||||||
|
- [ ] Each unsupported feature is detected during request translation, before any
|
||||||
|
upstream is contacted
|
||||||
|
- [ ] Rejection uses the Anthropic error shape
|
||||||
|
`{"type":"error","error":{"type":"invalid_request_error","message":"..."}}` and
|
||||||
|
the message names the specific unsupported feature, not just "unsupported"
|
||||||
|
- [ ] No unsupported feature is silently ignored, stripped, or partially honoured; a
|
||||||
|
request containing one never produces a 200
|
||||||
|
- [ ] A request combining a supported and an unsupported field is rejected, not
|
||||||
|
serviced with the unsupported part dropped
|
||||||
|
- [ ] The `/v1/*` OpenAI surface is unaffected: tool calling continues to work there
|
||||||
|
exactly as it does today
|
||||||
|
- [ ] Every entry in the unsupported set has a test asserting the rejection, so
|
||||||
|
widening scope forces a deliberate test change rather than a quiet code change
|
||||||
|
- [ ] The list is documented on the surface, so the frontend can see the boundary
|
||||||
|
without reading gateway code
|
||||||
|
|
||||||
|
Enabling any of these later must be an explicit config or code change accompanied by
|
||||||
|
its own translation work and tests. A silent partial implementation is the specific
|
||||||
|
failure this task exists to prevent.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":8,"tools":[{"name":"x"}],"messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
# expected: 4xx, error.message names tools, no upstream stub hit
|
||||||
|
|
||||||
|
curl -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":8,"messages":[{"role":"user","content":[{"type":"image","source":{}}]}]}'
|
||||||
|
# expected: 4xx, error.message names image content blocks, no upstream stub hit
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","messages":[{"role":"user","content":"hi"}],"tools":[{"type":"function","function":{"name":"x"}}]}'
|
||||||
|
# expected: 200, tool calling still works on the OpenAI surface
|
||||||
|
```
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# 2.2 — Body-based model dispatch (GREEN)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surface
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [2.1](2.1-model-registry.md), [1.1](1.1-reverse-proxy.md), [1.2](1.2-streaming-passthrough.md)
|
||||||
|
|
||||||
|
- [ ] `POST /v1/chat/completions` selects its upstream from the `model` field of the JSON request body
|
||||||
|
- [ ] The body reaching the upstream is byte-identical to the body received, `model` included — dispatch reads, it does not rewrite
|
||||||
|
- [ ] The upstream sees the canonical path `/v1/chat/completions`
|
||||||
|
- [ ] `"model":"reasoning"` reaches `reasoning-predictor.llm-serving:80`; `"model":"ornith:35b"` and `"model":"qwen2.5:3b-instruct"` both reach `ornith-predictor.llm-serving:80`
|
||||||
|
- [ ] `"stream":true` streams unbuffered — chunks reach the client as the upstream emits them, and are not accumulated in order to inspect the body
|
||||||
|
- [ ] A client disconnect mid-stream cancels the upstream request rather than orphaning it
|
||||||
|
- [ ] `reasoning_content` is passed through untouched alongside `content`; the gateway does not merge, reorder or strip either
|
||||||
|
- [ ] Reading the body to find `model` respects the route's body size cap and does not load an unbounded request into memory
|
||||||
|
|
||||||
|
This is the single capability Kong OSS lacked — `ai-proxy-advanced` is Enterprise-only —
|
||||||
|
and the entire reason this gateway exists. Everything else in this phase is a
|
||||||
|
consequence of it. Ports are 80, not 8080.
|
||||||
|
|
||||||
|
An orphaned generation holds one of only eight vLLM sequence slots in the cluster,
|
||||||
|
which is why disconnect cancellation belongs in the acceptance criteria of this task
|
||||||
|
and not only in the proxy layer.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
# expected: 200, stub for reasoning-predictor recorded the hit with path /v1/chat/completions and an unmodified body
|
||||||
|
|
||||||
|
curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"qwen2.5:3b-instruct","messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
# expected: 200, ornith-predictor stub hit, reasoning-predictor stub not hit
|
||||||
|
|
||||||
|
curl -N -s localhost:8080/v1/chat/completions -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","stream":true,"messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
# expected: 200, content-type text/event-stream, first data: chunk arrives before the stub finishes emitting
|
||||||
|
```
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# 2.3 — Unknown and missing model errors (RED)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surface
|
||||||
|
Stage: RED
|
||||||
|
Depends on: [2.1](2.1-model-registry.md), [2.2](2.2-body-based-dispatch.md)
|
||||||
|
|
||||||
|
- [ ] `POST /v1/chat/completions` with a `model` that is not in the configured registry returns a 4xx client error, never 5xx
|
||||||
|
- [ ] A request with no `model` field, a null `model`, or an empty-string `model` is the same class of client error
|
||||||
|
- [ ] A body that is not valid JSON is also a client error, distinguishable from an unknown model
|
||||||
|
- [ ] The response is RFC 9457 with `Content-Type: application/problem+json`
|
||||||
|
- [ ] The problem body names the rejected value and enumerates every currently configured model name, derived from the registry rather than written out by hand
|
||||||
|
- [ ] No fallback to a default model happens under any of these conditions, and no upstream is contacted
|
||||||
|
- [ ] The rejection is logged with its reason; the request body is not logged
|
||||||
|
|
||||||
|
Write the failing tests first. A silent fallback to a default model is the specific
|
||||||
|
failure mode this task exists to prevent: it turns a client typo into a bill against
|
||||||
|
the wrong GPU and hides the mistake from the caller.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -i localhost:8080/v1/chat/completions -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"gpt-4","messages":[]}'
|
||||||
|
# expected: 400 (or 404), content-type: application/problem+json, body lists reasoning, ornith:35b, qwen2.5:3b-instruct, and the embedding/rerank models
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' -d '{"messages":[]}'
|
||||||
|
# expected: 4xx, no upstream stub recorded any hit
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' -d 'not json'
|
||||||
|
# expected: 400, problem+json, reason distinct from unknown-model
|
||||||
|
```
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# 2.4 — Legacy path aliases (GREEN)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surface
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [2.2](2.2-body-based-dispatch.md)
|
||||||
|
|
||||||
|
- [ ] `POST /v1/reasoning/chat/completions` behaves as the canonical endpoint with `model` forced to `reasoning`
|
||||||
|
- [ ] `POST /v1/ornith/chat/completions` behaves as the canonical endpoint with `model` forced to `ornith:35b`
|
||||||
|
- [ ] `POST /v1/qwen/chat/completions` behaves as the canonical endpoint with `model` forced to `qwen2.5:3b-instruct`
|
||||||
|
- [ ] The forced value overrides whatever `model` the body carries, including a conflicting one, and the upstream receives the forced value
|
||||||
|
- [ ] The upstream sees the canonical path `/v1/chat/completions`, not the alias path
|
||||||
|
- [ ] Streaming, disconnect cancellation and `reasoning_content` passthrough behave identically to the canonical endpoint
|
||||||
|
- [ ] The alias set is configuration, so removal is a config change and needs no code change
|
||||||
|
- [ ] Each alias is marked temporary where it is defined, with the condition for removal stated: all callers migrated
|
||||||
|
|
||||||
|
These three paths exist only because Kong OSS could not dispatch on the request body.
|
||||||
|
They keep the cutover reversible — pi is a live caller today. They are deleted once
|
||||||
|
callers have moved to `POST /v1/chat/completions` with `model` in the body.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/reasoning/chat/completions \
|
||||||
|
-H 'content-type: application/json' -d '{"messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
# expected: 200, reasoning-predictor stub hit at /v1/chat/completions with body model=reasoning
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/qwen/chat/completions \
|
||||||
|
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}'
|
||||||
|
# expected: 200, ornith-predictor stub hit with body model=qwen2.5:3b-instruct — the body's "reasoning" is overridden
|
||||||
|
```
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# 2.5 — `GET /v1/models` (GREEN)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surface
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [2.1](2.1-model-registry.md)
|
||||||
|
|
||||||
|
- [ ] `GET /v1/models` returns 200 with `Content-Type: application/json`
|
||||||
|
- [ ] The response shape is `{"object":"list","data":[{"id","object":"model","owned_by","created"}]}`
|
||||||
|
- [ ] Every entry's `object` is the literal string `model`
|
||||||
|
- [ ] The `id` values are exactly the model names the configured registry will accept for dispatch — no more, no fewer
|
||||||
|
- [ ] Adding or removing a model in configuration changes this response with no code change
|
||||||
|
- [ ] No model list is hardcoded anywhere; the list cannot disagree with what routing accepts
|
||||||
|
- [ ] The endpoint contacts no upstream and stays cheap
|
||||||
|
|
||||||
|
Kong served a static list via `request-termination`, and its own manifest flags that
|
||||||
|
the list can drift from what the engines actually serve. Deriving from the registry
|
||||||
|
makes that drift structurally impossible: the same source answers this endpoint and
|
||||||
|
decides which `model` values dispatch.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s localhost:8080/v1/models
|
||||||
|
# expected: 200, {"object":"list","data":[...]} with ids reasoning, ornith:35b, qwen2.5:3b-instruct,
|
||||||
|
# nomic-ai/nomic-embed-text-v2-moe, BAAI/bge-reranker-base
|
||||||
|
|
||||||
|
# every advertised id must dispatch; nothing advertised may 404
|
||||||
|
for m in $(curl -s localhost:8080/v1/models | grep -o '"id":"[^"]*"' | cut -d'"' -f4); do
|
||||||
|
curl -s -o /dev/null -w "$m %{http_code}\n" localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' -d "{\"model\":\"$m\",\"messages\":[]}"
|
||||||
|
done
|
||||||
|
# expected: no unknown-model rejection for any advertised id
|
||||||
|
```
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# 2.6 — `POST /v1/embeddings` passthrough (GREEN)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surface
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [2.1](2.1-model-registry.md), [1.1](1.1-reverse-proxy.md)
|
||||||
|
|
||||||
|
- [ ] `POST /v1/embeddings` reaches `embeddings-predictor.llm-serving:80` at the path `/v1/embeddings`, unrewritten
|
||||||
|
- [ ] The request body is forwarded byte-identical, and the upstream response body is returned byte-identical
|
||||||
|
- [ ] The route's connect timeout is 10s and its read and write timeouts are 10m, explicit in configuration rather than inherited
|
||||||
|
- [ ] The route has an explicit body size cap
|
||||||
|
- [ ] Upstream error statuses are surfaced as-is; the gateway invents no retries and no substitute response
|
||||||
|
|
||||||
|
TEI already serves the canonical path, so this route rewrites nothing. The model
|
||||||
|
served here is `nomic-ai/nomic-embed-text-v2-moe`. Port is 80, not 8080. Whether this
|
||||||
|
route dispatches on the body's `model` or is pinned to the single embeddings upstream
|
||||||
|
is a design decision for whoever works it; either way the path must not change and an
|
||||||
|
unknown `model` must not silently reach the wrong upstream.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -i localhost:8080/v1/embeddings -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"nomic-ai/nomic-embed-text-v2-moe","input":"hello"}'
|
||||||
|
# expected: 200, embeddings stub hit at path /v1/embeddings, request body unmodified, response body byte-identical to the stub's
|
||||||
|
```
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# 2.7 — `POST /v1/rerank` path rewrite (GREEN)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surface
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [2.1](2.1-model-registry.md), [1.1](1.1-reverse-proxy.md)
|
||||||
|
|
||||||
|
- [ ] `POST /v1/rerank` reaches `reranker-predictor.llm-serving:80` at the path `/rerank`
|
||||||
|
- [ ] The rewrite is expressed in the route configuration, not special-cased in dispatch code
|
||||||
|
- [ ] The request body is forwarded byte-identical, and the upstream response body is returned byte-identical
|
||||||
|
- [ ] The route's connect timeout is 10s and its read and write timeouts are 10m, explicit in configuration
|
||||||
|
- [ ] The route has an explicit body size cap
|
||||||
|
- [ ] A request to `/v1/rerank` never reaches the upstream as `/v1/rerank`
|
||||||
|
|
||||||
|
TEI does not serve `/v1/rerank`: probing it returned 404, while `/rerank` returned 405
|
||||||
|
for the wrong method — that is how the correct upstream path was established. The
|
||||||
|
model served here is `BAAI/bge-reranker-base`. Port is 80, not 8080. This is the only
|
||||||
|
route in the LLM surface that rewrites its path.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -i localhost:8080/v1/rerank -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"BAAI/bge-reranker-base","query":"q","texts":["a","b"]}'
|
||||||
|
# expected: 200, reranker stub recorded exactly one hit at path /rerank and zero at /v1/rerank
|
||||||
|
```
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# 2.8 — Kong parity test (RED)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surface
|
||||||
|
Stage: RED
|
||||||
|
Depends on: [2.2](2.2-body-based-dispatch.md), [2.4](2.4-legacy-path-aliases.md), [2.5](2.5-models-endpoint.md), [2.6](2.6-embeddings-passthrough.md), [2.7](2.7-rerank-rewrite.md)
|
||||||
|
|
||||||
|
- [ ] A repeatable check issues the same request to Kong and to the gateway and compares the real HTTP responses: status, meaningful headers, and body
|
||||||
|
- [ ] It covers every migrated route: `GET /v1/models`, the three `/v1/{reasoning,ornith,qwen}/chat/completions` aliases, `POST /v1/embeddings`, `POST /v1/rerank`, and the new canonical `POST /v1/chat/completions`
|
||||||
|
- [ ] It covers a streaming chat request and asserts equivalent chunk sequencing, not just an equal final concatenation
|
||||||
|
- [ ] It covers a mid-stream client disconnect and asserts the upstream request is cancelled rather than left running
|
||||||
|
- [ ] It asserts `reasoning_content` and `content` are both present and untouched for the `reasoning` model
|
||||||
|
- [ ] Fields that legitimately differ per request — ids, timestamps, generated text — are normalised before comparison, and the normalisation is explicit rather than a blanket ignore
|
||||||
|
- [ ] Any difference fails the check loudly and names the route and the field
|
||||||
|
- [ ] `GET /v1/models` is expected to differ from Kong's static list only where the gateway's derived list is more accurate; that difference is recorded deliberately, not normalised away
|
||||||
|
- [ ] The check passes before cutover ([6.5](6.5-cutover.md)) and blocks it if it does not
|
||||||
|
|
||||||
|
Kong is serving live traffic while this runs. The check reads only; it changes no
|
||||||
|
cluster state. Eleven ReplicaSets exist on the Kong Deployment, so re-run this
|
||||||
|
immediately before cutover rather than trusting an old result.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./parity --kong https://api.riotpiao.com --gateway http://homelab-frontend.api.svc.cluster.local
|
||||||
|
# expected: exit 0, one PASS line per route, explicit report of the /v1/models difference
|
||||||
|
|
||||||
|
# spot-check by hand, same body to both
|
||||||
|
curl -s -o /dev/null -w 'kong %{http_code}\n' https://api.riotpiao.com/v1/rerank \
|
||||||
|
-H 'content-type: application/json' -d '{"model":"BAAI/bge-reranker-base","query":"q","texts":["a"]}'
|
||||||
|
# expected: identical status from both endpoints
|
||||||
|
```
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# 2.9 — Dialect-neutral canonical request (GREEN)
|
||||||
|
|
||||||
|
Phase: 2 — LLM surfaces
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [2.1](2.1-model-registry.md), [2.2](2.2-body-based-dispatch.md)
|
||||||
|
|
||||||
|
The gateway serves two permanent protocol dialects: `/v1/*` is OpenAI-compatible and
|
||||||
|
`/llm/*` is Anthropic Messages. Both translate into one internal shape before anything
|
||||||
|
else happens to them, and both translate back out of it on the way to the client.
|
||||||
|
|
||||||
|
- [ ] A single internal request representation exists that carries at minimum: the
|
||||||
|
resolved upstream, the model name as the client sent it, the ordered turns, an
|
||||||
|
optional system instruction, a maximum output token count, stop sequences, and a
|
||||||
|
streaming flag
|
||||||
|
- [ ] The representation names no dialect: nothing in it is called openai or anthropic,
|
||||||
|
and no field exists solely because one dialect happens to spell it that way
|
||||||
|
- [ ] Model dispatch, the shared slot controller, per-caller budgets, logging and
|
||||||
|
metrics all read the canonical request and never the raw dialect body
|
||||||
|
- [ ] The surface that received a request is recorded as one field on the canonical
|
||||||
|
request, used only to choose the response and error rendering, never to choose an
|
||||||
|
upstream or a slot
|
||||||
|
- [ ] An identical prompt sent to `/v1/chat/completions` and to `/llm/v1/messages`
|
||||||
|
produces the same resolved upstream, the same slot accounting and the same log
|
||||||
|
fields apart from that one surface label
|
||||||
|
- [ ] Adding a third dialect requires a new translator only; the slot controller,
|
||||||
|
dispatch and budget code are untouched
|
||||||
|
- [ ] Translation failure is a client error at the surface boundary, and no partially
|
||||||
|
populated canonical request ever reaches an upstream
|
||||||
|
|
||||||
|
`reasoning` maps to `reasoning-predictor.llm-serving:80`; `ornith:35b` and
|
||||||
|
`qwen2.5:3b-instruct` both map to `ornith-predictor.llm-serving:80`. Ports are 80.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
# expected: 200, reasoning stub hit once
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/llm/v1/messages \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
# expected: 200, reasoning stub hit once, same upstream and same slot counter as the /v1 call
|
||||||
|
|
||||||
|
grep -h 'upstream=' /tmp/gateway.log | tail -2
|
||||||
|
# expected: both lines show upstream=reasoning-predictor and model=reasoning, differing only in the surface label
|
||||||
|
```
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# 3.1 — JWKS fetch and rotation (GREEN)
|
||||||
|
|
||||||
|
Phase: 3 — Authentication
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [0.2](0.2-route-configuration.md), [0.3](0.3-health-endpoints.md)
|
||||||
|
|
||||||
|
- [ ] The signing key set is fetched from Authentik at `https://authentik.riotpiao.com` at runtime; no public key is pinned in configuration, in an image, or in a manifest
|
||||||
|
- [ ] The JWKS URL is configuration, so a local stub issuer can be pointed at instead — no cluster and no credentials needed to verify this task
|
||||||
|
- [ ] Fetched keys are cached and reused; a token verification does not trigger a network call per request
|
||||||
|
- [ ] A token whose key id is not in the cache triggers a refetch, and the refetch is rate-limited so an unknown-key flood cannot hammer Authentik
|
||||||
|
- [ ] After a key rotates at the issuer, tokens signed by the new key verify without restarting, redeploying, or editing configuration
|
||||||
|
- [ ] Tokens signed by a key that is no longer published stop verifying once the cache reflects that
|
||||||
|
- [ ] `GET /readyz` fails while JWKS has never been fetched successfully, and `GET /healthz` is unaffected
|
||||||
|
- [ ] A JWKS fetch failure while a valid cache exists does not take the gateway down; it is logged and retried
|
||||||
|
- [ ] No key material, token, or fetched secret appears in logs
|
||||||
|
|
||||||
|
This deletes the rotation runbook `AUTH-PLAN.md` was forced to propose. That runbook
|
||||||
|
existed only because Kong OSS needed a pinned `rsa_public_key`; it must not be
|
||||||
|
carried forward.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# stub issuer serving a JWKS, gateway pointed at it
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/readyz # expected: 200 once JWKS has been fetched
|
||||||
|
|
||||||
|
# stub returns 500 for JWKS from a cold start
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/readyz # expected: 503
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/healthz # expected: 200
|
||||||
|
|
||||||
|
# rotate the stub's key, then present a token signed by the new key, no restart
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/models -H "authorization: Bearer $NEW_TOKEN"
|
||||||
|
# expected: 200 — refetch happened on the unknown key id
|
||||||
|
```
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# 3.2 — Bearer token validation (GREEN)
|
||||||
|
|
||||||
|
Phase: 3 — Authentication
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [3.1](3.1-jwks-fetch-and-rotation.md)
|
||||||
|
|
||||||
|
- [ ] `Authorization: Bearer <jwt>` is the credential the gateway accepts on protected routes
|
||||||
|
- [ ] The scheme match is case-insensitive, as the HTTP spec requires — `bearer`, `Bearer` and `BEARER` all work
|
||||||
|
- [ ] A valid, unexpired token signed by a currently published Authentik key returns the upstream response
|
||||||
|
- [ ] Missing header, wrong scheme, malformed token, bad signature, expired token, and wrong issuer or audience each return 401 with `WWW-Authenticate` set
|
||||||
|
- [ ] Rejections are RFC 9457 `application/problem+json`, and the reason is logged without logging the token
|
||||||
|
- [ ] The `Authorization` header is not forwarded to upstreams
|
||||||
|
- [ ] The validated caller identity is available to later stages, since [3.5](3.5-capability-authorization.md) authorizes on it
|
||||||
|
- [ ] Signature verification is real: a token with a valid-looking payload and a forged signature is rejected
|
||||||
|
|
||||||
|
This is precisely what Kong OSS `key-auth` could not do. Verified live: a raw
|
||||||
|
`apikey:` header succeeded with 200 while `Authorization: Bearer` failed with 401,
|
||||||
|
which hard-blocked every OpenAI-compatible client and is why authentication is OFF
|
||||||
|
on the model API today.
|
||||||
|
|
||||||
|
Authentik is at `https://authentik.riotpiao.com`. A local stub issuer must be enough
|
||||||
|
to work this task — no cluster, no credentials.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/chat/completions \
|
||||||
|
-H "authorization: Bearer $VALID_TOKEN" -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","messages":[]}'
|
||||||
|
# expected: 200
|
||||||
|
|
||||||
|
curl -s -i localhost:8080/v1/chat/completions -H 'content-type: application/json' -d '{"model":"reasoning"}'
|
||||||
|
# expected: 401, WWW-Authenticate present, content-type application/problem+json
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/models -H "apikey: $OLD_KONG_KEY"
|
||||||
|
# expected: 401 — the retired Kong credential form is not accepted
|
||||||
|
```
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# 3.3 — Authentik service account and token provider (GREEN)
|
||||||
|
|
||||||
|
Phase: 3 — Authentication
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [3.2](3.2-bearer-validation.md)
|
||||||
|
|
||||||
|
- [ ] A service account exists in Authentik at `https://authentik.riotpiao.com` for machine callers of the model API
|
||||||
|
- [ ] An OAuth2 provider is configured so that account can obtain a token non-interactively, with no browser step
|
||||||
|
- [ ] The token endpoint returns a signed JWT whose issuer and audience match what the gateway validates, and which the gateway accepts on a protected route
|
||||||
|
- [ ] Token lifetime is set deliberately and recorded, not left at whatever the default is
|
||||||
|
- [ ] The client secret is stored as a Kubernetes Secret referenced from git, never committed in plaintext
|
||||||
|
- [ ] The Authentik configuration is captured in the repo as reproducible steps or declarative config, so it can be rebuilt rather than clicked together again
|
||||||
|
- [ ] The token carries whatever claim [3.5](3.5-capability-authorization.md) authorizes on
|
||||||
|
|
||||||
|
Open risk recorded in `AUTH-PLAN.md` and unresolved: Authentik 2026.x may require an
|
||||||
|
app-password or JWT-assertion flow rather than a plain `client_secret` POST to the
|
||||||
|
token endpoint. Verify which flow the running version accepts before wiring anything
|
||||||
|
that depends on it, and record the answer here.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -i -X POST https://authentik.riotpiao.com/application/o/token/ \
|
||||||
|
-d grant_type=client_credentials -d "client_id=$CID" -d "client_secret=$CSEC"
|
||||||
|
# expected: 200, JSON with access_token; if 400/401, the app-password / JWT-assertion risk above is real — record which
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' http://homelab-frontend.api.svc.cluster.local/v1/models \
|
||||||
|
-H "authorization: Bearer $ACCESS_TOKEN"
|
||||||
|
# expected: 200 — the gateway accepts a real Authentik-issued token, not only a stub one
|
||||||
|
```
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# 3.4 — Flag-gated auth rollout (GREEN)
|
||||||
|
|
||||||
|
Phase: 3 — Authentication
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [3.2](3.2-bearer-validation.md)
|
||||||
|
|
||||||
|
- [ ] Authentication is controlled by an explicit flag in configuration, and its default is OFF
|
||||||
|
- [ ] With the flag off, every route answers exactly as it did before Phase 3 existed — no 401s, no `WWW-Authenticate`, no behaviour change
|
||||||
|
- [ ] With the flag on, protected routes require `Authorization: Bearer` and reject anything else with 401
|
||||||
|
- [ ] `GET /healthz` and `GET /readyz` never require authentication in either state
|
||||||
|
- [ ] Which routes are protected is per-route configuration, so auth can be turned on for one surface at a time
|
||||||
|
- [ ] Turning the flag on is a git change synced by Argo; it is never toggled by hand against the cluster
|
||||||
|
- [ ] The flag's current state is visible in logs at startup and in metrics, so nobody has to guess whether auth is on
|
||||||
|
- [ ] Turning the flag off again fully restores unauthenticated access, making the rollout reversible
|
||||||
|
|
||||||
|
The model API is unauthenticated today — verified live, a request with no credentials
|
||||||
|
returns 200. Enabling this flag breaks every current caller until they hold a token,
|
||||||
|
pi included. That is why it defaults off and is enabled deliberately, after
|
||||||
|
[3.6](3.6-pi-client-migration.md).
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# flag off
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/models # expected: 200
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' # expected: 200
|
||||||
|
|
||||||
|
# flag on, no credentials
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' # expected: 401
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/healthz # expected: 200
|
||||||
|
```
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# 3.5 — Capability authorization (RED)
|
||||||
|
|
||||||
|
Phase: 3 — Authentication
|
||||||
|
Stage: RED
|
||||||
|
Depends on: [3.2](3.2-bearer-validation.md), [3.4](3.4-flag-gated-rollout.md)
|
||||||
|
|
||||||
|
- [ ] Beyond proving who the caller is, the gateway checks the token is permitted to invoke the capability being called
|
||||||
|
- [ ] Each capability prefix — `/v1/*` for the model surface, and the future `/sqs/*`, `/workflow/*`, `/cluster/*`, `/db/*` — maps to a required grant, and the mapping is configuration
|
||||||
|
- [ ] A token carrying a queue grant but no model grant is rejected on `POST /v1/chat/completions` with 403, not 401 — it authenticated fine, it is not permitted
|
||||||
|
- [ ] A token with no recognised grant at all is rejected on every protected route
|
||||||
|
- [ ] Rejections are RFC 9457 `application/problem+json` and state which capability was denied, without echoing the token or its claims verbatim
|
||||||
|
- [ ] Denials are logged with the caller identity and the capability, and counted as a distinct rejection reason
|
||||||
|
- [ ] A route with no required grant configured while auth is on fails startup rather than defaulting to allow-all
|
||||||
|
|
||||||
|
Write the failing tests first: a queue token must not invoke a GPU. The default on an
|
||||||
|
unconfigured route is deny, because a fail-open authorization layer is worse than none
|
||||||
|
— it reads as protection while granting everything.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -i localhost:8080/v1/chat/completions -H "authorization: Bearer $QUEUE_ONLY_TOKEN" \
|
||||||
|
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}'
|
||||||
|
# expected: 403, application/problem+json naming the denied capability, no upstream stub hit
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/chat/completions \
|
||||||
|
-H "authorization: Bearer $MODEL_TOKEN" -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","messages":[]}'
|
||||||
|
# expected: 200
|
||||||
|
|
||||||
|
./gateway --config testdata/auth-on-route-without-grant.yaml; echo "exit=$?"
|
||||||
|
# expected: non-zero exit, stderr names the route missing a required grant
|
||||||
|
```
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# 3.6 — Migrate pi to Bearer and one provider (REFACTOR)
|
||||||
|
|
||||||
|
Phase: 3 — Authentication
|
||||||
|
Stage: REFACTOR
|
||||||
|
Depends on: [2.2](2.2-body-based-dispatch.md), [3.3](3.3-authentik-service-account.md), [3.4](3.4-flag-gated-rollout.md)
|
||||||
|
|
||||||
|
- [ ] `~/.pi/agent/models.json` no longer contains a `customHeaders: {apikey: ...}` block anywhere
|
||||||
|
- [ ] pi authenticates with `Authorization: Bearer` carrying an Authentik-issued token
|
||||||
|
- [ ] The three provider entries `homelab-reasoning`, `homelab-ornith` and `homelab-qwen` collapse into ONE provider entry
|
||||||
|
- [ ] That single provider's base URL is the canonical OpenAI base, and the three models are listed under it — body-based dispatch makes per-model base URLs unnecessary
|
||||||
|
- [ ] pi reaches all three models through `POST /v1/chat/completions` with `model` set to `reasoning`, `ornith:35b`, and `qwen2.5:3b-instruct`
|
||||||
|
- [ ] Streaming still works from pi, and interrupting a generation cancels it upstream rather than leaving it running
|
||||||
|
- [ ] The old config is captured before editing, so a revert is one file restore
|
||||||
|
- [ ] Verified with auth ON, because that is the state the migration exists to survive
|
||||||
|
|
||||||
|
The `apikey` header exists only because Kong OSS `key-auth` rejected
|
||||||
|
`Authorization: Bearer`. Once this lands, nothing depends on the legacy aliases in
|
||||||
|
[2.4](2.4-legacy-path-aliases.md) and they can be removed.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
grep -c apikey ~/.pi/agent/models.json # expected: 0
|
||||||
|
grep -c '"homelab' ~/.pi/agent/models.json # expected: 1 provider entry, not 3
|
||||||
|
|
||||||
|
for m in reasoning ornith:35b qwen2.5:3b-instruct; do
|
||||||
|
curl -s -o /dev/null -w "$m %{http_code}\n" https://api.riotpiao.com/v1/chat/completions \
|
||||||
|
-H "authorization: Bearer $ACCESS_TOKEN" -H 'content-type: application/json' \
|
||||||
|
-d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}"
|
||||||
|
done
|
||||||
|
# expected: 200 for all three, with auth enabled
|
||||||
|
|
||||||
|
curl -N -s https://api.riotpiao.com/v1/chat/completions -H "authorization: Bearer $ACCESS_TOKEN" \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","stream":true,"messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
# expected: text/event-stream chunks arriving incrementally; reasoning_content and content both present
|
||||||
|
```
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
# 4.1 — GPU slot semaphore for `reasoning` (GREEN)
|
||||||
|
|
||||||
|
Phase: 4 — Limits and budgets
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [4.3](4.3-problem-json-errors.md), [2.9](2.9-canonical-request-model.md)
|
||||||
|
|
||||||
|
The `reasoning` upstream is 2 replicas x `--max-num-seqs=4` = 8 concurrent sequence
|
||||||
|
slots cluster-wide. That 8 is a vLLM concurrency setting, not a GPU count — worker-1
|
||||||
|
has 4 physical GPUs and the two numbers are unrelated. Exceeding 8 does not fail
|
||||||
|
fast; it silently queues inside vLLM where the gateway has no visibility and no
|
||||||
|
ability to cancel.
|
||||||
|
|
||||||
|
The gateway serves two permanent dialects over the same predictors: `/v1/*` is
|
||||||
|
OpenAI-compatible and `/llm/*` is Anthropic Messages. There is ONE controller, keyed by
|
||||||
|
upstream, sitting below both.
|
||||||
|
|
||||||
|
- [ ] Concurrent in-flight requests to `reasoning` are capped at a configured limit strictly below 8, leaving operator headroom
|
||||||
|
- [ ] The cap, the queue depth, and the queue wait timeout are all explicit in configuration with no silent defaults
|
||||||
|
- [ ] Requests over the cap wait in a bounded queue rather than being rejected immediately
|
||||||
|
- [ ] Once the queue is full, further requests are rejected with a retryable status and a `Retry-After`, as an `application/problem+json` document
|
||||||
|
- [ ] A slot is released when the response completes, when the upstream errors, and when the client disconnects mid-stream — no path leaks a slot
|
||||||
|
- [ ] A client that disconnects while still queued never reaches the upstream and never consumes a slot
|
||||||
|
- [ ] The cap applies only to `reasoning`; other upstreams are unaffected and are not blocked behind its queue
|
||||||
|
- [ ] The controller is keyed by UPSTREAM, never by route, path prefix or dialect — `reasoning-predictor.llm-serving:80` has exactly one cap and one queue
|
||||||
|
- [ ] A request arriving on `/v1/chat/completions` and one arriving on `/llm/v1/messages` contend for the same slots and the same queue, admitted in arrival order
|
||||||
|
- [ ] Total in-flight requests against `reasoning` never exceed the configured cap regardless of which surface they arrived through, including when both surfaces are saturated at once
|
||||||
|
- [ ] Per-dialect semaphores are explicitly wrong and must not exist: the 8 sequence slots are physical, so two independent gates would each believe they were within budget while together exceeding it
|
||||||
|
- [ ] The controller reads the dialect-neutral canonical request and has no knowledge of which surface produced it; adding a third dialect requires no change here
|
||||||
|
- [ ] Slot occupancy and queue depth are reported per upstream, and requests are attributable to a surface for logging only, never for admission
|
||||||
|
- [ ] Rejection at a full queue renders per surface — `application/problem+json` on `/v1`, the Anthropic error shape on `/llm` — from one shared decision
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Fire 40 concurrent chat requests at a stub reasoning upstream that holds each for 2s,
|
||||||
|
# with cap=6 and queue=8 configured.
|
||||||
|
seq 40 | xargs -P40 -I{} curl -s -o /dev/null -w '%{http_code}\n' \
|
||||||
|
-X POST localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' | sort | uniq -c
|
||||||
|
# expected: a mix of 200 and 503; zero 500s
|
||||||
|
|
||||||
|
# expected: the stub logged at most 6 simultaneous in-flight requests, never 7
|
||||||
|
grep -c 'max_concurrent=7' /tmp/stub-reasoning.log
|
||||||
|
# expected: 0
|
||||||
|
|
||||||
|
curl -s -D - -o /dev/null -X POST localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' | grep -i retry-after
|
||||||
|
# expected: Retry-After present on the rejected request
|
||||||
|
|
||||||
|
# Same cap=6 stub, but split the load across both dialects: 20 on /v1 and 20 on /llm.
|
||||||
|
seq 20 | xargs -P20 -I{} curl -s -o /dev/null -X POST localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' &
|
||||||
|
seq 20 | xargs -P20 -I{} curl -s -o /dev/null -X POST localhost:8080/llm/v1/messages \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}' &
|
||||||
|
wait
|
||||||
|
grep -c 'max_concurrent=7' /tmp/stub-reasoning.log
|
||||||
|
# expected: 0 — the two surfaces share one cap, they do not get 6 each
|
||||||
|
```
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# 4.2 — Per-caller request budgets (GREEN)
|
||||||
|
|
||||||
|
Phase: 4 — Limits and budgets
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [4.3](4.3-problem-json-errors.md)
|
||||||
|
|
||||||
|
No `rate-limiting` plugin exists anywhere in the cluster today. This is net-new work,
|
||||||
|
not a migration — there is no prior behaviour to preserve.
|
||||||
|
|
||||||
|
- [ ] An identified caller gets a bounded number of requests per configured time window
|
||||||
|
- [ ] Caller identity comes from the authenticated token's subject when auth is on, and from a documented fallback attribute when auth is off
|
||||||
|
- [ ] Budget size and window length are explicit in configuration, per caller class, with no silent defaults
|
||||||
|
- [ ] Exceeding the budget is rejected with a retryable status, an `application/problem+json` body, and a `Retry-After` naming when the window resets
|
||||||
|
- [ ] Remaining budget and reset time are observable to the caller on allowed requests, not only on rejections
|
||||||
|
- [ ] Budgets are enforced independently of the `reasoning` concurrency cap — a caller under budget can still be queued or rejected for slot pressure, and vice versa
|
||||||
|
- [ ] Two distinct callers do not consume each other's budget
|
||||||
|
- [ ] Budget state is per-replica-safe: the documented behaviour with 2 replicas is stated, not accidental
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# budget=5 per 60s window for the test caller
|
||||||
|
for i in $(seq 1 7); do
|
||||||
|
curl -s -o /dev/null -w '%{http_code} ' -H 'authorization: Bearer test-caller-a' \
|
||||||
|
localhost:8080/v1/models
|
||||||
|
done; echo
|
||||||
|
# expected: 200 200 200 200 200 429 429
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' -H 'authorization: Bearer test-caller-b' \
|
||||||
|
localhost:8080/v1/models
|
||||||
|
# expected: 200 — caller B has its own budget
|
||||||
|
```
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# 4.3 — RFC 9457 problem+json rejections (RED)
|
||||||
|
|
||||||
|
Phase: 4 — Limits and budgets
|
||||||
|
Stage: RED
|
||||||
|
|
||||||
|
Every rejection the gateway generates itself must be a machine-readable problem
|
||||||
|
document. Upstream responses are passed through untouched — this covers only errors
|
||||||
|
the gateway originates.
|
||||||
|
|
||||||
|
- [ ] Every gateway-originated rejection responds with `Content-Type: application/problem+json`
|
||||||
|
- [ ] The body carries at minimum `type`, `title`, `status`, and `detail`, and `status` equals the HTTP status line
|
||||||
|
- [ ] `type` is a stable, distinct URI per rejection reason, so a client can branch on it without parsing prose
|
||||||
|
- [ ] `detail` is human-useful and names the offending input where one exists
|
||||||
|
- [ ] `Retry-After` is set whenever a retry time is knowable — queue full, budget exhausted, upstream saturated
|
||||||
|
- [ ] `Retry-After` is absent when no retry will help — unknown model, malformed body, oversized body
|
||||||
|
- [ ] No token, credential, header value, or request body content appears in any field
|
||||||
|
- [ ] A rejection is emitted as a structured log line carrying the same reason identifier used in `type`
|
||||||
|
|
||||||
|
Tests come first and must fail because the shape does not exist yet, not because a
|
||||||
|
route is missing.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -D - -o /tmp/p.json -X POST localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' -d '{"model":"nope"}' | grep -i content-type
|
||||||
|
# expected: application/problem+json
|
||||||
|
|
||||||
|
python3 -c "import json;d=json.load(open('/tmp/p.json'));assert{'type','title','status','detail'}<=d.keys();assert d['status']==400;print('ok')"
|
||||||
|
# expected: ok
|
||||||
|
```
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# 5.1 — Prometheus parity with the retiring Kong plugin (RED)
|
||||||
|
|
||||||
|
Phase: 5 — Observability
|
||||||
|
Stage: RED
|
||||||
|
|
||||||
|
Kong runs a cluster-wide `prometheus` plugin today. It is deleted at teardown. If the
|
||||||
|
gateway does not carry equivalent signal, observability REGRESSES at cutover and
|
||||||
|
nobody notices until an incident.
|
||||||
|
|
||||||
|
- [ ] A Prometheus text-format endpoint is exposed and scrapeable without authentication from inside the cluster
|
||||||
|
- [ ] Request rate is observable, labelled by route and by upstream
|
||||||
|
- [ ] Request latency is observable as a distribution, not a mean, labelled by route and upstream
|
||||||
|
- [ ] Response status is observable, labelled by route, upstream, and status class
|
||||||
|
- [ ] Bandwidth in both directions is observable per route and upstream
|
||||||
|
- [ ] Upstream health is observable — whether each configured upstream is currently reachable and answering
|
||||||
|
- [ ] Label values are drawn from the configured route and upstream names, never from raw request paths or user input, so cardinality cannot be driven by a caller
|
||||||
|
- [ ] Streaming responses record their full byte count and full duration, not the time to first byte
|
||||||
|
- [ ] The metrics endpoint is not reachable through the public route surface
|
||||||
|
|
||||||
|
Write the assertions against the metric families first; they must fail because the
|
||||||
|
families are absent, not because the endpoint 404s.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s localhost:8080/metrics | grep -E '^# TYPE' | awk '{print $3}' | sort -u
|
||||||
|
# expected: includes counter and histogram families covering requests, duration, bytes, upstream health
|
||||||
|
|
||||||
|
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","messages":[]}' >/dev/null
|
||||||
|
curl -s localhost:8080/metrics | grep 'upstream="reasoning-predictor"' | head
|
||||||
|
# expected: request, duration and byte samples all carry route and upstream labels
|
||||||
|
```
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# 5.2 — Gateway-specific metrics Kong could not provide (GREEN)
|
||||||
|
|
||||||
|
Phase: 5 — Observability
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [5.1](5.1-prometheus-parity.md)
|
||||||
|
|
||||||
|
These are the signals that justify replacing Kong. Without them the concurrency cap
|
||||||
|
and budget layers are unfalsifiable — you cannot tell a saturated GPU from a broken
|
||||||
|
gateway.
|
||||||
|
|
||||||
|
- [ ] In-flight request count is observable per upstream, and returns to zero when traffic stops
|
||||||
|
- [ ] Queue depth for the `reasoning` concurrency queue is observable
|
||||||
|
- [ ] Occupancy of the `reasoning` slot cap is observable — how many of the configured slots are held right now
|
||||||
|
- [ ] The configured slot cap itself is observable, so occupancy can be read as a ratio without hardcoding the limit in a dashboard
|
||||||
|
- [ ] Rejections are counted and broken down by reason: queue full, budget exhausted, body too large, unknown model, auth failure
|
||||||
|
- [ ] Rejection reason label values match the stable reason identifiers used in the problem+json `type` field, so metrics and logs join cleanly
|
||||||
|
- [ ] Time spent waiting in the queue is observable as a distribution, separately from upstream latency
|
||||||
|
- [ ] In-flight and occupancy gauges are correct after a client disconnects mid-stream — no permanent drift upward
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Hold 3 long requests open against a stub reasoning upstream, then scrape.
|
||||||
|
curl -s localhost:8080/metrics | grep -E 'inflight|queue_depth|slots'
|
||||||
|
# expected: in-flight for the reasoning upstream reads 3, queue depth reads 0, slot cap is exported
|
||||||
|
|
||||||
|
# Kill the clients mid-stream, wait, scrape again.
|
||||||
|
curl -s localhost:8080/metrics | grep -E 'inflight'
|
||||||
|
# expected: back to 0 — disconnects released their slots
|
||||||
|
|
||||||
|
curl -s localhost:8080/metrics | grep 'reason='
|
||||||
|
# expected: rejection counters split by reason, matching the problem+json type identifiers
|
||||||
|
```
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# 5.3 — ServiceMonitor for gateway scraping (GREEN)
|
||||||
|
|
||||||
|
Phase: 5 — Observability
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [5.2](5.2-gateway-metrics.md)
|
||||||
|
|
||||||
|
Exposing metrics is not the same as having them collected. Kong's plugin was picked
|
||||||
|
up cluster-wide; the gateway must be explicitly registered or the dashboards go
|
||||||
|
blank at cutover.
|
||||||
|
|
||||||
|
- [ ] A ServiceMonitor selects the gateway Service and is committed to git, applied by Argo — never `kubectl apply`
|
||||||
|
- [ ] It carries whatever label the cluster's Prometheus uses to select ServiceMonitors, verified against the running Prometheus rather than assumed
|
||||||
|
- [ ] The scraped port is a named port on the gateway Service, matched by name not number
|
||||||
|
- [ ] Scrape interval and timeout are explicit
|
||||||
|
- [ ] The gateway appears as an `up == 1` target in Prometheus, in the expected namespace
|
||||||
|
- [ ] Metric labels identify the gateway pod and namespace, so 2 replicas are distinguishable
|
||||||
|
- [ ] Scraping works while the gateway is still unexposed to public traffic — this lands before cutover, not after
|
||||||
|
- [ ] Note: `prometheus-operated.monitoring` is headless with ClusterIP `None`, so any egress policy that touches it needs pod selectors, not a ClusterIP
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n monitoring get servicemonitor -l release --show-labels | grep -i frontend
|
||||||
|
# expected: the gateway ServiceMonitor exists and carries the selector label Prometheus uses
|
||||||
|
|
||||||
|
# Query Prometheus for the gateway target
|
||||||
|
curl -s 'http://localhost:9090/api/v1/query?query=up{job=~".*homelab-frontend.*"}' \
|
||||||
|
| python3 -c "import json,sys;r=json.load(sys.stdin)['data']['result'];print(len(r),[x['value'][1] for x in r])"
|
||||||
|
# expected: 2 targets, both reporting "1"
|
||||||
|
```
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# 6.1 — Hardened container image (GREEN)
|
||||||
|
|
||||||
|
Phase: 6 — Deploy and cutover
|
||||||
|
Stage: GREEN
|
||||||
|
|
||||||
|
The gateway is the public edge process. It gets no shell, no package manager, no
|
||||||
|
writable filesystem and no privileges it does not need.
|
||||||
|
|
||||||
|
- [ ] The runtime image is distroless or scratch — no shell, no package manager, no busybox
|
||||||
|
- [ ] The container runs as a non-root user, enforced by `runAsNonRoot` and an explicit non-zero UID
|
||||||
|
- [ ] The root filesystem is read-only; any writable path the process genuinely needs is an explicitly mounted volume, not a relaxed rootfs
|
||||||
|
- [ ] All Linux capabilities are dropped, and none are added back
|
||||||
|
- [ ] `seccompProfile` is `RuntimeDefault`
|
||||||
|
- [ ] Privilege escalation is disabled
|
||||||
|
- [ ] Image tags are the commit SHA of the source that built them. Never `:latest`, never a moving tag — Argo `selfHeal` cannot reconcile a mutable tag reliably, and a rollback needs a tag that still means what it meant yesterday
|
||||||
|
- [ ] The image is reproducible from a committed build definition; nothing is built by hand
|
||||||
|
- [ ] The image contains no credentials, kubeconfig or service-account token baked in (G2)
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run --rm --entrypoint sh <image>:<sha>
|
||||||
|
# expected: fails — no shell in the image
|
||||||
|
|
||||||
|
docker run --rm --read-only --user 65532 <image>:<sha> --version
|
||||||
|
# expected: starts and exits cleanly under a read-only rootfs as a non-root user
|
||||||
|
|
||||||
|
docker inspect <image>:<sha> --format '{{.Config.User}}'
|
||||||
|
# expected: a non-zero numeric UID, not empty and not "root"
|
||||||
|
|
||||||
|
grep -rn 'image:' k8s/ | grep -v '@sha256\|:[0-9a-f]\{7,\}'
|
||||||
|
# expected: no matches — every image reference is pinned to a SHA
|
||||||
|
```
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# 6.2 — Deployment, Service, NetworkPolicy (GREEN)
|
||||||
|
|
||||||
|
Phase: 6 — Deploy and cutover
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [6.1](6.1-hardened-image.md)
|
||||||
|
|
||||||
|
Manifests only. Nothing here exposes the gateway publicly — that is the cutover, and
|
||||||
|
it is a separate step.
|
||||||
|
|
||||||
|
- [ ] A Deployment runs the gateway with at least 2 replicas, matching Kong's current replica count
|
||||||
|
- [ ] The pod spec carries the hardening from 6.1: non-root, read-only rootfs, all capabilities dropped, `seccompProfile: RuntimeDefault`
|
||||||
|
- [ ] Liveness probes hit `/healthz` and readiness probes hit `/readyz`; readiness fails while config is invalid or JWKS has never been fetched
|
||||||
|
- [ ] Route configuration is mounted from a ConfigMap sourced from git — not a CRD, and not baked into the image
|
||||||
|
- [ ] A config change rolls the pods; a stale ConfigMap cannot be silently served by a long-lived pod
|
||||||
|
- [ ] Termination allows in-flight requests to drain, with a grace period long enough for the streaming timeouts in use
|
||||||
|
- [ ] A Service exposes the gateway in-cluster with a named port, resolvable at `http://<svc>.api.svc.cluster.local`
|
||||||
|
- [ ] No ServiceAccount with any RBAC is bound; the pod does not need or receive an API-server token (G2)
|
||||||
|
- [ ] A NetworkPolicy allows ingress only from `ingress-nginx`
|
||||||
|
- [ ] The same NetworkPolicy allows egress only to the proxied upstreams plus Authentik, plus DNS. Everything else is denied
|
||||||
|
- [ ] Resource requests and limits are set explicitly
|
||||||
|
- [ ] Every manifest is committed to git and applied by Argo. No `kubectl apply`, no `helm upgrade` (G7)
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n api get deploy homelab-frontend -o jsonpath='{.spec.replicas} {.spec.template.spec.securityContext}{"\n"}'
|
||||||
|
# expected: 2, with runAsNonRoot true and seccompProfile RuntimeDefault
|
||||||
|
|
||||||
|
kubectl -n api exec deploy/homelab-frontend -- true 2>&1
|
||||||
|
# expected: fails — distroless image has no shell to exec into
|
||||||
|
|
||||||
|
kubectl -n llm-serving run np-probe --rm -it --image=curlimages/curl --restart=Never -- \
|
||||||
|
curl -s -m 5 http://homelab-frontend.api.svc.cluster.local/healthz
|
||||||
|
# expected: times out or is refused — ingress is restricted to ingress-nginx only
|
||||||
|
```
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# 6.3 — Argo Application in the homelab-root GitOps repo (GREEN)
|
||||||
|
|
||||||
|
Phase: 6 — Deploy and cutover
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [6.2](6.2-kubernetes-manifests.md)
|
||||||
|
|
||||||
|
Verified live 2026-08-19: zero Argo Applications anywhere in the cluster source from
|
||||||
|
any Forgejo URL. `github.com/Riotpiaole/riotpiao.homelab.com` is authoritative — do
|
||||||
|
not introduce a second source of truth.
|
||||||
|
|
||||||
|
- [ ] An Argo Application for the gateway is committed under `k8s/argocd/apps/` in `github.com/Riotpiaole/riotpiao.homelab.com`
|
||||||
|
- [ ] `repoURL` is that GitHub repo. No Forgejo URL, no local path, no second remote
|
||||||
|
- [ ] It targets the `api` namespace, alongside the existing Kong deployment
|
||||||
|
- [ ] Its sync wave orders it so the gateway is healthy before anything that depends on it, and does not disturb Kong's existing wave-7 Application
|
||||||
|
- [ ] Automated sync with prune and selfHeal is enabled, so drift is corrected without a human
|
||||||
|
- [ ] The Application reaches `Synced` and `Healthy` and stays there across a resync
|
||||||
|
- [ ] Adding it changes nothing about live traffic — Kong still serves `api.riotpiao.com` after this lands
|
||||||
|
- [ ] The commit is pushed and Argo picks it up on its own. No `kubectl apply` of the Application itself (G7)
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n argocd get app homelab-frontend \
|
||||||
|
-o jsonpath='{.spec.source.repoURL}{" "}{.status.sync.status}{" "}{.status.health.status}{"\n"}'
|
||||||
|
# expected: the GitHub repo URL, Synced, Healthy
|
||||||
|
|
||||||
|
kubectl -n argocd get app -o json | grep -ci forgejo
|
||||||
|
# expected: 0 — GitHub remains the only Application source
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/v1/models
|
||||||
|
# expected: 200, still served by Kong — this task changed no live traffic
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# 6.4 — Deploy unexposed alongside Kong (GREEN)
|
||||||
|
|
||||||
|
Phase: 6 — Deploy and cutover
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [6.3](6.3-argocd-application.md)
|
||||||
|
|
||||||
|
The gateway runs in the cluster against the real predictors while Kong continues to
|
||||||
|
serve every byte of live traffic. This is the last step before anything user-visible
|
||||||
|
changes, and it is fully reversible — deleting the Application removes it.
|
||||||
|
|
||||||
|
- [ ] The gateway is running in namespace `api` and is reachable only in-cluster at `http://<svc>.api.svc.cluster.local`
|
||||||
|
- [ ] No Ingress points at the gateway. Ingress `api/api` still sends `/` on `api.riotpiao.com` to `kong-proxy:80`
|
||||||
|
- [ ] Its configured upstreams are the real Services: `reasoning-predictor.llm-serving:80`, `ornith-predictor.llm-serving:80`, `embeddings-predictor.llm-serving:80`, `reranker-predictor.llm-serving:80`, and `agent-hub.agent-pod:9090`
|
||||||
|
- [ ] Note `reasoning-predictor` listens on port 80, not 8080
|
||||||
|
- [ ] A real chat completion succeeds in-cluster against `reasoning`, returning `reasoning_content` and `content` as separate fields
|
||||||
|
- [ ] A streaming chat completion delivers tokens incrementally in-cluster, not as one buffered blob at completion
|
||||||
|
- [ ] Embeddings and rerank both answer correctly, with rerank reaching the upstream's `/rerank` path
|
||||||
|
- [ ] A client disconnect mid-stream is observed to cancel the upstream generation and release its sequence slot
|
||||||
|
- [ ] Prometheus is scraping the gateway and the metrics reflect this in-cluster traffic
|
||||||
|
- [ ] `api.riotpiao.com` is unaffected throughout — verified before and after
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n api get ingress api -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}{"\n"}'
|
||||||
|
# expected: kong-proxy — the gateway is still unexposed
|
||||||
|
|
||||||
|
kubectl -n api run probe --rm -it --image=curlimages/curl --restart=Never -- sh -c '
|
||||||
|
curl -sN -X POST http://homelab-frontend.api.svc.cluster.local/v1/chat/completions \
|
||||||
|
-H "content-type: application/json" \
|
||||||
|
-d "{\"model\":\"reasoning\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"count to 5\"}]}"'
|
||||||
|
# expected: multiple data: chunks arriving over time, terminated by data: [DONE]
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/v1/models
|
||||||
|
# expected: 200 — public traffic still served by Kong, untouched
|
||||||
|
```
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# 6.5 — Cutover: repoint Ingress `api/api` (GREEN)
|
||||||
|
|
||||||
|
Phase: 6 — Deploy and cutover
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [2.8](2.8-kong-parity-test.md), [6.4](6.4-deploy-alongside-kong.md)
|
||||||
|
|
||||||
|
THIS is the cutover. Ingress `api/api` catch-alls `/` on `api.riotpiao.com`; today its
|
||||||
|
backend is `kong-proxy:80`. Changing that one backend moves all live traffic to the
|
||||||
|
gateway. Reverting is the same one-line change back to `kong-proxy:80`, committed and
|
||||||
|
synced — Kong stays running and untouched, so the revert takes effect in seconds.
|
||||||
|
|
||||||
|
Preconditions, all required before starting:
|
||||||
|
|
||||||
|
- [ ] 2.8 Kong parity is passing — gateway and Kong return equivalent responses for every route in the migration inventory, including a streaming request and a mid-stream disconnect
|
||||||
|
- [ ] 6.4 is complete: the gateway is healthy in-cluster against the real predictors, and Prometheus is scraping it
|
||||||
|
- [ ] The inventory has been re-verified immediately beforehand. Kong's Deployment has eleven ReplicaSets and is being actively iterated; a stale inventory is a stale plan
|
||||||
|
- [ ] Kong remains deployed and serving-capable throughout. Nothing about Kong is deleted in this task
|
||||||
|
|
||||||
|
The change itself:
|
||||||
|
|
||||||
|
- [ ] The `api/api` Ingress backend becomes the gateway Service, changed in git and synced by Argo. No `kubectl apply`, no `kubectl edit` (G7)
|
||||||
|
- [ ] The nginx annotations stay exactly as they are: `proxy-read-timeout: 3600`, `proxy-send-timeout: 3600`, `proxy-buffering: off`, `proxy-body-size: 0`. These are what make token streaming work and the gateway needs the same treatment
|
||||||
|
- [ ] The revert is a single-line commit reverting the backend to `kong-proxy:80`, and it has been rehearsed at least once
|
||||||
|
- [ ] After the change, public streaming works end to end through nginx, unbuffered
|
||||||
|
- [ ] pi keeps working on the legacy `/v1/{reasoning,ornith,qwen}/chat/completions` aliases
|
||||||
|
- [ ] A soak period follows, watching gateway metrics, rejection counters and pi traffic. Do not proceed to 6.6 until the soak is clean
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n api get ingress api -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}{"\n"}'
|
||||||
|
# expected: the gateway Service, no longer kong-proxy
|
||||||
|
|
||||||
|
curl -sN -X POST https://api.riotpiao.com/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-d '{"model":"reasoning","stream":true,"messages":[{"role":"user","content":"count to 5"}]}'
|
||||||
|
# expected: incremental data: chunks over time through nginx, ending in data: [DONE]
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' -X POST https://api.riotpiao.com/v1/reasoning/chat/completions \
|
||||||
|
-H 'content-type: application/json' -d '{"messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
# expected: 200 — pi's legacy alias still works
|
||||||
|
|
||||||
|
kubectl -n api get deploy kong-kong -o jsonpath='{.status.readyReplicas}{"\n"}'
|
||||||
|
# expected: 2 — Kong still running and ready, so the revert is one commit away
|
||||||
|
```
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# 6.6 — Kong teardown (REFACTOR)
|
||||||
|
|
||||||
|
Phase: 6 — Deploy and cutover
|
||||||
|
Stage: REFACTOR
|
||||||
|
Depends on: [6.5](6.5-cutover.md)
|
||||||
|
|
||||||
|
IRREVERSIBLE. Every step before this one could be undone in seconds by repointing
|
||||||
|
Ingress `api/api` back to `kong-proxy:80`. Once the Kong Application is removed and
|
||||||
|
Argo prunes the Helm release and its CRDs, that escape hatch is gone — recovery means
|
||||||
|
reinstalling Kong from scratch and rebuilding six plugin CRs.
|
||||||
|
|
||||||
|
Do not start until the soak after 6.5 has been clean for a deliberate, agreed period.
|
||||||
|
|
||||||
|
Preconditions:
|
||||||
|
|
||||||
|
- [ ] 6.5 is complete: `api.riotpiao.com` has been served entirely by the gateway through the soak, with no reverts
|
||||||
|
- [ ] Gateway metrics over the soak show no elevated 5xx rate and no unexplained rejections
|
||||||
|
- [ ] pi and every other known caller have been confirmed working against the gateway
|
||||||
|
- [ ] The inventory has been re-verified immediately beforehand — Kong's config has been actively iterated
|
||||||
|
|
||||||
|
Order matters. Delete the routing objects first, the Application last:
|
||||||
|
|
||||||
|
- [ ] The 7 `ingressClassName: kong` Ingresses are deleted from git — 6 in `llm-serving`, 1 in `agent-pod` for `/console`, `/run`, `/sessions`
|
||||||
|
- [ ] The 6 `KongPlugin` CRs are deleted from git: the three `llm-rewrite-*` chat rewrites, `llm-rewrite-rerank`, `llm-models-list`, and the cluster-wide `prometheus` plugin
|
||||||
|
- [ ] Removing the cluster-wide `prometheus` plugin does not blind any dashboard, because 5.1 parity metrics are already being scraped from the gateway
|
||||||
|
- [ ] The `kong` Application is then removed from `k8s/argocd/apps/55-api-gateway.yaml`
|
||||||
|
- [ ] Argo prunes the Helm release, the Kong CRDs and the namespace leftovers on its own. No `helm uninstall`, no `kubectl delete` (G7)
|
||||||
|
- [ ] Public traffic is verified unaffected after each deletion, not only at the end
|
||||||
|
- [ ] No orphaned Kong CRDs, ReplicaSets or Services remain in namespace `api`
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl get ingress -A -o jsonpath='{range .items[*]}{.spec.ingressClassName}{"\n"}{end}' | sort | uniq -c
|
||||||
|
# expected: no kong entries remain; nginx and istio counts unchanged
|
||||||
|
|
||||||
|
kubectl get kongplugins -A 2>&1; kubectl -n argocd get app kong 2>&1
|
||||||
|
# expected: CRD not found, and Application "kong" not found
|
||||||
|
|
||||||
|
kubectl -n api get all | grep -i kong
|
||||||
|
# expected: no output — nothing Kong-related left
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/v1/models
|
||||||
|
# expected: 200, served by the gateway
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# 7.1 — `/cluster/*` proxies to atlas (GREEN)
|
||||||
|
|
||||||
|
Phase: 7 — Additional capability prefixes
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [6.5](6.5-cutover.md)
|
||||||
|
|
||||||
|
atlas lives in a separate repo, `riotpiao-backend`. It keeps its own informers and its
|
||||||
|
own RBAC. The gateway proxies to it and holds no Kubernetes credentials of its own —
|
||||||
|
this is G2, and it is the whole reason cluster-read capability lives behind atlas
|
||||||
|
rather than in the edge process.
|
||||||
|
|
||||||
|
- [ ] `/cluster/*` on `api.riotpiao.com` proxies to the atlas Service
|
||||||
|
- [ ] The gateway gains no ServiceAccount token, no kubeconfig and no RBAC as part of this. Any authorization decision about cluster data is atlas's, not the gateway's (G2)
|
||||||
|
- [ ] Path rewriting between the `/cluster` prefix and atlas's own paths is explicit in configuration
|
||||||
|
- [ ] Connect, read and write timeouts and a body cap are explicit for this route, with no silent defaults (G6)
|
||||||
|
- [ ] The route requires authentication, and the token is checked for cluster capability — a token minted for queue access must not read cluster state
|
||||||
|
- [ ] The NetworkPolicy is extended to allow egress to atlas and nothing more
|
||||||
|
- [ ] `/v1/*` behaviour is byte-identical before and after this route is added — the prefixes cannot collide
|
||||||
|
- [ ] Metrics and rejection counters cover this route with its own route label
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/cluster/healthz
|
||||||
|
# expected: 401 without a token
|
||||||
|
|
||||||
|
curl -s -H "authorization: Bearer $CLUSTER_TOKEN" https://api.riotpiao.com/cluster/healthz
|
||||||
|
# expected: atlas's own response body, proxied unmodified
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' -H "authorization: Bearer $QUEUE_ONLY_TOKEN" \
|
||||||
|
https://api.riotpiao.com/cluster/healthz
|
||||||
|
# expected: 403 — a queue token does not grant cluster capability
|
||||||
|
|
||||||
|
kubectl -n api get pod -l app=homelab-frontend -o jsonpath='{.items[0].spec.serviceAccountName}{"\n"}'
|
||||||
|
# expected: a ServiceAccount with no RBAC bindings; the gateway still holds no cluster credentials
|
||||||
|
```
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# 7.2 — `/sqs/*` to kmsvc management-service and Kafka (GREEN)
|
||||||
|
|
||||||
|
Phase: 7 — Additional capability prefixes
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [6.5](6.5-cutover.md)
|
||||||
|
|
||||||
|
`management-service` already exposes gRPC at `kmsvc.riotpiao.com`. This prefix is a
|
||||||
|
NEW surface, not a replacement — that gRPC endpoint stays exactly as it is and nothing
|
||||||
|
here migrates off it.
|
||||||
|
|
||||||
|
- [ ] `/sqs/*` on `api.riotpiao.com` proxies to kmsvc `management-service` and the Kafka/Strimzi surfaces in the `sqs` namespace
|
||||||
|
- [ ] `kmsvc.riotpiao.com` is unchanged and still serving after this lands
|
||||||
|
- [ ] Which sub-paths map to which upstream is explicit in configuration; there is no catch-all fallback
|
||||||
|
- [ ] Timeouts and body caps are explicit per sub-route, with no silent defaults (G6)
|
||||||
|
- [ ] The route requires authentication and the token is checked for queue capability
|
||||||
|
- [ ] The NetworkPolicy is extended to reach only the named `sqs` upstreams
|
||||||
|
- [ ] `kmsvc-redis-master.sqs:6379` has NO authentication — `ALLOW_EMPTY_PASSWORD=yes`, TLS off. Any workload with network reach has full unauthenticated read/write. It is not proxied, and the NetworkPolicy must not grant the gateway egress to it
|
||||||
|
- [ ] `/v1/*` behaviour is unchanged before and after
|
||||||
|
- [ ] Metrics and rejection counters cover this route with its own route label
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/sqs/healthz
|
||||||
|
# expected: 401 without a token
|
||||||
|
|
||||||
|
curl -s -H "authorization: Bearer $QUEUE_TOKEN" https://api.riotpiao.com/sqs/healthz
|
||||||
|
# expected: management-service's own response, proxied unmodified
|
||||||
|
|
||||||
|
kubectl -n api get networkpolicy -o yaml | grep -c 6379
|
||||||
|
# expected: 0 — no egress path from the gateway to unauthenticated Redis
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' https://kmsvc.riotpiao.com
|
||||||
|
# expected: unchanged from before this task — the existing gRPC surface is untouched
|
||||||
|
```
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# 7.3 — `/workflow/*` to Temporal (GREEN)
|
||||||
|
|
||||||
|
Phase: 7 — Additional capability prefixes
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [6.5](6.5-cutover.md)
|
||||||
|
|
||||||
|
Temporal runs in the `temporal` namespace. Temporal namespace registration is
|
||||||
|
automatic via queue-operator and is NEVER done manually — this route must not create,
|
||||||
|
register or mutate Temporal namespaces, only proxy to what queue-operator has already
|
||||||
|
provisioned.
|
||||||
|
|
||||||
|
- [ ] `/workflow/*` on `api.riotpiao.com` proxies to the Temporal Service in the `temporal` namespace
|
||||||
|
- [ ] Nothing in this route registers a Temporal namespace. Registration stays with queue-operator
|
||||||
|
- [ ] Path rewriting between the `/workflow` prefix and Temporal's own paths is explicit in configuration
|
||||||
|
- [ ] Timeouts and body caps are explicit, with no silent defaults (G6). Long-poll semantics are accounted for rather than truncated by a short read timeout
|
||||||
|
- [ ] The route requires authentication and the token is checked for workflow capability — a GPU token must not drive workflows
|
||||||
|
- [ ] The NetworkPolicy is extended to reach only Temporal
|
||||||
|
- [ ] Streaming or long-poll responses pass through unbuffered, and a client disconnect cancels the upstream call rather than orphaning it (G4)
|
||||||
|
- [ ] `/v1/*` behaviour is unchanged before and after
|
||||||
|
- [ ] Metrics and rejection counters cover this route with its own route label
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/workflow/health
|
||||||
|
# expected: 401 without a token
|
||||||
|
|
||||||
|
curl -s -H "authorization: Bearer $WORKFLOW_TOKEN" https://api.riotpiao.com/workflow/health
|
||||||
|
# expected: Temporal's own response, proxied unmodified
|
||||||
|
|
||||||
|
kubectl -n temporal exec svc/temporal-admintools -- tctl --ad temporal-frontend:7233 namespace list | sort > /tmp/ns.after
|
||||||
|
diff /tmp/ns.before /tmp/ns.after
|
||||||
|
# expected: no diff — the gateway registered nothing; queue-operator remains the only registrar
|
||||||
|
```
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# 7.4 — `/db/*` read surfaces (GREEN)
|
||||||
|
|
||||||
|
Phase: 7 — Additional capability prefixes
|
||||||
|
Stage: GREEN
|
||||||
|
Depends on: [6.5](6.5-cutover.md)
|
||||||
|
|
||||||
|
The most dangerous prefix in this phase. G2 says the gateway holds no cluster
|
||||||
|
credentials — so it cannot be the thing that authenticates to CloudNativePG, MinIO or
|
||||||
|
Prometheus on a caller's behalf. If designing this route makes you want to give the
|
||||||
|
gateway a secret, the design is wrong: put the credential-holding logic in a service
|
||||||
|
behind the gateway and proxy to that.
|
||||||
|
|
||||||
|
- [ ] `/db/*` on `api.riotpiao.com` exposes read surfaces backed by CloudNativePG, MinIO and monitoring
|
||||||
|
- [ ] The gateway holds no database password, no MinIO access key and no cluster credential of any kind (G2). Credentials, if any are needed, live in the service being proxied to
|
||||||
|
- [ ] Exposed operations are read-only. There is no write, no delete and no schema-changing path on this prefix
|
||||||
|
- [ ] Which sub-paths reach which upstream is enumerated explicitly in configuration. No catch-all, no pass-through of arbitrary query text
|
||||||
|
- [ ] Timeouts and body caps are explicit per sub-route, with no silent defaults (G6)
|
||||||
|
- [ ] The route requires authentication and the token is checked for a distinct read capability
|
||||||
|
- [ ] Result sets are paginated with a bounded page size; an unbounded read cannot be requested
|
||||||
|
- [ ] The NetworkPolicy is extended only to the specific upstreams reached. Note `prometheus-operated.monitoring` is headless with ClusterIP `None`, so it needs a pod selector, not a ClusterIP
|
||||||
|
- [ ] `/v1/*` behaviour is unchanged before and after
|
||||||
|
- [ ] Metrics and rejection counters cover this route with its own route label
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/db/healthz
|
||||||
|
# expected: 401 without a token
|
||||||
|
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' -X DELETE -H "authorization: Bearer $DB_READ_TOKEN" \
|
||||||
|
https://api.riotpiao.com/db/anything
|
||||||
|
# expected: 405 or 404 — no mutating method is routable on this prefix
|
||||||
|
|
||||||
|
kubectl -n api get pod -l app=homelab-frontend -o jsonpath='{range .items[0].spec.containers[0].env[*]}{.name}{"\n"}{end}' \
|
||||||
|
| grep -Ei 'password|secret|access_key'
|
||||||
|
# expected: no output — the gateway carries no database or object-store credentials
|
||||||
|
|
||||||
|
curl -s -H "authorization: Bearer $DB_READ_TOKEN" 'https://api.riotpiao.com/db/...?limit=100000' \
|
||||||
|
| python3 -c "import json,sys;d=json.load(sys.stdin);print(len(d['items']))"
|
||||||
|
# expected: capped at the configured maximum page size, not 100000
|
||||||
|
```
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
# Agent prompt template
|
||||||
|
|
||||||
|
Every task is worked by an agent starting from **zero context**. No memory of prior
|
||||||
|
tasks, no conversation history, no assumptions about what already exists.
|
||||||
|
|
||||||
|
This is deliberate. Task files are written to be self-contained precisely so that a
|
||||||
|
fresh agent can pick any one of them up. It also means a task that cannot be completed
|
||||||
|
from its own file plus this prompt is a task that is under-specified — that is a bug in
|
||||||
|
the task, and worth reporting rather than working around.
|
||||||
|
|
||||||
|
Rendered and invoked by [`scripts/run-task.sh`](../scripts/run-task.sh). Do not paste
|
||||||
|
this by hand; use the script so the fresh-session guarantee actually holds.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Template
|
||||||
|
|
||||||
|
`{{TASK_ID}}` and `{{TASK_FILE}}` are substituted by the runner.
|
||||||
|
|
||||||
|
```
|
||||||
|
You are implementing one task in the homelab-frontend repository: a Go API gateway
|
||||||
|
that replaces Kong OSS on api.riotpiao.com.
|
||||||
|
|
||||||
|
You are starting from zero context. Everything you need is below or in the files named
|
||||||
|
below. Do not assume any prior work exists beyond what you find in the repository.
|
||||||
|
|
||||||
|
## Your task
|
||||||
|
|
||||||
|
Read tasks/{{TASK_FILE}} and implement it. That file states what must be true; it does
|
||||||
|
not state how. The design is yours to reason out. The checkboxes are the contract.
|
||||||
|
|
||||||
|
## Before writing code
|
||||||
|
|
||||||
|
1. Read tasks/{{TASK_FILE}} in full.
|
||||||
|
2. Check its `Depends on:` line. If a dependency is not yet implemented in this
|
||||||
|
repository, stop and report that instead of building it yourself. One task per run.
|
||||||
|
3. Look at how the surrounding code is written and match it. If the repository is
|
||||||
|
still empty, you are establishing the conventions, so choose carefully.
|
||||||
|
|
||||||
|
## Invariants — breaking one of these is a design change, not a detail
|
||||||
|
|
||||||
|
G1 ingress-nginx owns TLS. The gateway never terminates TLS.
|
||||||
|
G2 The gateway holds no Kubernetes credentials. Config comes from git, not a CRD.
|
||||||
|
G3 Public surfaces use standard protocol shapes. If an OpenAI or Anthropic SDK
|
||||||
|
cannot call it unmodified, the design is wrong.
|
||||||
|
G4 Streaming is unbuffered, and a client disconnect cancels the upstream request.
|
||||||
|
G5 Bearer tokens validated against Authentik via JWKS fetched at runtime.
|
||||||
|
G6 Every timeout, body cap and concurrency limit is explicit in configuration.
|
||||||
|
G7 Deployment flows through git and Argo. Never run kubectl apply, helm upgrade,
|
||||||
|
or terraform apply.
|
||||||
|
|
||||||
|
## How to work
|
||||||
|
|
||||||
|
Test-driven. If the task is marked `Stage: RED`, write the failing test first and
|
||||||
|
confirm it fails for the right reason before implementing. If `GREEN`, write the
|
||||||
|
minimum code that passes. If `REFACTOR`, keep the tests green while improving shape.
|
||||||
|
|
||||||
|
Go standards for this repository:
|
||||||
|
- Never discard errors with `_ =`. Wrap them with context.
|
||||||
|
- Every upstream call carries a context.Context.
|
||||||
|
- No naked returns. Use `any`, not `interface{}`.
|
||||||
|
- Table-driven tests with subtests where there is more than one case.
|
||||||
|
|
||||||
|
## Definition of done
|
||||||
|
|
||||||
|
Run the `## Verify` block from the task file. It must pass.
|
||||||
|
|
||||||
|
Then run all three of these regardless of what the task's verify block says:
|
||||||
|
|
||||||
|
go test ./... -race
|
||||||
|
CGO_ENABLED=0 go build ./...
|
||||||
|
go vet ./...
|
||||||
|
|
||||||
|
`-race` is mandatory. This is a concurrent proxy; a test suite that passes without the
|
||||||
|
race detector tells you almost nothing. A detected race is a failure, not a warning.
|
||||||
|
|
||||||
|
Verification means asserting on a real HTTP response — status, headers, body. "It
|
||||||
|
compiles" and "it starts" are not verification. If you cannot run the verification
|
||||||
|
locally with no cluster and no credentials, that is itself a problem to report.
|
||||||
|
|
||||||
|
Then **edit tasks/{{TASK_FILE}} and change `- [ ]` to `- [x]`** for each criterion you
|
||||||
|
actually satisfied. This is a file edit, not something to state in your summary. Leave
|
||||||
|
unticked anything you did not complete. A summary that claims `[x]` while the file still
|
||||||
|
reads `[ ]` is a false report.
|
||||||
|
|
||||||
|
## Hard rules
|
||||||
|
|
||||||
|
- Kong is serving live traffic on api.riotpiao.com right now. Change no cluster state.
|
||||||
|
- Do not commit or push.
|
||||||
|
- Do not implement tasks other than {{TASK_ID}}. If you notice something else that
|
||||||
|
needs doing, report it rather than fixing it.
|
||||||
|
- Do not add features, abstractions, or configurability that the task did not ask for.
|
||||||
|
- If the task is ambiguous or appears wrong, stop and say so. Do not guess and proceed.
|
||||||
|
|
||||||
|
## Report when finished
|
||||||
|
|
||||||
|
- What you implemented, and the files you touched.
|
||||||
|
- The verification command you ran and its actual output.
|
||||||
|
- Which checkboxes you ticked and which you did not, with reasons.
|
||||||
|
- Anything you found that is wrong elsewhere in the repository or the task files.
|
||||||
|
```
|
||||||
+151
@@ -0,0 +1,151 @@
|
|||||||
|
# Task board — homelab-frontend
|
||||||
|
|
||||||
|
The Go API gateway replacing Kong OSS on `api.riotpiao.com`.
|
||||||
|
|
||||||
|
Contract: [REQUIREMENTS.md](../REQUIREMENTS.md).
|
||||||
|
Why: [ADR-0001](../docs/adr/ADR-0001-retire-kong-for-go-gateway.md).
|
||||||
|
What Kong does today and the cutover order: [docs/MIGRATION-kong.md](../docs/MIGRATION-kong.md).
|
||||||
|
|
||||||
|
## Rules carried from the ADR and requirements
|
||||||
|
|
||||||
|
- G1 — ingress-nginx owns TLS. The gateway never terminates TLS.
|
||||||
|
- G2 — the gateway holds no Kubernetes credentials. Config comes from git, not a CRD.
|
||||||
|
- G3 — public surfaces use standard protocol shapes. If an OpenAI SDK can't call it unmodified, it's wrong.
|
||||||
|
- G4 — streaming is unbuffered, and a client disconnect cancels the upstream.
|
||||||
|
- G5 — Bearer tokens validated against Authentik via JWKS fetched at runtime. No pinned keys.
|
||||||
|
- G6 — every timeout, body cap and concurrency limit is explicit in config.
|
||||||
|
- G7 — deployment flows through git and Argo. No `kubectl apply`, no `helm upgrade`.
|
||||||
|
|
||||||
|
## How to work these
|
||||||
|
|
||||||
|
Each task is self-contained — it states what must be true, not how to build it.
|
||||||
|
Reason out the implementation; the acceptance criteria are the contract.
|
||||||
|
|
||||||
|
Stages follow red-green-refactor. A task marked RED means the test comes first and
|
||||||
|
must fail for the right reason before any implementation exists.
|
||||||
|
|
||||||
|
**Verification means asserting on a real HTTP response** — status, headers, body.
|
||||||
|
"It compiles" and "it starts" are not verification. Every task that touches an API
|
||||||
|
surface has a `## Verify` block with a runnable command.
|
||||||
|
|
||||||
|
Kong is serving live traffic throughout phases 0–5. Nothing in those phases may
|
||||||
|
change cluster state.
|
||||||
|
|
||||||
|
## Phase 0 — Foundations
|
||||||
|
|
||||||
|
| Task | Description |
|
||||||
|
|---|---|
|
||||||
|
| [0.1](0.1-module-and-entrypoint.md) | Go module, entrypoint, graceful shutdown |
|
||||||
|
| [0.2](0.2-route-configuration.md) | Declarative route/upstream config from git, fail-loud validation |
|
||||||
|
| [0.3](0.3-health-endpoints.md) | `/healthz` and `/readyz` |
|
||||||
|
| [0.4](0.4-local-dev-harness.md) | Run with no cluster, no kubeconfig, no credentials — stub upstreams |
|
||||||
|
| [0.5](0.5-structured-logging.md) | Structured logs, no secrets or bodies |
|
||||||
|
| [0.6](0.6-ci-pipeline.md) | CI: build, vet, test, `govulncheck` |
|
||||||
|
|
||||||
|
## Phase 1 — Proxy core
|
||||||
|
|
||||||
|
| Task | Description |
|
||||||
|
|---|---|
|
||||||
|
| [1.1](1.1-reverse-proxy.md) | Reverse proxy to a configured upstream, connection reuse |
|
||||||
|
| [1.2](1.2-streaming-passthrough.md) | SSE and chunked responses pass through unbuffered |
|
||||||
|
| [1.3](1.3-disconnect-propagation.md) | Client disconnect cancels the upstream request |
|
||||||
|
| [1.4](1.4-per-route-timeouts.md) | Explicit connect/read/write timeouts per route |
|
||||||
|
| [1.5](1.5-header-hygiene.md) | Hop-by-hop stripping, `X-Forwarded-*` from nginx |
|
||||||
|
| [1.6](1.6-websocket-upgrade.md) | WebSocket upgrade — `agent-pod/console` needs it |
|
||||||
|
| [1.7](1.7-body-size-caps.md) | Per-route request body limits |
|
||||||
|
|
||||||
|
## Phase 2 — LLM surfaces (`/v1/*` and `/llm/*`)
|
||||||
|
|
||||||
|
Two protocol dialects over the same models and the same slot controller. Wire
|
||||||
|
formats are documented in [docs/API-llm.md](../docs/API-llm.md).
|
||||||
|
|
||||||
|
### OpenAI dialect — `/v1/*`
|
||||||
|
|
||||||
|
| Task | Description |
|
||||||
|
|---|---|
|
||||||
|
| [2.1](2.1-model-registry.md) | Model → upstream map from config |
|
||||||
|
| [2.2](2.2-body-based-dispatch.md) | `POST /v1/chat/completions` routes on the body's `model` — the reason this project exists |
|
||||||
|
| [2.3](2.3-unknown-model-errors.md) | Unknown/missing model → RFC 9457 problem+json listing valid values |
|
||||||
|
| [2.4](2.4-legacy-path-aliases.md) | Keep `/v1/{reasoning,ornith,qwen}/chat/completions` working during cutover |
|
||||||
|
| [2.5](2.5-models-endpoint.md) | `GET /v1/models` derived from config, never hardcoded |
|
||||||
|
| [2.6](2.6-embeddings-passthrough.md) | `POST /v1/embeddings` — no rewrite needed |
|
||||||
|
| [2.7](2.7-rerank-rewrite.md) | `POST /v1/rerank` → upstream `/rerank` |
|
||||||
|
| [2.8](2.8-kong-parity-test.md) | Gateway and Kong return equivalent responses for every migrated route |
|
||||||
|
|
||||||
|
### Anthropic dialect — `/llm/*`
|
||||||
|
|
||||||
|
| Task | Description |
|
||||||
|
|---|---|
|
||||||
|
| [2.9](2.9-canonical-request-model.md) | Dialect-neutral internal request both surfaces translate into |
|
||||||
|
| [2.10](2.10-anthropic-request-translation.md) | `POST /llm/v1/messages` request → canonical; `system`, blocks, required `max_tokens` |
|
||||||
|
| [2.11](2.11-anthropic-response-translation.md) | Upstream response → Messages shape; `reasoning_content` becomes a `thinking` block |
|
||||||
|
| [2.12](2.12-anthropic-sse-state-machine.md) | Named-event SSE with block indices — the hardest task in the phase |
|
||||||
|
| [2.13](2.13-anthropic-error-shape.md) | Anthropic error shape, not RFC 9457 — same rejection, two renderings |
|
||||||
|
| [2.14](2.14-queue-position-event.md) | Custom `event: queue` before `message_start` — deliberate non-standard extension |
|
||||||
|
| [2.15](2.15-dialect-scope-boundary.md) | Enforce what is deliberately unimplemented: tools, images, caching, batch |
|
||||||
|
|
||||||
|
## Phase 3 — Authentication (Authentik)
|
||||||
|
|
||||||
|
| Task | Description |
|
||||||
|
|---|---|
|
||||||
|
| [3.1](3.1-jwks-fetch-and-rotation.md) | Fetch and cache Authentik JWKS, handle rotation without a runbook |
|
||||||
|
| [3.2](3.2-bearer-validation.md) | Validate `Authorization: Bearer` — the thing Kong OSS could not do |
|
||||||
|
| [3.3](3.3-authentik-service-account.md) | Service account + `client_credentials` provider in Authentik |
|
||||||
|
| [3.4](3.4-flag-gated-rollout.md) | Auth defaults off; enabling it is deliberate |
|
||||||
|
| [3.5](3.5-capability-authorization.md) | A queue token must not invoke a GPU |
|
||||||
|
| [3.6](3.6-pi-client-migration.md) | Move pi off the `apikey` header onto Bearer |
|
||||||
|
|
||||||
|
## Phase 4 — Limits and budgets
|
||||||
|
|
||||||
|
| Task | Description |
|
||||||
|
|---|---|
|
||||||
|
| [4.1](4.1-gpu-slot-semaphore.md) | Cap concurrent `reasoning` requests below 8 slots, bounded queue |
|
||||||
|
| [4.2](4.2-per-caller-budgets.md) | Request budget per identified caller per window |
|
||||||
|
| [4.3](4.3-problem-json-errors.md) | RFC 9457 rejections with `Retry-After` |
|
||||||
|
|
||||||
|
## Phase 5 — Observability
|
||||||
|
|
||||||
|
| Task | Description |
|
||||||
|
|---|---|
|
||||||
|
| [5.1](5.1-prometheus-parity.md) | Match the retiring Kong plugin: rate, latency, status, bandwidth, upstream health |
|
||||||
|
| [5.2](5.2-gateway-metrics.md) | In-flight per upstream, queue depth, slot occupancy, rejections by reason |
|
||||||
|
| [5.3](5.3-servicemonitor.md) | ServiceMonitor so Prometheus scrapes it |
|
||||||
|
|
||||||
|
## Phase 6 — Deploy and cutover
|
||||||
|
|
||||||
|
| Task | Description |
|
||||||
|
|---|---|
|
||||||
|
| [6.1](6.1-hardened-image.md) | Distroless, non-root, read-only rootfs, no shell, SHA tags |
|
||||||
|
| [6.2](6.2-kubernetes-manifests.md) | Deployment, Service, NetworkPolicy |
|
||||||
|
| [6.3](6.3-argocd-application.md) | Argo Application in the homelab-root GitOps repo |
|
||||||
|
| [6.4](6.4-deploy-alongside-kong.md) | Deploy unexposed, verify in-cluster against the real upstreams |
|
||||||
|
| [6.5](6.5-cutover.md) | Repoint the nginx Ingress from `kong-proxy` to the gateway — reversible |
|
||||||
|
| [6.6](6.6-kong-teardown.md) | Delete kong Ingresses, plugins, Helm release. **Irreversible** |
|
||||||
|
|
||||||
|
## Phase 7 — Additional capability prefixes
|
||||||
|
|
||||||
|
Deliberately after cutover. Each is additive and must not disturb `/v1/*`.
|
||||||
|
|
||||||
|
| Task | Description |
|
||||||
|
|---|---|
|
||||||
|
| [7.1](7.1-cluster-prefix-atlas.md) | `/cluster/*` → atlas (`riotpiao-backend`) |
|
||||||
|
| [7.2](7.2-sqs-prefix.md) | `/sqs/*` → kmsvc management-service, Kafka |
|
||||||
|
| [7.3](7.3-workflow-prefix.md) | `/workflow/*` → Temporal |
|
||||||
|
| [7.4](7.4-db-prefix.md) | `/db/*` → CloudNativePG, MinIO, monitoring reads |
|
||||||
|
|
||||||
|
## Progress
|
||||||
|
|
||||||
|
Status as of 2026-08-19: scaffolded, nothing implemented. Kong is still serving all
|
||||||
|
live traffic on `api.riotpiao.com`, currently **unauthenticated**.
|
||||||
|
|
||||||
|
50 tasks. Suggested first slice: 0.1 → 0.2 → 0.4 → 1.1 → 1.2 → 2.1 → 2.2. That
|
||||||
|
reaches the single capability Kong could not provide — body-based model dispatch —
|
||||||
|
with a verification loop that needs no cluster.
|
||||||
|
|
||||||
|
The Anthropic dialect (2.9-2.15) can be worked in parallel with the OpenAI dialect
|
||||||
|
once 2.9 lands, since both translate into the same canonical request. Do not build
|
||||||
|
either surface's admission control separately — 4.1 owns it for both.
|
||||||
|
|
||||||
|
Decided 2026-08-19: authentication is `Authorization: Bearer` on **both** surfaces.
|
||||||
|
A stock Anthropic SDK sends `x-api-key` and will get a 401; that is accepted because
|
||||||
|
the `/llm` client is first-party. The 401 must say so rather than being bare.
|
||||||
+17
@@ -0,0 +1,17 @@
|
|||||||
|
routes:
|
||||||
|
- name: test-route
|
||||||
|
upstream:
|
||||||
|
address: host1.llm-serving:80
|
||||||
|
connectTimeout: "30s"
|
||||||
|
readTimeout: "5m"
|
||||||
|
writeTimeout: "1m"
|
||||||
|
maxBodySize: 10485760
|
||||||
|
authRequired: true
|
||||||
|
- name: test-route
|
||||||
|
upstream:
|
||||||
|
address: host2.llm-serving:80
|
||||||
|
connectTimeout: "30s"
|
||||||
|
readTimeout: "5m"
|
||||||
|
writeTimeout: "1m"
|
||||||
|
maxBodySize: 20971520
|
||||||
|
authRequired: false
|
||||||
Vendored
+31
@@ -0,0 +1,31 @@
|
|||||||
|
# Local development harness config. Every upstream points at the single stub
|
||||||
|
# server started by internal/testsupport, on loopback only — no cluster address,
|
||||||
|
# no *.riotpiao.com, no credentials. Route names match testdata/config/valid.yaml
|
||||||
|
# so the same assertions work against either fixture.
|
||||||
|
routes:
|
||||||
|
- name: reasoning-chat
|
||||||
|
upstream:
|
||||||
|
address: 127.0.0.1:9080
|
||||||
|
connectTimeout: "10s"
|
||||||
|
readTimeout: "1h"
|
||||||
|
writeTimeout: "1h"
|
||||||
|
maxBodySize: 10485760
|
||||||
|
authRequired: false
|
||||||
|
|
||||||
|
- name: ornith-chat
|
||||||
|
upstream:
|
||||||
|
address: 127.0.0.1:9080
|
||||||
|
connectTimeout: "5s"
|
||||||
|
readTimeout: "10m"
|
||||||
|
writeTimeout: "3m"
|
||||||
|
maxBodySize: 5242880
|
||||||
|
authRequired: false
|
||||||
|
|
||||||
|
- name: embeddings
|
||||||
|
upstream:
|
||||||
|
address: 127.0.0.1:9080
|
||||||
|
connectTimeout: "5s"
|
||||||
|
readTimeout: "30m"
|
||||||
|
writeTimeout: "2m"
|
||||||
|
maxBodySize: 5242880
|
||||||
|
authRequired: false
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
routes:
|
||||||
|
- name: test-route
|
||||||
|
upstream:
|
||||||
|
address: localhost
|
||||||
|
connectTimeout: "30s"
|
||||||
|
readTimeout: "5m"
|
||||||
|
writeTimeout: "1m"
|
||||||
|
maxBodySize: 10485760
|
||||||
|
authRequired: true
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
routes:
|
||||||
|
- name: test-route
|
||||||
|
upstream:
|
||||||
|
address: test-host.llm-serving:80
|
||||||
|
connectTimeout: "not-a-duration"
|
||||||
|
readTimeout: "5m"
|
||||||
|
writeTimeout: "1m"
|
||||||
|
maxBodySize: 10485760
|
||||||
|
authRequired: true
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user