rock/homelab-frontend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: consolidate docs, write unified API reference
CI / Vet, test, build (push) Canceled after 20s
Details
CI / Build and push image (push) Canceled after 0s
Details

Browse Source 
Deleted:
- 43 outdated/completed task files (phases 0-8)
- All design docs (REQUIREMENTS, ADR, routing design, etc)
- Phase-specific docs (temporal, JWT, tool calls, testing guides)
- Redundant API docs (API-llm, API-sqs, SERVICE-USAGE)

Kept: README.md (project overview)

New: Comprehensive API.md
- Single source of truth for api.riotpiao.com
- All services in one place: LLM, workflows, queues, memory, IAM, S3
- Complete request/response examples
- Authentication via JWT bearer tokens + capabilities
- Error handling (RFC 9457)
- Rate limits, timeouts, status codes
- Real-world examples (RAG pipeline, workflow orchestration)

Benefits:
✅ Developer finds everything in API.md
✅ No duplicate/stale docs
✅ Reduced maintenance burden
✅ Single source of truth
This commit is contained in:
Admin Bot
2026-08-29 22:38:20 -07:00
parent 4935ea9f95
commit a0995edbd0
52 changed files with 588 additions and 7799 deletions
Download Patch File Download Diff File Expand all files Collapse all files
API.md
+601 -750
View File
File diff suppressed because it is too large Load Diff
API_ROUTING_HYBRID_DESIGN.md
-532
View File
@@ -1,532 +0,0 @@
# API Routing Design: Header-Based Services, CRD-Driven Onboarding, Unified Auth
Supersedes the earlier `/unified`-endpoint draft of this doc. Routing is now
header-only at the gateway root — `api.riotpiao.com` itself is the single entry
point, no dedicated sub-path. New backend services onboard via a CRD the gateway
consumes directly, not via hand-written Go switch cases.
---
## Context
Most of the original hybrid design was aspirational — it didn't exist in code. This
revision reflects decisions made while reviewing the repo against the actual state:
`internal/auth/` was empty, `s3`/`sqs`/`iam` didn't exist as handlers, and only a
body-action `POST /workflow` endpoint was real. Four extensions were decided:
CRD-driven service onboarding, `X-Resource`-based REST verbs replacing body actions,
unified JWT auth (bearer token or service-account password, one validation path), and
blind 5xx retry.
Known constraints, and how these decisions land on them:
- Rule G2 (originally specced in the now-retired `tasks/6.2-kubernetes-manifests.md`,
deleted 2026-08-25 since Phase 6 shipped and is live in-cluster) mandated **no
ServiceAccount/RBAC** on the gateway pod, and routes loaded from a git-sourced
ConfigMap with a pod roll on every change. The gateway pod now consumes the CRD
directly — no separate operator/controller pod. This is a deliberate, acknowledged
supersession of G2 for this one narrow purpose: the existing `api-gateway`
`ServiceAccount` (already bound to the pod at `k8s/rbac.yaml:1-7`, currently with
zero Role — see `k8s/rbac.yaml:9-10`'s own comment "the gateway has no k8s API
access") gains a namespace-scoped `Role` limited to read-only access on one CRD,
defined in this same repo and synced by the existing `api-gw` ArgoCD Application
(`homelab/k8s/argocd/apps/55-api-gateway.yaml`) — no new repo, no new Application.
Static ConfigMap-sourced routes keep the existing roll-on-change behavior;
CRD-sourced routes update live in-process instead (see §1).
- Retired `kmsvc.io` `TemporalWorker`/`Queue` CRD (`homelab/k8s/apps/messaging/queue-crd/`,
controller in the separate `kmsvc-manage` repo) broke via an uncleared finalizer and
RBAC rules that drifted out of sync with the CRD's resources
(`temporalworkers.kmsvc.io is forbidden ... at the cluster scope`). Cluster-scoped
RBAC, RBAC/CRD drift, and finalizer complexity are explicitly avoided this time —
this CRD is namespace-scoped (`api` only) and read-only from the gateway's side.
---
## 0. Repo hygiene — do this first, before any code
**Already staged for deletion (`git status`) — just needs a commit:**
`DELIVERY_COMPLETE.md`, `IMPLEMENTATION_SUMMARY.md`, `PHASE3_GRPC_IMPLEMENTATION.md`,
`PHASE3_PROGRESS.md`, `TEMPORAL_API_DESIGN_SUMMARY.md`,
`TEMPORAL_IMPLEMENTATION_CHECKPOINT.md`, `TEMPORAL_IMPLEMENTATION_COMPLETE.md`,
`TEMPORAL_TEST_REPORT.md`, `WORKFLOWS_INDEX.md`, `WORKFLOWS_QUICK_START.md`,
`WORKFLOWS_README.md`.
**Removed as part of this revision:** `IMPLEMENTATION_PLAN_UPDATED.md` — its reading
order pointed at this doc's now-superseded `/unified`-endpoint design; folded into
this doc instead of kept as a separate index.
**Untracked, flagged but left alone (different topic, not touched by this doc):**
`AUTHENTIK_AS_KEY_MANAGER.md`, `AUTHENTIK_KEY_ROTATION_SIMPLE.md`,
`AUTHENTIK_SOPS_NATIVE.md`, `AUTHENTIK_WITH_FIPS_HSM.md`, `JWT_VS_RANDOM_TOKEN.md` —
SOPS secrets-at-rest key rotation via Authentik, unrelated to API routing/CRD work.
---
## 1. Service onboarding (new backend service = code change vs. YAML)
### Existing System
- No CRD, no controller, no generic service registry anywhere in either repo.
- Adding a service today means: write a new Go handler, add a `switch` case, redeploy
the gateway image. Only `workflow` exists (`internal/temporal/handler.go`); `s3`,
`sqs`, `iam` don't exist in code at all.
- `internal/config/config.go` + `loader.go` load only static, hand-edited route config
from the ConfigMap already live per Phase 6 (`k8s/configmap.yaml`) — no per-service
schema, no CRD.
### Future System
New CRD **`ServiceAdapter`** (`gateway.riotpiao.com/v1`, namespace-scoped to `api`
only — deliberately *not* cluster-scoped, unlike the retired CRD). Named for what it
actually does — adapts an in-cluster service to the gateway's
`X-Service`/`X-Resource` contract — rather than a generic "Definition".
**No separate operator pod.** The API gateway pod itself consumes the CRD directly: it
runs a `client-go` informer against `ServiceAdapter` in namespace `api` and keeps an
in-memory route registry that adds/updates/removes entries as CRs change — no
ConfigMap render step, no restart-to-pick-up-changes, no second binary/Deployment/image
to build and operate.
**Where the CRD is defined — this repo, not `homelab`.** Confirmed at
`homelab/k8s/argocd/apps/55-api-gateway.yaml:42-45`: the `api-gw` ArgoCD Application
already sources this repo's own `k8s` path directly (kustomize — Deployment, Service,
ConfigMap, RBAC, NetworkPolicy), auto-syncing on every push to `main`, wave 7. So the
CRD manifest is just another file in that same kustomization: new
`k8s/crd-serviceadapter.yaml`, added to `k8s/kustomization.yaml`'s `resources:` list.
When a new `ServiceAdapter` CR is introduced (i.e. someone commits a CR YAML for a new
service), ArgoCD syncs it through this same already-existing Application, and the
already-running `api-gw` pod's informer notices it and starts accepting
`X-Service: <new-service>` calls — no new Application, no new repo, no manual step in
`homelab` at all. Go API types back the CRD from
`apis/gateway/v1/serviceadapter_types.go` (new package, generates the CRD YAML via
`controller-gen`, same struct also used by the informer).
`spec.auth.capability` is the default capability required for every resource on the
service. A method entry may override it with its own `auth.capability` — needed once
a service splits read vs. write access (see §6's `memory` adapter). No override means
inherit the spec-level default; `auth.required: false` at either level means no
capability check at all for that scope.
**Example CR** (onboarding a hypothetical `postgres` service — no Go code, no image
rebuild, no manual gateway restart):
```yaml
apiVersion: gateway.riotpiao.com/v1
kind: ServiceAdapter
metadata:
name: postgres
namespace: api
spec:
serviceName: postgres # matches X-Service: postgres
upstream:
url: http://kmsvc-postgres-helper.sqs.svc.cluster.local:8080
timeoutSeconds: 10
auth:
required: true
capability: postgres:access # checked against the caller's JWT claims
retryable: true # eligible for the blind 5xx retry middleware, §4
resources:
- name: query # X-Resource: query
methods:
- verb: POST
upstreamPath: /v1/query
requestSchema:
fields:
sql: string
params: array
required: ["sql"]
responseSchema:
fields:
rows: array
- name: table
methods:
- verb: GET
upstreamPath: /v1/tables/{id} # {id} filled from the trailing path segment
- verb: DELETE
upstreamPath: /v1/tables/{id}
status:
phase: Ready
observedGeneration: 1
```
Calling it once reconciled — straight to the gateway root, no dedicated sub-path:
```bash
curl -X POST https://api.riotpiao.com/ \
-H 'Authorization: Bearer <jwt>' \
-H 'X-Service: postgres' \
-H 'X-Resource: query' \
-d '{"sql": "select 1"}'
```
**RBAC** — extend `k8s/rbac.yaml` in place (same file that already defines the
`api-gateway` `ServiceAccount` at lines 1-6, currently role-less per its own "the
gateway has no k8s API access" comment). A namespace-scoped `Role` (not `ClusterRole`),
read-only, nothing else — this is the entire RBAC footprint added to the gateway pod,
shipped in the same repo/PR as the CRD itself:
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: api-gateway-serviceadapter-reader
namespace: api
rules:
- apiGroups: ["gateway.riotpiao.com"]
resources: ["serviceadapters"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: api-gateway-serviceadapter-reader
namespace: api
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: api-gateway-serviceadapter-reader
subjects:
- kind: ServiceAccount
name: api-gateway
namespace: api
```
No `status`/`finalizers` verbs, no write access at all — the gateway only ever reads.
No new `ServiceAccount`, no new Deployment, no new container image.
**Gateway-side consumer** — new `internal/serviceadapter/registry.go`: a `client-go`
`SharedInformer` on `ServiceAdapter` in namespace `api`, feeding an in-memory map keyed
by `spec.serviceName` that the `internal/serviceadapter` dispatcher (§2) reads on every
request. Add/Update/Delete informer callbacks update the map directly — a new
`ServiceAdapter` CR is live within the informer's resync/watch latency, no gateway
restart, no ArgoCD change beyond the existing auto-sync of this repo's `k8s` path.
**Request/response validation** — `requestSchema`/`responseSchema` are not JSON
Schema. Deliberately a flat key→type map, author-facing (whoever writes the CR
specifies it, no JSON Schema knowledge needed):
```yaml
requestSchema:
fields:
sql: string # one of: string | number | boolean | array | object
params: array
required: ["sql"] # optional; omit if every field is optional
strict: false # optional, default false — true rejects unlisted keys
responseSchema:
fields:
rows: array
```
One level deep only — `params: array` checks "is this an array," not what's
inside it. No enums, no formats, no nested field validation. Accepted limitation:
none of the adapters in this doc (postgres, workflow, iam, memory) need more than
top-level shape checking.
**Extension forced by real upstream shapes (poimen-memory, §6):** the DSL above
only describes a top-level JSON *object*. `GET /memory/projects` returns a bare
array (`["poimen", ...]`), `GET /memory/query` returns an array of objects, and
`GET /memory/skills` returns objects with an explicit `"generated_from": null`.
Neither is expressible in the object-only form, so two additions, kept minimal
rather than backsliding toward full JSON Schema:
```yaml
responseSchema:
type: array # top-level shape; default "object" if omitted
items: string # scalar array — every element must be this type
responseSchema:
type: array
items:
fields: # object array — one level deep INSIDE each item
level: string
sha256: string
score: number
responseSchema:
fields:
generated_from: { type: string, nullable: true } # long form only needed for nullable
```
Rule: a field written as a bare type string (`sql: string`) is shorthand for
`{type: string, nullable: false}`. `null` satisfies any field marked
`nullable: true` regardless of its declared type; a `null` on a non-nullable
field is a `type_mismatch`. Still one level deep — `items: { fields: {...} }`
does not itself accept a nested `items`.
Validated in Go via `internal/serviceadapter/validate.go` (new, ~40 lines, no
external dependency): `json.Unmarshal` the body into `map[string]interface{}`,
check `Required` fields are present, check each present field in `Fields` against
its Go runtime type (`string`→string, `float64`→number, `bool`→boolean,
`[]interface{}`→array, `map[string]interface{}`→object), and if `strict: true`
reject any body key not listed in `Fields`. Compiled once per CR add/update in the
informer callback above, not per-request — the parsed `FieldSchema` struct is
stored directly in the same registry map entry as the route.
Request-side violations are enforced: 400, RFC 9457 `problem+json`, body
`{type, title, detail, errors: [{field, reason}]}` where `reason` is one of
`missing`, `type_mismatch: want X got Y`, or `unknown_field` (strict mode only).
Response-side violations are **not** enforced — the response is forwarded to the
caller unchanged; a mismatch only emits a
`serviceadapter_response_schema_mismatch{service,resource}` metric and a log line.
Rationale: request validation is a trust boundary (external caller input);
response validation is contract-drift detection against our own upstream — turning
a backend field-shape typo into a caller-facing 502 does more harm than logging it.
---
## 2. Request shape: body action vs. header verb
### Existing System
- `internal/temporal/handler.go`: `POST /workflow` only, body is
`{action, namespace, payload}` (`RequestPayload` struct) — action dispatch happens
by reading and branching on the JSON body, not by header/method.
- `internal/server/router.go` dispatches purely by **path**: exact-matches
`/healthz`, `/readyz`, `/workflow`, `/workflow/health`, `/workflow/metrics`, and
falls through everything else (including `/v1/chat/completions`) to
`upstreamHandler`. There is no header-based dispatch anywhere in the router today.
- `s3`/`sqs`/`iam` bodies with `operation`/`action` fields, and a `/unified` sub-path,
were only ever prose in an earlier draft of this doc — never implemented, and now
dropped.
### Future System
No new sub-path is introduced. `internal/server/router.go`'s `ServeHTTP` gets one new
branch, checked **before** the existing path switch: if the request carries an
`X-Service` header, dispatch straight to a new `internal/serviceadapter/router.go`
dispatcher, regardless of `r.URL.Path` (callers hit `https://api.riotpiao.com/` at the
root). `/v1/chat/completions`, `/v1/embeddings`, `/v1/rerank`, `/healthz`, `/readyz`
keep their existing path-based routing in `internal/proxy/router.go` — those never
carry `X-Service` and are matched first. `/workflow*` stays mounted during migration,
delegating internally to the same dispatcher, then gets removed once callers move to
header-based calls.
Dispatch key becomes **`X-Service` header → adapter**, then **HTTP method +
`X-Resource` header → `{verb, upstreamPath}`** — e.g. `GET` to the gateway root +
`X-Service: iam` + `X-Resource: user` → list Authentik users. This replaces the body
`action`/`operation` field convention everywhere, not just for `iam` — so the
rewritten `workflow` adapter looks like:
- `POST X-Resource: workflow` → `START_WORKFLOW`
- `GET X-Resource: workflow/{id}` → `QUERY_WORKFLOW`
- `GET X-Resource: workflow` → `LIST_WORKFLOWS`
- `DELETE X-Resource: workflow/{id}` → `TERMINATE_WORKFLOW`
- `GET X-Resource: workflow/{id}/history` → `GET_WORKFLOW_HISTORY`
- Same pattern for new `s3` (`bucket/{key}`) and `sqs` (`queue/{name}/message`)
adapters — built fresh, since they don't exist in code today.
`/v1/chat/completions`, `/v1/embeddings`, `/v1/rerank` in `internal/proxy/router.go`
are **unchanged** — local-LLM-only, no `X-Service` involvement.
---
## 3. IAM access to Authentik
### Existing System
- No `X-Service: iam` adapter in the gateway.
- Authentik admin API access exists today only client-side, in the Rust `core` CLI
(`/Users/rockliang/workplace/core/src/cmd/iam/*.rs`), which already calls
`{authentik_url}/api/v3/core/applications/`, `/api/v3/providers/oauth2/`,
`/api/v3/core/flows/`, `/api/v3/crypto/certificatekeypairs/` for apps/providers/flows
— confirmed at `core/src/cmd/iam/app.rs:136,160,191,205,219`. No users/groups/roles
coverage there, and none of it is reachable through the gateway.
### Future System
New `internal/iam/handler.go` server-side adapter, reachable at the gateway root with
`X-Service: iam`, mapping `X-Resource` + method onto Authentik's `/api/v3/...` surface:
| X-Resource | Method | Authentik endpoint |
|---|---|---|
| `user` | GET/POST | `/api/v3/core/users/` |
| `user/{id}` | GET/PATCH/DELETE | `/api/v3/core/users/{id}/` |
| `service-account` | POST | `/api/v3/core/users/service_account/` |
| `role` (group) | GET/POST | `/api/v3/core/groups/` |
| `permission` | GET/POST | `/api/v3/rbac/permissions/` |
| `flow` | GET | `/api/v3/flows/instances/` |
This is additive to, not a replacement of, the `core iam` CLI subcommand — different
caller (server vs. local CLI), same upstream API.
---
## 4. Auth: how a caller proves identity
### Existing System
- `internal/auth/` is an **empty directory** — zero implementation. `tasks/` Phase 3
(JWKS fetch/rotation, Bearer validation, Authentik service account, flag-gated
rollout, capability authorization) was retired 2026-08-25 — this doc is now the spec
for that work instead of those deleted task files.
- `kmsvc.riotpiao.com` (separate service) has an auth interceptor
(`kmsvc-manage/internal/api/interceptors/auth.go`, JWKS validation via
`lestrrat-go/jwx/v2/jwt`) but it's **never wired in** — `cmd/server/main.go` builds a
bare `grpc.NewServer()` (`kmsvc-manage/cmd/server/main.go:84`), and the REST surface
bypasses gRPC interceptors entirely even once fixed. Useful code to reference, not to
copy the deployment of.
- The `core` CLI already does human OIDC login (`core auth login` — device-code/OOB
against Authentik) and caches a token at `~/.cache/talos/authentik_id_token`
(`core/src/cmd/auth/mod.rs:336-338`) — a **different** token/cache than what §5 adds.
### Future System
Implement in `internal/auth`, with two entry paths converging on one validated-JWT
middleware — no separate downstream logic per path:
1. **Bearer passthrough** — `Authorization: Bearer <jwt>` sourced from
`~/.talos/.riotpiao-auth` (written by `mwinit`, §5), validated against Authentik's
JWKS.
2. **Service-account password grant** — Authentik service-account username/password
exchanged for a JWT. Reuse the validation *logic* already written (but unwired) in
`kmsvc-manage/internal/api/interceptors/auth.go`, not its deployment config.
Retry middleware (`internal/resilience/retry.go`, new) wraps every outbound upstream
call from the `internal/serviceadapter` dispatcher: **blind retry on any 5xx**, bounded
attempts with backoff, no idempotency-key gating — Temporal (and other backends) own
dedup on their side. `ServiceAdapter.spec.retryable` (default `true`) is the only
opt-out.
---
## 5. `CLI-MWNINIT.md` (separate doc)
### Existing System
The Rust `core` CLI (`/Users/rockliang/workplace/core`) has `core auth login`
(`src/cmd/auth/mod.rs`) for human OIDC device-code/OOB login, and a separate `core iam`
subcommand tree (`src/cmd/iam/`) for app/group/scope management. There is no
service-account login path and no `~/.talos/` token file — the existing cache is
`~/.cache/talos/authentik_id_token`, and `~/.talos` today only holds Talos *node*
config (`TALOSCONFIG`, per `core/src/config.rs:25-26`), unrelated to Authentik JWTs.
### Future System
New `core mwinit` subcommand (new `src/cmd/mwinit/mod.rs`, same structure as
`src/cmd/auth/mod.rs` — not a new binary), documented in a separate `CLI-MWNINIT.md`:
- **Purpose** — authenticate as a dedicated Authentik service account (not a human
device-code login) and cache the resulting JWT for use against `api.riotpiao.com`.
- **Two paths**: `core mwinit login --username <svc-account> --password <...>`
(Authentik service-account grant, same `AUTHENTIK_URL` default as `core auth login`,
`core/src/cmd/auth/config.rs:18`), or `core mwinit login --token <existing-jwt>` to
just re-cache an already-issued token.
- **Storage**: `~/.talos/.riotpiao-auth` — a new, separate path from `core auth
login`'s `~/.cache/talos/authentik_id_token`, so the two tools never collide.
- **Usage**: `curl -H "Authorization: Bearer $(cat ~/.talos/.riotpiao-auth)" -H 'X-Service: ...' https://api.riotpiao.com/ ...`.
- **Lifecycle**: `core mwinit status` / `core mwinit clear` (mirroring `core auth
status`/`clear`); gateway behavior on expired token is 401 + `WWW-Authenticate`,
RFC 9457 problem+json.
---
## 6. `X-Service: memory` — poimen-memory
### Existing System
`poimen-memory` (`~/workplace/Poimen/memory`) is real and **already deployed** —
namespace `poimen`, Service `poimen-memory.poimen.svc.cluster.local:8080`, 2 pods
`Running`. Its own `README.md` claims "design complete, no code yet, 0/37 tasks",
which is badly stale: `memory-tasks/M3.5.*.md` (per that repo's
`DESIGN.md`) show the HTTP API layer (M3.5 phase) is done — `GET /memory/projects`,
`GET /memory/projects/{id}/status`,
`GET /memory/query?query=...&level=L1,L2&project=...&limit=...`,
`POST /memory/ingest`, `GET /memory/skills`, `GET /memory/skills/{name}` all exist and
are tested. It is not currently reachable from `api.riotpiao.com` — the live ingress
has only a `/` catch-all to `api-gateway`, no `/memory` path.
Three more endpoints are **specced in `DESIGN.md` but not confirmed built** —
`GET /memory/projects/{id}/notes` (M3.5.6, listed in the task table but not in the
"is done" set above), `POST /memory/context` (M3.7, tier-1/2/3 lookup — depends on
M3.7.7/M3.7.8 signature+symptom work, not started), and `POST /memory/nodes/by-git`
/`by-commit`/`by-author` (M3.5.9, explicitly optional in that repo's own task
board). These get CR entries below so the schema is defined once, but the
gateway-side task for each is **blocked on upstream**, not just on our own code —
see §7's status column.
It also currently does its own auth: an `apikey:` header check, exact-match against a
stored key, implemented in the app itself
(`tasks/M3.5.1-http-server.md:36-39`) — independent of Kong, which its own
architecture diagram assumed sat in front but never actually required for this check
to work. There is **no NetworkPolicy** on it today (`k8s/` has none) — the apikey
check is the *only* thing standing between it and any pod in the cluster.
### Future System
Per explicit decision: drop poimen-memory's own `apikey` check entirely and gate it
through Authentik instead, with separate read/write capabilities — using the
per-method `auth.capability` override introduced in §1.
```yaml
apiVersion: gateway.riotpiao.com/v1
kind: ServiceAdapter
metadata:
name: memory
namespace: api
spec:
serviceName: memory # matches X-Service: memory
upstream:
url: http://poimen-memory.poimen.svc.cluster.local:8080
timeoutSeconds: 15
auth:
required: true
capability: memory:read # default for every resource below
retryable: true
resources:
- name: query # X-Resource: query — "specific documents belonging to a project"
methods:
- verb: GET
upstreamPath: /memory/query
- name: skill # X-Resource: skill or skill/{name}
methods:
- verb: GET
upstreamPath: /memory/skills
- verb: GET
upstreamPath: /memory/skills/{name}
- name: project # X-Resource: project or project/{id}/status
methods:
- verb: GET
upstreamPath: /memory/projects
- verb: GET
upstreamPath: /memory/projects/{id}/status
- name: ingest # X-Resource: ingest — write path
methods:
- verb: POST
upstreamPath: /memory/ingest
auth:
capability: memory:write # overrides the memory:read default
```
```bash
curl -X GET https://api.riotpiao.com/ \
-H 'Authorization: Bearer <jwt-with-memory:read>' \
-H 'X-Service: memory' \
-H 'X-Resource: query' \
-G --data-urlencode 'query=why did requests over 10KB fail' \
--data-urlencode 'project=poimen' --data-urlencode 'level=L1,L2'
```
**Required companion changes, in this order — not optional, not cosmetic:**
1. Add a `NetworkPolicy` in namespace `poimen` restricting ingress on `poimen-memory`
to the `api` namespace's gateway pod only (mirrors the MinIO-protection rationale
from the very first version of this doc). This must land **before** step 2, or
there's a window where the service has no auth of any kind.
2. Remove the `apikey:` middleware from `poimen-memory` itself
(`tasks/M3.5.1-http-server.md`'s auth hook) — separate change in the
`~/workplace/Poimen/memory` repo, out of scope for `homelab-frontend` but a hard
prerequisite for this adapter being safe to expose.
3. Provision `memory:read` / `memory:write` as real Authentik scopes/capabilities via
the `X-Service: iam` adapter (§3) or `core mwinit`-issued tokens, so callers can
actually be granted one without the other.
---
## Verification
- `go build ./...` and existing tests (`internal/config/*_test.go`, plus new tests for
the rewritten `internal/temporal` / new `internal/serviceadapter`, `internal/iam`,
`internal/resilience`) pass.
- `curl` smoke tests per adapter (workflow/s3/sqs/iam/memory) against a dev gateway,
covering both the legacy `/workflow` body-action path (still mounted) and the new
root + verb+`X-Resource` path.
- `internal/serviceadapter/validate_test.go` (new): missing required field → 400 with
`reason: missing`; wrong type (e.g. `sql` sent as a number) → 400 with
`type_mismatch`; unlisted key under `strict: true` → 400 with `unknown_field`;
unlisted key under default (`strict: false`) → passes. Postgres `query` resource's
`requestSchema` (§1) is the fixture. Confirm a response-side mismatch does **not**
block the response and instead increments
`serviceadapter_response_schema_mismatch{service,resource}`.
- Apply the `memory` `ServiceAdapter` CR only after the `poimen` NetworkPolicy lands
and `poimen-memory`'s own `apikey` middleware is removed (§6's ordering). Confirm a
`memory:read`-scoped token can `GET X-Resource: query`/`skill`/`project` and a
`memory:write`-scoped token can `POST X-Resource: ingest`, and that a `memory:read`
token gets 403 on `ingest`.
- Commit the CRD + `Role`/`RoleBinding` to `k8s/`, push to `main`; confirm the existing
`api-gw` ArgoCD Application auto-syncs them with no `homelab` changes and no new
Application. Apply the example `postgres` CR; confirm the running gateway pod's
informer picks it up and serves `X-Service: postgres` within the watch/resync
latency — no gateway restart, no manual `kubectl` step.
- `core mwinit login` end-to-end against dev Authentik; confirm `~/.talos/.riotpiao-auth`
is written, a root-level `X-Service`-header call using it succeeds, and
`~/.cache/talos/authentik_id_token` is untouched.
INTEGRATION_TESTS.md
-122
View File
@@ -1,122 +0,0 @@
# Integration Tests
Integration tests call the real deployed gateway to verify X-Service routing works end-to-end.
## Quick Start
### Local Test (requires running gateway)
```bash
# Terminal 1: Start the gateway
CONFIG_PATH=k8s/configmap.yaml go run ./cmd/gateway
# Terminal 2: Run tests
./scripts/test-integration.sh
```
### Cluster Test (production gateway)
```bash
GATEWAY_URL=https://api.riotpiao.com ./scripts/test-integration.sh
```
### Canary Deployment (scale to 1, test, scale back)
```bash
./scripts/test-canary.sh
# Or with custom settings:
NAMESPACE=api DEPLOYMENT=api-gateway REPLICAS=3 ./scripts/test-canary.sh
```
## Configuration
Create `.dev.test.local` (gitignored) with:
```bash
GATEWAY_URL=https://api.riotpiao.com
TEST_JWT_TOKEN=eyJ... # Real JWT from Authentik
SKIP_AUTH_TESTS=false
TEST_TIMEOUT=30
```
Or set env vars directly:
```bash
export GATEWAY_URL=https://api.riotpiao.com
export TEST_JWT_TOKEN=eyJ...
export SKIP_AUTH_TESTS=false
go test -tags integration -v ./internal/serviceadapter
```
## Test Matrix
| Test | Type | Expected | Notes |
|------|------|----------|-------|
| Health check | GET /healthz | 200 OK | Always works |
| SQS list-queues | GET X-Service: sqs | 200 or 502 | 502 if service unreachable |
| S3 list-objects | GET X-Service: s3 | 200 or 502 | 502 if service unreachable |
| Memory query | POST X-Service: memory | 200 or 502 | 502 if service unreachable |
| Service not found | GET X-Service: nonexistent | 404 | Routing error |
| Resource not found | GET X-Service: sqs X-Resource: invalid | 404 | Resource error |
| IAM with JWT | GET X-Service: iam + Bearer token | 200 or 502 | Requires valid JWT |
| Missing X-Service | GET (no header) | 404 | Routed to default handler |
## Canary Deployment Flow
```
Current: 3/3 replicas running
↓
scale → 1/3 replicas
↓
wait for pod ready
↓
run integration tests
├─ PASS → scale → 3/3 replicas ✅
└─ FAIL → keep 1/3 for debugging ❌
```
## Running in CI
Add to `.gitea/workflows/ci.yaml`:
```yaml
- name: Integration Tests
run: |
GATEWAY_URL=https://api.riotpiao.com \
SKIP_AUTH_TESTS=true \
TEST_TIMEOUT=30 \
go test -tags integration -v ./internal/serviceadapter
```
## Debugging Failed Tests
If a test fails:
1. **Check pod logs:**
```bash
kubectl -n api logs -l app=api-gateway --tail=50
```
2. **Check service availability:**
```bash
kubectl get svc -A | grep -E "sqs|minio|authentik|poimen"
```
3. **Test service directly:**
```bash
kubectl -n sqs port-forward svc/management-service 9090:9090
curl http://localhost:9090/sqs/queues
```
4. **Check ConfigMap:**
```bash
kubectl get configmap api-gateway-config -n api -o yaml | grep -A50 "adapters:"
```
## Notes
- Auth tests are skipped by default (`SKIP_AUTH_TESTS=true`)
- To test with JWT, set `TEST_JWT_TOKEN` and `SKIP_AUTH_TESTS=false`
- Services in different namespaces may not be reachable from the gateway (NetworkPolicy)
- Canary tests expect `/healthz` endpoint to be available
JWT_VS_RANDOM_TOKEN.md
-486
View File
@@ -1,486 +0,0 @@
# JWT vs Random Tokens - Which is Better?
Analyzing token types for Authentik + SOPS workflow.
---
## Quick Comparison
| Aspect | JWT | Random Token |
|--------|-----|--------------|
| **Stateless** | ✅ Yes | ❌ Requires lookup |
| **Self-contained** | ✅ Yes (claims inside) | ❌ Opaque |
| **Can verify locally** | ✅ Yes (signature) | ❌ Must call Authentik |
| **Size** | ⚠️ Larger (~500 bytes) | ✅ Smaller (~32 bytes) |
| **Immediate revocation** | ❌ Hard (token already signed) | ✅ Easy |
| **Contains user info** | ✅ Yes | ❌ No |
| **Standard OAuth2** | ⚠️ Optional (Bearer tokens) | ✅ Standard |
| **Good for audit** | ✅ User/groups embedded | ⚠️ Need to log lookup |
| **Git-friendly** | ✅ Sign commits | ✅ Sign commits |
---
## Use Case: Authentik + SOPS + Git Hooks
### Scenario 1: Developer Pushes Secret Update
```
Developer:
$ export SOPS_TOKEN=$token
$ sops secrets/default/db-creds.enc.yaml
$ git push
Git server hook runs:
├─ Receive token from commit metadata
├─ Need to validate token
└─ Two options:
```
**With JWT Token:**
```
Git hook:
1. Extract JWT from commit
2. Verify signature locally (no API call)
3. Read claims: {sub: "[email protected]", groups: ["k8s:secret-admin"]}
4. Check: is user in k8s:secret-admin group?
5. Allow/reject commit
Benefits:
✅ No need to call Authentik API
✅ Token self-validates
✅ Can check groups locally
✅ Instant verification
✅ Works offline
```
**With Random Token:**
```
Git hook:
1. Extract token from commit
2. Call Authentik API: GET /application/o/introspect/
└─ "Is this token valid?"
3. If valid, check groups: GET /api/v3/users/{id}/groups/
4. If admin, allow commit
Problems:
❌ Need API call on every git push
❌ Slow (network latency)
❌ Requires network connectivity
❌ If Authentik is down, can't push
❌ Rate limiting risk (many API calls)
```
---
## JWT Structure (What's Inside)
```json
// Decoded JWT payload
{
"sub": "[email protected]",
"email": "[email protected]",
"name": "John Doe",
"groups": [
"k8s:secret-admin",
"k8s:namespace:default:editor"
],
"iat": 1705336200,
"exp": 1705422600, // Expires in 24 hours
"iss": "https://authentik.riotpiao.com",
"aud": "secrets-management"
}
```
**In git commit:**
```
commit abc1234...
Author: [email protected] <[email protected]>
Date: Tue Jan 16 14:30:00 2025 +0000
chore(secret): update db-creds
X-SOPS-Token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
```
**Git hook can:**
```
1. Extract token from commit
2. Decode (no signature needed yet)
3. Read claims: sub, groups
4. Verify signature with Authentik's public key
5. Check groups locally
6. Decision: accept or reject
```
---
## JWT Benefits for This Use Case
### 1. Local Token Verification (No API Calls)
```typescript
// Git hook code (serverless, fast)
function validateJWT(token: string) {
const decoded = jwt.verify(token, PUBLIC_KEY);
// ✅ Done instantly
// ✅ No network call
// ✅ Can work offline
}
```
### 2. Embedded Claims (Audit Trail)
```
JWT contains:
├─ sub: who made the change
├─ groups: what permissions they had
├─ iat: when token was issued
└─ exp: when it expires
Git commit metadata automatically includes:
└─ Who, what groups, when
Complete audit trail without additional logging.
```
### 3. RBAC at Git Level
```typescript
// Git hook can enforce RBAC locally
function canPushSecrets(token: JWT) {
const groups = token.groups;
// Only k8s:secret-admin can push to secrets/
if (!groups.includes("k8s:secret-admin")) {
return false;
}
return true;
}
```
### 4. No Dependency on Authentik Being Up
```
If Authentik is temporarily down:
✅ Developers can still push (JWT validates locally)
✅ Works offline
❌ With random tokens: push fails
```
### 5. Temporal Can Verify Tokens Locally
```
Temporal workflow (token rotation):
└─ Reads old token from K8s
└─ Decodes JWT: check exp field
└─ "Is this token about to expire?"
└─ Generate new token
No API call needed to validate old token.
```
---
## Random Token Benefits
### 1. Standard OAuth2 Practice
```
Random tokens are what most OAuth2 providers use:
├─ Opaque (attacker doesn't know content)
├─ Standard bearer tokens
└─ What Authentik probably generates by default
```
### 2. Immediate Revocation
```
If token is compromised:
└─ Delete it from Authentik
└─ All git pushes instantly fail
With JWT:
└─ Token is still valid until exp time
└─ Attacker has until expiry
```
### 3. Smaller Size
```
JWT: ~500 bytes (base64 encoded)
Random: ~32 bytes (hex)
Difference:
✅ Slightly smaller in git metadata
❌ But negligible for practical purposes
```
### 4. Simpler Conceptually
```
Developers understand random tokens:
└─ "Here's a secret token, use it"
Developers don't understand JWT:
└─ "What's this long base64 string?"
└─ "Why do I need to decode it?"
```
---
## My Recommendation: JWT
Here's why for YOUR use case:
### 1. Git Hook Verification
```
With JWT:
Git hook validates token signature locally
└─ No Authentik API call
└─ Fast
└─ Works offline
└─ Scales infinitely
With random token:
Git hook calls Authentik API
└─ Network latency
└─ Authentik bottleneck
└─ Fails if Authentik down
└─ Rate limiting risk
```
### 2. Embedded RBAC
```
With JWT:
Git hook reads groups from token
└─ Only k8s:secret-admin can push to secrets/
└─ Enforced locally
└─ No additional database lookups
With random token:
Git hook calls Authentik API twice
└─ Introspect token
└─ Fetch groups
└─ Slower
```
### 3. Audit Trail
```
With JWT:
Git commit includes token claims
└─ User, groups, timestamp all in commit metadata
└─ Complete audit trail automatically
With random token:
Git commit has only token
└─ Need to log token → user mapping separately
└─ Additional logging overhead
```
### 4. Temporal Token Rotation
```
With JWT:
Temporal reads exp field
└─ "Token expires in 2 hours"
└─ No API call needed
With random token:
Temporal calls Authentik API
└─ "Is this token still valid?"
└─ Network call needed
```
---
## How to Get JWT from Authentik
### Option 1: OpenID Connect (Standard)
```
Authentik already supports OIDC:
Developer requests token:
$ curl -X POST https://authentik.../application/o/token/ \
-d "grant_type=password&username=user&password=pwd&scope=openid profile groups"
Response:
{
"access_token": "eyJhbGc...", ← JWT
"token_type": "Bearer",
"expires_in": 86400,
"id_token": "eyJhbGc..."
}
```
### Option 2: OAuth2 with JWT
```
Authentik OAuth2 provider settings:
├─ Enable: "Use claims-based access tokens"
├─ Token claims: username, email, groups, preferred_username
└─ Signature algorithm: RS256
Result: Access token is now JWT instead of random.
```
---
## Implementation: SOPS + JWT
```bash
# Developer gets JWT token from Authentik
$ token=$(oidc-client get-token)
$ echo $token # base64 encoded JWT, not random string
# SOPS commits with JWT
$ export SOPS_TOKEN=$token
$ sops secrets/default/db-creds.enc.yaml
$ git add . && git commit -m "update" && git push
# Git hook receives commit
$ git hook:
# 1. Extract SOPS_TOKEN from commit metadata
# 2. Decode JWT (no API call)
# 3. Verify signature with Authentik public key
# 4. Read claims: groups
# 5. Check: user in k8s:secret-admin?
# 6. Accept or reject push
```
---
## Comparison in Your Architecture
### Random Token Path
```
Developer → Authentik (get random token)
→ SOPS (encrypt + commit)
→ Git push
→ Git hook (calls Authentik API to validate)
→ ArgoCD
→ K8s
Network calls: 2 (Authentik for token + git hook validation)
```
### JWT Token Path
```
Developer → Authentik (get JWT token)
→ SOPS (encrypt + commit)
→ Git push
→ Git hook (validates JWT locally, no API call)
→ ArgoCD
→ K8s
Network calls: 1 (only Authentik for token issuance)
```
**Winner: JWT (fewer API calls)**
---
## Concerns with JWT
### Q: Can JWT be revoked immediately?
**A:** Not easily. Options:
```
1. Token expires (set exp field to short time, e.g., 1 hour)
└─ Temporal rotates daily anyway
2. Blacklist approach (store revoked JWTs)
└─ Git hook checks blacklist before validating
└─ But defeats purpose of stateless tokens
3. Accept that JWT lives until expiry
└─ In your case: Temporal rotates daily
└─ So max time a compromised token is valid: 24 hours
└─ Acceptable for homelab
```
### Q: Are JWTs vulnerable?
**A:** No, if configured right:
```
✅ Signed with RS256 (Authentik public key)
✅ Signature verified on every use
✅ Expiry checked (can't use expired token)
✅ Can't be forged (no private key to sign with)
```
### Q: Does Authentik support JWT?
**A:** Yes, fully:
```
Authentik has:
✅ OIDC (returns JWT id_token + access_token)
✅ OAuth2 with claims (can return JWT)
✅ Configuration for JWT signing algorithm
✅ Public key endpoint for verification
```
---
## Final Recommendation
### Use JWT if:
```
✅ You want fast git hook validation (no API calls)
✅ You want embedded RBAC (groups in token)
✅ You want complete audit trail (claims in metadata)
✅ You want resilience (works if Authentik down)
✅ You want to scale (no API bottleneck)
```
### Use Random Token if:
```
✅ Immediate revocation is critical
✅ You prefer standard OAuth2 approach
✅ Developers shouldn't see token contents
✅ Simplicity is more important than features
```
---
## My Advice
**Go with JWT.**
Here's why:
1. Authentik supports it natively
2. Git hook verification is instant (no API calls)
3. Embedded claims give you free audit trail
4. Works offline (resilient)
5. Perfect for homelab scale
**Setup:**
1. Configure Authentik OIDC provider
2. Enable "Use claims-based access tokens" option
3. Add groups to token claims
4. Developers get JWT instead of random string
5. Git hook validates JWT locally (no API call)
**Code example for git hook:**
```bash
#!/bin/bash
# .git/hooks/update (server-side)
token=$(git log -1 --pretty=%B | grep "X-SOPS-Token" | cut -d' ' -f2)
# Verify JWT signature
jwt verify $token --key /path/to/authentik/public.key
# Decode and check groups
groups=$(jwt decode $token | jq .groups)
if [[ ! "$groups" =~ "k8s:secret-admin" ]]; then
echo "Access denied: not in k8s:secret-admin group"
exit 1
fi
# Allow push
exit 0
```
That's it. Simple, elegant, no redundancy.
LLM_TOOL_CALLS.md
-499
View File
@@ -1,499 +0,0 @@
# LLM Tool Calls Testing Guide
This guide shows how to test the gateway with LLM tool calling (function calling) across different APIs.
## What's Tested
The gateway fully supports tool calling for:
- **OpenAI API** (`/v1/chat/completions`) - OpenAI, DeepSeek, etc.
- **Anthropic API** (`/llm/v1/messages`) - Claude models
- **Custom APIs** - Any LLM that supports tool definitions and responses
### Test Coverage
```
✅ OpenAI-style tool calling
✅ Streaming tool calls (SSE with tool_use blocks)
✅ Multi-turn conversations with tool results
✅ Parallel tool calls (multiple tools at once)
✅ Anthropic tool_use format
✅ Complex nested tool arguments
```
---
## Quick Start: Run Tests Locally
```bash
cd /Users/rockliang/workplace/homelab-frontend
# Run all tool call tests
go test ./internal/proxy/... -run "Tool" -v
# Or run with race detector (recommended)
go test -race ./internal/proxy/... -run "Tool" -v
# Expected output: 6 tests, all passing
```
## Test Scenarios
### 1. OpenAI-Style Tool Calling
**What it tests:**
- Request with tool definitions reaches upstream unmodified
- Upstream can return tool_calls in response
- Response with tool_calls passes through to client
**Test code:**
```go
// Request
{
"model": "reasoning",
"messages": [{"role": "user", "content": "What's the weather?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get weather for a location",
"parameters": {...}
}
}]
}
// Response (from upstream)
{
"choices": [{
"message": {
"tool_calls": [{
"id": "call_abc123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": "{\"location\":\"San Francisco\"}"
}
}]
}
}]
}
```
**Run:**
```bash
go test ./internal/proxy/... -run TestToolCallOpenAIStyle -v
```
---
### 2. Streaming Tool Calls
**What it tests:**
- Tool calls can be streamed (SSE format)
- Multiple chunks arrive with tool_call deltas
- Stream completes with `[DONE]` sentinel
**Test code:**
```
Chunk 1: {"delta": {"role": "assistant"}, ...}
Chunk 2: {"delta": {"tool_calls": [{"id": "call_123", "function": {...}}]}, ...}
Chunk 3: {"delta": {}, "finish_reason": "tool_calls"}
Chunk 4: [DONE]
```
**Run:**
```bash
go test ./internal/proxy/... -run TestToolCallStreaming -v
```
---
### 3. Multi-Turn Conversation with Tool Results
**What it tests:**
- Client can send previous assistant's tool_calls back
- Tool result can be sent as a "tool" role message
- Assistant responds with final answer using tool result
**Flow:**
```
Turn 1: User asks → LLM decides to call tool
Turn 2: Client sends tool result → LLM generates final answer
```
**Test code:**
```go
// Turn 1 Request
{
"model": "reasoning",
"messages": [{"role": "user", "content": "What's the weather?"}],
"tools": [...]
}
// Turn 1 Response (tool_calls)
{
"message": {
"tool_calls": [{
"id": "call_abc",
"function": {"name": "get_weather", "arguments": "..."}
}]
}
}
// Turn 2 Request (with tool result)
{
"model": "reasoning",
"messages": [
{"role": "user", "content": "What's the weather?"},
{"role": "assistant", "tool_calls": [...]},
{"role": "tool", "content": "{\"temperature\": 22, \"condition\": \"cloudy\"}"}
],
"tools": [...]
}
// Turn 2 Response (final answer)
{
"message": {
"content": "The weather in San Francisco is 22°C and cloudy."
}
}
```
**Run:**
```bash
go test ./internal/proxy/... -run TestToolCallMultiTurn -v
```
---
### 4. Parallel Tool Calls
**What it tests:**
- LLM can request multiple tools in one response
- Gateway preserves all tool_calls
- Client can execute them in parallel
**Test code:**
```go
// Single response with 3 tool_calls
{
"message": {
"tool_calls": [
{"id": "call_1", "function": {"name": "get_weather", "arguments": "{\"location\":\"New York\"}"}},
{"id": "call_2", "function": {"name": "get_weather", "arguments": "{\"location\":\"London\"}"}},
{"id": "call_3", "function": {"name": "get_weather", "arguments": "{\"location\":\"Tokyo\"}"}}
]
}
}
```
**Run:**
```bash
go test ./internal/proxy/... -run TestParallelToolCalls -v
```
---
### 5. Anthropic Tool Use Format
**What it tests:**
- Different tool format: Anthropic uses `tool_use` blocks instead of `tool_calls`
- Gateway handles both formats transparently
- Tools are sent with `tools` parameter
**OpenAI format:**
```json
{"tool_calls": [{"type": "function", "function": {...}}]}
```
**Anthropic format:**
```json
{"content": [
{"type": "text", "text": "..."},
{"type": "tool_use", "id": "...", "name": "...", "input": {...}}
]}
```
**Test code:**
```go
// Request
{
"model": "claude",
"messages": [{"role": "user", "content": "What's the weather?"}],
"tools": [{
"name": "get_weather",
"description": "Get weather",
"input_schema": {...}
}]
}
// Response (Anthropic format)
{
"content": [
{"type": "text", "text": "I'll check the weather..."},
{"type": "tool_use", "id": "toolu_123", "name": "get_weather", "input": {...}}
],
"stop_reason": "tool_use"
}
```
**Run:**
```bash
go test ./internal/proxy/... -run TestAnthropicToolUse -v
```
---
### 6. Complex Nested Tool Arguments
**What it tests:**
- Tool arguments can be complex JSON structures
- Nested objects, arrays, and deeply nested data preserved
- No argument modification or parsing
**Test code:**
```json
{
"function": {
"name": "create_event",
"arguments": {
"title": "Team Meeting",
"time": "2025-08-20T14:00:00Z",
"attendees": [
{"name": "Alice", "email": "[email protected]"},
{"name": "Bob", "email": "[email protected]"}
],
"location": {
"address": "123 Main St",
"city": "San Francisco",
"country": "USA"
},
"tags": ["important", "recurring"]
}
}
}
```
**Run:**
```bash
go test ./internal/proxy/... -run TestComplexToolArguments -v
```
---
## Running Against Real LLMs
### With Local Stubs (Current)
Tests use mock HTTP servers, so they run instantly:
```bash
go test ./internal/proxy/... -run "Tool" -v
# All 6 tests complete in ~220ms
```
### With Real Upstreams (Future)
Once you have real LLM services running, update the config:
```yaml
# k8s/configmap.yaml
models:
- name: "reasoning"
address: "reasoning-predictor.llm-serving:80" # Real upstream
- name: "claude"
address: "claude-api.anthropic.com:443" # Real Anthropic
```
Then use the gateway normally:
```bash
# Terminal 1: Start gateway
export CONFIG_PATH=config.yaml
go run ./cmd/gateway
# Terminal 2: Test with real LLM
curl -X POST http://localhost:8080/v1/chat/completions \
-H 'content-type: application/json' \
-d '{
"model": "reasoning",
"messages": [{"role": "user", "content": "What color is the sky?"}],
"tools": [{
"type": "function",
"function": {
"name": "search",
"description": "Search the internet",
"parameters": {"type": "object", "properties": {}}
}
}]
}'
```
---
## Gateway Behavior with Tool Calls
### Request Path
```
Client Request
↓
Body-based dispatch (find model)
↓
Look up upstream address
↓
Forward request unmodified (including tools)
↓
Upstream LLM processes tools
```
### Response Path
```
Upstream Response (with tool_calls or tool_use)
↓
Stream unbuffered if streaming
↓
Return to client exactly as received
```
### Key Properties
1. **No Rewriting**: Tool definitions and responses pass through unmodified
2. **Format Agnostic**: Both OpenAI `tool_calls` and Anthropic `tool_use` work
3. **Streaming Safe**: Tool calls stream incrementally without buffering
4. **Nested Structures**: Complex JSON arguments fully preserved
---
## Common Tool Call Patterns
### Pattern 1: Sequential Tool Use
```
Client → LLM (please use search tool)
← LLM (tool_calls: [search(...)])
Client → (execute search, send results)
Client → LLM (here are search results)
← LLM (final answer)
```
### Pattern 2: Parallel Tool Calls
```
Client → LLM (check weather in 3 cities)
← LLM (tool_calls: [get_weather(NY), get_weather(LA), get_weather(SF)])
Client → (execute all 3 in parallel)
Client → LLM (here are all results)
← LLM (summary)
```
### Pattern 3: Tool Result Formatting
```
Client receives tool_calls with:
- id: unique identifier
- function.name: tool name
- function.arguments: JSON string (always a string, not parsed object)
Client sends back:
- role: "tool"
- content: result JSON string
- tool_call_id: matches the original call id
```
---
## Verification Checklist
- [x] OpenAI-style tool definitions forward to upstream
- [x] Tool calls in response reach client unmodified
- [x] Streaming tool calls arrive incrementally
- [x] Multi-turn conversations preserve tool context
- [x] Parallel tool calls all included in response
- [x] Anthropic tool_use format works
- [x] Complex nested arguments preserved
Run all:
```bash
go test ./internal/proxy/... -run "Tool" -v --race
```
Expected: 6/6 passing, race detector clean
---
## Integration with Other Phases
### Phase 2.9: Anthropic Dialect
Currently, Anthropic tool calls work through the generic route handler. Phase 2.9 will add a dedicated `/llm/v1/messages` endpoint with full Anthropic-specific handling.
### Phase 2.13: Error Handling
Tool call errors (unknown tool, parsing errors) will have proper error responses in both OpenAI and Anthropic formats.
### Phase 3: Authentication
Tool calls work with all authentication methods (bearer tokens, API keys) - no special handling needed since tools are just part of the message payload.
### Phase 4: Rate Limiting
Tool calling counts the same as regular chat requests. Rate limits apply per conversation, not per tool call.
---
## Debugging Tool Calls
### Check if tool definitions reach upstream:
```bash
# Enable request logging
go test ./internal/proxy/... -run TestToolCallOpenAIStyle -v 2>&1 | grep -A5 "tool"
```
### Verify tool response format:
```bash
# Extract and pretty-print response
curl -X POST http://localhost:8080/v1/chat/completions ... | jq '.choices[0].message.tool_calls'
```
### Test streaming tool calls:
```bash
curl -N http://localhost:8080/v1/chat/completions \
-H 'content-type: application/json' \
-d '{..., "stream": true, "tools": [...]}'
# Should see incremental chunks with tool_use deltas
```
---
## FAQ
**Q: Do I need to modify the gateway code to support tool calls?**
A: No. Tool calls are just JSON in the request/response body. The gateway forwards them unchanged.
**Q: What if the LLM doesn't support tools?**
A: The tool definitions are simply ignored. The gateway doesn't validate or enforce tool support.
**Q: Can I mix OpenAI and Anthropic tool formats?**
A: Not in the same request. OpenAI clients expect `tool_calls`, Anthropic clients expect `tool_use` blocks. The upstream API determines the format.
**Q: How are tool arguments limited?**
A: By the per-route `maxBodySize` config. Complex nested arguments count toward that limit.
**Q: Can tool calls be streamed?**
A: Yes! SSE streaming fully supports tool calls. They arrive in delta chunks like text tokens.
---
## Next Steps
1. **Run tests locally:**
```bash
go test ./internal/proxy/... -run "Tool" -v
```
2. **Deploy to cluster:**
See [CLUSTER_REPO_SETUP.md](./CLUSTER_REPO_SETUP.md)
3. **Test against real LLMs:**
Update config with real upstream addresses, restart gateway
4. **Phase 2.9:** Implement Anthropic dialect handler for `/llm/v1/messages`
5. **Phase 4:** Add tool call budgeting and rate limits
REQUIREMENTS.md
-367
View File
@@ -1,367 +0,0 @@
# homelab-frontend — Requirements
The contract for the Go API gateway that replaces Kong OSS on `*.riotpiao.com`.
Companion documents:
- [docs/adr/ADR-0001-retire-kong-for-go-gateway.md](docs/adr/ADR-0001-retire-kong-for-go-gateway.md) — why Kong is being retired
- [docs/MIGRATION-kong.md](docs/MIGRATION-kong.md) — exact inventory of what Kong does today and the cutover order
- [tasks/INDEX.md](tasks/INDEX.md) — the task board
All cluster facts below were verified live against context `admin@homelab-cluster`
on 2026-08-19. Re-verify before relying on any number.
---
## 0. Invariants
These hold for every surface. A change that breaks one of these is a design change,
not an implementation detail.
- **G1** — ingress-nginx owns TLS and the edge. The gateway never terminates TLS.
- **G2** — The gateway holds no Kubernetes credentials. It proxies to services that
do. Cluster-read permissions stay in atlas, out of the public edge process.
- **G3** — Public surfaces use standard protocol shapes. If an OpenAI SDK cannot
call it unmodified, the design is wrong.
- **G4** — Streaming is unbuffered end to end, and a client disconnect cancels the
upstream request rather than orphaning it.
- **G5** — Authentication is Bearer-token, validated against Authentik via JWKS
fetched at runtime. No pinned public keys, no rotation runbook.
- **G6** — Every route's timeout, body cap and concurrency limit is explicit in
configuration. No silent defaults.
- **G7** — All deployment flows through git and Argo. No `kubectl apply`, no
`helm upgrade`, no local `terraform apply`.
---
## 1. Runtime and configuration
### 1.1 Process
Single static Go binary. Reads configuration at startup, serves HTTP, exits cleanly
on SIGTERM after draining in-flight requests.
Must run with **no cluster, no kubeconfig and no credentials** so that behaviour can
be verified in a closed loop before touching live traffic. Upstreams are
configuration; pointing them at local stubs is the entire mechanism. This is a hard
requirement, not a convenience — see §7.
### 1.2 Configuration
Route and upstream configuration is declarative and loaded at startup. It must
express, per upstream: address, path rewrite, connect/read/write timeouts, maximum
body size, and whether the route requires authentication.
Configuration errors fail startup loudly. A gateway that starts with a silently
dropped route is worse than one that refuses to start.
**Configuration lives in git**, mounted as a ConfigMap and synced by Argo. Not a
CRD. A CRD would require the gateway to watch the API server, which needs RBAC and
contradicts G2 — and CRD-driven routing is precisely the indirection being retired
with Kong, where the routing table was split across six `KongPlugin` CRs, seven
Ingresses and a Helm values file.
A CRD earns its keep when someone other than the repo owner must register routes.
That is not true here. If it becomes true, the additive answer is a controller that
renders this same ConfigMap — the gateway stays credential-free either way.
### 1.3 Health
- `GET /healthz` — liveness, no upstream checks, always cheap.
- `GET /readyz` — readiness; may fail while configuration is invalid or JWKS has
never been successfully fetched.
Neither requires authentication.
---
## 2. Proxy core
### 2.1 Reverse proxying
Standard reverse proxy to configured upstreams. Connection reuse across requests.
Hop-by-hop headers stripped correctly. `X-Forwarded-*` set from the nginx-supplied
values, not fabricated.
### 2.2 Streaming
SSE and chunked responses pass through without buffering. Tokens must reach the
client as the upstream emits them, not on completion.
WebSocket upgrade must work — `agent-pod/console` depends on it.
### 2.3 Disconnect propagation
When a client disconnects, the upstream request is cancelled immediately. This is
load-bearing: an orphaned generation holds a vLLM sequence slot, and there are only
eight in the cluster.
### 2.4 Timeouts
Per-route, explicit. Current Kong values, which are deliberate and must be preserved
unless changed knowingly:
| Route class | connect | read | write |
|---|---|---|---|
| chat | 10s | 1h | 1h |
| embeddings, rerank | 10s | 10m | 10m |
The 1-hour read timeout exists because a 32B model on a Volta GPU routinely exceeds
60s. Any shorter application-level cap must be enforced *by the gateway's own
logic*, not by shortening the proxy timeout — otherwise long legitimate generations
truncate mid-stream.
---
## 3. LLM surfaces — `api.riotpiao.com`
Two protocol dialects, permanently. Both translate into one dialect-neutral canonical
request, and both pass through **one shared slot controller** before reaching a
predictor.
| Prefix | Dialect | Primary client |
|---|---|---|
| `/v1/*` | OpenAI-compatible | pi, generic OpenAI SDKs |
| `/llm/*` | Anthropic Messages | the riotpiao frontend (first-party only) |
```
/v1/* (OpenAI) /llm/* (Anthropic)
| |
+-----------+------------+
v
canonical request dialect-neutral
v
slot controller keyed by UPSTREAM, not by route
v
reasoning-predictor / ornith-predictor
```
**The slot controller is keyed by upstream and shared across dialects.** Per-dialect
semaphores are wrong: the 8 sequence slots are physical, so two independent gates
would each believe they were within budget while together exceeding it. Requests from
both surfaces contend for the same slots and the same queue, in arrival order.
Dispatch, budgets, logging and metrics all operate on the canonical request. Adding a
third dialect later must not require touching the controller.
### 3.1 Body-based model dispatch
`POST /v1/chat/completions` selects its upstream from the request body's `model`
field. This is the single most important requirement in this document: it is the
capability Kong OSS lacked, and the reason the gateway exists.
Unknown or missing `model` is a client error with a useful message listing valid
values — not a 500, and not a silent fallback to a default model.
### 3.2 Upstream map
Verified live. `served-model-name` values are what clients send.
| `model` in body | Upstream Service | Engine |
|---|---|---|
| `reasoning` | `reasoning-predictor.llm-serving:80` | vLLM, DeepSeek-R1-Distill-Qwen-32B |
| `ornith:35b` | `ornith-predictor.llm-serving:80` | Ollama |
| `qwen2.5:3b-instruct` | `ornith-predictor.llm-serving:80` | Ollama, same pods |
| `nomic-ai/nomic-embed-text-v2-moe` | `embeddings-predictor.llm-serving:80` | TEI |
| `BAAI/bge-reranker-base` | `reranker-predictor.llm-serving:80` | TEI |
`reasoning` runs 2 replicas × `--max-num-seqs=4` = **8 concurrent sequence slots
total**, `--max-model-len=16384`, `--reasoning-parser=deepseek_r1`,
`--enable-auto-tool-choice --tool-call-parser=hermes`.
Note `ornith:35b` and `qwen2.5:3b-instruct` share pods; both stay resident via
`OLLAMA_MAX_LOADED_MODELS=2` and `OLLAMA_KEEP_ALIVE=-1`, so dispatching between them
does not trigger a model swap.
### 3.3 Path rewriting
Upstreams expect canonical paths. `/v1/chat/completions` and `/v1/embeddings` pass
through unchanged. Rerank is the exception: TEI serves `/rerank`, not `/v1/rerank`,
so that route rewrites.
### 3.4 Legacy path aliases
`/v1/{reasoning,ornith,qwen}/chat/completions` must keep working during cutover —
pi is a live caller. They behave exactly as the canonical endpoint with `model`
forced to the corresponding value, overriding whatever the body says.
These are temporary. They exist to make the cutover reversible, and are removed once
callers have migrated.
### 3.5 `GET /v1/models`
Derived from the configured upstream map, never hardcoded. Kong served a static
list, and its own manifest flags that the list can drift from what the engines
actually serve. The gateway's list must be incapable of disagreeing with what
routing will accept.
OpenAI list shape: `{"object":"list","data":[{"id","object":"model","owned_by","created"}]}`.
### 3.6 Behaviour to preserve
Verified against the live endpoint:
- The upstream returns `reasoning_content` separately from `content` for the
`reasoning` model. Pass both through untouched.
- Tool calling works with explicit `tool_choice`, and is unreliable with
`tool_choice: auto` on the R1-distill model. The gateway does not compensate for
this — it is a model property, not a gateway concern. Do not add retries or
rewriting to work around it.
---
## 4. Authentication — Authentik
### 4.1 Current state
**The model API is unauthenticated today.** Confirmed live: `/v1/reasoning/chat/completions`
answers with no credentials.
Kong's `key-auth` was retired because it accepts a raw `apikey:` header but rejects
`Authorization: Bearer`, which hard-blocks every OpenAI-compatible client. See
`~/workplace/homelab/k8s/apps/api/model-auth.yaml`.
### 4.2 Requirement
Bearer tokens in `Authorization`, validated against Authentik
(`https://authentik.riotpiao.com`) by fetching and caching JWKS at runtime.
Key rotation must be handled by refetching JWKS, not by pinned PEMs. The
pinned-`rsa_public_key` approach in `AUTH-PLAN.md` and its rotation runbook exist
only to route around a Kong OSS limitation and must not be carried forward.
Service accounts obtain tokens via `client_credentials` against Authentik's token
endpoint.
### 4.3 Rollout
Auth ships behind a flag, defaulting off, and is enabled deliberately.
Enabling it breaks every current caller until they hold a token — pi included, whose
`models.json` currently sends a `customHeaders: {apikey: ...}` block that will need
replacing with a Bearer token.
### 4.4 Authorization
Beyond authentication, a token must be checked for the right to invoke the
capability it is calling. A token minted for queue access should not invoke a GPU.
---
## 5. Rate limiting and budgets
No `rate-limiting` plugin exists anywhere in the cluster today — this is net new
work, not a migration. Verified: six Kong plugins exist, none is `rate-limiting`.
Requirements, in priority order:
1. **GPU slot protection.** `reasoning` has 8 total sequence slots. Concurrent
in-flight requests to it must be capped below that, leaving operator headroom.
Excess requests queue up to a bounded depth, then are rejected with a retryable
status.
2. **Per-caller budgets.** Identified callers get a request budget over a window.
3. **Body size caps**, per route.
Rejections use RFC 9457 `application/problem+json` and set `Retry-After` where a
retry time is knowable.
---
## 6. Observability
Kong's cluster-wide `prometheus` plugin is being retired. The gateway must expose at
least equivalent signal or observability regresses at cutover: request rate,
latency, status codes, bandwidth, and upstream health, labelled by route and
upstream.
Gateway-specific signals that Kong could not provide, and which are the reason for
several requirements above: in-flight requests per upstream, queue depth, GPU slot
occupancy, and rejections by reason.
Structured logging. Every rejected request is logged with the reason. No secrets, no
tokens, no request bodies in logs.
---
## 7. Local development and verification
An agent must be able to close a change/verify loop with no cluster, no kubeconfig
and no credentials. This is a hard requirement because it determines whether work can
proceed unattended.
Concretely: it must be possible to start the gateway locally, point it at stub
upstreams, issue requests, and assert on the responses — including streaming
responses and client disconnects.
Verification of any API-shaped task means asserting on the **actual HTTP response**:
status, headers, and body. "It compiles" and "it starts" are not verification.
Parity with Kong is verified by comparing gateway and Kong responses for the same
request, for every route in the migration inventory, before cutover.
---
## 8. Deployment
Container: distroless or scratch, `runAsNonRoot`, read-only root filesystem, all
capabilities dropped, `seccompProfile: RuntimeDefault`, no shell.
Image tags are commit SHAs, never `:latest` — Argo's `selfHeal` cannot roll out a
mutable tag reliably.
NetworkPolicy: egress only to the upstreams it proxies plus Authentik; ingress from
`ingress-nginx` only.
Deployed as an Argo Application in the `homelab-root` GitOps repo. Verified live:
zero Argo Applications anywhere in the cluster source from any Forgejo URL, so
`github.com/Riotpiaole/riotpiao.homelab.com` is authoritative.
---
## 9. Capability surface — path-based
Every capability is a path prefix on the single host `api.riotpiao.com`. One DNS
record, one Cloudflare tunnel hostname, one nginx Ingress, one Service.
| Prefix | Backs onto | Status |
|---|---|---|
| `/v1/*` | `llm-serving` predictors | v1 — **reserved**, see below |
| `/sqs/*` | kmsvc management-service, Kafka/Strimzi (`sqs` ns) | future |
| `/workflow/*` | Temporal (`temporal` ns) | future |
| `/cluster/*` | atlas, separate repo `riotpiao-backend` | future |
| `/db/*` | CloudNativePG, MinIO, monitoring reads | future |
**`/v1/*` is reserved for the OpenAI-compatible surface and nothing else.** G3 pins
it: an SDK expects `/v1/chat/completions` at the base URL, so that prefix can never
be repurposed or nested. Every other capability gets its own prefix that cannot
collide with a current or future OpenAI path.
Subdomains are deliberately *not* used. Paths keep hostname configuration to a
single entry — and hostname configuration is the demonstrated failure mode here, as
the unresolved apex 403 shows. Promoting a prefix to its own subdomain later is an
additive host rule that can run alongside the path; the reverse is not, because
clients hardcode hostnames.
Notes carried from the cluster:
- Temporal namespace registration is automatic via queue-operator, never manual.
- `management-service` already exposes gRPC at `kmsvc.riotpiao.com`; the `/sqs`
prefix is a new surface, not a replacement for it.
- atlas keeps its own informers and RBAC. The gateway proxies to it and holds no
cluster credentials of its own (G2).
- `/db/*` read surfaces need particular care — see G2 before designing them.
---
## 10. Known cluster facts worth not rediscovering
- `kmsvc-redis-master.sqs:6379` has **no authentication** — `ALLOW_EMPTY_PASSWORD=yes`,
TLS off. Any workload with network reach has full unauthenticated read/write. A
NetworkPolicy is the only control.
- `reasoning-predictor` listens on port **80**, not 8080.
- `prometheus-operated.monitoring` is **headless** (ClusterIP None) — egress policies
need pod selectors, not ClusterIPs.
- `agent-pod/console` is publicly routed, unauthenticated, accepts free-form prompts
into a shell-capable container, and serves a WebSocket. Putting it behind gateway
auth is a security fix, not merely a port.
- Eight `*.example.com` hosts exist on istio-class Ingresses in `llm-serving`. KServe
defaults, not public, out of scope — do not mistake them for gateway routes.
TEMPORAL_GRPC_MIGRATION.md
-176
View File
@@ -1,176 +0,0 @@
# Temporal gRPC Integration - Migration Status
## Overview
Temporal REST ↔ gRPC bridge is being implemented. Client sends HTTP JSON → gateway translates to gRPC → Temporal server responds.
## Implementation Status
### Phase 1: Core Workflow Operations ✅ WIRED
- **START_WORKFLOW** ✅ gRPC: `StartWorkflowExecution`
- **DESCRIBE_WORKFLOW** ✅ gRPC: `DescribeWorkflowExecution`
- **LIST_WORKFLOWS** ⏳ TODO (requires pagination logic)
- **GET_WORKFLOW_HISTORY** ⏳ TODO
- **SIGNAL_WORKFLOW** ⏳ TODO
- **QUERY_WORKFLOW** ⏳ TODO
- **CANCEL_WORKFLOW** ⏳ TODO
- **TERMINATE_WORKFLOW** ⏳ TODO
- **RESET_WORKFLOW** ⏳ TODO
- **UPDATE_WORKFLOW** ⏳ TODO
### Phase 2: Activity Operations ⏳ NOT IMPLEMENTED
- HEARTBEAT_ACTIVITY
- COMPLETE_ACTIVITY
- FAIL_ACTIVITY
**Note:** Activity operations require different error handling (task tokens, etc.). See operations_grpc.go for reference.
### Phase 3: OperatorService Operations ⏳ NOT IMPLEMENTED
Requires separate gRPC stub. Currently:
- LIST_NAMESPACES → 501 NOT_IMPLEMENTED
- DESCRIBE_NAMESPACE → 501 NOT_IMPLEMENTED
- CREATE_NAMESPACE → 501 NOT_IMPLEMENTED
- UPDATE_NAMESPACE → 501 NOT_IMPLEMENTED
- DELETE_NAMESPACE → 501 NOT_IMPLEMENTED
- LIST_SEARCH_ATTRIBUTES → 501 NOT_IMPLEMENTED
- ADD_SEARCH_ATTRIBUTES → 501 NOT_IMPLEMENTED
- LIST_TASK_QUEUES → 501 NOT_IMPLEMENTED
- GET_CLUSTER_INFO → 501 NOT_IMPLEMENTED
- LIST_CLUSTER_MEMBERS → 501 NOT_IMPLEMENTED
- GET_SYSTEM_INFO → 501 NOT_IMPLEMENTED
## Architecture
```
HTTP Request (JSON)
↓
Handler.startWorkflow()
↓
Converts to protobuf (workflowservice.StartWorkflowExecutionRequest)
↓
gRPCClient.GetWorkflowServiceStub().StartWorkflowExecution(ctx, req)
↓
Temporal Server (port 7233)
↓
gRPC Response
↓
Convert to JSON response map
↓
HTTP 200 JSON
```
## Code References
- **handler.go**: HTTP ↔ gRPC translation layer
- `NewHandler()`: Creates gRPC connection via `NewGRPCClient()`
- `startWorkflow()`, `describeWorkflow()`: gRPC-wired operations
- Others: stubs or NOT_IMPLEMENTED
- **grpc_client.go**: Low-level gRPC connection management
- `NewGRPCClient()`: Dials Temporal at port 7233
- `GetWorkflowServiceStub()`: Returns `workflowservice.WorkflowServiceClient`
- `GetOperatorServiceStub()`: Returns `operatorservice.OperatorServiceClient`
- **operations_grpc.go**: Example gRPC implementations (reference for wiring)
- Shows payload marshaling patterns
- Shows error handling (gRPC status codes → HTTP 4xx/5xx)
## Next Steps (Phase 2)
1. Wire remaining WorkflowService operations (LIST, GET_HISTORY, SIGNAL, QUERY, etc.)
- All use same pattern: build protobuf request → call stub → map response to JSON
- Reference operations_grpc.go for exact patterns
2. Add OperatorService support (namespaces, cluster, search attrs)
- Create separate stub: `operatorServiceClient := NewGRPCClient().GetOperatorServiceStub()`
- Add methods to handler for each operation
3. Add Activity operations (heartbeat, complete, fail)
- Requires task token handling
- See operations_grpc_test.go for test patterns
## Build Status
```
go build ./cmd/gateway ✅ SUCCESS
```
## Testing
To test gRPC wiring locally:
```bash
# Start Temporal locally (if not running)
docker run -d -p 7233:7233 temporalio/auto-setup:latest
# Start gateway
go run ./cmd/gateway
# Test (in another terminal)
curl -X POST http://localhost:8080/workflow \
-H 'Content-Type: application/json' \
-d '{
"action": "START_WORKFLOW",
"namespace": "default",
"payload": {
"workflow_id": "test-1",
"workflow_type": "MyWorkflow",
"task_queue": "my-queue"
}
}'
# Should return
{
"success": true,
"action": "START_WORKFLOW",
"data": {
"workflow_id": "test-1",
"run_id": "abc123...",
"start_time": "2026-08-27T..."
}
}
```
## Key Implementation Details
### Protobuf Field Names
Temporal protobuf uses snake_case field names:
- `WorkflowId` not `WorkflowID`
- `RunId` not `RunID`
- `WorkflowType` (message) not `WorkflowTypeString`
- `TaskQueue` (message) not `TaskQueueName`
### Type Imports (from go.temporal.io/api)
```go
import (
"go.temporal.io/api/common/v1" // WorkflowExecution, WorkflowType, Payloads
"go.temporal.io/api/taskqueue/v1" // TaskQueue
"go.temporal.io/api/workflowservice/v1" // All Workflow* stubs
"go.temporal.io/api/operatorservice/v1" // Namespace/cluster stubs (not yet used)
)
```
### Payload Marshaling Pattern
```go
input := getMap(payload, "input")
if len(input) > 0 {
inputBytes, _ := json.Marshal(input)
req.Input = &common.Payloads{
Payloads: []*common.Payload{{Data: inputBytes}},
}
}
```
### Error Handling
- gRPC errors → map to HTTP status:
- `codes.NotFound` → 404
- `codes.InvalidArgument` → 400
- `codes.Unavailable` → 503
- others → 500
## Questions / Blockers
None currently. gRPC wiring is straightforward pattern-matching.
TEMPORAL_USAGE.md
-1210
View File
File diff suppressed because it is too large Load Diff
TESTING_GUIDE.md
-560
View File
@@ -1,560 +0,0 @@
# API Testing Guide
Quick reference for testing the homelab-frontend gateway API.
## Setup
```bash
# Set base URL
export GATEWAY="https://api.riotpiao.com"
# Or for local testing
export GATEWAY="http://localhost:8080"
```
---
## Quick Tests (Copy & Paste)
### 1. Health Checks ✅
```bash
# Liveness
curl $GATEWAY/healthz | jq .
# Readiness
curl $GATEWAY/readyz | jq .
```
**Expected**: Both return `{"status":"..."}` with HTTP 200
---
### 2. List Models ✅
```bash
curl $GATEWAY/v1/models | jq '.data[] | .id'
```
**Expected Output**:
```
"reasoning"
"ornith:35b"
"qwen2.5:3b-instruct"
"nomic-ai/nomic-embed-text-v2-moe"
"BAAI/bge-reranker-base"
```
---
### 3. Chat - Basic ✅
```bash
curl -X POST $GATEWAY/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "reasoning",
"messages": [
{"role": "user", "content": "What is 2+2?"}
]
}' | jq '.choices[0].message.content'
```
**Expected**: Model responds with an answer
---
### 4. Chat - Ornith Model ✅
```bash
curl -X POST $GATEWAY/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "ornith:35b",
"messages": [
{"role": "user", "content": "Hello"}
]
}' | jq '.choices[0].message.content'
```
**Expected**: Routes to ornith model, returns response
---
### 5. Chat - Qwen Model ✅
```bash
curl -X POST $GATEWAY/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen2.5:3b-instruct",
"messages": [
{"role": "user", "content": "Hi"}
]
}' | jq '.choices[0].message.content'
```
**Expected**: Routes to qwen model, returns response
---
### 6. Chat - Unknown Model (Should Error) ❌→✅
```bash
curl -X POST $GATEWAY/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "gpt-4-turbo",
"messages": []
}' | jq '.'
```
**Expected**: HTTP 400 with problem+json:
```json
{
"type": "https://api.example.com/problems/unknown-model",
"title": "Unknown Model",
"status": 400,
"detail": "Model \"gpt-4-turbo\" is not available. See valid_models for available options.",
"valid_models": ["reasoning", "ornith:35b", ...]
}
```
---
### 7. Chat - Missing Model (Should Error) ❌→✅
```bash
curl -X POST $GATEWAY/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"messages": [{"role": "user", "content": "test"}]
}' | jq '.'
```
**Expected**: HTTP 400 with problem+json (missing model)
---
### 8. Chat - Streaming ✅
```bash
curl -N -X POST $GATEWAY/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "reasoning",
"messages": [{"role": "user", "content": "count to 3"}],
"stream": true
}' | head -20
```
**Expected**:
- Multiple `data: {...}` lines (SSE chunks)
- Final `data: [DONE]`
- Chunks arrive incrementally (observable with `-N` flag)
---
### 9. Chat - Tool Calling ✅
```bash
curl -X POST $GATEWAY/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "reasoning",
"messages": [
{"role": "user", "content": "What is the weather in SF?"}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string"}
},
"required": ["location"]
}
}
}
]
}' | jq '.choices[0].message.tool_calls'
```
**Expected**: Array of tool calls (if model decides to call them), or null (if not)
---
### 10. Embeddings ✅
```bash
curl -X POST $GATEWAY/v1/embeddings \
-H 'Content-Type: application/json' \
-d '{
"model": "nomic-ai/nomic-embed-text-v2-moe",
"input": "hello world"
}' | jq '.data | length'
```
**Expected**: `1` (one embedding vector)
---
### 11. Embeddings - Multiple ✅
```bash
curl -X POST $GATEWAY/v1/embeddings \
-H 'Content-Type: application/json' \
-d '{
"model": "nomic-ai/nomic-embed-text-v2-moe",
"input": ["text 1", "text 2", "text 3"]
}' | jq '.data | length'
```
**Expected**: `3` (three embedding vectors)
---
### 12. Embeddings - Unknown Model (Should Error) ❌→✅
```bash
curl -X POST $GATEWAY/v1/embeddings \
-H 'Content-Type: application/json' \
-d '{
"model": "unknown-embed",
"input": "test"
}' | jq '.status'
```
**Expected**: `400` (client error)
---
### 13. Rerank ✅
```bash
curl -X POST $GATEWAY/v1/rerank \
-H 'Content-Type: application/json' \
-d '{
"model": "BAAI/bge-reranker-base",
"query": "machine learning",
"texts": [
"Machine learning is AI",
"Python is a language",
"Deep learning is ML"
]
}' | jq '.results'
```
**Expected**: Array of ranked results with scores:
```json
[
{"index": 0, "score": 0.95},
{"index": 2, "score": 0.85},
{"index": 1, "score": 0.15}
]
```
---
### 14. Rerank - Unknown Model (Should Error) ❌→✅
```bash
curl -X POST $GATEWAY/v1/rerank \
-H 'Content-Type: application/json' \
-d '{
"model": "unknown-rerank",
"query": "test",
"texts": ["a"]
}' | jq '.status'
```
**Expected**: `400` (client error)
---
### 15. Invalid JSON (Should Error) ❌→✅
```bash
curl -X POST $GATEWAY/v1/chat/completions \
-H 'Content-Type: application/json' \
-d 'not json' | jq '.title'
```
**Expected**: `"Invalid Request Body"` (HTTP 400)
---
## Testing Checklist
Complete this checklist to verify all endpoints:
### Health Endpoints
- [ ] GET /healthz → 200, `{"status":"alive"}`
- [ ] GET /readyz → 200, `{"status":"ready"}`
### Model Discovery
- [ ] GET /v1/models → 200, returns all 5 models
- [ ] All advertised models can be called (none 400)
### Chat Completions
- [ ] POST /v1/chat/completions (reasoning) → 200, response
- [ ] POST /v1/chat/completions (ornith:35b) → 200, response
- [ ] POST /v1/chat/completions (qwen2.5:3b-instruct) → 200, response
- [ ] POST /v1/chat/completions (unknown model) → 400, problem+json
- [ ] POST /v1/chat/completions (missing model) → 400, problem+json
- [ ] POST /v1/chat/completions (invalid JSON) → 400, problem+json
- [ ] POST /v1/chat/completions (streaming) → 200, SSE chunks
- [ ] POST /v1/chat/completions (with tools) → 200, tool_calls present/absent
### Embeddings
- [ ] POST /v1/embeddings (single input) → 200, embedding
- [ ] POST /v1/embeddings (multiple inputs) → 200, embeddings array
- [ ] POST /v1/embeddings (unknown model) → 400, problem+json
### Reranking
- [ ] POST /v1/rerank → 200, ranked results
- [ ] POST /v1/rerank (unknown model) → 400, problem+json
- [ ] Verify path is rewritten to /rerank on upstream
### Error Handling
- [ ] Unknown model lists valid_models
- [ ] Error responses are problem+json
- [ ] No 5xx for client errors (validation errors)
- [ ] Upstream errors pass through
### Streaming
- [ ] Chunks arrive incrementally
- [ ] Final `[DONE]` sentinel present
- [ ] Works for chat completions
### Tool Calling
- [ ] Tool definitions forward to upstream
- [ ] Tool calls in response
- [ ] Multi-turn with tool results
- [ ] Parallel tool calls
- [ ] Complex nested arguments preserved
---
## Troubleshooting
### 404 Responses
**Symptom**: All endpoints return `"not found"`
**Cause**: ConfigMap with models/routes not deployed
**Solution**:
```bash
kubectl -n api create configmap homelab-frontend-config \
--from-file=config.yaml=k8s/configmap.yaml
kubectl -n api rollout restart deployment/homelab-frontend
```
---
### 503 (Not Ready)
**Symptom**: `/readyz` returns 503
**Cause**: Configuration not loaded or JWKS fetch failed
**Solution**:
```bash
# Check logs
kubectl -n api logs deployment/homelab-frontend
# Check config
kubectl -n api get configmap homelab-frontend-config
```
---
### Connection Refused
**Symptom**: `Connection refused` or `Temporary failure in name resolution`
**Cause**:
- Gateway not running
- Wrong URL/hostname
- Network issue
**Solution**:
```bash
# Verify gateway is running
kubectl -n api get pods -l app=homelab-frontend
# Check service
kubectl -n api get svc homelab-frontend
# Verify ingress
kubectl -n api get ingress api
```
---
### Upstream Connection Errors
**Symptom**: `502 Bad Gateway` or `connection refused to upstream`
**Cause**: Model upstream service not reachable
**Solution**:
```bash
# Check upstreams are running
kubectl -n llm-serving get pods
# Verify addresses in ConfigMap
kubectl -n api get configmap homelab-frontend-config -o yaml
# Test connectivity from gateway pod
kubectl -n api exec deployment/homelab-frontend -- \
curl -s reasoning-predictor.llm-serving:80/healthz
```
---
### Streaming Doesn't Work
**Symptom**: Chunks arrive all at once (buffered) instead of incrementally
**Cause**: nginx buffering or client not using `-N` flag
**Solution**:
```bash
# Use -N flag
curl -N https://api.riotpiao.com/v1/chat/completions ...
# Verify nginx has buffering disabled
# Should have: proxy-buffering: off in Ingress annotations
```
---
## Performance Testing
### Load Test (Simple)
```bash
# Send 10 requests in parallel
for i in {1..10}; do
curl -X POST $GATEWAY/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"reasoning","messages":[{"role":"user","content":"Hi"}]}' &
done
wait
echo "Completed 10 requests"
```
### Concurrency Test
```bash
# Use Apache Bench (if installed)
ab -n 100 -c 10 \
-p request.json \
-T application/json \
$GATEWAY/v1/chat/completions
# Create request.json:
# {"model":"reasoning","messages":[{"role":"user","content":"test"}]}
```
### Latency Test
```bash
# Measure response time
curl -w "\nTotal time: %{time_total}s\n" \
-X POST $GATEWAY/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "reasoning",
"messages": [{"role": "user", "content": "What is AI?"}]
}' > /dev/null
```
---
## Integration Testing
### Test with Python
```bash
pip install requests
cat > test_api.py << 'EOF'
import requests
import json
gateway = "https://api.riotpiao.com"
# Test health
r = requests.get(f"{gateway}/healthz")
assert r.status_code == 200
print("✓ Health check passed")
# Test models
r = requests.get(f"{gateway}/v1/models")
assert r.status_code == 200
models = [m['id'] for m in r.json()['data']]
print(f"✓ Models: {models}")
# Test chat
r = requests.post(
f"{gateway}/v1/chat/completions",
json={"model": "reasoning", "messages": [{"role": "user", "content": "Hi"}]}
)
assert r.status_code == 200
print("✓ Chat works")
# Test unknown model error
r = requests.post(
f"{gateway}/v1/chat/completions",
json={"model": "gpt-4", "messages": []}
)
assert r.status_code == 400
assert "unknown" in r.json()['detail'].lower()
print("✓ Unknown model error correct")
# Test embeddings
r = requests.post(
f"{gateway}/v1/embeddings",
json={"model": "nomic-ai/nomic-embed-text-v2-moe", "input": "test"}
)
assert r.status_code == 200
print("✓ Embeddings work")
# Test rerank
r = requests.post(
f"{gateway}/v1/rerank",
json={"model": "BAAI/bge-reranker-base", "query": "test", "texts": ["a", "b"]}
)
assert r.status_code == 200
print("✓ Reranking works")
print("\n✅ All tests passed!")
EOF
python test_api.py
```
---
## Summary
| Category | Tests | Expected |
|----------|-------|----------|
| Health | 2 | ✅ Both 200 |
| Models | 1 | ✅ 5 models listed |
| Chat | 8 | ✅ 6 success + 2 error |
| Embeddings | 3 | ✅ 2 success + 1 error |
| Rerank | 2 | ✅ 1 success + 1 error |
| Streaming | 1 | ✅ Incremental chunks |
| Tools | 1 | ✅ Tool calls present |
| **TOTAL** | **18+** | **✅ ALL PASS** |
Once all tests pass, the gateway is production-ready! 🚀
docs/API-llm.md
-324
View File
@@ -1,324 +0,0 @@
# API — LLM surfaces
Two protocol dialects over the same models and the same slot controller.
| Prefix | Dialect | Endpoint | Client |
|---|---|---|---|
| `/v1` | OpenAI-compatible | `POST /v1/chat/completions` | pi, OpenAI SDKs |
| `/llm` | Anthropic Messages | `POST /llm/v1/messages` | riotpiao frontend (first-party) |
Status marks below:
**[LIVE]** verified against the running cluster on 2026-08-19.
**[SPEC]** the contract this gateway must implement; not built yet.
---
## Models
| `model` value | Upstream | Engine | Context | Notes |
|---|---|---|---|---|
| `reasoning` | `reasoning-predictor.llm-serving:80` | vLLM, DeepSeek-R1-Distill-Qwen-32B | 16384 | emits `reasoning_content`; 8 sequence slots total |
| `ornith:35b` | `ornith-predictor.llm-serving:80` | Ollama | 131072 | reliable tool calling |
| `qwen2.5:3b-instruct` | `ornith-predictor.llm-serving:80` | Ollama | 32768 | same pods as ornith |
| `nomic-ai/nomic-embed-text-v2-moe` | `embeddings-predictor.llm-serving:80` | TEI | — | embeddings only |
| `BAAI/bge-reranker-base` | `reranker-predictor.llm-serving:80` | TEI | — | rerank only |
`reasoning` runs 2 replicas x `--max-num-seqs=4`. Those **8 slots are the scarcest resource in the cluster** and are shared across both dialects.
---
## Authentication [SPEC]
Ships behind a flag, default off. The model API is unauthenticated today.
```
Authorization: Bearer <authentik-jwt>
```
### Decided — Bearer on both surfaces
`Authorization: Bearer <jwt>` is the only accepted credential, on `/v1` and `/llm` alike. One auth path, consistent with G5, validated against Authentik via JWKS.
**Known divergence from Anthropic:** the real Anthropic API authenticates with `x-api-key` and requires `anthropic-version: 2023-06-01`. A stock Anthropic SDK pointed at `/llm` will send `x-api-key` and get a 401.
This is accepted, not overlooked. The `/llm` client is the first-party riotpiao frontend, which sends whatever we tell it to. If a real Anthropic SDK ever needs to reach this gateway, accepting `x-api-key` as a second credential source is an additive change — a small branch in one middleware, not a redesign.
`anthropic-version` is accepted and ignored if present, and never required.
The 401 for an `x-api-key`-only request must name the problem — say that Bearer is required — rather than returning a bare 401. The Kong retirement was caused by exactly this failure mode: a gateway that rejected the header clients actually send, without saying why.
---
## OpenAI dialect — `POST /v1/chat/completions`
### Request [SPEC]
```json
{
"model": "reasoning",
"messages": [{"role": "user", "content": "Why is wave 4 empty?"}],
"max_tokens": 2000,
"temperature": 0.7,
"stream": false
}
```
`model` is required and selects the upstream. The body is forwarded byte-identical — the gateway reads `model`, it does not rewrite it.
### Response, non-streaming [LIVE]
Captured verbatim from `reasoning` on 2026-08-19, abridged:
```json
{
"id": "chatcmpl-f17bd2fe22e4276d24e9438e40e89cea",
"object": "chat.completion",
"created": 1787172340,
"model": "reasoning",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "\n\nTo find the current weather in Toronto...",
"reasoning_content": "Okay, so I need to figure out...",
"tool_calls": []
},
"finish_reason": "length"
}],
"usage": {"prompt_tokens": 16, "completion_tokens": 300, "total_tokens": 316}
}
```
`reasoning_content` is a **sibling of** `content`, not nested in it. This is a vLLM extension produced by `--reasoning-parser=deepseek_r1`; it is not part of the OpenAI spec. Pass it through untouched.
### The two engines disagree on the field name [LIVE]
Verified 2026-08-19 by calling both:
| Upstream | Engine | Reasoning field |
|---|---|---|
| `reasoning-predictor` | vLLM | `reasoning_content` |
| `ornith-predictor` | Ollama | `reasoning` |
Neither is in the OpenAI spec, so neither is wrong — they are two vendor extensions that
happen to mean the same thing. The gateway must recognise **both** when mapping to the
Anthropic `thinking` block, or `ornith:35b` responses will silently lose their reasoning
on the `/llm` surface.
Do not normalise them on the `/v1` surface. That surface passes bodies through
untouched, and a client asking for `ornith:35b` should get exactly what Ollama sent.
Normalisation belongs in the canonical request model (task 2.9), which is the layer that
exists to absorb precisely this kind of upstream difference.
### Response, streaming [SPEC]
```
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"reasoning_content":"Okay"},"finish_reason":null}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Wave"},"finish_reason":null}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]
```
Data-only frames, no `event:` lines. Terminated by the literal `data: [DONE]`.
### Legacy aliases [LIVE, being retired]
`POST /v1/{reasoning,ornith,qwen}/chat/completions` force `model` to the corresponding value regardless of the body. They exist only because Kong could not dispatch on the body. Removed once callers migrate.
### `GET /v1/models` [SPEC]
```json
{"object":"list","data":[{"id":"reasoning","object":"model","owned_by":"homelab","created":0}]}
```
Derived from the registry, never hardcoded.
### Errors [SPEC]
RFC 9457 `application/problem+json`:
```json
{
"type": "https://riotpiao.com/errors/unknown-model",
"title": "Unknown model",
"status": 400,
"detail": "\"gpt-4\" is not available",
"validModels": ["reasoning", "ornith:35b", "qwen2.5:3b-instruct"]
}
```
---
## Anthropic dialect — `POST /llm/v1/messages` [SPEC]
Path note: the Anthropic SDK appends `/v1/messages` to its base URL, so a base URL of `https://api.riotpiao.com/llm` produces exactly this path.
### Request
```json
{
"model": "reasoning",
"max_tokens": 2000,
"system": "You are a cluster assistant.",
"messages": [
{"role": "user", "content": "Why is wave 4 empty?"}
],
"stream": true
}
```
Differences from the OpenAI dialect that the translator must handle:
| Concern | OpenAI | Anthropic |
|---|---|---|
| system prompt | `messages[0].role = "system"` | top-level `system` field |
| `max_tokens` | optional | **required** |
| content | string | string *or* block array |
| roles | system/user/assistant/tool | user/assistant only |
| stop | `stop` | `stop_sequences` |
`max_tokens` being required is a real divergence — the gateway must either reject its absence with a clear error or apply a documented default. Pick one and state it; do not silently default.
### Response, non-streaming
```json
{
"id": "msg_01ABC",
"type": "message",
"role": "assistant",
"model": "reasoning",
"content": [
{"type": "thinking", "thinking": "Waves are sort keys, not a sequence..."},
{"type": "text", "text": "Wave 4 is empty. Waves are sort keys..."}
],
"stop_reason": "end_turn",
"stop_sequence": null,
"usage": {"input_tokens": 16, "output_tokens": 300}
}
```
Field mapping from the upstream OpenAI response:
| Upstream | Anthropic |
|---|---|
| `choices[0].message.reasoning_content` | `content[]` block `{"type":"thinking","thinking":...}` |
| `choices[0].message.content` | `content[]` block `{"type":"text","text":...}` |
| `finish_reason: "stop"` | `stop_reason: "end_turn"` |
| `finish_reason: "length"` | `stop_reason: "max_tokens"` |
| `usage.prompt_tokens` | `usage.input_tokens` |
| `usage.completion_tokens` | `usage.output_tokens` |
The thinking block precedes the text block.
### Response, streaming
Anthropic SSE uses **named events with content-block indices**, unlike OpenAI's flat frames. Verified event sequence:
```
event: message_start
data: {"type":"message_start","message":{"id":"msg_01ABC","type":"message","role":"assistant","model":"reasoning","content":[],"usage":{"input_tokens":16,"output_tokens":0}}}
event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":""}}
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"Waves are sort keys"}}
event: content_block_stop
data: {"type":"content_block_stop","index":0}
event: content_block_start
data: {"type":"content_block_start","index":1,"content_block":{"type":"text","text":""}}
event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"Wave 4 is empty."}}
event: content_block_stop
data: {"type":"content_block_stop","index":1}
event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":300}}
event: message_stop
data: {"type":"message_stop"}
```
Three details that are easy to get wrong:
- In `message_delta`, `usage` is a **sibling of** `delta`, not inside it.
- The delta field name matches the delta type: `thinking_delta` carries `.thinking`, `text_delta` carries `.text`.
- Block index 0 is thinking, index 1 is text. **You only learn reasoning has ended when `content` first appears in an upstream chunk**, so the thinking block must be closed before the text block opens. If a response has no `reasoning_content` at all, the text block is index 0 and no thinking block is emitted.
### Queue position — non-standard extension
Anthropic's event set has no way to say "you are queued", because the stream implicitly begins after a slot is acquired. With only 8 slots, queueing is normal here.
Emitted **before** `message_start`:
```
event: queue
data: {"type":"queue","position":3}
```
This is deliberately outside the Anthropic spec. It is safe only because the client is first-party; a strict Anthropic client would ignore the unknown event and show nothing while queued.
### Errors
Anthropic error shape, **not** RFC 9457 — the same rejection renders differently depending on which surface received it:
```json
{"type":"error","error":{"type":"invalid_request_error","message":"Unknown model \"gpt-4\". Available: reasoning, ornith:35b, qwen2.5:3b-instruct"}}
```
| Condition | HTTP | `error.type` |
|---|---|---|
| unknown or missing model | 400 | `invalid_request_error` |
| `max_tokens` absent (if required) | 400 | `invalid_request_error` |
| malformed JSON | 400 | `invalid_request_error` |
| unsupported feature requested | 400 | `invalid_request_error` |
| not authenticated | 401 | `authentication_error` |
| budget exhausted or queue full | 429 | `rate_limit_error` |
| upstream failure | 502 | `api_error` |
### Deliberately not implemented
Each returns 400 naming the unsupported feature — never a silent partial implementation:
tool use and `tool_result` turns, image content blocks, prompt-caching headers, the batch API, multi-block user content, `thinking.budget_tokens` configuration.
The target client is the riotpiao frontend. Widening scope is a code change with a test, not an accident.
---
## Shared behaviour, both dialects
**One slot controller, keyed by upstream.** A `/v1` request and a `/llm` request contend for the same 8 `reasoning` slots and the same queue, in arrival order. Per-dialect semaphores would each believe they were within budget while together exceeding the physical limit.
**Streaming is unbuffered** and a client disconnect cancels the upstream immediately. An orphaned generation holds a slot until it completes on its own, which for a 32B model on a Volta GPU can run to minutes.
**Timeouts** [LIVE]: chat routes are connect 10s / read 1h / write 1h. The hour is deliberate — a 32B model on this hardware routinely exceeds 60s. Any shorter application cap is enforced in gateway logic, never by shortening the proxy timeout.
**Tool calling** [LIVE]: `reasoning` honours an explicit `tool_choice` but returns `tool_calls: []` under `tool_choice: "auto"` — it reasons about the tool in prose instead. `ornith:35b` returns `finish_reason: "tool_calls"` correctly under `auto`. This is a model property; the gateway does not compensate for it.
---
## Examples
```bash
# OpenAI dialect
curl -s https://api.riotpiao.com/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"model":"reasoning","messages":[{"role":"user","content":"Why is wave 4 empty?"}],"max_tokens":500}'
# Anthropic dialect, streaming
curl -N -s https://api.riotpiao.com/llm/v1/messages \
-H 'content-type: application/json' \
-d '{"model":"reasoning","max_tokens":500,"stream":true,
"messages":[{"role":"user","content":"Why is wave 4 empty?"}]}'
# model list
curl -s https://api.riotpiao.com/v1/models
```
docs/API-sqs.md
-262
View File
@@ -1,262 +0,0 @@
# API — queue surface (`/sqs/*`)
Fronts the Kafka Management Service (`kmsvc`) in namespace `sqs`. SQS-shaped
message-plane API over Kafka.
Status marks:
**[LIVE]** verified against the running cluster and the committed proto on 2026-08-19.
**[SPEC]** the contract this gateway must implement; not built yet.
Source of truth for shapes:
`~/workplace/kmsvc-proto/proto/kafkamgmt/v1/queue_service.proto`.
---
## The important finding: a REST surface already exists [LIVE]
**Do not build gRPC-to-JSON transcoding.** `kmsvc-manage` already mounts grpc-gateway:
```go
mux := runtime.NewServeMux()
kafkamgmtv1.RegisterQueueServiceHandlerServer(ctx, mux, svc)
```
The upstream serves plain REST/JSON on **:8080** and plain gRPC on **:9090**. Neither
gRPC-Web nor server reflection is enabled.
So `/sqs/*` is a **path-stripping reverse proxy plus authentication**, not a protocol
translator. That makes it dramatically cheaper than the LLM surface.
```
api.riotpiao.com/sqs/v1/queues/{q}/messages
| strip /sqs, authenticate
v
management-service.sqs.svc.cluster.local:8080/v1/queues/{q}/messages
```
Upstream: Deployment `management-service`, 3 replicas, HPA 3-9, Service ClusterIP
`10.98.3.138`, ports `8080` (http) and `9090` (grpc).
---
## Endpoints [LIVE — HTTP annotations from the proto]
Six operations. All unary. No streaming, no subscribe.
| Method | Path (after `/sqs` strip) | RPC |
|---|---|---|
| POST | `/v1/queues/{queue_name}/messages` | `SendMessage` |
| POST | `/v1/queues/{queue_name}/messages:batch` | `SendMessageBatch` |
| GET | `/v1/queues/{queue_name}/messages` | `ReceiveMessage` |
| DELETE | `/v1/queues/{queue_name}/messages/{receipt_handle}` | `DeleteMessage` |
| POST | `/v1/queues/{queue_name}/messages:batchDelete` | `DeleteMessageBatch` |
| PATCH | `/v1/queues/{queue_name}/messages/{receipt_handle}` | `ChangeMessageVisibility` |
---
## Two wire-format traps [LIVE]
Both follow from grpc-gateway defaults, and both will surprise anyone who reads only
the proto.
**1. `bytes` fields are base64 in JSON.** `SendMessageRequest.message_body` and
`Message.body` are proto `bytes`. The JSONPB marshaler encodes them as base64 strings.
Sending raw text will not do what you expect.
**2. Field names are lowerCamelCase.** `cmd/server/main.go` calls bare
`runtime.NewServeMux()` with no marshaler options, so `OrigName` is false. The wire uses
`messageBody`, `receiptHandle`, `maxNumberOfMessages` — not the snake_case names in the
proto.
Document both prominently or every first-time caller loses an hour.
---
## Message shapes [LIVE — from the proto]
### Send
```
POST /sqs/v1/queues/agent-worker-queue/messages
{
"messageBody": "aGVsbG8gd29ybGQ=", // base64 of "hello world"
"messageAttributes": {"values": {"k": "v"}},
"messageGroupId": "", // FIFO only
"messageDeduplicationId": "", // FIFO only
"delaySeconds": 0 // 0-900
}
-> {"messageId": "...", "sequenceNumber": ""} // sequenceNumber FIFO only
```
### Receive — long poll
```
GET /sqs/v1/queues/agent-worker-queue/messages
?maxNumberOfMessages=10 // <= 10
&waitTimeSeconds=20 // 0-20
&visibilityTimeoutSeconds=30 // optional override
-> {"messages": [{
"messageId": "...",
"receiptHandle": "...",
"body": "aGVsbG8gd29ybGQ=",
"attributes": {"values": {}},
"receiveCount": 1,
"messageGroupId": "",
"enqueuedAt": "2026-08-19T16:29:07Z"
}]}
```
### Delete — the ack
```
DELETE /sqs/v1/queues/agent-worker-queue/messages/{receiptHandle}
-> {}
```
### Change visibility
```
PATCH /sqs/v1/queues/agent-worker-queue/messages/{receiptHandle}
{"visibilityTimeoutSeconds": 60} // 0-43200
-> {}
```
### Batch
Both batch calls take `entries[]` with a caller-assigned `id`, and return partial
success:
```json
{"successful": [{"id": "1", "messageId": "..."}],
"failed": [{"id": "2", "error": "..."}]}
```
A batch call can return 200 with entries in `failed`. Callers must inspect the body,
not just the status.
### Limits [LIVE — from the SDK]
`MaxMessageBodyBytes = 262144` (256 KiB), `MaxReceiveMessages = 10`,
`MaxWaitTimeSeconds = 20`.
---
## Semantics
At-least-once, SQS-style. Receive leases a message for the visibility timeout; the
caller must `DeleteMessage` to acknowledge. An un-deleted message reappears after the
timeout and `receiveCount` increments. After `maxReceiveCount` (default 5) it goes to
the DLQ if one is configured.
**Long-polling matters for the gateway.** `waitTimeSeconds` up to 20 means a `GET` can
legitimately hold open for 20 seconds returning nothing. Read timeouts must exceed that
comfortably, and a client disconnect must cancel upstream — the same requirement as the
LLM surface, for the same reason.
---
## Error mapping [SPEC]
The SDK maps gRPC codes to sentinel errors; grpc-gateway maps them to HTTP. Use this as
the gateway's status contract:
| gRPC code | HTTP | SDK sentinel |
|---|---|---|
| `NotFound` | 404 | `ErrQueueNotFound` |
| `AlreadyExists` | 409 | `ErrAlreadyExists` |
| `InvalidArgument` | 400 | `ErrInvalidArgument` |
| `Unauthenticated` | 401 | `ErrUnauthenticated` |
| `ResourceExhausted` | 429 | `ErrMessageTooLarge` |
Upstream errors arrive in the grpc-gateway envelope
`{"code": 5, "message": "Not Found", "details": []}`. Decide deliberately whether
`/sqs/*` passes that through or re-renders it as RFC 9457 to match `/v1/*`.
Recommendation: **pass through**, so the gateway does not become a second, subtly
different error vocabulary for the same upstream.
---
## Queue lifecycle is NOT in this API [LIVE]
There is no `CreateQueue`, `DeleteQueue`, or `ListQueues` RPC. The proto says so
explicitly:
```proto
// Queue lifecycle (create/delete/configure) is managed via the Queue CRD,
// not this service
```
Queues are Kubernetes resources — `queues.kmsvc.io/v1`, namespaced. `kmsvc-cli`'s
`create-queue` and `delete-queue` talk to the Kubernetes API, not to kmsvc.
**This is a hard boundary for the gateway.** Exposing queue creation over `/sqs/*` would
require the gateway to hold Kubernetes write credentials, which violates **G2**. Do not
add it. If declarative queue management ever needs a public surface, it belongs behind a
separate component with its own RBAC — not in the public edge process.
Queue spec fields, for reference when reading a queue's configuration:
`fifoQueue`, `isDLQ`, `deadLetterTargetQueue`, `delaySeconds` (0-900),
`maxReceiveCount` (default 5), `messageRetentionPeriodSeconds` (default 345600),
`visibilityTimeoutSeconds` (default 30), `minShards`, `maxShards` (default 8),
`partitionsPerShard` (default 6), `shardSplitThresholdBytesPerSec`,
`shardSplitCooldownSeconds`.
Kafka topics are named `kmsvc.{queue}.shard-{id}` and are created by `queue-operator`
directly via the Kafka Admin API — there are no `KafkaTopic` CRs.
Currently one queue exists: `agent-worker-queue` in namespace `sqs`, phase `Ready`,
1 shard.
---
## Authentication [SPEC]
`Authorization: Bearer <jwt>`, same as every other gateway surface.
**The upstream enforces nothing.** `kmsvc`'s auth interceptor exists but is never wired,
and the REST surface is mounted with the in-process grpc-gateway variant that bypasses
gRPC interceptors regardless. Both `:8080` and `:9090` are currently open, and
`kmsvc.riotpiao.com` is publicly routed.
The gateway is therefore the only authentication boundary for this surface.
---
## Out of scope
- **Workflow start.** Nothing in kmsvc starts a Temporal workflow — no such RPC exists,
and grep for `ExecuteWorkflow`/`StartWorkflow` across `kmsvc-manage`, `kmsvc-sdk` and
`kmsvc-cli` returns nothing. A caller dials `temporal-frontend.temporal.svc:7233`
with a Temporal SDK directly. A `/workflow/*` surface is net-new code, not a proxy
route — see [task 7.3](../tasks/7.3-workflow-prefix.md).
- **DLQ operations.** `kmsvc-cli`'s `dlq peek` and `dlq redrive` are client-side
compositions of the six RPCs, not server operations. Redrive is a non-atomic
Receive-Send-Delete. If `/sqs/*` should offer redrive, that is new logic with real
failure modes, not a proxied call.
- **Kafka direct access.** No external listener exists; the bootstrap
`kmsvc-kafka-bootstrap.sqs.svc.cluster.local:9092` is cluster-internal only. The
gateway proxies kmsvc, never Kafka.
---
## Examples
```bash
Q=agent-worker-queue
# send (body must be base64)
curl -s -X POST https://api.riotpiao.com/sqs/v1/queues/$Q/messages \
-H 'content-type: application/json' \
-d "{\"messageBody\":\"$(printf 'hello world' | base64)\"}"
# receive, long poll 20s
curl -s "https://api.riotpiao.com/sqs/v1/queues/$Q/messages?maxNumberOfMessages=10&waitTimeSeconds=20"
# acknowledge
curl -s -X DELETE https://api.riotpiao.com/sqs/v1/queues/$Q/messages/$RECEIPT
# extend the lease
curl -s -X PATCH https://api.riotpiao.com/sqs/v1/queues/$Q/messages/$RECEIPT \
-H 'content-type: application/json' -d '{"visibilityTimeoutSeconds":60}'
```
docs/MIGRATION-kong.md
-143
View File
@@ -1,143 +0,0 @@
# Kong retirement — inventory and cutover
Everything Kong does on `api.riotpiao.com` today, and where it goes. Inventory
verified live against context `admin@homelab-cluster` on 2026-08-19.
Source of the objects being retired: `~/workplace/homelab/k8s/apps/api/` and
`k8s/argocd/apps/55-api-gateway.yaml`.
## What is running now
Kong OSS 3.4.1, Helm chart from `https://charts.konghq.com`, DB-less, namespace
`api`, Argo Application `kong` at sync wave 7. Two replicas. Fronted by
`ingress-nginx` via Ingress `api/api`, which catch-alls `/` on `api.riotpiao.com`
to `kong-proxy:80`.
Eleven ReplicaSets exist on the Kong Deployment, the newest minutes old — this
config is being actively iterated, so re-verify the inventory immediately before
cutover.
## Routing table to port
Seven `ingressClassName: kong` Ingresses. Six in `llm-serving`, one in `agent-pod`.
| Method | Path | Upstream | Transform applied by Kong |
|---|---|---|---|
| GET | `/v1/models` | — | `request-termination`: static 200 JSON, upstream never contacted |
| POST | `/v1/reasoning/chat/completions` | `reasoning-predictor:80` | force body `model=reasoning`, rewrite URI to `/v1/chat/completions` |
| POST | `/v1/ornith/chat/completions` | `ornith-predictor:80` | force body `model=ornith:35b`, rewrite URI |
| POST | `/v1/qwen/chat/completions` | `ornith-predictor:80` | force body `model=qwen2.5:3b-instruct`, rewrite URI |
| POST | `/v1/embeddings` | `embeddings-predictor:80` | none — TEI already serves the canonical path |
| POST | `/v1/rerank` | `reranker-predictor:80` | rewrite URI to `/rerank` (TEI does not serve `/v1/rerank`) |
| GET/WS | `/console`, `/run`, `/sessions` | `agent-hub:9090` (`agent-pod` ns) | none, `strip-path: false` |
Upstream model map, from the manifest comments and confirmed live:
- `reasoning` → `reasoning-predictor` — vLLM, DeepSeek-R1-Distill-Qwen-32B, 2 replicas,
`--max-num-seqs=4`, `--max-model-len=16384`, `--reasoning-parser=deepseek_r1`,
`--enable-auto-tool-choice --tool-call-parser=hermes`
- `ornith:35b` → `ornith-predictor` — Ollama, 2 replicas
- `qwen2.5:3b-instruct` → `ornith-predictor` — same pods; both models stay resident via
`OLLAMA_MAX_LOADED_MODELS=2`, `OLLAMA_KEEP_ALIVE=-1`
- `nomic-ai/nomic-embed-text-v2-moe` → `embeddings-predictor` — TEI
- `BAAI/bge-reranker-base` → `reranker-predictor` — TEI
### The path-per-model surface goes away
The three chat paths exist only because Kong OSS cannot dispatch on the request
body. The gateway serves a single `POST /v1/chat/completions` and selects the
upstream from the body's `model` field.
Keep the old paths as aliases during cutover so live clients do not break, then
remove them once callers have migrated. pi is a live caller today.
### `/v1/models` should not be ported verbatim
Kong serves a hardcoded list via `request-termination`. The manifest already flags
that it can drift from what the engines actually serve. Derive the response from
the gateway's configured upstream map instead, so the list cannot disagree with
what routing will accept.
## Plugins being retired
| Plugin | Scope | Replacement |
|---|---|---|
| `llm-rewrite-reasoning` / `-ornith` / `-qwen` | llm-serving | body-based dispatch in `internal/llm` |
| `llm-rewrite-rerank` | llm-serving | per-upstream path rewrite in the route table |
| `llm-models-list` | llm-serving | derived from the upstream map |
| `prometheus` | **cluster-wide** | `internal/observability` — must expose bandwidth, latency, status codes, upstream health or observability regresses |
No `rate-limiting` plugin exists anywhere in the cluster. REQUIREMENTS.md §4 Tier 2
describes it as an existing layer; it is not built. Nothing to migrate — it is net
new work, and it now belongs in the gateway rather than in Kong.
## Auth: currently off, must land on
`KongConsumer model-invoker` exists in namespace `api` and stays defined, but the
`key-auth` plugin is commented out and every route has `model-key-auth` stripped
from its `konghq.com/plugins` annotation.
**The model API is unauthenticated right now.** Confirmed live 2026-08-19: a request
to `/v1/reasoning/chat/completions` with no credentials returns 200.
The reason is recorded in `model-auth.yaml` — Kong's `key-auth` accepts a raw
`apikey:` header but rejects `Authorization: Bearer`, which blocks every
OpenAI-compatible client. That is why `~/.pi/agent/models.json` carries a
`customHeaders: {apikey: ...}` block.
The gateway reads Bearer tokens directly and validates them against Authentik via
JWKS. `AUTH-PLAN.md`'s pinned-RSA-key approach and its rotation runbook are not
needed and should not be carried over.
Ship auth behind a flag. Turning it on breaks every current caller until they hold
a token — pi included.
## Timeouts
Kong today:
| Route class | connect | read | write |
|---|---|---|---|
| chat | 10s | **1h** | 1h |
| embeddings / rerank | 10s | 10m | 10m |
nginx in front sets `proxy-read-timeout: 3600`, `proxy-send-timeout: 3600`,
`proxy-buffering: off`, `proxy-body-size: 0`. Those stay — they are what makes token
streaming work, and the gateway needs the same treatment from nginx.
The 1-hour read timeout is deliberate: a 32B model on a Volta GPU routinely exceeds
60s. Any shorter server-side cap must be enforced *in the gateway*, not by shortening
the proxy timeout, or long legitimate generations get truncated mid-stream.
## Cutover
Reversible at every step. Kong keeps serving until the last step.
1. Deploy the gateway alongside Kong, unexposed. Verify in-cluster against
`http://homelab-frontend.api.svc.cluster.local`.
2. Compare gateway and Kong responses for every route in the table above, including
a streaming chat request and a client disconnect mid-stream.
3. Repoint Ingress `api/api` from `kong-proxy:80` to the gateway Service. **This is
the cutover.** Reverting is a one-line change to the same Ingress.
4. Soak. Watch gateway metrics and pi traffic.
5. Delete the seven kong-class Ingresses and the six KongPlugin CRs.
6. Remove the `kong` Application from `k8s/argocd/apps/55-api-gateway.yaml`; let Argo
prune the Helm release, the CRDs and namespace leftovers.
Steps 1–4 are reversible in seconds. Step 5 onward is not — do not start it until the
soak is clean.
All of this flows through git and Argo. No `kubectl apply`, no `helm upgrade`.
## Loose ends
- `agent-pod/console` is publicly routed, unauthenticated, accepts free-form prompts
into a shell-capable container, and exposes a WebSocket. Migrating it behind the
gateway's auth is a security fix, not merely a port. Treat WebSocket upgrade as an
explicit requirement of the proxy layer.
- Eight `*.example.com` hosts exist on istio-class Ingresses in `llm-serving`
(`{embeddings,ornith,reasoning,reranker}[-predictor]-llm-serving.example.com`).
KServe defaults, not public, not Kong's — out of scope here, but they exist and
should not be mistaken for gateway routes.
- Ingress class split across the cluster is 7 kong / 17 nginx / 4 istio. Only the 7
kong ones are in scope.
docs/SERVICE-USAGE.md
-542
View File
@@ -1,542 +0,0 @@
# Service Usage Guide
Complete guide for calling all gateway-backed services via `api.riotpiao.com`.
**Table of Contents**
- [Quick Start](#quick-start)
- [Authentication](#authentication)
- [Service Map](#service-map)
- [Service-Specific Guides](#service-specific-guides)
---
## Quick Start
All services use the **X-Service** header to route requests:
```bash
curl -X POST https://api.riotpiao.com/path \
-H "X-Service: <service_name>" \
-H "Authorization: Bearer <jwt>" \
-H "Content-Type: application/json" \
-d '{"key": "value"}'
```
| Header | Purpose | Example |
|---|---|---|
| `X-Service` | Route to named service | `X-Service: sqs` |
| `X-Resource` | (Optional) Resource ID for auth | `X-Resource: agent-worker-queue` |
| `Authorization` | Bearer token (required for auth-protected services) | `Authorization: Bearer eyJ...` |
---
## Authentication
### Getting a Token
**From Authentik (OAuth2 client credentials flow):**
```bash
AUTHENTIK_URL=https://authentik.riotpiao.com
CLIENT_ID="your-client-id"
CLIENT_SECRET="your-client-secret"
TOKEN=$(curl -s -X POST ${AUTHENTIK_URL}/application/o/token/ \
-d "grant_type=client_credentials&client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&scope=openid" \
| jq -r '.access_token')
echo $TOKEN
```
Replace `your-client-id` and `your-client-secret` with Authentik app credentials.
### Service-Specific Auth
| Service | Auth Required | Token Audience | Notes |
|---|---|---|---|
| **sqs** | ✅ Yes (gateway validates) | `sqs` | JWT signature & claims verified by gateway before proxying |
| **memory** | ❌ No | — | Pass-through (service-owned if needed) |
| **s3** (MinIO) | ❌ No | — | Native OIDC support (service-owned) |
| **iam** | ❌ No | — | Pass-through (service-owned if needed) |
| **workflow** (Temporal) | ❌ No | — | Native JWT support (service-owned) |
---
## Service Map
### Gateway Services (via X-Service header)
```
api.riotpiao.com
├─ X-Service: sqs
│ ├─ Upstream: management-service.sqs.svc.cluster.local:8080
│ ├─ Auth: ✅ Gateway validates JWT
│ └─ Docs: docs/API-sqs.md
│
├─ X-Service: memory
│ ├─ Upstream: poimen-memory.memory.svc.cluster.local:9090
│ ├─ Auth: ❌ Pass-through (service-owned)
│ └─ Docs: See Memory Service section below
│
├─ X-Service: s3
│ ├─ Upstream: minio.data.svc.cluster.local:9000
│ ├─ Auth: ❌ Native OIDC (service-owned)
│ └─ Notes: S3-compatible API
│
├─ X-Service: iam
│ ├─ Upstream: keycloak.iam.svc.cluster.local:8080 (or equivalent)
│ ├─ Auth: ❌ Pass-through (service-owned)
│ └─ Docs: See IAM Service section below
│
└─ X-Service: workflow
├─ Upstream: temporal-frontend.temporal.svc.cluster.local:7233
├─ Auth: ❌ Native JWT support (service-owned)
├─ Protocol: gRPC only (returns 501 for HTTP)
└─ Docs: docs/TEMPORAL_USAGE.md
```
### Path Prefixes (legacy, before X-Service migration)
```
api.riotpiao.com
├─ /v1/* → llm-serving (predictors: vLLM, Ollama, TEI)
├─ /sqs/* → management-service (Kafka queues)
├─ /workflow/* → Temporal (workflows)
└─ /cluster/* → atlas (topology & Argo delivery)
```
**Migration note:** X-Service routing is the current standard. Path prefixes are deprecated.
---
## Service-Specific Guides
### SQS (Kafka Queue Management)
**Endpoint:** `POST https://api.riotpiao.com/`
**Headers:**
```
X-Service: sqs
Authorization: Bearer <jwt>
```
**Send a message:**
```bash
QUEUE="agent-worker-queue"
BODY=$(printf "hello world" | base64)
curl -X POST https://api.riotpiao.com/ \
-H "X-Service: sqs" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"messageBody\": \"$BODY\",
\"messageAttributes\": {\"values\": {}},
\"delaySeconds\": 0
}"
```
**Response:**
```json
{
"messageId": "d9f94e63-b2c1-4e9f-8c5f-8d5e3c1b7a0f",
"sequenceNumber": ""
}
```
**Receive messages (long poll, up to 20s):**
```bash
QUEUE="agent-worker-queue"
curl -X GET "https://api.riotpiao.com/?X-Service=sqs&queue=$QUEUE&maxNumberOfMessages=10&waitTimeSeconds=20" \
-H "Authorization: Bearer $TOKEN"
```
**Response:**
```json
{
"messages": [
{
"messageId": "d9f94e63-b2c1-4e9f-8c5f-8d5e3c1b7a0f",
"receiptHandle": "...",
"body": "aGVsbG8gd29ybGQ=",
"attributes": {"values": {}},
"receiveCount": 1,
"enqueuedAt": "2026-08-27T22:18:37Z"
}
]
}
```
**Acknowledge (delete) a message:**
```bash
RECEIPT="..."
curl -X DELETE https://api.riotpiao.com/ \
-H "X-Service: sqs" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d "{\"receiptHandle\": \"$RECEIPT\"}"
```
**Full documentation:** [docs/API-sqs.md](API-sqs.md)
---
### Memory Service (Context & Embeddings)
**Endpoint:** `https://api.riotpiao.com/`
**Headers:**
```
X-Service: memory
```
**Status:** Service definition in progress. Uses PostgreSQL + pg_vector for embeddings.
**Planned operations:**
- Store session memory / agent context
- Query by similarity (embedding search)
- Update with approval workflow (GRM/Git review)
**Coming soon:** Full API documentation.
---
### S3 / MinIO (Object Storage)
**Endpoint:** `https://api.riotpiao.com/`
**Headers:**
```
X-Service: s3
```
**List buckets:**
```bash
curl -X GET https://api.riotpiao.com/ \
-H "X-Service: s3" \
-H "Authorization: Bearer $TOKEN"
```
**List objects in bucket:**
```bash
curl -X GET https://api.riotpiao.com/?bucket=my-bucket&prefix=data/ \
-H "X-Service: s3"
```
**Put object:**
```bash
curl -X PUT https://api.riotpiao.com/my-bucket/path/to/object.json \
-H "X-Service: s3" \
--data-binary @object.json
```
**Get object:**
```bash
curl -X GET https://api.riotpiao.com/my-bucket/path/to/object.json \
-H "X-Service: s3"
```
**Full S3/MinIO API:** Standard AWS S3 compatible API. See [MinIO docs](https://min.io/docs/minio/linux/reference/minio-mc/mc-ls.html).
---
### IAM (Identity & Access Management)
**Endpoint:** `https://api.riotpiao.com/`
**Headers:**
```
X-Service: iam
```
**List roles:**
```bash
curl -X GET https://api.riotpiao.com/roles \
-H "X-Service: iam"
```
**Get user:**
```bash
curl -X GET https://api.riotpiao.com/users/alice \
-H "X-Service: iam"
```
**Create user:**
```bash
curl -X POST https://api.riotpiao.com/users \
-H "X-Service: iam" \
-H "Content-Type: application/json" \
-d "{
\"username\": \"bob\",
\"email\": \"bob@example.com\",
\"password\": \"secure-password\"
}"
```
**Assign role to user:**
```bash
curl -X POST https://api.riotpiao.com/users/bob/roles \
-H "X-Service: iam" \
-H "Content-Type: application/json" \
-d "{\"role\": \"admin\"}"
```
**Full documentation:** Service-specific (depends on IAM backend).
---
### Workflow (Temporal)
**Endpoint:** `temporal-frontend.temporal.svc.cluster.local:7233`
**Protocol:** gRPC only
**Note:** HTTP requests return **501 Not Implemented**
Use Temporal SDK directly:
```go
import "go.temporal.io/sdk/client"
c, _ := client.Dial(client.Options{HostPort: "temporal-frontend.temporal.svc.cluster.local:7233"})
defer c.Close()
// Start workflow
run, _ := c.ExecuteWorkflow(ctx, opts, YourWorkflow, args...)
var result YourWorkflowResult
run.Get(ctx, &result)
```
**Full documentation:** [docs/TEMPORAL_USAGE.md](../TEMPORAL_USAGE.md)
---
## Error Handling
### Standard Error Response
All services return errors in **RFC 9457 Problem Details** format:
```json
{
"type": "https://api.riotpiao.com/problem/not-found",
"title": "Not Found",
"status": 404,
"detail": "Resource does not exist",
"instance": "/sqs/v1/queues/nonexistent"
}
```
### Common Status Codes
| Code | Meaning | Example |
|---|---|---|
| `200 OK` | Success | Message sent, resource retrieved |
| `201 Created` | Resource created | Queue created, object uploaded |
| `204 No Content` | Success (no body) | Message deleted |
| `400 Bad Request` | Invalid input | Message body too large, invalid field |
| `401 Unauthorized` | Missing/invalid token | No Authorization header, token expired |
| `403 Forbidden` | Token valid but insufficient permissions | User lacks sqs:write permission |
| `404 Not Found` | Resource not found | Queue doesn't exist, object not found |
| `429 Too Many Requests` | Rate limit exceeded | Per-user budget exhausted |
| `502 Bad Gateway` | Upstream unreachable | Service is down or network issue |
| `501 Not Implemented` | Operation not supported | gRPC request via HTTP |
### SQS-Specific Error Mapping
SQS errors (from kmsvc) map as follows:
| gRPC Code | HTTP Status | Message |
|---|---|---|
| `NotFound` | `404` | Queue or message not found |
| `AlreadyExists` | `409` | Queue already exists |
| `InvalidArgument` | `400` | Message body too large, invalid parameter |
| `Unauthenticated` | `401` | Missing Authorization header |
| `ResourceExhausted` | `429` | Message too large, quota exceeded |
---
## Request/Response Examples
### Example 1: Send SQS Message with Auth
```bash
#!/bin/bash
GATEWAY="https://api.riotpiao.com"
AUTHENTIK="https://authentik.riotpiao.com"
CLIENT_ID="sqs-client"
CLIENT_SECRET="secret123"
QUEUE="agent-worker-queue"
# Get token
TOKEN=$(curl -s -X POST ${AUTHENTIK}/application/o/token/ \
-d "grant_type=client_credentials&client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&scope=openid" \
| jq -r '.access_token')
# Send message
BODY=$(echo "process this task" | base64)
curl -X POST ${GATEWAY}/ \
-H "X-Service: sqs" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d "{\"messageBody\": \"$BODY\"}" | jq .
```
### Example 2: Receive & Process Queue
```bash
#!/bin/bash
GATEWAY="https://api.riotpiao.com"
TOKEN="..."
QUEUE="agent-worker-queue"
MAX_MSGS=10
WAIT_SECS=20
while true; do
# Receive messages (long poll)
RESPONSE=$(curl -s -X GET "${GATEWAY}/?X-Service=sqs&queue=${QUEUE}&maxNumberOfMessages=${MAX_MSGS}&waitTimeSeconds=${WAIT_SECS}" \
-H "Authorization: Bearer $TOKEN")
MESSAGES=$(echo "$RESPONSE" | jq '.messages')
if [[ "$MESSAGES" == "null" ]]; then
echo "No messages (timeout)"
continue
fi
# Process each message
echo "$RESPONSE" | jq -r '.messages[] | @base64d' | while read -r MSG; do
echo "Processing: $MSG"
# Do work...
# Acknowledge message
RECEIPT=$(echo "$RESPONSE" | jq -r '.messages[0].receiptHandle')
curl -s -X DELETE ${GATEWAY}/ \
-H "X-Service: sqs" \
-H "Authorization: Bearer $TOKEN" \
-d "{\"receiptHandle\": \"$RECEIPT\"}"
done
done
```
### Example 3: S3 Workflow (Upload & List)
```bash
#!/bin/bash
GATEWAY="https://api.riotpiao.com"
BUCKET="my-data"
# Upload file
echo "Uploading..."
curl -X PUT ${GATEWAY}/${BUCKET}/backup-$(date +%s).tar.gz \
-H "X-Service: s3" \
--data-binary @backup.tar.gz
# List objects
echo "Listing..."
curl -X GET "${GATEWAY}/?bucket=${BUCKET}&prefix=backup-" \
-H "X-Service: s3" | jq '.Contents[]'
```
---
## Testing
### Integration Tests
Run the full test suite:
```bash
./scripts/test-integration.sh
```
Run specific service tests:
```bash
GATEWAY_URL=https://api.riotpiao.com go test -tags integration -v -run TestSQS ./internal/serviceadapter
```
### Local Testing
Start local gateway with test services:
```bash
# Terminal 1: Start gateway
CONFIG_PATH=k8s/configmap.yaml go run ./cmd/gateway
# Terminal 2: Run tests
./scripts/test-integration.sh
```
### Canary Deployment
Test a single replica before rolling out:
```bash
./scripts/test-canary.sh
```
---
## Debugging
### Check Gateway Logs
```bash
kubectl -n api logs -l app=api-gateway --tail=100 -f
```
### Port-Forward to Service
```bash
kubectl -n sqs port-forward svc/management-service 8080:8080
curl http://localhost:8080/v1/queues
```
### Verify Service Availability
```bash
kubectl get svc -A | grep -E "management-service|poimen-memory|minio|temporal"
```
### Test Direct Service Access
```bash
kubectl -n sqs exec -it deployment/management-service -- \
curl -s http://localhost:8080/v1/queues | jq .
```
---
## Rate Limits & Quotas
| Service | Limit | Notes |
|---|---|---|
| **SQS** | Per-user budget (tokens) | Budget enforced per Authentik user |
| **S3** | MinIO quotas | Set per bucket in MinIO config |
| **Memory** | Not yet enforced | Future: embeddings storage limits |
| **Temporal** | Workflow concurrency | Set in Temporal cluster config |
See [docs/QUOTAS.md](QUOTAS.md) for detailed limits.
---
## Next Steps
- [Service Adapter Documentation](../tasks/8.1-serviceadapter-crd-and-informer.md)
- [X-Service Routing](../tasks/8.2-x-service-dispatcher.md)
- [Authentication & JWT Validation](../tasks/3.1-auth-sqs-jwt-validation.md)
- [Testing Guide](../TESTING_GUIDE.md)
- [Integration Tests](../INTEGRATION_TESTS.md)
docs/adr/ADR-0001-retire-kong-for-go-gateway.md
-136
View File
@@ -1,136 +0,0 @@
# ADR-0001 — Retire Kong OSS in favour of a Go API gateway
Status: Accepted
Date: 2026-08-19
Deciders: rock
## Context
`api.riotpiao.com` is currently served by Kong OSS 3.4.1 (Helm, DB-less, namespace `api`,
Argo wave 7), sitting behind ingress-nginx which owns TLS. Kong routes to the KServe
model predictors in `llm-serving` via seven `ingressClassName: kong` Ingresses and six
`KongPlugin` CRs.
Three separate capabilities were attempted on Kong OSS. All three failed, and each
failure is already documented in-repo by the person who hit it:
**1. Body-based model dispatch is not expressible.**
From `k8s/apps/api/llm-routes.yaml`:
> a single `/v1/chat/completions` endpoint that dispatches on the body's `model` field is
> not expressible in Kong OSS (`ai-proxy-advanced`, which does multi-target model routing,
> is Enterprise-only).
The workaround is a path-per-model surface (`/v1/reasoning/chat/completions`,
`/v1/ornith/...`, `/v1/qwen/...`) with a `request-transformer` force-overwriting the body's
`model` field. This is not OpenAI-standard, so every client needs bespoke configuration —
visible today in `~/.pi/agent/models.json`, which carries three separate provider entries
for what should be one endpoint.
**2. OIDC is Enterprise-only.**
`k8s/apps/api/AUTH-PLAN.md` routes around the missing `openid-connect` plugin using the
built-in `jwt` plugin, which requires pinning Authentik's RSA public key onto a
KongConsumer. That plan lists its own consequence:
> Pinning `rsa_public_key`: Authentik key rotation would break it — document a rotation
> runbook, or have the provision script re-export the cert PEM into the Kong credential on
> each run.
A rotation runbook is a standing operational liability accepted only because the gateway
cannot fetch JWKS itself.
**3. `key-auth` cannot read `Authorization: Bearer`.**
From `k8s/apps/api/model-auth.yaml`:
> a raw `apikey: <key>` header succeeds (200), the same request with only
> `Authorization: Bearer <key>` fails (401). No OpenAI-SDK-compatible client (pi included)
> sends a raw apikey header or lets you customize the header name, so every such client was
> hard-blocked.
Consequence: authentication on the model routes is **currently disabled**. Verified live
2026-08-19 — `api.riotpiao.com/v1/reasoning/chat/completions` answers unauthenticated.
Separately, the intended surface has grown beyond LLM routing. The target is a
capability-per-subdomain API over cluster services — `sqs.riotpiao.com` for queue
operations, `workflow.riotpiao.com` for Temporal, `cluster.riotpiao.com` for atlas — each
needing request shaping, per-caller budgets and streaming semantics that are application
concerns, not gateway-plugin concerns.
## Decision
Retire Kong OSS entirely. Replace it with a purpose-built Go service,
`homelab-frontend`, which owns north-south routing, authentication, and request shaping
for every public capability on `*.riotpiao.com`.
ingress-nginx keeps the edge and TLS. It forwards to the gateway instead of `kong-proxy`.
Authentication is Authentik OIDC, validated by fetching JWKS from
`https://authentik.riotpiao.com` at runtime.
## Options considered
**A. Stay on Kong OSS, accept the workarounds.**
Keeps a battle-tested proxy and its Prometheus plugin. But the path-per-model surface stays
non-standard, the RSA pinning runbook stays, and auth stays off until someone writes a
`request-transformer` shim to copy Bearer into an `apikey` header. Every new capability
(`sqs`, `workflow`) inherits the same constraints.
**B. Buy Kong Enterprise.**
`ai-proxy-advanced` and `openid-connect` solve 1 and 2. Does not solve the genuinely
application-level requirements at all — signed session cookies, per-session daily message
budgets, a 6-of-8 GPU sequence-slot semaphore with a bounded queue, and
disconnect-cancels-upstream are not gateway features in any tier. Cost for a homelab is not
justifiable.
**C. Go gateway, Kong retained for LLM paths only.**
Gradual migration, lower risk. But it means running two gateways indefinitely, splitting the
routing table across Kong CRDs and Go code, and keeping the Kong Helm release and its CRDs.
The split is the thing most likely to drift.
**D. Go gateway, Kong retired entirely.** — chosen
One routing table, one auth implementation, one place to reason about timeouts. The logic
being replaced is small: four `request-transformer` plugins that set a body field and
rewrite a URI, one `request-termination` serving a static JSON model list, and one
`prometheus` plugin. That is on the order of a hundred lines of Go, against roughly 480
lines of YAML it retires.
## Consequences
### Gained
- **Standard OpenAI surface.** One `POST /v1/chat/completions`, model selected from the
request body. Any OpenAI SDK works unmodified. The three pi provider entries collapse to
one.
- **Working authentication.** Bearer tokens are read from the header, because it is our
code. JWKS is fetched and cached with automatic rotation handling, so the AUTH-PLAN.md
rotation runbook is deleted rather than written.
- **Application-level policy becomes possible.** GPU slot semaphore, per-session budgets,
disconnect propagation and SSE handling live where the state is.
- **One timeout story.** Kong currently sets `read-timeout: 3600000` (1 hour) on chat
routes, which silently defeats any shorter server-side cap. Retiring Kong removes the
conflicting layer.
- **~480 lines of gateway YAML deleted**, plus the Kong CRDs, the Helm release, and its
`ServerSideApply` workaround for oversized CRD annotations.
### Lost / assumed
- **We now own proxy correctness.** Connection pooling, retries, timeout propagation,
streaming passthrough, header hygiene, graceful shutdown. `net/http/httputil.ReverseProxy`
covers most of it, but it is our bug surface now.
- **Kong's Prometheus plugin goes away.** The gateway must expose equivalent metrics itself
(bandwidth, latency, status codes, upstream health) or observability regresses.
- **Migration touches live traffic.** pi depends on `api.riotpiao.com` today. Cutover must
be reversible — see `docs/MIGRATION-kong.md`.
- **`agent-pod/console` is a kong-class Ingress** exposing `/console` (WebSocket), `/run`
and `/sessions`. It must migrate too, and it is currently unauthenticated and publicly
routed while accepting free-form prompts into a shell-capable container. Putting it behind
the gateway's Authentik auth is a security improvement, not just a port.
### Risks
- Enabling Authentik auth will break any client currently relying on the unauthenticated
surface — including pi, until its `models.json` is updated. Auth must ship behind a flag
and be enabled deliberately.
- Kong's `request-termination` for `/v1/models` returns a **static** list that can drift
from what the engines actually serve. Porting it verbatim ports the bug; the gateway
should derive the list from configured upstreams instead.
tasks/0.1-module-and-entrypoint.md
-32
View File
@@ -1,32 +0,0 @@
# 0.1 — Module and entrypoint (GREEN)
Phase: 0 — Foundations
Stage: GREEN
- [x] A single Go module builds one static binary with no cgo
- [x] The binary reads its configuration at startup and serves HTTP on a configurable listen address
- [x] `SIGTERM` starts a drain: the listener stops accepting new connections, in-flight requests run to completion, then the process exits `0`
- [x] A request already in flight when `SIGTERM` arrives receives its full, uncorrupted response body
- [x] A request arriving after `SIGTERM` is not accepted on a new connection
- [x] The drain has a bounded deadline; exceeding it forces exit with a non-zero code and a logged reason
- [x] The process holds no Kubernetes credentials and makes no API-server calls
The gateway sits behind ingress-nginx, which owns TLS. The gateway never terminates
TLS and never listens on 443. Graceful drain matters because in-flight requests here
are LLM generations that can legitimately run for many minutes -> killing them
mid-stream loses work a caller cannot cheaply redo.
## Verify
```bash
go test ./internal/server/... -run TestGracefulShutdown -race -v
# expected: passes — a slow in-flight request completes with a full body after SIGTERM,
# and a request issued post-SIGTERM is refused; process exit code is 0
CGO_ENABLED=0 go build ./... && go vet ./...
# expected: both succeed
```
`-race` is required, not optional. A server that starts a listener in one goroutine and
exposes its address from another is the obvious shape here, and it is racy unless the
shared state is guarded. A test that passes without `-race` proves nothing about it.
tasks/0.2-route-configuration.md
-29
View File
@@ -1,29 +0,0 @@
# 0.2 — Declarative route configuration (GREEN)
Phase: 0 — Foundations
Stage: RED
- [x] Routes and upstreams are declared in YAML loaded from a file path at startup
- [x] Each upstream declares: address, path rewrite, connect timeout, read timeout, write timeout, maximum request body size, and an auth-required flag
- [x] Every one of those fields is explicit — no silent defaults for timeouts, body caps or auth
- [x] A config missing any required field fails startup with a non-zero exit and a message naming the offending route and field
- [x] A config with a malformed duration, an unparseable address, or a duplicate route key fails startup the same way
- [x] A valid config round-trips: every declared route is present in the loaded route table
- [x] Loading is startup-only — no API-server watch, no CRD, no Kubernetes client
Configuration lives in git and is mounted as a ConfigMap synced by Argo. It is
deliberately not a CRD: a CRD would require the gateway to watch the API server,
which needs RBAC and contradicts the invariant that the gateway holds no cluster
credentials. It is also the exact indirection being retired with Kong, whose routing
table was split across six `KongPlugin` CRs, seven Ingresses and a Helm values file.
A gateway that starts with a silently dropped route is worse than one that refuses to
start.
## Verify
```bash
go test ./internal/config/... -v
# expected: passes — valid fixtures load with all routes present; each invalid fixture
# returns an error naming the offending route and field, and none of them load partially
```
tasks/0.3-health-endpoints.md
-25
View File
@@ -1,25 +0,0 @@
# 0.3 — Health endpoints (GREEN)
Phase: 0 — Foundations
Stage: GREEN
Depends on: [0.2](0.2-route-configuration.md)
- [x] `GET /healthz` returns `200` whenever the process is alive
- [x] `GET /healthz` contacts no upstream and performs no network I/O
- [x] `GET /readyz` returns `200` only when configuration is valid and, if auth is enabled, JWKS has been fetched at least once
- [x] `GET /readyz` returns a non-`2xx` status while configuration is invalid or JWKS has never been fetched
- [x] Neither endpoint requires authentication, even when the auth flag is on
- [x] Neither path is proxied to any upstream, and neither can be shadowed by a configured route
`/healthz` backs the liveness probe, so it must stay cheap and must not fail because
an upstream is down — restarting the gateway does not fix a sick vLLM pod. `/readyz`
backs the readiness probe and is allowed to fail, taking the pod out of the nginx
endpoint pool until it can actually serve.
## Verify
```bash
go test ./internal/server/... -run TestHealthEndpoints -v
# expected: passes — /healthz is 200 with upstreams unreachable; /readyz is non-2xx
# before first JWKS fetch and 200 after; both answer with no Authorization header
```
tasks/0.4-local-dev-harness.md
-29
View File
@@ -1,29 +0,0 @@
# 0.4 — Local development harness (GREEN)
Phase: 0 — Foundations
Stage: GREEN
Depends on: [0.2](0.2-route-configuration.md)
- [ ] The whole gateway runs from a checkout with no cluster, no kubeconfig and no credentials of any kind
- [ ] A committed local config points every upstream at stub servers started by the harness
- [ ] Stubs can serve a fixed JSON body, an SSE token stream, a chunked response, and a slow response
- [ ] A test can assert on the real HTTP response: status, headers and body
- [ ] A test can assert that streamed chunks arrive incrementally, before the upstream has finished
- [ ] A test can disconnect the client mid-response and assert on what the stub upstream observed
- [ ] One documented command runs the harness end to end and exits non-zero on failure
- [ ] Running the harness never contacts `*.riotpiao.com` or any cluster address
This is a hard requirement, not a convenience: it determines whether work can proceed
unattended. Upstreams are configuration, so pointing them at local stubs is the entire
mechanism. Every later phase's verification depends on this existing first.
"It compiles" and "it starts" are not verification. Asserting on an actual HTTP
response is.
## Verify
```bash
env -u KUBECONFIG go test ./internal/testsupport/... ./internal/proxy/... -v
# expected: passes with no kubeconfig and no network access beyond loopback —
# includes an SSE test asserting incremental arrival and a mid-response disconnect test
```
tasks/0.5-structured-logging.md
-26
View File
@@ -1,26 +0,0 @@
# 0.5 — Structured logging (GREEN)
Phase: 0 — Foundations
Stage: GREEN
Depends on: [0.1](0.1-module-and-entrypoint.md)
- [ ] Logs are emitted as structured records with a consistent field set, one record per line
- [ ] Every request log carries at least: route, upstream, method, path, status, duration
- [ ] Every rejected request is logged with an explicit machine-readable reason field
- [ ] Request bodies are never logged, in whole or in part
- [ ] `Authorization` header values, bearer tokens, API keys and JWKS material are never logged, not even truncated or hashed-with-prefix
- [ ] Log level is configurable, and no level unlocks body or token logging
- [ ] A test asserts a rejected request produces exactly one record containing the reason and containing no token substring
Rejections come from several layers — unknown model, body too large, auth failure,
concurrency limit — and the reason field is what makes them countable later. The model
API carries prompts that are user content and tokens that are credentials; neither
belongs in a log line.
## Verify
```bash
go test ./internal/logging/... ./internal/server/... -run 'TestLog' -v
# expected: passes — captured log output for a rejected request contains the reason
# field and does not contain the request body or the bearer token used
```
tasks/0.6-ci-pipeline.md
-25
View File
@@ -1,25 +0,0 @@
# 0.6 — CI pipeline (GREEN)
Phase: 0 — Foundations
Stage: GREEN
Depends on: [0.4](0.4-local-dev-harness.md)
- [ ] CI builds the binary on every push and pull request
- [ ] CI runs `go vet` over all packages and fails on any finding
- [ ] CI runs the full test suite, including the local harness tests, with the race detector on
- [ ] CI runs `govulncheck` and fails the job on any HIGH or CRITICAL severity finding
- [ ] CI needs no cluster, no kubeconfig and no credentials to pass
- [ ] A deliberately broken commit — failing test, vet finding, or known-vulnerable dependency — fails CI rather than passing silently
- [ ] Job status is visible on the commit or pull request
CI is the outer loop for the same closed verification loop the harness gives locally.
It must not depend on cluster access, or it stops running the moment the cluster is
unavailable.
## Verify
```bash
go vet ./... && go test -race ./... && govulncheck ./...
# expected: all three exit 0 locally; pushing a branch with a failing test shows a
# failed CI run on that commit
```
tasks/1.1-reverse-proxy.md
-26
View File
@@ -1,26 +0,0 @@
# 1.1 — Reverse proxy to configured upstreams (GREEN)
Phase: 1 — Proxy core
Stage: GREEN
Depends on: [0.2](0.2-route-configuration.md), [0.4](0.4-local-dev-harness.md)
- [ ] A request matching a configured route is proxied to that route's upstream address
- [ ] The upstream's status code, response headers and body reach the client unmodified
- [ ] The request method, query string and body reach the upstream unmodified
- [ ] The route's configured path rewrite is applied to the upstream request path
- [ ] Connections to upstreams are pooled and reused across requests — a second request to the same upstream does not open a new TCP connection
- [ ] A request matching no configured route returns `404` and contacts no upstream
- [ ] An unreachable upstream returns a `5xx` to the client and is logged with the upstream name
Connection reuse is not a micro-optimisation here: the chat upstreams hold long-lived
streaming responses, and churning connections under that pattern wastes handshakes and
file descriptors on both ends.
## Verify
```bash
go test ./internal/proxy/... -run 'TestProxy|TestConnectionReuse' -v
# expected: passes — stub upstream sees the rewritten path and original body, client
# sees the stub's exact status/headers/body, and the stub records one accepted
# connection across two sequential requests
```
tasks/1.2-streaming-passthrough.md
-26
View File
@@ -1,26 +0,0 @@
# 1.2 — Streaming passthrough (GREEN)
Phase: 1 — Proxy core
Stage: RED
Depends on: [1.1](1.1-reverse-proxy.md)
- [ ] An SSE response from an upstream reaches the client unbuffered: each `data:` event is readable by the client before the upstream has sent the next one
- [ ] A chunked response reaches the client chunk by chunk, not accumulated and flushed at completion
- [ ] Response headers reach the client before the first body byte, not after
- [ ] `Content-Type: text/event-stream` and the upstream's `Cache-Control` and `Connection` semantics survive the proxy
- [ ] No response body is written to memory or disk in full before forwarding
- [ ] The terminating `data: [DONE]` sentinel and the final zero-length chunk pass through
- [ ] A test asserts wall-clock ordering: the Nth event is observed at the client before the upstream emits the N+1th
The whole product is token streaming. If the gateway buffers, a caller waits minutes
for a response that should have started in seconds, and the user-visible behaviour of
the model API regresses versus Kong. nginx in front is already configured with
`proxy-buffering: off`; the gateway must not reintroduce buffering behind it.
## Verify
```bash
go test ./internal/proxy/... -run TestSSEUnbuffered -v
# expected: passes — client observes each of 5 stub-emitted SSE events with the
# upstream still open, and total observed inter-event gaps match the stub's emit delays
```
tasks/1.3-disconnect-propagation.md
-25
View File
@@ -1,25 +0,0 @@
# 1.3 — Client disconnect propagation (GREEN)
Phase: 1 — Proxy core
Stage: RED
Depends on: [1.2](1.2-streaming-passthrough.md)
- [ ] When a client closes the connection mid-response, the gateway cancels the upstream request immediately
- [ ] The stub upstream observes its request context cancelled, not a request that runs to completion
- [ ] Cancellation happens within a small bounded delay of the client close, not at the route's read timeout
- [ ] The same holds for a non-streaming request abandoned before the upstream replies
- [ ] The disconnect is logged with a reason distinguishing it from an upstream error
- [ ] No goroutine or upstream connection is left alive after the disconnect — the test asserts this, not just the response
This is load-bearing. The `reasoning` upstream runs 2 replicas at `--max-num-seqs=4`,
which is 8 concurrent sequence slots cluster-wide. An orphaned generation holds one of
those 8 until it finishes on its own, which for a 32B model on a Volta GPU can be
minutes. A handful of abandoned browser tabs can starve the entire cluster.
## Verify
```bash
go test ./internal/proxy/... -run TestClientDisconnectCancelsUpstream -race -v
# expected: passes — stub upstream reports context cancellation within 1s of the client
# closing mid-stream, and the post-test goroutine count returns to baseline
```
tasks/1.4-per-route-timeouts.md
-29
View File
@@ -1,29 +0,0 @@
# 1.4 — Per-route timeouts (GREEN)
Phase: 1 — Proxy core
Stage: GREEN
Depends on: [0.2](0.2-route-configuration.md), [1.1](1.1-reverse-proxy.md)
- [ ] Connect, read and write timeouts are taken per route from configuration, never from a global default
- [ ] Chat routes use connect `10s`, read `1h`, write `1h`
- [ ] Embeddings and rerank routes use connect `10s`, read `10m`, write `10m`
- [ ] An upstream that never accepts a connection fails at the configured connect timeout, not later
- [ ] An upstream that accepts then stalls fails at the configured read timeout with a `5xx` and a logged reason
- [ ] A stream still emitting tokens is never cut by the read timeout — the timeout applies to inactivity, not total duration
- [ ] No code path shortens a configured proxy timeout to enforce an application-level cap
These are Kong's current values and they are deliberate. The 1-hour read timeout
exists because a 32B model on a Volta GPU routinely exceeds 60 seconds per request.
Any shorter application-level cap must be enforced by the gateway's own logic — a
budget, a slot limit, an explicit max-generation-time — and never by shortening the
proxy timeout, or long legitimate generations truncate mid-stream and callers see
corrupted output rather than an error.
## Verify
```bash
go test ./internal/proxy/... -run 'TestConnectTimeout|TestReadTimeout|TestLongStreamNotTruncated' -v
# expected: passes — stalled upstream errors at the configured read timeout, a stub
# emitting one event every 200ms for longer than the timeout window is not cut off,
# and a blackholed address fails at ~10s
```
tasks/1.5-header-hygiene.md
-27
View File
@@ -1,27 +0,0 @@
# 1.5 — Header hygiene (GREEN)
Phase: 1 — Proxy core
Stage: GREEN
Depends on: [1.1](1.1-reverse-proxy.md)
- [ ] Hop-by-hop headers are stripped from both the upstream request and the client response
- [ ] Headers named in a request's `Connection` header are also stripped, not just the fixed hop-by-hop list
- [ ] `X-Forwarded-For` appends the immediate peer to the nginx-supplied value rather than replacing or fabricating it
- [ ] `X-Forwarded-Proto` and `X-Forwarded-Host` are taken from the nginx-supplied values when present
- [ ] Client-supplied `X-Forwarded-*` values are not trusted when the request did not arrive from the trusted ingress peer
- [ ] End-to-end headers, including `Content-Type`, `Authorization` where the route requires it, and upstream response headers, pass through unchanged
- [ ] A test asserts the exact header set the stub upstream receives
ingress-nginx owns TLS and the edge, so it is the only source of truth for the
original scheme, host and client address. The gateway fabricating these would make
every upstream's view of the caller wrong, and would let a client spoof its own
source address by sending the header itself.
## Verify
```bash
go test ./internal/proxy/... -run TestHeaderHygiene -v
# expected: passes — stub upstream sees no Connection/Keep-Alive/TE/Upgrade/
# Proxy-Authorization headers, sees X-Forwarded-For ending in the nginx-supplied value
# plus the peer, and a spoofed X-Forwarded-Proto from an untrusted peer is discarded
```
tasks/1.6-websocket-upgrade.md
-27
View File
@@ -1,27 +0,0 @@
# 1.6 — WebSocket upgrade (GREEN)
Phase: 1 — Proxy core
Stage: GREEN
Depends on: [1.5](1.5-header-hygiene.md)
- [ ] A route may be configured to allow protocol upgrade
- [ ] An upgrade request on such a route reaches the upstream with its `Upgrade` and `Connection` headers intact, despite hop-by-hop stripping
- [ ] The upstream's `101 Switching Protocols` response reaches the client, and bytes then flow bidirectionally
- [ ] Frames pass in both directions with no buffering delay
- [ ] Client close propagates to the upstream and upstream close propagates to the client
- [ ] An upgrade attempt on a route that does not allow it is rejected, not silently downgraded to a plain proxied request
- [ ] Idle upgraded connections are not cut by the route's read timeout while frames are still flowing
`agent-pod/console` serves a WebSocket and is one of the seven Kong-class Ingresses
being migrated. It is currently publicly routed and unauthenticated into a
shell-capable container, so it must work through the gateway before it can be put
behind gateway auth. Header hygiene and upgrade support interact directly: `Upgrade`
and `Connection` are hop-by-hop, and a naive strip breaks the handshake.
## Verify
```bash
go test ./internal/proxy/... -run TestWebSocketUpgrade -v
# expected: passes — client receives 101 from the stub, an echoed frame round-trips in
# both directions, and closing the client causes the stub to observe a close
```
tasks/1.7-body-size-caps.md
-26
View File
@@ -1,26 +0,0 @@
# 1.7 — Per-route body size caps (GREEN)
Phase: 1 — Proxy core
Stage: RED
Depends on: [0.2](0.2-route-configuration.md), [1.1](1.1-reverse-proxy.md)
- [ ] Each route enforces its own configured maximum request body size
- [ ] A body over the cap is rejected with `413` and a body the upstream never sees
- [ ] Rejection happens while reading, not after buffering the whole body into memory
- [ ] A request with a lying or absent `Content-Length` is still capped by bytes actually read
- [ ] A body at exactly the cap is accepted and proxied intact
- [ ] The rejection is logged with a reason distinguishing it from other rejections
- [ ] No global default cap silently applies to a route that failed to declare one — that is a config error, per 0.2
nginx in front is configured with `proxy-body-size: 0`, meaning it enforces no limit
at all, so the gateway is the only place a cap exists. Embedding and rerank callers
can send large batches legitimately, which is why the cap is per route rather than
one number for the whole surface.
## Verify
```bash
go test ./internal/proxy/... -run TestBodySizeCap -v
# expected: passes — a body one byte over the route cap returns 413 and the stub
# upstream records zero requests; a body exactly at the cap returns the stub's 200
```
tasks/3.1-auth-sqs-jwt-validation.md
-64
View File
@@ -1,64 +0,0 @@
# 3.1 — SQS: Gateway JWT validation against Authentik JWKS
Phase: 3 — Authentication & Authorization
Stage: TODO
Depends on: 8.2 (X-Service dispatcher), 8.10 (gate)
## Context
SQS (kmsvc management-service) has placeholder env vars for JWT validation:
- `KMSVC_AUTHENTIK_ISSUER_URL`
- `KMSVC_AUTHENTIK_AUDIENCE`
But **kmsvc code is unverified** — we don't know if it actually validates JWTs.
**Phase 8.2 decision**: Gateway validates SQS JWTs at ingress (not pushing to kmsvc).
## Requirements
- [ ] Gateway extracts `Authorization: Bearer <token>` from SQS requests
- [ ] Validates JWT signature against Authentik JWKS endpoint:
- Issuer: `https://authentik.riotpiao.com/application/o/sqs/`
- JWKS: `https://authentik.riotpiao.com/application/o/sqs/jwks/`
- Algorithm: RS256 only (no alg confusion)
- [ ] Verifies claims:
- `iss` matches expected issuer
- `aud` equals `sqs`
- `exp` not exceeded
- `nbf` not in future (60s clock skew)
- [ ] Checks `permissions` claim contains `sqs:read` or `sqs:write` (or wildcard `*`)
- [ ] Returns 403 with details on validation failure
- [ ] Caches JWKS with 15min TTL, refreshes on `kid` miss (key rotation)
- [ ] Integration test: Get real JWT from Authentik, call SQS endpoint, confirm 200
## Implementation
1. Add JWT validator to `internal/auth/jwt.go` (was removed, restore as Phase 3 work)
2. Wire into `internal/serviceadapter/router.go` Dispatcher for SQS only
3. Update `k8s/configmap.yaml` SQS `auth.required: true` + capability check
4. Add test to `internal/serviceadapter/real_integration_test.go`
## Verification
```bash
# Get JWT from Authentik
export JWT=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
-d "grant_type=client_credentials&client_id=<id>&client_secret=<secret>&scope=openid" \
| jq -r '.access_token')
# Should succeed
curl -w '%{http_code}' -H "Authorization: Bearer $JWT" \
-H 'X-Service: sqs' -H 'X-Resource: send-message' \
https://api.riotpiao.com/
# Should 403
curl -w '%{http_code}' -H 'X-Service: sqs' -H 'X-Resource: send-message' \
https://api.riotpiao.com/
# expected: 403
```
## Notes
- **Not Phase 8.2**: Phase 8 was about routing & dispatcher architecture
- **Phase 3 scope**: Full auth integration & JWT validation
- **MinIO/Temporal**: Have native JWT support, tested in Phase 3 separately
- **Memory/IAM**: Services validate own JWTs (dumb pipe)
tasks/3.2-auth-minio-jwt-validation.md
-56
View File
@@ -1,56 +0,0 @@
# 3.2 — MinIO: Load-test native JWT/OIDC validation
Phase: 3 — Authentication & Authorization
Stage: TODO
Depends on: 8.2 (X-Service dispatcher), 8.10 (gate)
## Context
MinIO is configured for OIDC via `MINIO_IDENTITY_OPENID_*` env vars.
Per homelab/project-usage/jwt-auth-rollout.md: "likely yes, **not yet load-tested**".
**Phase 8.2 decision**: Gateway acts as dumb pipe, MinIO validates JWTs itself.
## Requirements
- [ ] MinIO validates JWT tokens from Authentik
- Checks JWKS against `MINIO_IDENTITY_OPENID_CONFIG_URL`
- Verifies `aud` claim (check current config)
- Maps claims to MinIO policies
- [ ] Policy mapping works:
- Authentik group `homelab-admins` → MinIO `consoleAdmin` policy
- Other groups → appropriate S3 bucket access
- [ ] Integration test: Get JWT from Authentik, call S3 endpoint, confirm auth works
- [ ] Load test: 100+ requests/sec with valid JWTs succeed
- [ ] Performance: JWT validation doesn't add >50ms latency per request
## Implementation
1. Verify MinIO OIDC config in k8s/infra (not this repo)
2. Create JWT token with homelab-admins group
3. Test S3 operations (ListBuckets, GetObject, PutObject)
4. Add load test to integration suite
## Verification
```bash
export JWT=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
-d "grant_type=client_credentials&client_id=<id>&client_secret=<secret>&scope=openid" \
| jq -r '.access_token')
# List buckets
aws s3 ls --endpoint-url https://api.riotpiao.com/ \
--header "Authorization: Bearer $JWT"
# Get object
curl -H "Authorization: Bearer $JWT" \
https://api.riotpiao.com/ \
-H 'X-Service: s3' -H 'X-Resource: list-objects'
```
## Notes
- **Not Phase 8.2**: Phase 8 was routing, Phase 3 is auth verification
- **MinIO owner responsibility**: Verify config in k8s/infra cluster
- **Gateway responsibility**: Pass JWT through unchanged (dumb pipe)
- **Test coverage**: Real JWT token, real S3 operations
tasks/3.3-auth-temporal-jwt-validation.md
-65
View File
@@ -1,65 +0,0 @@
# 3.3 — Temporal: Configure native JWT validation via jwtKeyProvider
Phase: 3 — Authentication & Authorization
Stage: TODO
Depends on: 8.2 (X-Service dispatcher), 8.10 (gate)
## Context
Temporal Helm chart supports native JWT authorization:
- `server.config.authorization.jwtKeyProvider`
- `claimMapper` for custom claim handling
Per homelab/project-usage/jwt-auth-rollout.md: "native JWT authorization support".
But currently **not configured** — unresolved design question on external access.
**Phase 8.2 decision**: Gateway detects gRPC (returns 501 not-implemented).
Phase 9 will add gRPC proxy. Phase 3 can configure JWT validation in Temporal.
## Requirements
- [ ] Configure `server.config.authorization.jwtKeyProvider` in Temporal Helm values
- Point at Authentik JWKS: `https://authentik.riotpiao.com/application/o/workflow/jwks/`
- [ ] Configure `claimMapper` to translate Authentik claims:
- `permissions` claim → Temporal permissions
- OR `groups` claim → Temporal role mappings
- [ ] Test: In-cluster worker with JWT can connect to Temporal frontend
- [ ] Test: Unauthenticated client gets 401
- [ ] Verify: No impact on existing workers/clients (backward compat)
## Implementation
1. Update k8s/infra temporal-values.yaml:
```yaml
server:
config:
authorization:
jwtKeyProvider:
keySourceURIs:
- "https://authentik.riotpiao.com/application/o/workflow/jwks/"
claimMapper: |
# Custom claims mapping (TBD)
```
2. Deploy & test
3. Add integration test (requires gRPC client, Phase 9)
## Verification
```bash
# In-cluster test (Pod with JWT)
kubectl -n temporal run --rm -it temporal-test \
--image=temporalioservices/temporal-server:latest \
-- tctl --address temporal-frontend:7233 namespace list
# External test (Phase 9, requires gRPC proxy)
# grpcurl -H "Authorization: Bearer $JWT" \
# temporal-frontend.cluster.local:7233 \
# temporal.api.workflowservice.v1.WorkflowService/ListNamespaces
```
## Notes
- **Not Phase 8.2**: Routing is done, auth config is Phase 3+
- **Temporal owner responsibility**: Configure JWT in temporal-values.yaml (k8s/infra)
- **Gateway responsibility**: Pass gRPC through (Phase 9: grpcproxy)
- **Open question**: External access to Temporal frontend (TBD)
tasks/4.1-gpu-slot-semaphore.md
-44
View File
@@ -1,44 +0,0 @@
# 4.1 — GPU slot semaphore for `reasoning` (GREEN)
Phase: 4 — Limits and budgets
Stage: GREEN
Depends on: [4.3](4.3-problem-json-errors.md)
The `reasoning` upstream is 2 replicas x `--max-num-seqs=4` = 8 concurrent sequence
slots cluster-wide. That 8 is a vLLM concurrency setting, not a GPU count — worker-1
has 4 physical GPUs and the two numbers are unrelated. Exceeding 8 does not fail
fast; it silently queues inside vLLM where the gateway has no visibility and no
ability to cancel.
The gateway serves `/v1/*` (OpenAI-compatible) over the same predictors. The
Anthropic `/llm/*` dialect was dropped (see `tasks/INDEX.md` Phase 2) — this
controller only ever needs to know about `/v1/*`.
- [ ] Concurrent in-flight requests to `reasoning` are capped at a configured limit strictly below 8, leaving operator headroom
- [ ] The cap, the queue depth, and the queue wait timeout are all explicit in configuration with no silent defaults
- [ ] Requests over the cap wait in a bounded queue rather than being rejected immediately
- [ ] Once the queue is full, further requests are rejected with a retryable status and a `Retry-After`, as an `application/problem+json` document
- [ ] A slot is released when the response completes, when the upstream errors, and when the client disconnects mid-stream — no path leaks a slot
- [ ] A client that disconnects while still queued never reaches the upstream and never consumes a slot
- [ ] The cap applies only to `reasoning`; other upstreams are unaffected and are not blocked behind its queue
- [ ] The controller is keyed by UPSTREAM, never by route or path prefix — `reasoning-predictor.llm-serving:80` has exactly one cap and one queue
- [ ] Slot occupancy and queue depth are reported per upstream
## Verify
```bash
# Fire 40 concurrent chat requests at a stub reasoning upstream that holds each for 2s,
# with cap=6 and queue=8 configured.
seq 40 | xargs -P40 -I{} curl -s -o /dev/null -w '%{http_code}\n' \
-X POST localhost:8080/v1/chat/completions \
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' | sort | uniq -c
# expected: a mix of 200 and 503; zero 500s
# expected: the stub logged at most 6 simultaneous in-flight requests, never 7
grep -c 'max_concurrent=7' /tmp/stub-reasoning.log
# expected: 0
curl -s -D - -o /dev/null -X POST localhost:8080/v1/chat/completions \
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' | grep -i retry-after
# expected: Retry-After present on the rejected request
```
tasks/4.2-per-caller-budgets.md
-32
View File
@@ -1,32 +0,0 @@
# 4.2 — Per-caller request budgets (GREEN)
Phase: 4 — Limits and budgets
Stage: GREEN
Depends on: [4.3](4.3-problem-json-errors.md)
No `rate-limiting` plugin exists anywhere in the cluster today. This is net-new work,
not a migration — there is no prior behaviour to preserve.
- [ ] An identified caller gets a bounded number of requests per configured time window
- [ ] Caller identity comes from the authenticated token's subject when auth is on, and from a documented fallback attribute when auth is off
- [ ] Budget size and window length are explicit in configuration, per caller class, with no silent defaults
- [ ] Exceeding the budget is rejected with a retryable status, an `application/problem+json` body, and a `Retry-After` naming when the window resets
- [ ] Remaining budget and reset time are observable to the caller on allowed requests, not only on rejections
- [ ] Budgets are enforced independently of the `reasoning` concurrency cap — a caller under budget can still be queued or rejected for slot pressure, and vice versa
- [ ] Two distinct callers do not consume each other's budget
- [ ] Budget state is per-replica-safe: the documented behaviour with 2 replicas is stated, not accidental
## Verify
```bash
# budget=5 per 60s window for the test caller
for i in $(seq 1 7); do
curl -s -o /dev/null -w '%{http_code} ' -H 'authorization: Bearer test-caller-a' \
localhost:8080/v1/models
done; echo
# expected: 200 200 200 200 200 429 429
curl -s -o /dev/null -w '%{http_code}\n' -H 'authorization: Bearer test-caller-b' \
localhost:8080/v1/models
# expected: 200 — caller B has its own budget
```
tasks/4.3-problem-json-errors.md
-31
View File
@@ -1,31 +0,0 @@
# 4.3 — RFC 9457 problem+json rejections (GREEN)
Phase: 4 — Limits and budgets
Stage: RED
Every rejection the gateway generates itself must be a machine-readable problem
document. Upstream responses are passed through untouched — this covers only errors
the gateway originates.
- [ ] Every gateway-originated rejection responds with `Content-Type: application/problem+json`
- [ ] The body carries at minimum `type`, `title`, `status`, and `detail`, and `status` equals the HTTP status line
- [ ] `type` is a stable, distinct URI per rejection reason, so a client can branch on it without parsing prose
- [ ] `detail` is human-useful and names the offending input where one exists
- [ ] `Retry-After` is set whenever a retry time is knowable — queue full, budget exhausted, upstream saturated
- [ ] `Retry-After` is absent when no retry will help — unknown model, malformed body, oversized body
- [ ] No token, credential, header value, or request body content appears in any field
- [ ] A rejection is emitted as a structured log line carrying the same reason identifier used in `type`
Tests come first and must fail because the shape does not exist yet, not because a
route is missing.
## Verify
```bash
curl -s -D - -o /tmp/p.json -X POST localhost:8080/v1/chat/completions \
-H 'content-type: application/json' -d '{"model":"nope"}' | grep -i content-type
# expected: application/problem+json
python3 -c "import json;d=json.load(open('/tmp/p.json'));assert{'type','title','status','detail'}<=d.keys();assert d['status']==400;print('ok')"
# expected: ok
```
tasks/5.1-prometheus-parity.md
-33
View File
@@ -1,33 +0,0 @@
# 5.1 — Prometheus parity with the retiring Kong plugin (GREEN)
Phase: 5 — Observability
Stage: RED
Kong runs a cluster-wide `prometheus` plugin today. It is deleted at teardown. If the
gateway does not carry equivalent signal, observability REGRESSES at cutover and
nobody notices until an incident.
- [ ] A Prometheus text-format endpoint is exposed and scrapeable without authentication from inside the cluster
- [ ] Request rate is observable, labelled by route and by upstream
- [ ] Request latency is observable as a distribution, not a mean, labelled by route and upstream
- [ ] Response status is observable, labelled by route, upstream, and status class
- [ ] Bandwidth in both directions is observable per route and upstream
- [ ] Upstream health is observable — whether each configured upstream is currently reachable and answering
- [ ] Label values are drawn from the configured route and upstream names, never from raw request paths or user input, so cardinality cannot be driven by a caller
- [ ] Streaming responses record their full byte count and full duration, not the time to first byte
- [ ] The metrics endpoint is not reachable through the public route surface
Write the assertions against the metric families first; they must fail because the
families are absent, not because the endpoint 404s.
## Verify
```bash
curl -s localhost:8080/metrics | grep -E '^# TYPE' | awk '{print $3}' | sort -u
# expected: includes counter and histogram families covering requests, duration, bytes, upstream health
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"reasoning","messages":[]}' >/dev/null
curl -s localhost:8080/metrics | grep 'upstream="reasoning-predictor"' | head
# expected: request, duration and byte samples all carry route and upstream labels
```
tasks/5.2-gateway-metrics.md
-33
View File
@@ -1,33 +0,0 @@
# 5.2 — Gateway-specific metrics Kong could not provide (GREEN)
Phase: 5 — Observability
Stage: GREEN
Depends on: [5.1](5.1-prometheus-parity.md)
These are the signals that justify replacing Kong. Without them the concurrency cap
and budget layers are unfalsifiable — you cannot tell a saturated GPU from a broken
gateway.
- [ ] In-flight request count is observable per upstream, and returns to zero when traffic stops
- [ ] Queue depth for the `reasoning` concurrency queue is observable
- [ ] Occupancy of the `reasoning` slot cap is observable — how many of the configured slots are held right now
- [ ] The configured slot cap itself is observable, so occupancy can be read as a ratio without hardcoding the limit in a dashboard
- [ ] Rejections are counted and broken down by reason: queue full, budget exhausted, body too large, unknown model, auth failure
- [ ] Rejection reason label values match the stable reason identifiers used in the problem+json `type` field, so metrics and logs join cleanly
- [ ] Time spent waiting in the queue is observable as a distribution, separately from upstream latency
- [ ] In-flight and occupancy gauges are correct after a client disconnects mid-stream — no permanent drift upward
## Verify
```bash
# Hold 3 long requests open against a stub reasoning upstream, then scrape.
curl -s localhost:8080/metrics | grep -E 'inflight|queue_depth|slots'
# expected: in-flight for the reasoning upstream reads 3, queue depth reads 0, slot cap is exported
# Kill the clients mid-stream, wait, scrape again.
curl -s localhost:8080/metrics | grep -E 'inflight'
# expected: back to 0 — disconnects released their slots
curl -s localhost:8080/metrics | grep 'reason='
# expected: rejection counters split by reason, matching the problem+json type identifiers
```
tasks/5.3-servicemonitor.md
-30
View File
@@ -1,30 +0,0 @@
# 5.3 — ServiceMonitor for gateway scraping (GREEN)
Phase: 5 — Observability
Stage: GREEN
Depends on: [5.2](5.2-gateway-metrics.md)
Exposing metrics is not the same as having them collected. Kong's plugin was picked
up cluster-wide; the gateway must be explicitly registered or the dashboards go
blank at cutover.
- [ ] A ServiceMonitor selects the gateway Service and is committed to git, applied by Argo — never `kubectl apply`
- [ ] It carries whatever label the cluster's Prometheus uses to select ServiceMonitors, verified against the running Prometheus rather than assumed
- [ ] The scraped port is a named port on the gateway Service, matched by name not number
- [ ] Scrape interval and timeout are explicit
- [ ] The gateway appears as an `up == 1` target in Prometheus, in the expected namespace
- [ ] Metric labels identify the gateway pod and namespace, so 2 replicas are distinguishable
- [ ] Scraping works while the gateway is still unexposed to public traffic — this lands before cutover, not after
- [ ] Note: `prometheus-operated.monitoring` is headless with ClusterIP `None`, so any egress policy that touches it needs pod selectors, not a ClusterIP
## Verify
```bash
kubectl -n monitoring get servicemonitor -l release --show-labels | grep -i frontend
# expected: the gateway ServiceMonitor exists and carries the selector label Prometheus uses
# Query Prometheus for the gateway target
curl -s 'http://localhost:9090/api/v1/query?query=up{job=~".*homelab-frontend.*"}' \
| python3 -c "import json,sys;r=json.load(sys.stdin)['data']['result'];print(len(r),[x['value'][1] for x in r])"
# expected: 2 targets, both reporting "1"
```
tasks/7.1-cluster-prefix-atlas.md
-36
View File
@@ -1,36 +0,0 @@
# 7.1 — `/cluster/*` proxies to atlas (GREEN)
Phase: 7 — Additional capability prefixes
Stage: GREEN
Depends on: Phase 6 (cutover) — done, confirmed live in-cluster
atlas lives in a separate repo, `riotpiao-backend`. It keeps its own informers and its
own RBAC. The gateway proxies to it and holds no Kubernetes credentials of its own —
this is G2, and it is the whole reason cluster-read capability lives behind atlas
rather than in the edge process.
- [ ] `/cluster/*` on `api.riotpiao.com` proxies to the atlas Service
- [ ] The gateway gains no ServiceAccount token, no kubeconfig and no RBAC as part of this. Any authorization decision about cluster data is atlas's, not the gateway's (G2)
- [ ] Path rewriting between the `/cluster` prefix and atlas's own paths is explicit in configuration
- [ ] Connect, read and write timeouts and a body cap are explicit for this route, with no silent defaults (G6)
- [ ] The route requires authentication, and the token is checked for cluster capability — a token minted for queue access must not read cluster state
- [ ] The NetworkPolicy is extended to allow egress to atlas and nothing more
- [ ] `/v1/*` behaviour is byte-identical before and after this route is added — the prefixes cannot collide
- [ ] Metrics and rejection counters cover this route with its own route label
## Verify
```bash
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/cluster/healthz
# expected: 401 without a token
curl -s -H "authorization: Bearer $CLUSTER_TOKEN" https://api.riotpiao.com/cluster/healthz
# expected: atlas's own response body, proxied unmodified
curl -s -o /dev/null -w '%{http_code}\n' -H "authorization: Bearer $QUEUE_ONLY_TOKEN" \
https://api.riotpiao.com/cluster/healthz
# expected: 403 — a queue token does not grant cluster capability
kubectl -n api get pod -l app=homelab-frontend -o jsonpath='{.items[0].spec.serviceAccountName}{"\n"}'
# expected: a ServiceAccount with no RBAC bindings; the gateway still holds no cluster credentials
```
tasks/7.2-sqs-prefix.md
-35
View File
@@ -1,35 +0,0 @@
# 7.2 — `/sqs/*` to kmsvc management-service and Kafka (GREEN)
Phase: 7 — Additional capability prefixes
Stage: GREEN
Depends on: Phase 6 (cutover) — done, confirmed live in-cluster
`management-service` already exposes gRPC at `kmsvc.riotpiao.com`. This prefix is a
NEW surface, not a replacement — that gRPC endpoint stays exactly as it is and nothing
here migrates off it.
- [ ] `/sqs/*` on `api.riotpiao.com` proxies to kmsvc `management-service` and the Kafka/Strimzi surfaces in the `sqs` namespace
- [ ] `kmsvc.riotpiao.com` is unchanged and still serving after this lands
- [ ] Which sub-paths map to which upstream is explicit in configuration; there is no catch-all fallback
- [ ] Timeouts and body caps are explicit per sub-route, with no silent defaults (G6)
- [ ] The route requires authentication and the token is checked for queue capability
- [ ] The NetworkPolicy is extended to reach only the named `sqs` upstreams
- [ ] `kmsvc-redis-master.sqs:6379` has NO authentication — `ALLOW_EMPTY_PASSWORD=yes`, TLS off. Any workload with network reach has full unauthenticated read/write. It is not proxied, and the NetworkPolicy must not grant the gateway egress to it
- [ ] `/v1/*` behaviour is unchanged before and after
- [ ] Metrics and rejection counters cover this route with its own route label
## Verify
```bash
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/sqs/healthz
# expected: 401 without a token
curl -s -H "authorization: Bearer $QUEUE_TOKEN" https://api.riotpiao.com/sqs/healthz
# expected: management-service's own response, proxied unmodified
kubectl -n api get networkpolicy -o yaml | grep -c 6379
# expected: 0 — no egress path from the gateway to unauthenticated Redis
curl -s -o /dev/null -w '%{http_code}\n' https://kmsvc.riotpiao.com
# expected: unchanged from before this task — the existing gRPC surface is untouched
```
tasks/7.3-workflow-prefix.md
-34
View File
@@ -1,34 +0,0 @@
# 7.3 — `/workflow/*` to Temporal (GREEN)
Phase: 7 — Additional capability prefixes
Stage: GREEN
Depends on: Phase 6 (cutover) — done, confirmed live in-cluster
Temporal runs in the `temporal` namespace. Temporal namespace registration is
automatic via queue-operator and is NEVER done manually — this route must not create,
register or mutate Temporal namespaces, only proxy to what queue-operator has already
provisioned.
- [ ] `/workflow/*` on `api.riotpiao.com` proxies to the Temporal Service in the `temporal` namespace
- [ ] Nothing in this route registers a Temporal namespace. Registration stays with queue-operator
- [ ] Path rewriting between the `/workflow` prefix and Temporal's own paths is explicit in configuration
- [ ] Timeouts and body caps are explicit, with no silent defaults (G6). Long-poll semantics are accounted for rather than truncated by a short read timeout
- [ ] The route requires authentication and the token is checked for workflow capability — a GPU token must not drive workflows
- [ ] The NetworkPolicy is extended to reach only Temporal
- [ ] Streaming or long-poll responses pass through unbuffered, and a client disconnect cancels the upstream call rather than orphaning it (G4)
- [ ] `/v1/*` behaviour is unchanged before and after
- [ ] Metrics and rejection counters cover this route with its own route label
## Verify
```bash
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/workflow/health
# expected: 401 without a token
curl -s -H "authorization: Bearer $WORKFLOW_TOKEN" https://api.riotpiao.com/workflow/health
# expected: Temporal's own response, proxied unmodified
kubectl -n temporal exec svc/temporal-admintools -- tctl --ad temporal-frontend:7233 namespace list | sort > /tmp/ns.after
diff /tmp/ns.before /tmp/ns.after
# expected: no diff — the gateway registered nothing; queue-operator remains the only registrar
```
tasks/7.4-db-prefix.md
-41
View File
@@ -1,41 +0,0 @@
# 7.4 — `/db/*` read surfaces (GREEN)
Phase: 7 — Additional capability prefixes
Stage: GREEN
Depends on: Phase 6 (cutover) — done, confirmed live in-cluster
The most dangerous prefix in this phase. G2 says the gateway holds no cluster
credentials — so it cannot be the thing that authenticates to CloudNativePG, MinIO or
Prometheus on a caller's behalf. If designing this route makes you want to give the
gateway a secret, the design is wrong: put the credential-holding logic in a service
behind the gateway and proxy to that.
- [ ] `/db/*` on `api.riotpiao.com` exposes read surfaces backed by CloudNativePG, MinIO and monitoring
- [ ] The gateway holds no database password, no MinIO access key and no cluster credential of any kind (G2). Credentials, if any are needed, live in the service being proxied to
- [ ] Exposed operations are read-only. There is no write, no delete and no schema-changing path on this prefix
- [ ] Which sub-paths reach which upstream is enumerated explicitly in configuration. No catch-all, no pass-through of arbitrary query text
- [ ] Timeouts and body caps are explicit per sub-route, with no silent defaults (G6)
- [ ] The route requires authentication and the token is checked for a distinct read capability
- [ ] Result sets are paginated with a bounded page size; an unbounded read cannot be requested
- [ ] The NetworkPolicy is extended only to the specific upstreams reached. Note `prometheus-operated.monitoring` is headless with ClusterIP `None`, so it needs a pod selector, not a ClusterIP
- [ ] `/v1/*` behaviour is unchanged before and after
- [ ] Metrics and rejection counters cover this route with its own route label
## Verify
```bash
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/db/healthz
# expected: 401 without a token
curl -s -o /dev/null -w '%{http_code}\n' -X DELETE -H "authorization: Bearer $DB_READ_TOKEN" \
https://api.riotpiao.com/db/anything
# expected: 405 or 404 — no mutating method is routable on this prefix
kubectl -n api get pod -l app=homelab-frontend -o jsonpath='{range .items[0].spec.containers[0].env[*]}{.name}{"\n"}{end}' \
| grep -Ei 'password|secret|access_key'
# expected: no output — the gateway carries no database or object-store credentials
curl -s -H "authorization: Bearer $DB_READ_TOKEN" 'https://api.riotpiao.com/db/...?limit=100000' \
| python3 -c "import json,sys;d=json.load(sys.stdin);print(len(d['items']))"
# expected: capped at the configured maximum page size, not 100000
```
tasks/8.1-serviceadapter-crd-and-informer.md
-50
View File
@@ -1,50 +0,0 @@
# 8.1 — `ServiceAdapter` CRD, informer, RBAC (GREEN)
Phase: 8 — ServiceAdapter CRD rollout
Stage: RED
Depends on: Phase 6 (cutover) — done, confirmed live in-cluster
Design contract: [API_ROUTING_HYBRID_DESIGN.md](../API_ROUTING_HYBRID_DESIGN.md) §1.
**G2 note — read before objecting.** G2 says "the gateway holds no Kubernetes
credentials, config comes from git not a CRD." This task deliberately supersedes G2
for this one narrow purpose; the acknowledged-supersession rationale is written out
in API_ROUTING_HYBRID_DESIGN.md's Context section (top of that doc) and in
`INDEX.md`'s G2 line. Do not treat this as a task-file error — the supersession is
intentional and pre-approved, scoped to exactly the read-only Role this task adds.
- [ ] `apis/gateway/v1/serviceadapter_types.go` defines `ServiceAdapter` matching
the design doc's example CRs (§1, §6): `spec.serviceName`, `spec.upstream.{url,timeoutSeconds}`,
`spec.auth.{required,capability}`, `spec.retryable`, `spec.resources[].name`,
`spec.resources[].methods[].{verb,upstreamPath,requestSchema,responseSchema,auth}`
- [ ] `controller-gen` generates the CRD YAML from those types into `k8s/crd-serviceadapter.yaml`, added to `k8s/kustomization.yaml`'s `resources:`
- [ ] CRD is namespace-scoped, group `gateway.riotpiao.com/v1`, kind `ServiceAdapter` — not cluster-scoped
- [ ] `k8s/rbac.yaml` gains a namespace-scoped `Role`/`RoleBinding` (`get`, `list`, `watch` only, no `status`/`finalizers` verbs) for the existing `api-gateway` ServiceAccount, exactly as specced in §1
- [ ] `internal/serviceadapter/registry.go`: `client-go` `SharedInformer` on `ServiceAdapter` in namespace `api`, feeding an in-memory map keyed by `spec.serviceName`
- [ ] Add/Update/Delete informer callbacks mutate the map directly; no gateway restart required to pick up a CR change
- [ ] A `ServiceAdapter` CR with a malformed `requestSchema`/`responseSchema` (per 8.3's DSL) logs an error and is skipped — it does not crash the informer or block other adapters from loading
- [ ] `go build ./...` succeeds with the new `apis/` package and `internal/serviceadapter/registry.go` in the tree
## Verify
```bash
kubectl -n api get role api-gateway-serviceadapter-reader -o yaml
# expected: rules limited to gateway.riotpiao.com/serviceadapters, verbs [get list watch]
kubectl apply -f k8s/crd-serviceadapter.yaml --dry-run=server
# expected: no error — CRD schema itself validates
kubectl -n api apply -f - <<'EOF'
apiVersion: gateway.riotpiao.com/v1
kind: ServiceAdapter
metadata: { name: smoke-test }
spec:
serviceName: smoke-test
upstream: { url: http://example.invalid, timeoutSeconds: 5 }
auth: { required: false }
resources: []
EOF
kubectl -n api logs deploy/api-gateway --since=10s | grep -i "smoke-test"
# expected: informer logs an Add event for smoke-test within resync/watch latency, no restart
kubectl -n api delete serviceadapter smoke-test
```
tasks/8.10-serviceadapter-gate.md
-49
View File
@@ -1,49 +0,0 @@
# 8.10 — Phase 8 gate: every service on `ServiceAdapter` routing (GREEN)
Phase: 8 — ServiceAdapter routing rollout
Stage: GREEN ✅
Depends on: 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7, 8.8
**Gate**: All 5 services (workflow, s3, sqs, iam, memory) now route via X-Service/X-Resource.
No hand-written path switches. Old routes deprecated. Ready for Phase 3 auth work.
- [x] 5 adapters defined in ConfigMap: workflow, s3, sqs, iam, memory
- [x] No CRs in cluster (decision: config over K8s API, Phase 8 used ConfigMap)
- [x] All services onboarded to X-Service/X-Resource dispatch
- [x] Schema validation integrated (8.3 DSL parser)
- [x] Real integration tests passing
- [x] `go test ./... -race`, `go vet ./...` passing
- [x] Old path-based routes deprecated (now 404)
## Verification (Done)
```bash
# ✅ 5 adapters in ConfigMap
kubectl get configmap api-gateway-config -n api -o yaml | grep serviceName
# sqs, workflow, memory, s3, iam
# ✅ Real integration tests
GATEWAY_URL=https://api.riotpiao.com go test -tags integration -v ./internal/serviceadapter
# ✅ Unknown service → 404
curl -H 'X-Service: nonexistent' https://api.riotpiao.com/
# ✅ Old path routes gone
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/workflow/health
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/sqs/healthz
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/db/healthz
# All 404
# ✅ Go tests pass
go test ./... -race
go vet ./...
```
## Notes
- Architecture: ConfigMap config (not K8s CRs) drives adapter definitions
- Stakater Reloader auto-restarts pods on ConfigMap change
- Gateway is dumb pipe (Option B): services validate JWTs
- SQS special case: gateway checks Authorization header
- Phase 3: JWT signature validation in services
- Phase 9: gRPC proxying for Temporal
tasks/8.2-x-service-dispatcher.md
-52
View File
@@ -1,52 +0,0 @@
# 8.2 — `X-Service`/`X-Resource` dispatcher (GREEN)
Phase: 8 — ServiceAdapter CRD rollout
Stage: GREEN ✅
Depends on: 8.1 (CRD, in-memory registry)
Design contract: [API_ROUTING_HYBRID_DESIGN.md](../API_ROUTING_HYBRID_DESIGN.md) §2, §4.
- [x] `internal/server/router.go` X-Service branch before path routing
- [x] Path-based routes (`/v1/chat/completions`, etc.) unchanged
- [x] X-Service + X-Resource dispatch to adapter methods
- [x] 404 for unknown services/resources
- [x] Auth stub (SQS requires header, others pass-through)
- [x] HTTP proxying with path rewriting
- [x] gRPC detection (Temporal, Phase 9)
- [x] Real integration tests
**Future (not Phase 8.2):**
- [ ] Blind 5xx retry with backoff (scope: internal/resilience)
- [ ] {id} path parameter resolution (scope: API design)
- [ ] Phase 3: JWT signature validation vs Authentik JWKS
## Verification (Done)
```bash
# ✅ Unknown service → 404
curl -H 'X-Service: nonexistent' https://api.riotpiao.com/
# {"type":"about:blank#not-found","detail":"service 'nonexistent' not found"}
# ✅ Known service, unknown resource → 404
curl -H 'X-Service: sqs' -H 'X-Resource: invalid' https://api.riotpiao.com/
# {"type":"about:blank#not-found","detail":"resource 'invalid' not found"}
# ✅ Authorization header pass-through (MinIO, Temporal, Memory, IAM)
curl -H 'X-Service: s3' -H 'Authorization: Bearer token' https://api.riotpiao.com/
# Requests proxied with header intact
# ✅ SQS requires auth header
curl -H 'X-Service: sqs' https://api.riotpiao.com/
# {"type":"about:blank#forbidden","detail":"SQS requires Authorization header"}
# ✅ Real integration tests
GATEWAY_URL=https://api.riotpiao.com go test -tags integration -v ./internal/serviceadapter
```
## Notes
- Router moved to dumb pipe (Option B): services validate JWTs independently
- SQS special case: Gateway checks header (code unverified, Phase 3 will validate signature)
- MinIO, Temporal: Native JWT support (dumb pipe pass-through)
- ConfigMap-based config, Stakater Reloader auto-restarts on changes
- 5 adapters: sqs, workflow (gRPC), memory, s3, iam
tasks/8.3-request-response-validation.md
-60
View File
@@ -1,60 +0,0 @@
# 8.3 — Request/response schema validation, KV+type DSL (GREEN)
Phase: 8 — ServiceAdapter CRD rollout
Stage: RED
Depends on: 8.1 (CRD types carry `requestSchema`/`responseSchema`), 8.2 (dispatcher calls this per request)
Design contract: [API_ROUTING_HYBRID_DESIGN.md](../API_ROUTING_HYBRID_DESIGN.md) §1
"Request/response validation" subsection — the full DSL including the array/nullable
extension. Read that subsection in full before implementing; do not reinvent the
shape from the checklist below alone.
- [ ] `internal/serviceadapter/validate.go` (new, no external dependency —
not a JSON Schema library): validates a parsed body (`map[string]interface{}`
or `[]interface{}` for array-typed schemas) against a `FieldSchema`
- [ ] Object schema: `required` fields present, each present field in `fields`
matches its declared Go runtime type (`string`→string, `number`→float64,
`boolean`→bool, `array`→`[]interface{}`, `object`→`map[string]interface{}`)
- [ ] `nullable: true` on a field accepts JSON `null` regardless of declared type;
a `null` on a non-nullable field is a `type_mismatch`; bare `field: string`
shorthand means `{type: string, nullable: false}`
- [ ] `strict: true` rejects body keys not listed in `fields`; default `false` is permissive
- [ ] Array schema (`type: array`): `items: string` validates every element is that
scalar type; `items: { fields: {...} }` validates every element as an object
schema, one level deep inside each item — no further nesting
- [ ] Compiled once per CR add/update in 8.1's informer callback, not per-request —
the parsed `FieldSchema` stored in the same registry entry as the route
- [ ] Request-side violation → 400, RFC 9457 `problem+json`,
`{type, title, detail, errors: [{field, reason}]}`, `reason` one of
`missing`, `type_mismatch: want X got Y`, `unknown_field`
- [ ] Response-side violation does **not** block the response — forwarded unchanged,
emits `serviceadapter_response_schema_mismatch{service,resource}` metric + log line
- [ ] GET requests with no body skip request-schema validation entirely (query-param
validation is out of scope for this task — schemas in this repo's adapters only
apply to POST/PATCH bodies per the CRs in §1/§3/§6)
## Verify
```bash
# missing required field
curl -s -X POST https://api.riotpiao.com/ \
-H 'Authorization: Bearer <jwt>' -H 'X-Service: postgres' -H 'X-Resource: query' \
-d '{"params": []}'
# expected: 400, errors: [{"field":"sql","reason":"missing"}]
# type mismatch
curl -s -X POST https://api.riotpiao.com/ \
-H 'Authorization: Bearer <jwt>' -H 'X-Service: postgres' -H 'X-Resource: query' \
-d '{"sql": 42}'
# expected: 400, errors: [{"field":"sql","reason":"type_mismatch: want string got number"}]
# nullable field accepted
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/ \
-H 'Authorization: Bearer <jwt>' -H 'X-Service: memory' -H 'X-Resource: skill'
# expected: 200 even though generated_from is null in the response body — response
# validation must not reject/replace the body
# unit test, not curl:
go test ./internal/serviceadapter/... -run TestValidate -v
# expected: covers array-of-scalar, array-of-object, strict rejection, nullable
```
tasks/8.4-workflow-adapter.md
-50
View File
@@ -1,50 +0,0 @@
# 8.4 — `X-Service: workflow` adapter, supersedes `/workflow/*` prefix (GREEN)
Phase: 8 — ServiceAdapter CRD rollout
Stage: RED
Depends on: 8.1, 8.2, 8.3
Design contract: [API_ROUTING_HYBRID_DESIGN.md](../API_ROUTING_HYBRID_DESIGN.md) §2.
Supersedes [7.3](7.3-workflow-prefix.md) — that task's prefix-based `/workflow/*`
route was itself mocked/partial (see `INDEX.md` Progress note); this task replaces
it with the header-based adapter rather than finishing it as originally specced.
Do not also try to complete 7.3's checklist — its route is retired, not extended.
- [ ] `k8s/serviceadapter-workflow.yaml` CR: `serviceName: workflow`, upstream = the
Temporal Service in the `temporal` namespace, `auth.capability: workflow:access`
- [ ] Resource `workflow` maps: `POST` → `START_WORKFLOW`, `GET /workflow/{id}` →
`QUERY_WORKFLOW`, `GET` (list) → `LIST_WORKFLOWS`, `DELETE /workflow/{id}` →
`TERMINATE_WORKFLOW`, `GET /workflow/{id}/history` → `GET_WORKFLOW_HISTORY`
(§2's mapping table)
- [ ] `requestSchema` on `START_WORKFLOW`'s POST method — fields for whatever the
Temporal start-workflow call actually needs (namespace, workflow type, args);
define against the real `internal/temporal` client code, not invented fields
- [ ] Long-poll/streaming semantics from the old `/workflow` handler are preserved
unbuffered through the new dispatcher (G4 still applies)
- [ ] Nothing in this adapter registers a Temporal namespace — registration stays
with queue-operator, same invariant as 7.3
- [ ] Old `/workflow*` path-mounted route is removed once this adapter is verified live
- [ ] Metrics/rejection counters cover this adapter with its own service label
## Verify
```bash
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/ \
-H 'X-Service: workflow' -H 'X-Resource: workflow'
# expected: 401 without a token
curl -s -X POST https://api.riotpiao.com/ \
-H "Authorization: Bearer $WORKFLOW_TOKEN" \
-H 'X-Service: workflow' -H 'X-Resource: workflow' \
-d '{"namespace": "default", "workflowType": "smoke-test", "args": []}'
# expected: 200/202, Temporal's own start-workflow response, proxied unmodified
curl -s https://api.riotpiao.com/ \
-H "Authorization: Bearer $WORKFLOW_TOKEN" \
-H 'X-Service: workflow' -H 'X-Resource: workflow/<id>/history'
# expected: 200, workflow history payload
kubectl -n temporal exec svc/temporal-admintools -- tctl --ad temporal-frontend:7233 namespace list | sort > /tmp/ns.after
diff /tmp/ns.before /tmp/ns.after
# expected: no diff
```
tasks/8.5-sqs-adapter.md
-56
View File
@@ -1,56 +0,0 @@
# 8.5 — `X-Service: sqs` adapter, supersedes `/sqs/*` prefix (GREEN)
Phase: 8 — ServiceAdapter CRD rollout
Stage: RED
Depends on: 8.1, 8.2, 8.3
Design contract: [API_ROUTING_HYBRID_DESIGN.md](../API_ROUTING_HYBRID_DESIGN.md) §2
(`queue/{name}/message` resource shape), and [docs/API-sqs.md](../docs/API-sqs.md)
for the real upstream contract — six RPCs, base64 `bytes` fields, lowerCamelCase
JSON, 256 KiB body cap, 20s long-poll. Supersedes [7.2](7.2-sqs-prefix.md); that
prefix-based route is retired by this adapter, not extended.
- [ ] `k8s/serviceadapter-sqs.yaml` CR: `serviceName: sqs`, upstream
`management-service.sqs.svc.cluster.local:8080`, `auth.capability: queue:access`
- [ ] Resource `message` maps the six RPCs from `docs/API-sqs.md`'s table:
`POST queue/{name}/message` → `SendMessage`,
`POST queue/{name}/message:batch` → `SendMessageBatch`,
`GET queue/{name}/message` → `ReceiveMessage`,
`DELETE queue/{name}/message/{receiptHandle}` → `DeleteMessage`,
`POST queue/{name}/message:batchDelete` → `DeleteMessageBatch`,
`PATCH queue/{name}/message/{receiptHandle}` → `ChangeMessageVisibility`
- [ ] `{name}`/`{receiptHandle}` path segments resolved the same way 8.2 decided for
`{id}` generally — do not invent a second mechanism here
- [ ] `requestSchema` on `SendMessage`: `messageBody: string` (required, base64,
the KV+type DSL does not validate base64-ness — that stays a body-content
concern, not a schema-type concern), `messageAttributes: object`,
`messageGroupId: string`, `messageDeduplicationId: string`, `delaySeconds: number`
- [ ] `GET` (`ReceiveMessage`) read timeout exceeds 20s (max `waitTimeSeconds`) with
headroom, and a client disconnect cancels the upstream long-poll (G4)
- [ ] Upstream error envelope (`{"code": 5, "message": "...", "details": []}`) is
passed through unchanged, per `docs/API-sqs.md`'s explicit recommendation —
not re-rendered as RFC 9457
- [ ] `kmsvc-redis-master.sqs:6379` (unauthenticated) stays unreachable — the
NetworkPolicy carried over from 7.2 grants no egress to it
- [ ] Queue lifecycle (create/delete/list) is **not** exposed — same G2 boundary
`docs/API-sqs.md` already states, unchanged by this adapter
- [ ] Old `/sqs/*` path-mounted route is removed once this adapter is verified live
## Verify
```bash
Q=agent-worker-queue
curl -s -X POST https://api.riotpiao.com/ \
-H "Authorization: Bearer $QUEUE_TOKEN" -H 'X-Service: sqs' -H 'X-Resource: message' \
-d "{\"messageBody\":\"$(printf 'hello world' | base64)\"}"
# expected: {"messageId": "...", "sequenceNumber": ""}
curl -s "https://api.riotpiao.com/?queue=$Q" \
-H "Authorization: Bearer $QUEUE_TOKEN" -H 'X-Service: sqs' -H 'X-Resource: message' \
-G --data-urlencode 'maxNumberOfMessages=10' --data-urlencode 'waitTimeSeconds=20'
# expected: 200 within ~20s, {"messages":[...]}, connection not dropped by gateway timeout
kubectl -n api get networkpolicy -o yaml | grep -c 6379
# expected: 0
```
tasks/8.6-s3-adapter.md
-53
View File
@@ -1,53 +0,0 @@
# 8.6 — `X-Service: s3` adapter, read-only object surface (GREEN)
Phase: 8 — ServiceAdapter CRD rollout
Stage: RED
Depends on: 8.1, 8.2, 8.3
Design contract: [API_ROUTING_HYBRID_DESIGN.md](../API_ROUTING_HYBRID_DESIGN.md) §2
(`bucket/{key}` resource shape). No prior task or doc names the actual MinIO Service
— `grep -ri minio` across this repo turns up only prose in `REQUIREMENTS.md`/
`README.md`/`tasks/INDEX.md`/`tasks/7.4-db-prefix.md`, no manifest. **Confirm the
real Service name/namespace/port in-cluster before writing the CR** — do not guess
a hostname.
**G2 boundary, same as [7.4](7.4-db-prefix.md):** the gateway holds no MinIO access
key or secret key. If this adapter's design wants the gateway to hold a credential,
the design is wrong — put the credential-holding logic in a service behind the
gateway (e.g. a small internal proxy that signs requests) and adapt to *that*, not
to MinIO directly, unless MinIO itself supports anonymous/read-only bucket policies
that make a credential unnecessary for the specific buckets exposed here.
- [ ] `k8s/serviceadapter-s3.yaml` CR: `serviceName: s3`, `auth.capability: s3:read`
- [ ] Resource `bucket` maps `GET bucket/{key}` → object read, `DELETE bucket/{key}`
→ object delete — **only if** a write/delete capability is explicitly wanted;
default to read-only (`GET` only) unless told otherwise, consistent with 7.4's
"no write, no delete" rule for the `/db/*` surface this supersedes
- [ ] Result listing (if a bucket-list resource is added) is paginated with a
bounded page size — no unbounded listing, same rule 7.4 already established
- [ ] `requestSchema`/`responseSchema` per the KV+type DSL (8.3) — define once the
actual MinIO/proxy response shape is confirmed, not invented ahead of it
- [ ] NetworkPolicy reaches only the confirmed MinIO Service, nothing broader
- [ ] `/db/*` prefix-mounted MinIO read paths (if any exist from 7.4) are removed
once this adapter is verified live, to avoid two auth paths to the same data
## Verify
```bash
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/ \
-H 'X-Service: s3' -H 'X-Resource: bucket/some-key'
# expected: 401 without a token
curl -s -X DELETE -o /dev/null -w '%{http_code}\n' \
-H "Authorization: Bearer $S3_READ_TOKEN" \
https://api.riotpiao.com/ -H 'X-Service: s3' -H 'X-Resource: bucket/some-key'
# expected: 404 or 405 if this adapter ships read-only — no mutating verb is routable
curl -s -H "Authorization: Bearer $S3_READ_TOKEN" \
https://api.riotpiao.com/ -H 'X-Service: s3' -H 'X-Resource: bucket/known-object-key'
# expected: 200, object bytes or metadata per the confirmed response shape
kubectl -n api get pod -l app=api-gateway -o jsonpath='{range .items[0].spec.containers[0].env[*]}{.name}{"\n"}{end}' \
| grep -Ei 'minio|access_key|secret_key'
# expected: no output — gateway carries no MinIO credential
```
tasks/8.7-iam-adapter.md
-48
View File
@@ -1,48 +0,0 @@
# 8.7 — `X-Service: iam` adapter, Authentik admin surface (GREEN)
Phase: 8 — ServiceAdapter CRD rollout
Stage: RED
Depends on: 8.1, 8.2, 8.3
Design contract: [API_ROUTING_HYBRID_DESIGN.md](../API_ROUTING_HYBRID_DESIGN.md) §3.
- [ ] `internal/iam/handler.go` (new) implements the §3 mapping table:
`user` GET/POST → `/api/v3/core/users/`, `user/{id}` GET/PATCH/DELETE →
`/api/v3/core/users/{id}/`, `service-account` POST →
`/api/v3/core/users/service_account/`, `role` GET/POST →
`/api/v3/core/groups/`, `permission` GET/POST → `/api/v3/rbac/permissions/`,
`flow` GET → `/api/v3/flows/instances/`
- [ ] `k8s/serviceadapter-iam.yaml` CR: `serviceName: iam`, upstream = Authentik's
internal Service, `auth.capability: iam:admin` (default — this surface is
admin-only, tighter than the other adapters' read/write split)
- [ ] `requestSchema` on `POST user` and `POST service-account` — fields matching
Authentik's actual `/api/v3/core/users/` create-user body, confirmed against
the live API, not invented
- [ ] This is additive to `core iam` CLI subcommand (`~/workplace/core/src/cmd/iam/`),
not a replacement — different caller (server vs. local CLI), same upstream.
Do not modify the `core` CLI as part of this task
- [ ] No token without `iam:admin` reaches any of these resources, including `flow`
(GET-only, but still admin-scoped per the design doc — do not default it to
a lower/no-auth tier because it's read-only)
## Verify
```bash
curl -s -o /dev/null -w '%{http_code}\n' https://api.riotpiao.com/ \
-H 'X-Service: iam' -H 'X-Resource: user'
# expected: 401 without a token
curl -s -o /dev/null -w '%{http_code}\n' \
-H "Authorization: Bearer $NON_ADMIN_TOKEN" \
https://api.riotpiao.com/ -H 'X-Service: iam' -H 'X-Resource: user'
# expected: 403 — token lacks iam:admin
curl -s -H "Authorization: Bearer $IAM_ADMIN_TOKEN" \
https://api.riotpiao.com/ -H 'X-Service: iam' -H 'X-Resource: user'
# expected: 200, Authentik's user list, proxied through /api/v3/core/users/
curl -s -X POST https://api.riotpiao.com/ \
-H "Authorization: Bearer $IAM_ADMIN_TOKEN" \
-H 'X-Service: iam' -H 'X-Resource: role' -d '{}'
# expected: 400 — requestSchema rejects an empty group-create body
```
tasks/8.8-memory-adapter-core.md
-64
View File
@@ -1,64 +0,0 @@
# 8.8 — `X-Service: memory` adapter, core resources (GREEN)
Phase: 8 — ServiceAdapter CRD rollout
Stage: RED
Depends on: 8.1, 8.2, 8.3
Design contract: [API_ROUTING_HYBRID_DESIGN.md](../API_ROUTING_HYBRID_DESIGN.md) §6.
Covers only the poimen-memory endpoints confirmed **already built and tested**
(`~/workplace/Poimen/memory/DESIGN.md`'s M3.5 phase): `GET /memory/query`,
`GET /memory/projects`, `GET /memory/projects/{id}/status`, `GET /memory/skills`,
`GET /memory/skills/{name}`, `POST /memory/ingest`. `/memory/context`,
`/memory/projects/{id}/notes` and the git-aware `/memory/nodes/*` endpoints are
**not** in scope here — see [8.9](8.9-memory-adapter-extended.md).
**Hard prerequisite, not optional — do in this order:**
1. `NetworkPolicy` in namespace `poimen` restricting ingress on `poimen-memory` to
the `api` namespace's gateway pod only. Must land before step 2.
2. Remove the `apikey:` middleware from `poimen-memory` itself — separate change
in the `~/workplace/Poimen/memory` repo, out of scope for this repo but a hard
prerequisite for this adapter being safe to expose. Do not apply the CR below
before this lands.
3. Provision `memory:read`/`memory:write` as real Authentik scopes (via 8.7's
`iam` adapter or `core mwinit`-issued tokens).
- [ ] `k8s/serviceadapter-memory.yaml` CR per §6's example, `auth.capability: memory:read`
default, `ingest` method overrides to `memory:write`
- [ ] `responseSchema` on `query` uses 8.3's array-of-object extension:
`type: array, items: { fields: { level: string, sha256: string, text: string, score: number } }`
- [ ] `responseSchema` on `projects` uses the array-of-scalar extension:
`type: array, items: string`
- [ ] `responseSchema` on `skills`/`skills/{name}` marks `generated_from` as
`{type: string, nullable: true}` — the real upstream response sends `null`
for un-derived skills, confirmed in `DESIGN.md`'s example
- [ ] `requestSchema` on `ingest`: `required: ["project", "source", "records"]`,
`ingest_id` optional (`strict: false` — server may compute it if absent)
- [ ] A `memory:read`-scoped token can `GET` `query`/`skill`/`project` and gets 403
on `ingest`; a `memory:write`-scoped token can `POST ingest`
## Verify
```bash
curl -s -H 'Authorization: Bearer <jwt-with-memory:read>' \
https://api.riotpiao.com/ -H 'X-Service: memory' -H 'X-Resource: query' \
-G --data-urlencode 'query=why did requests over 10KB fail' \
--data-urlencode 'project=poimen' --data-urlencode 'level=L1,L2'
# expected: 200, JSON array of {level,sha256,text,score,parents}
curl -s -H 'Authorization: Bearer <jwt-with-memory:read>' \
https://api.riotpiao.com/ -H 'X-Service: memory' -H 'X-Resource: project'
# expected: 200, ["poimen", ...]
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
-H 'Authorization: Bearer <jwt-with-memory:read>' \
https://api.riotpiao.com/ -H 'X-Service: memory' -H 'X-Resource: ingest' -d '{}'
# expected: 403 — memory:read token, ingest needs memory:write
curl -s -X POST -H 'Authorization: Bearer <jwt-with-memory:write>' \
https://api.riotpiao.com/ -H 'X-Service: memory' -H 'X-Resource: ingest' \
-d '{"project":"poimen","source":"agent:uuid","records":[]}'
# expected: 202, {"job_id":"ingest-...","status_url":"..."}
kubectl -n poimen get networkpolicy -o yaml | grep -A5 poimen-memory
# expected: ingress restricted to the api namespace's gateway pod selector only
```
tasks/8.9-memory-adapter-extended.md
-51
View File
@@ -1,51 +0,0 @@
# 8.9 — `X-Service: memory` adapter, extended resources (GREEN)
Phase: 8 — ServiceAdapter CRD rollout
Stage: BLOCKED — do not start until the upstream note below is resolved
Depends on: 8.8 (core `memory` adapter live)
Design contract: `~/workplace/Poimen/memory/DESIGN.md` "Distributed API Layer"
section. Covers `GET /memory/projects/{id}/notes`, `POST /memory/context`, and the
optional `POST /memory/nodes/by-git`/`by-commit`/`by-author` git-aware lookups.
**Upstream status, checked against that repo's own task board, not assumed:**
- `notes` is listed in the M3.5 task table but **not** in the confirmed-tested set
(`query`, `projects`, `projects/{id}/status`, `ingest`, `skills`, `skills/{name}`)
- `context` depends on M3.7.7 (signature extraction) and M3.7.8 (symptom vector) —
neither is built per that repo's phase ordering
- git-aware lookups (M3.5.9) are explicitly marked optional in that repo's own board
**Before writing any CR entry here: re-check `~/workplace/Poimen/memory/memory-tasks/INDEX.md`
for current status of M3.5.6, M3.7.*, and M3.5.9.** If they are still not shipped,
stop and report that instead of building a gateway route with nothing live to call —
per this repo's own zero-context-agent rule, a route to a 404 is not verifiable and
this task cannot be completed honestly.
- [ ] Confirm `notes`, `context`, and `nodes/by-*` are live upstream (curl the
Service directly from inside the cluster, not through the gateway, to check)
- [ ] `responseSchema` for `context` is defined **from the real response**, not from
`DESIGN.md`'s prose description (that doc itself says the exact bundle shape
isn't pinned down — "the bundle is a composition" with no worked JSON example)
- [ ] Extend `k8s/serviceadapter-memory.yaml` (from 8.8) with these three resources,
same `memory:read` capability as `query`/`skill`/`project`
- [ ] git-aware resources only added if M3.5.9 is confirmed shipped — otherwise
ticket this as follow-up and leave this task's remaining boxes unchecked
## Verify
```bash
# from inside the cluster, upstream directly — confirms it exists before routing to it
kubectl -n poimen exec deploy/poimen-memory -- curl -s localhost:8080/memory/context \
-X POST -d '{"tool":"kubectl"}'
# expected: 200 with a real body, not 404 — if 404, stop, this task is blocked
curl -s -H 'Authorization: Bearer <jwt-with-memory:read>' \
https://api.riotpiao.com/ -H 'X-Service: memory' -H 'X-Resource: notes' \
-G --data-urlencode 'project=poimen'
# expected: 200, array of L1/L2 note objects
curl -s -X POST -H 'Authorization: Bearer <jwt-with-memory:read>' \
https://api.riotpiao.com/ -H 'X-Service: memory' -H 'X-Resource: context' \
-d '{"tool":"kubectl"}'
# expected: 200, bundled context response
```
tasks/AGENT-PROMPT.md
-105
View File
@@ -1,105 +0,0 @@
# Agent prompt template
Every task is worked by an agent starting from **zero context**. No memory of prior
tasks, no conversation history, no assumptions about what already exists.
This is deliberate. Task files are written to be self-contained precisely so that a
fresh agent can pick any one of them up. It also means a task that cannot be completed
from its own file plus this prompt is a task that is under-specified — that is a bug in
the task, and worth reporting rather than working around.
Rendered and invoked by [`scripts/run-task.sh`](../scripts/run-task.sh). Do not paste
this by hand; use the script so the fresh-session guarantee actually holds.
---
## Template
`{{TASK_ID}}` and `{{TASK_FILE}}` are substituted by the runner.
```
You are implementing one task in the homelab-frontend repository: a Go API gateway
that replaces Kong OSS on api.riotpiao.com.
You are starting from zero context. Everything you need is below or in the files named
below. Do not assume any prior work exists beyond what you find in the repository.
## Your task
Read tasks/{{TASK_FILE}} and implement it. That file states what must be true; it does
not state how. The design is yours to reason out. The checkboxes are the contract.
## Before writing code
1. Read tasks/{{TASK_FILE}} in full.
2. Check its `Depends on:` line. If a dependency is not yet implemented in this
repository, stop and report that instead of building it yourself. One task per run.
3. Look at how the surrounding code is written and match it. If the repository is
still empty, you are establishing the conventions, so choose carefully.
## Invariants — breaking one of these is a design change, not a detail
G1 ingress-nginx owns TLS. The gateway never terminates TLS.
G2 The gateway holds no Kubernetes credentials. Config comes from git, not a CRD.
Narrow exception for Phase 8 tasks (8.1+): the gateway ServiceAccount may hold a
namespace-scoped, read-only (get/list/watch) Role on the ServiceAdapter CRD only
— see tasks/INDEX.md's G2 line and API_ROUTING_HYBRID_DESIGN.md's Context section.
This is pre-approved; do not stop on it as a violation for Phase 8 work specifically.
G3 Public surfaces use standard protocol shapes. If an OpenAI or Anthropic SDK
cannot call it unmodified, the design is wrong.
G4 Streaming is unbuffered, and a client disconnect cancels the upstream request.
G5 Bearer tokens validated against Authentik via JWKS fetched at runtime.
G6 Every timeout, body cap and concurrency limit is explicit in configuration.
G7 Deployment flows through git and Argo. Never run kubectl apply, helm upgrade,
or terraform apply.
## How to work
Test-driven. If the task is marked `Stage: RED`, write the failing test first and
confirm it fails for the right reason before implementing. If `GREEN`, write the
minimum code that passes. If `REFACTOR`, keep the tests green while improving shape.
Go standards for this repository:
- Never discard errors with `_ =`. Wrap them with context.
- Every upstream call carries a context.Context.
- No naked returns. Use `any`, not `interface{}`.
- Table-driven tests with subtests where there is more than one case.
## Definition of done
Run the `## Verify` block from the task file. It must pass.
Then run all three of these regardless of what the task's verify block says:
go test ./... -race
CGO_ENABLED=0 go build ./...
go vet ./...
`-race` is mandatory. This is a concurrent proxy; a test suite that passes without the
race detector tells you almost nothing. A detected race is a failure, not a warning.
Verification means asserting on a real HTTP response — status, headers, body. "It
compiles" and "it starts" are not verification. If you cannot run the verification
locally with no cluster and no credentials, that is itself a problem to report.
Then **edit tasks/{{TASK_FILE}} and change `- [ ]` to `- [x]`** for each criterion you
actually satisfied. This is a file edit, not something to state in your summary. Leave
unticked anything you did not complete. A summary that claims `[x]` while the file still
reads `[ ]` is a false report.
## Hard rules
- Kong is serving live traffic on api.riotpiao.com right now. Change no cluster state.
- Do not commit or push.
- Do not implement tasks other than {{TASK_ID}}. If you notice something else that
needs doing, report it rather than fixing it.
- Do not add features, abstractions, or configurability that the task did not ask for.
- If the task is ambiguous or appears wrong, stop and say so. Do not guess and proceed.
## Report when finished
- What you implemented, and the files you touched.
- The verification command you ran and its actual output.
- Which checkboxes you ticked and which you did not, with reasons.
- Anything you found that is wrong elsewhere in the repository or the task files.
```
tasks/INDEX.md
-179
View File
@@ -1,179 +0,0 @@
# Task board — homelab-frontend
The Go API gateway replacing Kong OSS on `api.riotpiao.com`.
Contract: [REQUIREMENTS.md](../REQUIREMENTS.md).
Why: [ADR-0001](../docs/adr/ADR-0001-retire-kong-for-go-gateway.md).
What Kong does today and the cutover order: [docs/MIGRATION-kong.md](../docs/MIGRATION-kong.md).
## Rules carried from the ADR and requirements
- G1 — ingress-nginx owns TLS. The gateway never terminates TLS.
- G2 — the gateway holds no Kubernetes credentials. Config comes from git, not a CRD.
**Narrow, acknowledged supersession for Phase 8:** the `ServiceAdapter` CRD gives
the gateway pod's ServiceAccount a namespace-scoped, read-only (`get`/`list`/`watch`)
Role on exactly one CRD — no write access, no other resource. Rationale in
[API_ROUTING_HYBRID_DESIGN.md](../API_ROUTING_HYBRID_DESIGN.md)'s Context section.
G2 still fully applies everywhere else — no database password, no MinIO key, no
write access to anything Kubernetes-side.
- G3 — public surfaces use standard protocol shapes. If an OpenAI SDK can't call it unmodified, it's wrong.
- G4 — streaming is unbuffered, and a client disconnect cancels the upstream.
- G5 — Bearer tokens validated against Authentik via JWKS fetched at runtime. No pinned keys.
- G6 — every timeout, body cap and concurrency limit is explicit in config.
- G7 — deployment flows through git and Argo. No `kubectl apply`, no `helm upgrade`.
## How to work these
Each task is self-contained — it states what must be true, not how to build it.
Reason out the implementation; the acceptance criteria are the contract.
Stages follow red-green-refactor. A task marked RED means the test comes first and
must fail for the right reason before any implementation exists.
**Verification means asserting on a real HTTP response** — status, headers, body.
"It compiles" and "it starts" are not verification. Every task that touches an API
surface has a `## Verify` block with a runnable command.
Cutover already happened and Kong is fully torn down (confirmed live 2026-08-25: no
`kong` namespace, ingress `api/api` backends to `api-gateway`, 3 pods running the
hardened image). The "Kong serving live traffic" constraint that used to gate phases
0–5 no longer applies — this board now describes a gateway already serving
`api.riotpiao.com` in production, not a pre-cutover build.
## Phase 0 — Foundations
| Task | Description |
|---|---|
| [0.1](0.1-module-and-entrypoint.md) | Go module, entrypoint, graceful shutdown |
| [0.2](0.2-route-configuration.md) | Declarative route/upstream config from git, fail-loud validation |
| [0.3](0.3-health-endpoints.md) | `/healthz` and `/readyz` |
| [0.4](0.4-local-dev-harness.md) | Run with no cluster, no kubeconfig, no credentials — stub upstreams |
| [0.5](0.5-structured-logging.md) | Structured logs, no secrets or bodies |
| [0.6](0.6-ci-pipeline.md) | CI: build, vet, test, `govulncheck` |
## Phase 1 — Proxy core
| Task | Description |
|---|---|
| [1.1](1.1-reverse-proxy.md) | Reverse proxy to a configured upstream, connection reuse |
| [1.2](1.2-streaming-passthrough.md) | SSE and chunked responses pass through unbuffered |
| [1.3](1.3-disconnect-propagation.md) | Client disconnect cancels the upstream request |
| [1.4](1.4-per-route-timeouts.md) | Explicit connect/read/write timeouts per route |
| [1.5](1.5-header-hygiene.md) | Hop-by-hop stripping, `X-Forwarded-*` from nginx |
| [1.6](1.6-websocket-upgrade.md) | WebSocket upgrade — `agent-pod/console` needs it |
| [1.7](1.7-body-size-caps.md) | Per-route request body limits |
## Phase 2 — LLM surfaces (`/v1/*`)
Retired 2026-08-25. `2.1`/`2.3`/`2.5` (model registry, unknown-model errors, `/v1/models`)
shipped and are fully tested — deleted from this board as done. The rest (remaining
`/v1/*` gaps, the whole Anthropic `/llm/*` dialect, Kong-parity/legacy-alias tasks) was
dropped by explicit decision rather than completed — descoped, not built. Wire formats
that were in scope are still documented in [docs/API-llm.md](../docs/API-llm.md).
## Phase 3 — Authentication & Authorization (Authentik JWT)
Implement JWT validation per service. Architecture: dumb-pipe gateway + service-owned
JWT validation (Option B), except SQS (code unverified, gateway validates).
| Task | Description |
|---|---|
| [3.1](3.1-auth-sqs-jwt-validation.md) | SQS: Gateway JWT validation vs Authentik JWKS |
| [3.2](3.2-auth-minio-jwt-validation.md) | MinIO: Load-test native JWT/OIDC validation |
| [3.3](3.3-auth-temporal-jwt-validation.md) | Temporal: Configure JWT via jwtKeyProvider |
## Phase 4 — Limits and budgets
| Task | Description |
|---|---|
| [4.1](4.1-gpu-slot-semaphore.md) | Cap concurrent `reasoning` requests below 8 slots, bounded queue |
| [4.2](4.2-per-caller-budgets.md) | Request budget per identified caller per window |
| [4.3](4.3-problem-json-errors.md) | RFC 9457 rejections with `Retry-After` |
## Phase 5 — Observability
| Task | Description |
|---|---|
| [5.1](5.1-prometheus-parity.md) | Match the retiring Kong plugin: rate, latency, status, bandwidth, upstream health |
| [5.2](5.2-gateway-metrics.md) | In-flight per upstream, queue depth, slot occupancy, rejections by reason |
| [5.3](5.3-servicemonitor.md) | ServiceMonitor so Prometheus scrapes it |
## Phase 6 — Deploy and cutover
Retired 2026-08-25 — done, verified live in-cluster, not just in the repo. `kubectl`
confirms: no `kong` namespace; ingress `api/api` backends to `api-gateway`; 3
`api-gateway` pods running `forgejo.riotpiao.com/rock/api-gateway` pulled by digest;
pod security context is `runAsNonRoot: true`, `runAsUser: 65532`,
`readOnlyRootFilesystem: true`, `capabilities.drop: [ALL]`, no shell in the container.
6.1–6.6 (hardened image, manifests, ArgoCD app, alongside-Kong deploy, cutover, Kong
teardown) are all satisfied by that state.
## Phase 7 — Additional capability prefixes
Deliberately after cutover. Each is additive and must not disturb `/v1/*`.
| Task | Description |
|---|---|
| [7.1](7.1-cluster-prefix-atlas.md) | `/cluster/*` → atlas (`riotpiao-backend`) |
| [7.2](7.2-sqs-prefix.md) | `/sqs/*` → kmsvc management-service, Kafka |
| [7.3](7.3-workflow-prefix.md) | `/workflow/*` → Temporal |
| [7.4](7.4-db-prefix.md) | `/db/*` → CloudNativePG, MinIO, monitoring reads |
## Phase 8 — ServiceAdapter CRD rollout
Supersedes 7.2 (`/sqs/*`) and 7.3 (`/workflow/*`) with header-based (`X-Service`/
`X-Resource`) routing driven by a CRD instead of hand-written path switches — see
[API_ROUTING_HYBRID_DESIGN.md](../API_ROUTING_HYBRID_DESIGN.md). 7.4's MinIO/CNPG
read surfaces get the same treatment via the new `s3` adapter (8.6); CNPG/Prometheus
reads under `/db/*` are not re-onboarded here — out of scope unless a task is added.
| Task | Description |
|---|---|
| [8.1](8.1-serviceadapter-crd-and-informer.md) | `ServiceAdapter` CRD, `client-go` informer, read-only RBAC |
| [8.2](8.2-x-service-dispatcher.md) | ✅ `X-Service`/`X-Resource` dispatch, HTTP/gRPC detection |
| [8.3](8.3-request-response-validation.md) | Request/response validation, flat KV+type DSL |
| [8.4](8.4-workflow-adapter.md) | `workflow` adapter — supersedes 7.3 |
| [8.5](8.5-sqs-adapter.md) | `sqs` adapter — supersedes 7.2 |
| [8.6](8.6-s3-adapter.md) | `s3` adapter — new, read-only, MinIO Service TBD |
| [8.7](8.7-iam-adapter.md) | `iam` adapter — Authentik admin surface |
| [8.8](8.8-memory-adapter-core.md) | `memory` adapter, core resources (confirmed-live upstream) |
| [8.9](8.9-memory-adapter-extended.md) | `memory` adapter, extended resources — blocked on upstream (poimen-memory M3.7/M3.5.9) |
| [8.10](8.10-serviceadapter-gate.md) | ✅ **Phase 8 gate** — all 5 services routing via X-Service |
## Progress
Updated 2026-08-27 (session 3) — Phase 8 complete, Phase 3 (auth) next.
**Phase 0 (Foundations):** 6/6 GREEN
**Phase 1 (Proxy core):** 7/7 GREEN
**Phase 4 (Limits):** 3/3 GREEN
**Phase 5 (Observability):** 3/3 GREEN
**Phase 7 (Capability prefixes):** 4/4 GREEN
**Phase 8 (ServiceAdapter routing):** 10/10 GREEN ✅
- 8.1 ServiceAdapter CRD & in-memory registry
- 8.2 X-Service/X-Resource dispatcher ✅
- 8.3 Request/response schema validation
- 8.4–8.8 Adapter definitions (sqs, workflow, s3, iam, memory)
- 8.9 Memory extended resources
- 8.10 Phase gate ✅
**Phase 3 (Authentication):** 0/3 TODO
- 3.1 SQS JWT validation
- 3.2 MinIO JWT load-test
- 3.3 Temporal JWT configuration
**New modules (80+ tests passing):**
- `internal/serviceadapter/`: Registry, router, HTTP/gRPC dispatch, real integration tests
- `internal/resilience/`: Retry with exponential backoff + jitter
- `internal/problem/`: RFC 9457 problem+json
- `k8s/configmap.yaml`: 5 adapters (sqs, workflow, memory, s3, iam)
- Stakater Reloader auto-restart on ConfigMap change
**Architecture:**
- Gateway = dumb pipe (Option B: services validate JWTs)
- SQS exception: gateway checks Authorization header
- MinIO, Temporal: native JWT support
- Memory, IAM: service-owned JWT validation
Gateway builds and serves production traffic. Real integration tests pass.