homelab-frontend/API_ROUTING_HYBRID_DESIGN.md

API Routing Design: Header-Based Services, CRD-Driven Onboarding, Unified Auth

Supersedes the earlier /unified-endpoint draft of this doc. Routing is now header-only at the gateway root — api.riotpiao.com itself is the single entry point, no dedicated sub-path. New backend services onboard via a CRD the gateway consumes directly, not via hand-written Go switch cases.

---

Context

Most of the original hybrid design was aspirational — it didn't exist in code. This revision reflects decisions made while reviewing the repo against the actual state: internal/auth/ was empty, s3/sqs/iam didn't exist as handlers, and only a body-action POST /workflow endpoint was real. Four extensions were decided: CRD-driven service onboarding, X-Resource-based REST verbs replacing body actions, unified JWT auth (bearer token or service-account password, one validation path), and blind 5xx retry.

Known constraints, and how these decisions land on them:

• Rule G2 (originally specced in the now-retired tasks/6.2-kubernetes-manifests.md, deleted 2026-08-25 since Phase 6 shipped and is live in-cluster) mandated no ServiceAccount/RBAC on the gateway pod, and routes loaded from a git-sourced ConfigMap with a pod roll on every change. The gateway pod now consumes the CRD directly — no separate operator/controller pod. This is a deliberate, acknowledged supersession of G2 for this one narrow purpose: the existing api-gateway ServiceAccount (already bound to the pod at k8s/rbac.yaml:1-7, currently with zero Role — see k8s/rbac.yaml:9-10's own comment "the gateway has no k8s API access") gains a namespace-scoped Role limited to read-only access on one CRD, defined in this same repo and synced by the existing api-gw ArgoCD Application (homelab/k8s/argocd/apps/55-api-gateway.yaml) — no new repo, no new Application. Static ConfigMap-sourced routes keep the existing roll-on-change behavior; CRD-sourced routes update live in-process instead (see §1).
• Retired kmsvc.io TemporalWorker/Queue CRD (homelab/k8s/apps/messaging/queue-crd/, controller in the separate kmsvc-manage repo) broke via an uncleared finalizer and RBAC rules that drifted out of sync with the CRD's resources (temporalworkers.kmsvc.io is forbidden ... at the cluster scope). Cluster-scoped RBAC, RBAC/CRD drift, and finalizer complexity are explicitly avoided this time — this CRD is namespace-scoped (api only) and read-only from the gateway's side.

---

0. Repo hygiene — do this first, before any code

Already staged for deletion (git status) — just needs a commit: DELIVERY_COMPLETE.md, IMPLEMENTATION_SUMMARY.md, PHASE3_GRPC_IMPLEMENTATION.md, PHASE3_PROGRESS.md, TEMPORAL_API_DESIGN_SUMMARY.md, TEMPORAL_IMPLEMENTATION_CHECKPOINT.md, TEMPORAL_IMPLEMENTATION_COMPLETE.md, TEMPORAL_TEST_REPORT.md, WORKFLOWS_INDEX.md, WORKFLOWS_QUICK_START.md, WORKFLOWS_README.md.

Removed as part of this revision: IMPLEMENTATION_PLAN_UPDATED.md — its reading order pointed at this doc's now-superseded /unified-endpoint design; folded into this doc instead of kept as a separate index.

Untracked, flagged but left alone (different topic, not touched by this doc): AUTHENTIK_AS_KEY_MANAGER.md, AUTHENTIK_KEY_ROTATION_SIMPLE.md, AUTHENTIK_SOPS_NATIVE.md, AUTHENTIK_WITH_FIPS_HSM.md, JWT_VS_RANDOM_TOKEN.md — SOPS secrets-at-rest key rotation via Authentik, unrelated to API routing/CRD work.

---

1. Service onboarding (new backend service = code change vs. YAML)

Existing System

• No CRD, no controller, no generic service registry anywhere in either repo.
• Adding a service today means: write a new Go handler, add a switch case, redeploy the gateway image. Only workflow exists (internal/temporal/handler.go); s3, sqs, iam don't exist in code at all.
• internal/config/config.go + loader.go load only static, hand-edited route config from the ConfigMap already live per Phase 6 (k8s/configmap.yaml) — no per-service schema, no CRD.

Future System

New CRD ServiceAdapter (gateway.riotpiao.com/v1, namespace-scoped to api only — deliberately not cluster-scoped, unlike the retired CRD). Named for what it actually does — adapts an in-cluster service to the gateway's X-Service/X-Resource contract — rather than a generic "Definition".

No separate operator pod. The API gateway pod itself consumes the CRD directly: it runs a client-go informer against ServiceAdapter in namespace api and keeps an in-memory route registry that adds/updates/removes entries as CRs change — no ConfigMap render step, no restart-to-pick-up-changes, no second binary/Deployment/image to build and operate.

Where the CRD is defined — this repo, not homelab. Confirmed at homelab/k8s/argocd/apps/55-api-gateway.yaml:42-45: the api-gw ArgoCD Application already sources this repo's own k8s path directly (kustomize — Deployment, Service, ConfigMap, RBAC, NetworkPolicy), auto-syncing on every push to main, wave 7. So the CRD manifest is just another file in that same kustomization: new k8s/crd-serviceadapter.yaml, added to k8s/kustomization.yaml's resources: list. When a new ServiceAdapter CR is introduced (i.e. someone commits a CR YAML for a new service), ArgoCD syncs it through this same already-existing Application, and the already-running api-gw pod's informer notices it and starts accepting X-Service: <new-service> calls — no new Application, no new repo, no manual step in homelab at all. Go API types back the CRD from apis/gateway/v1/serviceadapter_types.go (new package, generates the CRD YAML via controller-gen, same struct also used by the informer).

spec.auth.capability is the default capability required for every resource on the service. A method entry may override it with its own auth.capability — needed once a service splits read vs. write access (see §6's memory adapter). No override means inherit the spec-level default; auth.required: false at either level means no capability check at all for that scope.

Example CR (onboarding a hypothetical postgres service — no Go code, no image rebuild, no manual gateway restart):

apiVersion: gateway.riotpiao.com/v1
kind: ServiceAdapter
metadata:
  name: postgres
  namespace: api
spec:
  serviceName: postgres          # matches X-Service: postgres
  upstream:
    url: http://kmsvc-postgres-helper.sqs.svc.cluster.local:8080
    timeoutSeconds: 10
  auth:
    required: true
    capability: postgres:access   # checked against the caller's JWT claims
  retryable: true                 # eligible for the blind 5xx retry middleware, §4
  resources:
    - name: query                 # X-Resource: query
      methods:
        - verb: POST
          upstreamPath: /v1/query
          requestSchema:
            fields:
              sql: string
              params: array
            required: ["sql"]
          responseSchema:
            fields:
              rows: array
    - name: table
      methods:
        - verb: GET
          upstreamPath: /v1/tables/{id}   # {id} filled from the trailing path segment
        - verb: DELETE
          upstreamPath: /v1/tables/{id}
status:
  phase: Ready
  observedGeneration: 1

Calling it once reconciled — straight to the gateway root, no dedicated sub-path:

curl -X POST https://api.riotpiao.com/ \
  -H 'Authorization: Bearer <jwt>' \
  -H 'X-Service: postgres' \
  -H 'X-Resource: query' \
  -d '{"sql": "select 1"}'

RBAC — extend k8s/rbac.yaml in place (same file that already defines the api-gateway ServiceAccount at lines 1-6, currently role-less per its own "the gateway has no k8s API access" comment). A namespace-scoped Role (not ClusterRole), read-only, nothing else — this is the entire RBAC footprint added to the gateway pod, shipped in the same repo/PR as the CRD itself:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: api-gateway-serviceadapter-reader
  namespace: api
rules:
  - apiGroups: ["gateway.riotpiao.com"]
    resources: ["serviceadapters"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: api-gateway-serviceadapter-reader
  namespace: api
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: api-gateway-serviceadapter-reader
subjects:
  - kind: ServiceAccount
    name: api-gateway
    namespace: api

No status/finalizers verbs, no write access at all — the gateway only ever reads. No new ServiceAccount, no new Deployment, no new container image.

Gateway-side consumer — new internal/serviceadapter/registry.go: a client-go SharedInformer on ServiceAdapter in namespace api, feeding an in-memory map keyed by spec.serviceName that the internal/serviceadapter dispatcher (§2) reads on every request. Add/Update/Delete informer callbacks update the map directly — a new ServiceAdapter CR is live within the informer's resync/watch latency, no gateway restart, no ArgoCD change beyond the existing auto-sync of this repo's k8s path.

Request/response validation — requestSchema/responseSchema are not JSON Schema. Deliberately a flat key→type map, author-facing (whoever writes the CR specifies it, no JSON Schema knowledge needed):

requestSchema:
  fields:
    sql: string        # one of: string | number | boolean | array | object
    params: array
  required: ["sql"]    # optional; omit if every field is optional
  strict: false         # optional, default false — true rejects unlisted keys
responseSchema:
  fields:
    rows: array

One level deep only — params: array checks "is this an array," not what's inside it. No enums, no formats, no nested field validation. Accepted limitation: none of the adapters in this doc (postgres, workflow, iam, memory) need more than top-level shape checking.

Extension forced by real upstream shapes (poimen-memory, §6): the DSL above only describes a top-level JSON object. GET /memory/projects returns a bare array (["poimen", ...]), GET /memory/query returns an array of objects, and GET /memory/skills returns objects with an explicit "generated_from": null. Neither is expressible in the object-only form, so two additions, kept minimal rather than backsliding toward full JSON Schema:

responseSchema:
  type: array            # top-level shape; default "object" if omitted
  items: string           # scalar array — every element must be this type
responseSchema:
  type: array
  items:
    fields:                # object array — one level deep INSIDE each item
      level: string
      sha256: string
      score: number
responseSchema:
  fields:
    generated_from: { type: string, nullable: true }   # long form only needed for nullable

Rule: a field written as a bare type string (sql: string) is shorthand for {type: string, nullable: false}. null satisfies any field marked nullable: true regardless of its declared type; a null on a non-nullable field is a type_mismatch. Still one level deep — items: { fields: {...} } does not itself accept a nested items.

Validated in Go via internal/serviceadapter/validate.go (new, ~40 lines, no external dependency): json.Unmarshal the body into map[string]interface{}, check Required fields are present, check each present field in Fields against its Go runtime type (string→string, float64→number, bool→boolean, []interface{}→array, map[string]interface{}→object), and if strict: true reject any body key not listed in Fields. Compiled once per CR add/update in the informer callback above, not per-request — the parsed FieldSchema struct is stored directly in the same registry map entry as the route.

Request-side violations are enforced: 400, RFC 9457 problem+json, body {type, title, detail, errors: [{field, reason}]} where reason is one of missing, type_mismatch: want X got Y, or unknown_field (strict mode only). Response-side violations are not enforced — the response is forwarded to the caller unchanged; a mismatch only emits a serviceadapter_response_schema_mismatch{service,resource} metric and a log line. Rationale: request validation is a trust boundary (external caller input); response validation is contract-drift detection against our own upstream — turning a backend field-shape typo into a caller-facing 502 does more harm than logging it.

---

2. Request shape: body action vs. header verb

Existing System

• internal/temporal/handler.go: POST /workflow only, body is {action, namespace, payload} (RequestPayload struct) — action dispatch happens by reading and branching on the JSON body, not by header/method.
• internal/server/router.go dispatches purely by path: exact-matches /healthz, /readyz, /workflow, /workflow/health, /workflow/metrics, and falls through everything else (including /v1/chat/completions) to upstreamHandler. There is no header-based dispatch anywhere in the router today.
• s3/sqs/iam bodies with operation/action fields, and a /unified sub-path, were only ever prose in an earlier draft of this doc — never implemented, and now dropped.

Future System

No new sub-path is introduced. internal/server/router.go's ServeHTTP gets one new branch, checked before the existing path switch: if the request carries an X-Service header, dispatch straight to a new internal/serviceadapter/router.go dispatcher, regardless of r.URL.Path (callers hit https://api.riotpiao.com/ at the root). /v1/chat/completions, /v1/embeddings, /v1/rerank, /healthz, /readyz keep their existing path-based routing in internal/proxy/router.go — those never carry X-Service and are matched first. /workflow* stays mounted during migration, delegating internally to the same dispatcher, then gets removed once callers move to header-based calls.

Dispatch key becomes X-Service header → adapter, then HTTP method + X-Resource header → {verb, upstreamPath} — e.g. GET to the gateway root + X-Service: iam + X-Resource: user → list Authentik users. This replaces the body action/operation field convention everywhere, not just for iam — so the rewritten workflow adapter looks like:

• POST X-Resource: workflow → START_WORKFLOW
• GET X-Resource: workflow/{id} → QUERY_WORKFLOW
• GET X-Resource: workflow → LIST_WORKFLOWS
• DELETE X-Resource: workflow/{id} → TERMINATE_WORKFLOW
• GET X-Resource: workflow/{id}/history → GET_WORKFLOW_HISTORY
• Same pattern for new s3 (bucket/{key}) and sqs (queue/{name}/message) adapters — built fresh, since they don't exist in code today.

/v1/chat/completions, /v1/embeddings, /v1/rerank in internal/proxy/router.go are unchanged — local-LLM-only, no X-Service involvement.

---

3. IAM access to Authentik

Existing System

• No X-Service: iam adapter in the gateway.
• Authentik admin API access exists today only client-side, in the Rust core CLI (/Users/rockliang/workplace/core/src/cmd/iam/*.rs), which already calls {authentik_url}/api/v3/core/applications/, /api/v3/providers/oauth2/, /api/v3/core/flows/, /api/v3/crypto/certificatekeypairs/ for apps/providers/flows — confirmed at core/src/cmd/iam/app.rs:136,160,191,205,219. No users/groups/roles coverage there, and none of it is reachable through the gateway.

Future System

New internal/iam/handler.go server-side adapter, reachable at the gateway root with X-Service: iam, mapping X-Resource + method onto Authentik's /api/v3/... surface:

X-Resource	Method	Authentik endpoint
user	GET/POST	/api/v3/core/users/
user/{id}	GET/PATCH/DELETE	/api/v3/core/users/{id}/
service-account	POST	/api/v3/core/users/service_account/
role (group)	GET/POST	/api/v3/core/groups/
permission	GET/POST	/api/v3/rbac/permissions/
flow	GET	/api/v3/flows/instances/

This is additive to, not a replacement of, the core iam CLI subcommand — different caller (server vs. local CLI), same upstream API.

---

4. Auth: how a caller proves identity

Existing System

• internal/auth/ is an empty directory — zero implementation. tasks/ Phase 3 (JWKS fetch/rotation, Bearer validation, Authentik service account, flag-gated rollout, capability authorization) was retired 2026-08-25 — this doc is now the spec for that work instead of those deleted task files.
• kmsvc.riotpiao.com (separate service) has an auth interceptor (kmsvc-manage/internal/api/interceptors/auth.go, JWKS validation via lestrrat-go/jwx/v2/jwt) but it's never wired in — cmd/server/main.go builds a bare grpc.NewServer() (kmsvc-manage/cmd/server/main.go:84), and the REST surface bypasses gRPC interceptors entirely even once fixed. Useful code to reference, not to copy the deployment of.
• The core CLI already does human OIDC login (core auth login — device-code/OOB against Authentik) and caches a token at ~/.cache/talos/authentik_id_token (core/src/cmd/auth/mod.rs:336-338) — a different token/cache than what §5 adds.

Future System

Implement in internal/auth, with two entry paths converging on one validated-JWT middleware — no separate downstream logic per path:

1. Bearer passthrough — Authorization: Bearer <jwt> sourced from ~/.talos/.riotpiao-auth (written by mwinit, §5), validated against Authentik's JWKS.
2. Service-account password grant — Authentik service-account username/password exchanged for a JWT. Reuse the validation logic already written (but unwired) in kmsvc-manage/internal/api/interceptors/auth.go, not its deployment config.

Retry middleware (internal/resilience/retry.go, new) wraps every outbound upstream call from the internal/serviceadapter dispatcher: blind retry on any 5xx, bounded attempts with backoff, no idempotency-key gating — Temporal (and other backends) own dedup on their side. ServiceAdapter.spec.retryable (default true) is the only opt-out.

---

5. CLI-MWNINIT.md (separate doc)

Existing System

The Rust core CLI (/Users/rockliang/workplace/core) has core auth login (src/cmd/auth/mod.rs) for human OIDC device-code/OOB login, and a separate core iam subcommand tree (src/cmd/iam/) for app/group/scope management. There is no service-account login path and no ~/.talos/ token file — the existing cache is ~/.cache/talos/authentik_id_token, and ~/.talos today only holds Talos node config (TALOSCONFIG, per core/src/config.rs:25-26), unrelated to Authentik JWTs.

Future System

New core mwinit subcommand (new src/cmd/mwinit/mod.rs, same structure as src/cmd/auth/mod.rs — not a new binary), documented in a separate CLI-MWNINIT.md:

• Purpose — authenticate as a dedicated Authentik service account (not a human device-code login) and cache the resulting JWT for use against api.riotpiao.com.
• Two paths: core mwinit login --username <svc-account> --password <...> (Authentik service-account grant, same AUTHENTIK_URL default as core auth login, core/src/cmd/auth/config.rs:18), or core mwinit login --token <existing-jwt> to just re-cache an already-issued token.
• Storage: ~/.talos/.riotpiao-auth — a new, separate path from core auth login's ~/.cache/talos/authentik_id_token, so the two tools never collide.
• Usage: curl -H "Authorization: Bearer $(cat ~/.talos/.riotpiao-auth)" -H 'X-Service: ...' https://api.riotpiao.com/ ....
• Lifecycle: core mwinit status / core mwinit clear (mirroring core auth status/clear); gateway behavior on expired token is 401 + WWW-Authenticate, RFC 9457 problem+json.

---

6. X-Service: memory — poimen-memory

Existing System

poimen-memory (~/workplace/Poimen/memory) is real and already deployed — namespace poimen, Service poimen-memory.poimen.svc.cluster.local:8080, 2 pods Running. Its own README.md claims "design complete, no code yet, 0/37 tasks", which is badly stale: memory-tasks/M3.5.*.md (per that repo's DESIGN.md) show the HTTP API layer (M3.5 phase) is done — GET /memory/projects, GET /memory/projects/{id}/status, GET /memory/query?query=...&level=L1,L2&project=...&limit=..., POST /memory/ingest, GET /memory/skills, GET /memory/skills/{name} all exist and are tested. It is not currently reachable from api.riotpiao.com — the live ingress has only a / catch-all to api-gateway, no /memory path.

Three more endpoints are specced in DESIGN.md but not confirmed built — GET /memory/projects/{id}/notes (M3.5.6, listed in the task table but not in the "is done" set above), POST /memory/context (M3.7, tier-1/2/3 lookup — depends on M3.7.7/M3.7.8 signature+symptom work, not started), and POST /memory/nodes/by-git /by-commit/by-author (M3.5.9, explicitly optional in that repo's own task board). These get CR entries below so the schema is defined once, but the gateway-side task for each is blocked on upstream, not just on our own code — see §7's status column.

It also currently does its own auth: an apikey: header check, exact-match against a stored key, implemented in the app itself (tasks/M3.5.1-http-server.md:36-39) — independent of Kong, which its own architecture diagram assumed sat in front but never actually required for this check to work. There is no NetworkPolicy on it today (k8s/ has none) — the apikey check is the only thing standing between it and any pod in the cluster.

Future System

Per explicit decision: drop poimen-memory's own apikey check entirely and gate it through Authentik instead, with separate read/write capabilities — using the per-method auth.capability override introduced in §1.

apiVersion: gateway.riotpiao.com/v1
kind: ServiceAdapter
metadata:
  name: memory
  namespace: api
spec:
  serviceName: memory              # matches X-Service: memory
  upstream:
    url: http://poimen-memory.poimen.svc.cluster.local:8080
    timeoutSeconds: 15
  auth:
    required: true
    capability: memory:read        # default for every resource below
  retryable: true
  resources:
    - name: query                  # X-Resource: query — "specific documents belonging to a project"
      methods:
        - verb: GET
          upstreamPath: /memory/query
    - name: skill                  # X-Resource: skill or skill/{name}
      methods:
        - verb: GET
          upstreamPath: /memory/skills
        - verb: GET
          upstreamPath: /memory/skills/{name}
    - name: project                # X-Resource: project or project/{id}/status
      methods:
        - verb: GET
          upstreamPath: /memory/projects
        - verb: GET
          upstreamPath: /memory/projects/{id}/status
    - name: ingest                 # X-Resource: ingest — write path
      methods:
        - verb: POST
          upstreamPath: /memory/ingest
          auth:
            capability: memory:write   # overrides the memory:read default

curl -X GET https://api.riotpiao.com/ \
  -H 'Authorization: Bearer <jwt-with-memory:read>' \
  -H 'X-Service: memory' \
  -H 'X-Resource: query' \
  -G --data-urlencode 'query=why did requests over 10KB fail' \
  --data-urlencode 'project=poimen' --data-urlencode 'level=L1,L2'

Required companion changes, in this order — not optional, not cosmetic:

1. Add a NetworkPolicy in namespace poimen restricting ingress on poimen-memory to the api namespace's gateway pod only (mirrors the MinIO-protection rationale from the very first version of this doc). This must land before step 2, or there's a window where the service has no auth of any kind.
2. Remove the apikey: middleware from poimen-memory itself (tasks/M3.5.1-http-server.md's auth hook) — separate change in the ~/workplace/Poimen/memory repo, out of scope for homelab-frontend but a hard prerequisite for this adapter being safe to expose.
3. Provision memory:read / memory:write as real Authentik scopes/capabilities via the X-Service: iam adapter (§3) or core mwinit-issued tokens, so callers can actually be granted one without the other.

---

Verification

• go build ./... and existing tests (internal/config/*_test.go, plus new tests for the rewritten internal/temporal / new internal/serviceadapter, internal/iam, internal/resilience) pass.
• curl smoke tests per adapter (workflow/s3/sqs/iam/memory) against a dev gateway, covering both the legacy /workflow body-action path (still mounted) and the new root + verb+X-Resource path.
• internal/serviceadapter/validate_test.go (new): missing required field → 400 with reason: missing; wrong type (e.g. sql sent as a number) → 400 with type_mismatch; unlisted key under strict: true → 400 with unknown_field; unlisted key under default (strict: false) → passes. Postgres query resource's requestSchema (§1) is the fixture. Confirm a response-side mismatch does not block the response and instead increments serviceadapter_response_schema_mismatch{service,resource}.
• Apply the memory ServiceAdapter CR only after the poimen NetworkPolicy lands and poimen-memory's own apikey middleware is removed (§6's ordering). Confirm a memory:read-scoped token can GET X-Resource: query/skill/project and a memory:write-scoped token can POST X-Resource: ingest, and that a memory:read token gets 403 on ingest.
• Commit the CRD + Role/RoleBinding to k8s/, push to main; confirm the existing api-gw ArgoCD Application auto-syncs them with no homelab changes and no new Application. Apply the example postgres CR; confirm the running gateway pod's informer picks it up and serves X-Service: postgres within the watch/resync latency — no gateway restart, no manual kubectl step.
• core mwinit login end-to-end against dev Authentik; confirm ~/.talos/.riotpiao-auth is written, a root-level X-Service-header call using it succeeds, and ~/.cache/talos/authentik_id_token is untouched.