homelab-frontend/tasks/6.5-cutover.md

6.5 — Cutover: repoint Ingress api/api (GREEN)

Phase: 6 — Deploy and cutover Stage: GREEN Depends on: 2.8, 6.4

THIS is the cutover. Ingress api/api catch-alls / on api.riotpiao.com; today its backend is kong-proxy:80. Changing that one backend moves all live traffic to the gateway. Reverting is the same one-line change back to kong-proxy:80, committed and synced — Kong stays running and untouched, so the revert takes effect in seconds.

Preconditions, all required before starting:

• 2.8 Kong parity is passing — gateway and Kong return equivalent responses for every route in the migration inventory, including a streaming request and a mid-stream disconnect
• 6.4 is complete: the gateway is healthy in-cluster against the real predictors, and Prometheus is scraping it
• The inventory has been re-verified immediately beforehand. Kong's Deployment has eleven ReplicaSets and is being actively iterated; a stale inventory is a stale plan
• Kong remains deployed and serving-capable throughout. Nothing about Kong is deleted in this task

The change itself:

• The api/api Ingress backend becomes the gateway Service, changed in git and synced by Argo. No kubectl apply, no kubectl edit (G7)
• The nginx annotations stay exactly as they are: proxy-read-timeout: 3600, proxy-send-timeout: 3600, proxy-buffering: off, proxy-body-size: 0. These are what make token streaming work and the gateway needs the same treatment
• The revert is a single-line commit reverting the backend to kong-proxy:80, and it has been rehearsed at least once
• After the change, public streaming works end to end through nginx, unbuffered
• pi keeps working on the legacy /v1/{reasoning,ornith,qwen}/chat/completions aliases
• A soak period follows, watching gateway metrics, rejection counters and pi traffic. Do not proceed to 6.6 until the soak is clean

Verify

kubectl -n api get ingress api -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}{"\n"}'
# expected: the gateway Service, no longer kong-proxy

curl -sN -X POST https://api.riotpiao.com/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"reasoning","stream":true,"messages":[{"role":"user","content":"count to 5"}]}'
# expected: incremental data: chunks over time through nginx, ending in data: [DONE]

curl -s -o /dev/null -w '%{http_code}\n' -X POST https://api.riotpiao.com/v1/reasoning/chat/completions \
  -H 'content-type: application/json' -d '{"messages":[{"role":"user","content":"hi"}]}'
# expected: 200 — pi's legacy alias still works

kubectl -n api get deploy kong-kong -o jsonpath='{.status.readyReplicas}{"\n"}'
# expected: 2 — Kong still running and ready, so the revert is one commit away