Ingress only covered /console -- agent-console run got a 404 hitting
/run since Kong had no route for it. Same Service, same port, just
more paths on the one Ingress.
agent-console complained about the two-step workflow: kubectl exec a
script to spawn an agent, separate terminal to watch it. Root cause
was hub being a separate Go sidecar with no access to the pi binary
or its config, so it could only watch, never trigger.
hub.js now runs inside the pi container itself (same filesystem, same
PATH) and exposes POST /run, which spawns `pi -p --mode json` directly
and streams it over the same /console WebSocket everyone else is
already watching. agent-console gained a `run <agent> <prompt>`
subcommand that POSTs then watches -- no separate exec step, no
separate terminal. Drops the golang:1.25-alpine sidecar container and
agent-run-configmap.yaml entirely (both superseded).
ConfigMaps regenerated via local-harness/hub/sync-to-homelab.sh
instead of hand-copied, so hub.js/config.json/settings.json each have
exactly one source of truth.
JSON.stringify({id, event: line}) treated the already-JSON `line` as a
plain string, so every event landed in the hub double-encoded (a JSON
string containing escaped JSON, not an object) -- agent-console's
json.Unmarshal into a struct silently failed on every single event.
Now the raw JSON line is spliced directly into the request body.
Also await all in-flight event posts before posting /agent/end, since
those POSTs were fire-and-forget and could reorder past it on the wire.
Hub relays pi's own --mode json session protocol (session/agent_start/
turn_start/message_*/turn_end/agent_end -- the same event shape Claude
Code sessions use) to agent-console over WebSocket, so concurrent pi
runs inside the pod are observable as real transcripts instead of log
tails. agent-run.js wraps `pi -p --mode json` and relays its stdout
lines to the hub; each invocation gets its own session id, so N
concurrent agents show up as N sessions. Source mounted via ConfigMap
and run with `go run .` (no registry yet, same as the pi container).
Cluster-wide prometheus KongClusterPlugin (kong-metrics.yaml) +
chart-native ServiceMonitor (kong-values.yaml) expose
kong_http_requests_total/kong_latency_bucket/kong_bandwidth_bytes for
every route, LLM and otherwise. Dashboard filters to route=~"llm-.*"
for request rate, error rate, p95 upstream latency, and bandwidth.
Token-count metrics still need ai-proxy-advanced (Enterprise-only);
not attempted.
Resources (CPU/mem/restarts) + Loki logs across llm-serving, agent-pod,
and api (Kong) namespaces, mirroring the svc-*.yaml dashboard pattern.
No request-rate/latency/token panels yet -- no ServiceMonitor exists
for Kong or the KServe predictors, so there's no metric to query.
pi's openai-completions handler (pi-ai/dist/api/openai-completions.js)
reads model.headers (populated from a provider-level "headers" field
in the config), not compat.customHeaders -- that field only gets read
by the Bedrock handler. Without it, pi used the openai SDK's default
Authorization: Bearer <apiKey>, which Kong's key-auth plugin rejects
(verified via direct fetch: apikey header -> 200, Bearer -> 401,
plugin does not strip the Bearer prefix). Moved the apikey header to
the correct provider-level "headers" field.
api.riotpiao.com has no in-cluster DNS record (getaddrinfo ENOTFOUND
from inside the pod, confirmed against google.com resolving fine and
kong-proxy.api.svc.cluster.local resolving fine) -- it only resolves
via the home network's own DNS. Node's fetch/undici also silently
drops a manually-set Host header (forbidden header per WHATWG fetch
spec), so overriding Host per-request isn't viable either. hostAliases
pinning the hostname to ingress-nginx-controller's ClusterIP lets
pi's models.json baseUrl work completely unchanged -- TLS SNI/Host
still say api.riotpiao.com so cert validation and Kong's Host-based
routing both still work.
pi-coding-agent requires node >=22.19.0 (undici's
webidl.util.markAsUncloneable, added in Node 22) -- node:20-slim
installed fine but crashed on every invocation.
0.1.23 was pi's own internal version string (from local pi --version),
not the npm package's semver -- npm has no such release, so the
container crash-looped on ETARGET. Latest published is 0.84.2.
Secret was created but never listed in secret-generator.yaml's files
allowlist, so ksops never decrypted it — pod stuck on FailedMount
waiting for secret "pi-models".
Namespace + Deployment (2-4 CPU, 4-8Gi mem) running node:20-slim with
pi installed at startup, wired to the homelab-ornith/reasoning/qwen
model providers via the existing model-invoke-apikey. Placeholder
node:20-slim image for now, real harness image to follow.
StorageClass.parameters is immutable and mkfsParams was added after creation,
so every sync failed. Replace=true recreates it instead of patching. Existing
volumes keep their format; only new ones get mkfsParams.
Longhorn names its own disk key and writes storageReserved into it, so git's
default-disk never matched and selfHeal kept trying to add a second disk on the
same path. Dropped spec.disks from git, added ignoreDifferences.
Two bugs. Kong timeouts were on the Ingress; it reads them from the Service, so
its 60s default applied. Moved to the isvc, which KServe propagates.
Probes ran 'ollama list' — models on disk, not in VRAM — so the pod went Ready
before it could serve. Now 'ollama ps', and both models are warmed at startup.
Five model servers were applied by hand and tracked nowhere. Exported live,
kubectl diff empty on all five, so the first sync adopts without restarting.
prune: false — KServe copies isvc labels to its child Deployment, so ArgoCD
would prune children it does not own and KServe would recreate them.
Two independent bugs, both silent, both found while pointing an agent harness
at api.riotpiao.com.
1. Requests over ~10.6KB failed with HTTP 400
{"error":{"message":"[] is too short - 'messages'"}}.
The request-transformer plugin on the llm-chat-* routes rewrites the JSON
body, which means it reads it via kong.request.get_body(). That returns
nothing once nginx spills the body past client_body_buffer_size into a temp
file, so the plugin re-serialized a body with no `messages` and the upstream
rejected it. Measured on /v1/ornith/chat/completions: 10588 B -> 200,
11088 B -> 400. Isolated by size-sweeping /v1/embeddings, the one route with
no request-transformer, which passed an 18057 B body straight through to a
semantic 413 from TEI.
Raises the Kong http-block buffer to 16m. Any agent request carrying tool
schemas clears the old ceiling in a single turn.
2. key-auth was never applied to any model route.
The model-key-auth KongPlugin sat in namespace `api` while all five routes
that annotate it live in `llm-serving`. The ingress controller resolves
konghq.com/plugins against the annotated object's own namespace, so the
reference dangled and the plugin never bound. Verified before the fix:
unauthenticated GET /v1/models and POST /v1/ornith/chat/completions both
returned 200. A dangling plugin reference fails open and logs nothing.
Re-test both without a key after this syncs; expect 401.
Note for follow-up: llm-embeddings and llm-score carry no plugins annotation at
all, so they stay unauthenticated even after this change.
Co-Authored-By: Claude Opus 5 <[email protected]>
llm-serving-default-deny admits port 8080 only from pods carrying
llm-client=true. Kong lacked it, so every route that actually contacts an
upstream timed out. /v1/models masked the problem: request-termination answers
inside Kong and never touches an upstream, so it returned 200 throughout.
Opting in via podLabels rather than relaxing the policy — it is a compensating
control, not hygiene, since vLLM v0.11.0 is frozen on Volta and will not receive
patches for several remote/unauthenticated advisories.
podLabels land only in the pod template, not spec.selector.matchLabels, so this
is not an immutable-field change.
Kong matches routes on host/path/method/header, never on the request body, so a
single /v1/chat/completions dispatching on body.model is not expressible in Kong
OSS (ai-proxy-advanced, which does multi-target model routing, is Enterprise).
Model therefore goes in the path:
GET /v1/models static list (request-termination)
POST /v1/reasoning/chat/completions reasoning-predictor (vLLM)
POST /v1/ornith/chat/completions ornith-predictor (Ollama)
POST /v1/qwen/chat/completions ornith-predictor (Ollama, same pod)
POST /v1/embeddings embeddings-predictor (TEI)
POST /v1/rerank reranker-predictor (TEI)
POST /v1/score verifier-predictor (vLLM pooling)
- each chat route force-overwrites body.model via request-transformer add+replace:
ornith:35b and qwen2.5:3b-instruct share one Ollama pod, so without this a
client hitting /v1/qwen with "model":"ornith:35b" would silently get the 35B
- routes live in ns llm-serving, not api: an Ingress can only reference a Service
in its own namespace, and KIC watches all namespaces
- embeddings and score need no rewrite (TEI/vLLM already serve the canonical
paths); rerank does, since /v1/rerank 404s and only /rerank exists
- read/write timeouts 1h: Kong defaults to 60s, which a 32B model on Volta
exceeds mid-generation and returns 504
- nginx_proxy_proxy_buffering=off: buffered responses lump or stall SSE, and both
hops (nginx Ingress and Kong) must be unbuffered or the buffered one wins
- no auth for now, per decision; api.riotpiao.com is reachable through nginx, so
GPU time is currently unauthenticated
- namespace: PodSecurity privileged, needed for /dev/kvm + privileged QEMU
- storageclass: 1 replica, strict-local, WaitForFirstConsumer
- deployment: nodeSelector workload=imessage + matching NoSchedule toleration,
Recreate strategy (two QEMU procs on one qcow2 corrupts it), no readiness
probe (guest install is interactive and takes many minutes)
- services: ClusterIP only; VNC is an unauthenticated console, reach it with
port-forward, never an Ingress
- networkpolicy: default-deny, opt-in via sms-client=true on port 1234
Story Crater Bot