coordinator.js now runs one long-lived planner/investigator/implementer/judge
session per repo (reused across every task via tmux send-keys) instead of a
fresh spawn per task per stage. Tasks within a repo run sequentially against
that pool; concurrency is now REPO_CONCURRENCY (default 3) concurrent repos
via a new --repos flag, not concurrent tasks in one repo's phase.
deepseek_v3 400s on this checkpoint: "could not locate tool call start/end tokens in the tokenizer". unsloth/DeepSeek-R1-Distill-Qwen-32B is a Qwen2.5 base distilled on R1 reasoning traces -- it kept R1's <think> format but never got DeepSeek-V3's own special tool-call tokens registered in its tokenizer. hermes parses from text patterns instead of special tokens, so it works against the underlying Qwen tokenizer.
pi sends tool_choice="auto" for every session (Read/Bash/etc.) -- vLLM 400s on that without --enable-auto-tool-choice and a --tool-call-parser. Verified this deployed vLLM v0.11.0's registered parsers directly; deepseek_v3 matches, same family as the deepseek_r1 reasoning-parser already set (this Qwen-base distillation still emits DeepSeek's own tool-call format).
process.exit() right after console.log() can drop buffered stdout when it's piped (not a TTY) -- exactly kubectl exec's case. Explains the silent empty-output-exit-1 failures. process.exitCode + natural exit lets the event loop drain and flush first.
git clone is mechanical -- routing it through spawnPi meant a crash gave zero diagnostic output, just a silent exit code. Direct runGit call now, same as commitPending/the squash-merge sequence. Drops the now-unused runStageWithResolver.
The pod's coordinator-src ConfigMap still had the pre-auto-discovery version -- --tasks was required, no task-board parsing, no self-chained stages, no judge model routing. Regenerated from the current source.
Both skills already have Bash in allowed-tools -- no new pi capability needed. For UI/frontend work, implementer screenshots/clicks through the golden path via npx playwright instead of trusting that code compiling means it renders correctly; judge does the same as review evidence, FAILing on visual defects a diff alone wouldn't show. Doesn't apply to non-UI work.
reasoning keeps its 2 GPUs untouched. verifier's freed GPU goes to a second ornith replica instead of a standalone qwen-only pod -- both replicas load ornith:35b + qwen2.5:3b-instruct, k8s Service load-balances across them, so 2 concurrent implementer-style calls get independent instances.
Frees verifier's GPU from an underused vLLM PRM deployment. qwen2.5:3b-instruct moves off ornith-predictor's shared pod onto its own dedicated GPU (grm.yaml), so verification/judge traffic stops contending with ornith:35b's agent traffic. /v1/qwen/chat/completions now points at grm-predictor; path unchanged.
Kong key-auth rejected the Authorization: Bearer header every OpenAI-SDK-compatible client sends (verified: raw apikey header works, Bearer doesn't), so it's commented out and stripped from every llm-routes.yaml annotation until there's a Bearer-compatible fix. agent-pod now clones and builds the agent-manager fork from source at container start (no prebuilt binary shipped -- wrong arch and over ConfigMap's size cap) and ships coordinator.js alongside hub.js, so multiple repos can run the pipeline concurrently in one pod via kubectl exec. hub.js keeps its existing role as the container's foreground process, unchanged.
llm-serving-default-deny admits port 8080 only from pods carrying
llm-client=true. Kong lacked it, so every route that actually contacts an
upstream timed out. /v1/models masked the problem: request-termination answers
inside Kong and never touches an upstream, so it returned 200 throughout.
Opting in via podLabels rather than relaxing the policy — it is a compensating
control, not hygiene, since vLLM v0.11.0 is frozen on Volta and will not receive
patches for several remote/unauthenticated advisories.
podLabels land only in the pod template, not spec.selector.matchLabels, so this
is not an immutable-field change.
Kong matches routes on host/path/method/header, never on the request body, so a
single /v1/chat/completions dispatching on body.model is not expressible in Kong
OSS (ai-proxy-advanced, which does multi-target model routing, is Enterprise).
Model therefore goes in the path:
GET /v1/models static list (request-termination)
POST /v1/reasoning/chat/completions reasoning-predictor (vLLM)
POST /v1/ornith/chat/completions ornith-predictor (Ollama)
POST /v1/qwen/chat/completions ornith-predictor (Ollama, same pod)
POST /v1/embeddings embeddings-predictor (TEI)
POST /v1/rerank reranker-predictor (TEI)
POST /v1/score verifier-predictor (vLLM pooling)
- each chat route force-overwrites body.model via request-transformer add+replace:
ornith:35b and qwen2.5:3b-instruct share one Ollama pod, so without this a
client hitting /v1/qwen with "model":"ornith:35b" would silently get the 35B
- routes live in ns llm-serving, not api: an Ingress can only reference a Service
in its own namespace, and KIC watches all namespaces
- embeddings and score need no rewrite (TEI/vLLM already serve the canonical
paths); rerank does, since /v1/rerank 404s and only /rerank exists
- read/write timeouts 1h: Kong defaults to 60s, which a 32B model on Volta
exceeds mid-generation and returns 504
- nginx_proxy_proxy_buffering=off: buffered responses lump or stall SSE, and both
hops (nginx Ingress and Kong) must be unbuffered or the buffered one wins
- no auth for now, per decision; api.riotpiao.com is reachable through nginx, so
GPU time is currently unauthenticated
- namespace: PodSecurity privileged, needed for /dev/kvm + privileged QEMU
- storageclass: 1 replica, strict-local, WaitForFirstConsumer
- deployment: nodeSelector workload=imessage + matching NoSchedule toleration,
Recreate strategy (two QEMU procs on one qcow2 corrupts it), no readiness
probe (guest install is interactive and takes many minutes)
- services: ClusterIP only; VNC is an unauthenticated console, reach it with
port-forward, never an Ingress
- networkpolicy: default-deny, opt-in via sms-client=true on port 1234
argocd-server ran --insecure (plain HTTP :8080) while its Helm-managed
Ingress set ssl-passthrough: true, which sends nginx's raw TLS handshake
straight to the pod - HTTP server can't complete a TLS handshake, nginx
logged 502 (peer closed connection in SSL handshake). Compounded by a
second, conflicting Ingress for the same host in
k8s/bootstrap/ingress/ingress.yaml - two Ingress objects on one host is
undefined nginx routing behavior. Disabled the Helm-managed Ingress
(enabled: false) so ingress.yaml's passthrough Ingress is the sole
source of truth, and set server.insecure: false so argocd-server
actually terminates TLS itself, matching passthrough's requirement.
ArgoCD directory.include uses Go filepath.Match glob syntax, not shell
brace expansion - {a,b,c} silently matched nothing, only the original 2
files stayed tracked.
homelab-ca was referenced by 6 manifests (authentik, forgejo-runner,
blackbox-exporter, management-service) as a CA trust ConfigMap but never
existed anywhere - not in git, not live in cluster. Generated a new
10-year self-signed root CA, wired it as a ClusterIssuer (cert-manager
namespace) and distributed the public cert as a ConfigMap to every
consuming namespace (iam, cicd, monitoring, sqs). Private key lives only
in the encrypted Secret. Widened cert-manager-issuers' directory include
glob rather than creating a new Application - destination.namespace is
just a fallback default on a plain directory source, not a transformer,
so it doesn't fight with each ConfigMap's own explicit namespace.
Also adds grafana-oidc secret (GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET),
same pre-existing gap as grafana-admin - was meant to come from a deleted
manual script, value already available in .env.
nodeSelector still targeted az-a/talos-cp-1 from before the 3-CP topology
change. talos-cp-2 (az-b) has the dedicated Longhorn disks now, so the
pod's zone pin and the PVC's only viable replica location never matched
- ReplicaSchedulingFailure: disks are unavailable, pod stuck
ContainerCreating waiting on AttachVolume.
Vault's S3 storage backend needs AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY
from vault-minio-creds, previously generated by a helmfile presync hook
that no longer exists post-Terraform/helmfile removal. Sourced from the
same MINIO_ROOT_USER/PASSWORD already in .env. vault-unseal-keys still
missing separately — needs a live 'vault operator init' run, deferred.
PVC sat Pending for 17 days — storageclass.storage.k8s.io "longhorn-wffc"
not found. Only longhorn, longhorn-cnpg, longhorn-static exist. Straight
naming drift, no such class was ever created.
Story Crater Bot