Gateway pods are ErrImagePull — nodes resolve forgejo.riotpiao.com to a
ClusterIP and time out, so the Service had no endpoints and the host was
returning 503. Kong is still running; this restores it.
Ingress api/api now backs onto api-gateway:8080; the kong Application, its
Helm values, plugins and llm-routes are removed. Gateway image v0.0.0 is in
the Forgejo registry and the pull secret is in the api namespace.
Values changes were inert as a bootstrap release, so the proxy-body-size fix
never reached the live Ingress. First sync is manual — the chart owns the
Forgejo PVC.
Two Ingresses claim forgejo.riotpiao.com and nginx honours the older chart one,
so the annotation on the other never applied and OCI pushes over 1m got 413.
- Runner egress: allow 192.168.1.160/32:443. forgejo.riotpiao.com resolves to
the ingress LB, inside the 192.168.1.0/24 block the NetworkPolicy denies, so
docker push hung until timeout.
- dind CA: also mount homelab-ca at /etc/docker/certs.d/forgejo.riotpiao.com/,
the path dockerd actually reads for per-registry trust.
- Pull secret: dockerconfigjson for the api namespace; /v2/ answers 401.
- AppProject: allow the Forgejo repo as a source for api-gw.
Old i===0 && !resuming gate meant this only ran on a fresh start -- every
run this session was a resume, so poiman's branch never got the harness
gitignore rules, and portfolio's PLAN.md stayed tracked from before the
rule existed (gitignore doesn't affect already-tracked files). Now checks
and fixes both on every phase instead of once at genesis.
Dead sessions were only caught after a full 10-min stall timeout; now
polled via agent-manager status and respawned (retry once). spawnPi had
no timeout and could hang a repo's whole pipeline forever -- bounded to
5 minutes now.
32 blocks was a ~1GB safety-valve leftover from the num_cpu_blocks=2000
hang incident, not meaningful offload capacity. This model's KV cache is
~32MB/128-token block (64 layers, 8 KV heads x 128 head_dim, fp16) --
256 blocks gives ~8GB of real DRAM offload (32,768 tokens), comfortably
under the pod's 36Gi limit alongside the ~20GB bnb-4bit weights.
DeepSeek-R1-distill's tool_choice=auto narration bug needed a real fix,
not a workaround -- Qwen3's native tool-call format (hermes-compatible
chat template) solves it at the source instead of parsing around it.
Dense Qwen3-32B avoids the MoE arch/quantization pitfalls hit by the two
prior swap attempts (Kimi-distilled Qwen3.6 MoE, AWQ Qwen3-30B-A3B) --
same bnb-4bit path already proven working on this sm70 (V100) node.
Three straight failures on worker-1: Kimi-K2.6-distilled Qwen3.6-35B-A3B
had an unrecognized model type (qwen3_5_moe); the AWQ-4bit fallback needed
compute capability 80+ (marlin INT4 kernels) but this node's GPU is sm70
(V100); on-the-fly bitsandbytes against the full-precision Qwen3-30B-A3B
kept crash-looping. Reverting to the last known-good config (596b5cb) --
tool-call narration bug on judge remains open, to revisit separately.
cpatonn's pre-quantized build failed with a real hardware constraint:
"Quantization scheme not supported for current GPU. Min capability: 80.
Current capability: 70." AWQ/GPTQ/compressed-tensors marlin INT4 kernels
all need sm80+ -- this node's GPU can't run any of them. Only bitsandbytes
or full precision work here. Switching to the official full-precision
Qwen/Qwen3-30B-A3B-Thinking-2507 with --quantization=bitsandbytes
on-the-fly, and bumping the memory limit (36Gi->48Gi, request unchanged)
for the transient bf16-shard staging during load.
cpatonn's "AWQ-4bit" repo is actually quantized via llm-compressor --
config.json declares compressed-tensors. Passing awq_marlin explicitly
conflicted with the checkpoint's own declared format and 400d at
config-validation time.
Kimi-K2.6-distilled Qwen3.6-35B-A3B crashed on boot -- model type
qwen3_5_moe unrecognized by transformers/vLLM 0.11.0, a genuinely
unsupported architecture, not a config issue. Using cpatonn's pre-quantized
AWQ-4bit build of the official Qwen3-30B-A3B-Thinking-2507 instead: native
vLLM support confirmed, no Kimi distillation but Qwen3's own tool-call
format is natively supported (the actual root problem being solved).
Restored max-num-seqs=4 since AWQ-4bit weight footprint leaves more KV
headroom than the bnb attempts did.
New model's weight footprint (35B total MoE at on-the-fly bnb-4bit) leaves
less confirmed KV-cache headroom on the 32GB card than the old one had --
reducing concurrent-sequence worst case until real memory use is verified.
R1-family tool_choice=auto is a documented vLLM architecture conflict --
the model narrates fake tool_calls in <think> instead of emitting real
ones, regardless of parser (deepseek_v3 400s, hermes parses but the model
still doesn't call out). Qwen3's native tool-call format sidesteps this.
No pre-quantized AWQ/GPTQ/bnb checkpoint exists for this specific distill
(only GGUF, llama.cpp/Ollama-only) -- using on-the-fly bitsandbytes
quantization against the full bf16 checkpoint instead.
Observed live: phase-judge (on homelab-reasoning) wrote a full page of
'I should check X, then Y' reasoning, declared VERDICT: PASS, and showed
the touch command as a fenced code block in its own text -- never ran
git diff, never wrote the result file, never touched the sentinel.
Coordinator timed out waiting on a file that was never going to appear.
poiman is Rust, portfolio is Go -- neither toolchain was reachable from an
interactive kubectl exec session (go's PATH export was local to its own
install script; rust was entirely absent, and cargo needs gcc as a linker
which also wasn't present).
brave-search was just a curl one-liner wrapped in its own skill file --
inlined the same curl command directly into info-collector/investigator's
instructions instead of dispatching to a separate skill for it.
vLLM 0.11.0's native OffloadingConnector -- spills KV blocks to CPU RAM on
preemption instead of discarding them, avoiding recompute. Built into vLLM
core, no extra dependency. Bumped memory request/limit (+4Gi/replica) to
give the CPU block pool real room; worker-1 had ~18Gi of request headroom
across both replicas.
Resuming the phase branch alone only recovers the code -- the task loop
still walked from the first task, re-verifying every already-done one
through a full planner call before reaching the first task that actually
needed work. .agent-progress is committed (not gitignored) and appended
per completed task, so a resumed run reads it once and skips straight
past known-done tasks with zero LLM calls. Validated locally against a
throwaway repo: second run skipped both tasks instantly (resumed: true)
instead of re-running planner on them.
- agent-manager spawn now gets --group repoId, so the TUI clusters
planner/investigator/implementer/judge under one repo heading instead of
4 unrelated sessions.
- runPhase was called with phaseBranch where it needed the true baseBranch,
so every per-task judge review compared phaseBranch...HEAD -- always
empty, since HEAD is phaseBranch while checked out. Judges only produced
real verdicts anyway because they fell back to their own git log/show.
- Every restart re-cloned baseBranch fresh and started a new phase branch,
discarding whatever a prior run had already committed mid-phase. Now:
fetch+resume an existing phase branch if origin has one, push after every
task instead of only at phase-end, and delete the phase branch (local +
origin) once its milestone squash-merges into base.
A pooled session's shell cwd drifts as it explores the repo between turns.
Seen live: a repo whose internal workspace dir is one letter off from the
repo's own directory name was enough for the agent to touch its sentinel
one level off from where coordinator watches for it -- coordinator waited
out the full timeout for a file that existed, just in the wrong place.
coordinator.js now runs one long-lived planner/investigator/implementer/judge
session per repo (reused across every task via tmux send-keys) instead of a
fresh spawn per task per stage. Tasks within a repo run sequentially against
that pool; concurrency is now REPO_CONCURRENCY (default 3) concurrent repos
via a new --repos flag, not concurrent tasks in one repo's phase.
deepseek_v3 400s on this checkpoint: "could not locate tool call start/end tokens in the tokenizer". unsloth/DeepSeek-R1-Distill-Qwen-32B is a Qwen2.5 base distilled on R1 reasoning traces -- it kept R1's <think> format but never got DeepSeek-V3's own special tool-call tokens registered in its tokenizer. hermes parses from text patterns instead of special tokens, so it works against the underlying Qwen tokenizer.
pi sends tool_choice="auto" for every session (Read/Bash/etc.) -- vLLM 400s on that without --enable-auto-tool-choice and a --tool-call-parser. Verified this deployed vLLM v0.11.0's registered parsers directly; deepseek_v3 matches, same family as the deepseek_r1 reasoning-parser already set (this Qwen-base distillation still emits DeepSeek's own tool-call format).
process.exit() right after console.log() can drop buffered stdout when it's piped (not a TTY) -- exactly kubectl exec's case. Explains the silent empty-output-exit-1 failures. process.exitCode + natural exit lets the event loop drain and flush first.
git clone is mechanical -- routing it through spawnPi meant a crash gave zero diagnostic output, just a silent exit code. Direct runGit call now, same as commitPending/the squash-merge sequence. Drops the now-unused runStageWithResolver.
The pod's coordinator-src ConfigMap still had the pre-auto-discovery version -- --tasks was required, no task-board parsing, no self-chained stages, no judge model routing. Regenerated from the current source.
Both skills already have Bash in allowed-tools -- no new pi capability needed. For UI/frontend work, implementer screenshots/clicks through the golden path via npx playwright instead of trusting that code compiling means it renders correctly; judge does the same as review evidence, FAILing on visual defects a diff alone wouldn't show. Doesn't apply to non-UI work.
reasoning keeps its 2 GPUs untouched. verifier's freed GPU goes to a second ornith replica instead of a standalone qwen-only pod -- both replicas load ornith:35b + qwen2.5:3b-instruct, k8s Service load-balances across them, so 2 concurrent implementer-style calls get independent instances.
Frees verifier's GPU from an underused vLLM PRM deployment. qwen2.5:3b-instruct moves off ornith-predictor's shared pod onto its own dedicated GPU (grm.yaml), so verification/judge traffic stops contending with ornith:35b's agent traffic. /v1/qwen/chat/completions now points at grm-predictor; path unchanged.
Kong key-auth rejected the Authorization: Bearer header every OpenAI-SDK-compatible client sends (verified: raw apikey header works, Bearer doesn't), so it's commented out and stripped from every llm-routes.yaml annotation until there's a Bearer-compatible fix. agent-pod now clones and builds the agent-manager fork from source at container start (no prebuilt binary shipped -- wrong arch and over ConfigMap's size cap) and ships coordinator.js alongside hub.js, so multiple repos can run the pipeline concurrently in one pod via kubectl exec. hub.js keeps its existing role as the container's foreground process, unchanged.
llm-serving-default-deny admits port 8080 only from pods carrying
llm-client=true. Kong lacked it, so every route that actually contacts an
upstream timed out. /v1/models masked the problem: request-termination answers
inside Kong and never touches an upstream, so it returned 200 throughout.
Opting in via podLabels rather than relaxing the policy — it is a compensating
control, not hygiene, since vLLM v0.11.0 is frozen on Volta and will not receive
patches for several remote/unauthenticated advisories.
podLabels land only in the pod template, not spec.selector.matchLabels, so this
is not an immutable-field change.
Story Crater Bot