Fixes controlplane.tftpl's install.wipe:true (should be false, live CPs already run false) and syncs coredns Corefile back to what's actually deployed (drops an unrolled-out, stale Kong-era rewrite).
The poiman repo was renamed to poimen on Forgejo; the stale repoURL made
poimen-root fail with a 301 redirect ComparisonError (ArgoCD's git
client doesn't follow redirects on smart-HTTP fetch), blocking sync for
poimen-root and everything under it.
Every workflow in the cluster has been silently dead. app.ini carried no
[actions] section, so Forgejo never created a run: the API returns
total_count: 0 for rock/homelab and rock/homelab-frontend alike, despite both
repos reporting has_actions: true, cluster-ci.yaml and build.yaml sitting on
their default branches, and forgejo-runner having registered successfully.
Registration does not go through the dispatcher, which is why the runner looks
healthy -- it logs "declared successfully" and "[poller 0] launched" and then
picks up nothing, forever. That reads like a runner or label problem and is
neither.
This also explains why the api-gateway images in the registry were all built
by hand: the pipeline that was supposed to build them has never once run.
Forgejo restarts on this values change; git and the container registry are
briefly unavailable.
A1: Replace per-repo Forgejo entries with https://forgejo.riotpiao.com/rock/*
wildcard so onboarding never requires touching AppProject.
A2: Add wave -1 Application for k8s/argocd/projects/ so it syncs before
any Application references the AppProject.
Also add kustomization.yaml to k8s/argocd/projects/ to make it renderable.
Enabled by Stage 1 (A1, A2).
Kong was retired 2026-08-19, replaced by the rock/homelab-frontend Go
gateway (single /v1/chat/completions endpoint, model routed via the
request body's "model" field per API.md). Old per-model baseUrls
(/v1/ornith, /v1/reasoning, /v1/qwen) all 404 against the new gateway.
Also flipping reasoning's supportsTools to true -- confirmed working via
live test now that reasoning runs Qwen3-32B instead of DeepSeek-R1.
Gateway pods are ErrImagePull — nodes resolve forgejo.riotpiao.com to a
ClusterIP and time out, so the Service had no endpoints and the host was
returning 503. Kong is still running; this restores it.
Ingress api/api now backs onto api-gateway:8080; the kong Application, its
Helm values, plugins and llm-routes are removed. Gateway image v0.0.0 is in
the Forgejo registry and the pull secret is in the api namespace.
Values changes were inert as a bootstrap release, so the proxy-body-size fix
never reached the live Ingress. First sync is manual — the chart owns the
Forgejo PVC.
Two Ingresses claim forgejo.riotpiao.com and nginx honours the older chart one,
so the annotation on the other never applied and OCI pushes over 1m got 413.
- Runner egress: allow 192.168.1.160/32:443. forgejo.riotpiao.com resolves to
the ingress LB, inside the 192.168.1.0/24 block the NetworkPolicy denies, so
docker push hung until timeout.
- dind CA: also mount homelab-ca at /etc/docker/certs.d/forgejo.riotpiao.com/,
the path dockerd actually reads for per-registry trust.
- Pull secret: dockerconfigjson for the api namespace; /v2/ answers 401.
- AppProject: allow the Forgejo repo as a source for api-gw.
Old i===0 && !resuming gate meant this only ran on a fresh start -- every
run this session was a resume, so poiman's branch never got the harness
gitignore rules, and portfolio's PLAN.md stayed tracked from before the
rule existed (gitignore doesn't affect already-tracked files). Now checks
and fixes both on every phase instead of once at genesis.
Dead sessions were only caught after a full 10-min stall timeout; now
polled via agent-manager status and respawned (retry once). spawnPi had
no timeout and could hang a repo's whole pipeline forever -- bounded to
5 minutes now.
32 blocks was a ~1GB safety-valve leftover from the num_cpu_blocks=2000
hang incident, not meaningful offload capacity. This model's KV cache is
~32MB/128-token block (64 layers, 8 KV heads x 128 head_dim, fp16) --
256 blocks gives ~8GB of real DRAM offload (32,768 tokens), comfortably
under the pod's 36Gi limit alongside the ~20GB bnb-4bit weights.
DeepSeek-R1-distill's tool_choice=auto narration bug needed a real fix,
not a workaround -- Qwen3's native tool-call format (hermes-compatible
chat template) solves it at the source instead of parsing around it.
Dense Qwen3-32B avoids the MoE arch/quantization pitfalls hit by the two
prior swap attempts (Kimi-distilled Qwen3.6 MoE, AWQ Qwen3-30B-A3B) --
same bnb-4bit path already proven working on this sm70 (V100) node.
Three straight failures on worker-1: Kimi-K2.6-distilled Qwen3.6-35B-A3B
had an unrecognized model type (qwen3_5_moe); the AWQ-4bit fallback needed
compute capability 80+ (marlin INT4 kernels) but this node's GPU is sm70
(V100); on-the-fly bitsandbytes against the full-precision Qwen3-30B-A3B
kept crash-looping. Reverting to the last known-good config (596b5cb) --
tool-call narration bug on judge remains open, to revisit separately.
cpatonn's pre-quantized build failed with a real hardware constraint:
"Quantization scheme not supported for current GPU. Min capability: 80.
Current capability: 70." AWQ/GPTQ/compressed-tensors marlin INT4 kernels
all need sm80+ -- this node's GPU can't run any of them. Only bitsandbytes
or full precision work here. Switching to the official full-precision
Qwen/Qwen3-30B-A3B-Thinking-2507 with --quantization=bitsandbytes
on-the-fly, and bumping the memory limit (36Gi->48Gi, request unchanged)
for the transient bf16-shard staging during load.
cpatonn's "AWQ-4bit" repo is actually quantized via llm-compressor --
config.json declares compressed-tensors. Passing awq_marlin explicitly
conflicted with the checkpoint's own declared format and 400d at
config-validation time.
Kimi-K2.6-distilled Qwen3.6-35B-A3B crashed on boot -- model type
qwen3_5_moe unrecognized by transformers/vLLM 0.11.0, a genuinely
unsupported architecture, not a config issue. Using cpatonn's pre-quantized
AWQ-4bit build of the official Qwen3-30B-A3B-Thinking-2507 instead: native
vLLM support confirmed, no Kimi distillation but Qwen3's own tool-call
format is natively supported (the actual root problem being solved).
Restored max-num-seqs=4 since AWQ-4bit weight footprint leaves more KV
headroom than the bnb attempts did.
Story Crater Bot