feat(llm-serving): add coder, serving Qwen3-Coder-30B-A3B on one V100

The name now matches the model. `coder` serves Qwen3-Coder-30B-A3B-Instruct, a 30.5B MoE with 3.3B active, quantized to 4-bit GPTQ. It is measured working on this node, not assumed. Measured, on worker-1, driver 580.159.04, compute capability 7.0: GET /v1/models -> id=coder, root=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A POST /v1/chat/completions model=coder -> 200 in 3.18s, "fn main() { println!(\"hi\"); }" POST /v1/chat/completions model=reasoning -> 200 in 0.21s Weights loaded in 15.61 GiB. torch.compile took 78s. The pod reached 1/1 Ready. `kubectl diff -f coder.yaml` returns empty, so the manifest matches live state. Two names are served on purpose: `coder` and `reasoning`. The rename therefore needs no change to poimen-memory's LLM_MODEL, only its host. That keeps the client working through the cutover. Why not Qwen3-Coder-Next, the model in the original plan. It cannot run here, and that is measured too. A v0.15.0 image on this free V100 gave: TORCH 2.9.1+cu129 ARCH_LIST [] <- empty RuntimeError: cudaGetDeviceCount() ... Error 803 fla: ModuleNotFoundError <- no GatedDeltaNet kernels vLLM 0.15.0 requires compute capability 7.5 or higher and fails here before it loads a model. Qwen3-Next's hybrid attention needs the `fla` module, which the image does not ship. So this stays on the pinned v0.11.0 image, which reports sm_70..sm_120 and works. Qwen publishes this model as BF16 (61.1 GB) and FP8 only. V100 has no FP8, and 61.1 GB does not fit 32 GB, so 4-bit GPTQ is the only form that fits one card. The checkpoint is a community quant; the manifest says so and names the reason. reasoning.yaml stays listed until poimen-memory's LLM_CHAT_BASE moves from reasoning-predictor to coder-predictor. Two GPUs are free, so coder rolled out beside reasoning with no downtime.
fix(llm-serving): allow poimen namespace ingress so memory ingest works
2026-10-31 15:26:08 -07:00 · 2026-10-31 14:58:39 -07:00
17 changed files with 163 additions and 535 deletions
@@ -4,9 +4,7 @@ metadata:
  name: comfyui
  namespace: comfyui
  annotations:
-    # No cert-manager annotation and no tls: block. This host is covered by
-    # the *.riotpiao.com wildcard, which nginx serves as its
-    # default-ssl-certificate (ingress-nginx/riotpiao-com-tls).
+    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
@@ -18,6 +16,10 @@ metadata:
      proxy_set_header Connection "upgrade";
 spec:
  ingressClassName: nginx
+  tls:
+    - secretName: comfyui-tls
+      hosts:
+        - comfyui.riotpiao.com
  rules:
    - host: comfyui.riotpiao.com
      http:
@@ -0,0 +1,118 @@
+apiVersion: serving.kserve.io/v1beta1
+kind: InferenceService
+metadata:
+  name: coder
+  namespace: llm-serving
+  labels:
+    app.kubernetes.io/name: llm-coder
+    app.kubernetes.io/part-of: llm-serving
+  annotations:
+    serving.kserve.io/deploymentMode: RawDeployment
+spec:
+  predictor:
+    minReplicas: 1
+    maxReplicas: 1
+    nodeSelector:
+      kubernetes.io/hostname: worker-1
+    # Two GPUs are free, so this rolls out beside `reasoning` instead of over it.
+    # Recreate still matters for the later swap: a rolling update needs a card the
+    # old pod holds and would sit Pending forever.
+    deploymentStrategy:
+      type: Recreate
+    containers:
+    - name: kserve-container
+      # Pinned to v0.11.0 on purpose. This is the last image measured to work on this
+      # node: torch 2.8.0+cu128, arch list sm_70..sm_120, driver 580.159.04.
+      # v0.15.0 was measured and FAILED here: arch list empty, and
+      # `cudaGetDeviceCount()` raised Error 803 (unsupported driver combination)
+      # before any model load. Do not raise this tag without re-running that probe.
+      image: vllm/vllm-openai:v0.11.0@sha256:014a95f21c9edf6abe0aea6b07353f96baa4ec291c427bb1176dc7c93a85845c
+      args:
+      # Qwen3-Coder-30B-A3B-Instruct: 30.5B total, 3.3B active. Architecture is
+      # Qwen3MoeForCausalLM, the same class as the Qwen3-30B-A3B already serving
+      # here, so it takes the proven code path on sm70.
+      #
+      # Qwen ships this model only as BF16 (61.1 GB) and FP8 (no FP8 on Volta).
+      # 4-bit GPTQ is the only form that fits one 32 GB card. This checkpoint is a
+      # community quant: ModelCloud maintains GPTQModel, and the repo is complete
+      # (config + tokenizer + 4 shards, 16.9 GB). Verified by loading, not by trust.
+      #
+      # Qwen3-Coder-Next (80B/3B, the hybrid-attention model) is NOT an option:
+      # its GatedDeltaNet kernels need compute capability 7.5+ and the `fla` module,
+      # and sm70 has neither. Measured, not assumed.
+      - --model=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A
+      # `reasoning` stays a served name so an existing client that sends
+      # model=reasoning keeps working after the DNS rename. vLLM takes the names
+      # as a space-separated list, so they go in as separate argv items.
+      - --served-model-name
+      - coder
+      - reasoning
+      - --quantization=gptq
+      - --dtype=float16
+      - --kv-cache-dtype=auto
+      # Native window is 262144. 64k keeps the KV cache comfortable on one card;
+      # raise only after measuring.
+      - --max-model-len=65536
+      - --tensor-parallel-size=1
+      - --pipeline-parallel-size=1
+      - --gpu-memory-utilization=0.90
+      - --max-num-seqs=16
+      - --max-num-batched-tokens=8192
+      - --long-prefill-token-threshold=2048
+      - --enable-chunked-prefill
+      - --enable-prefix-caching
+      - --reasoning-parser=qwen3
+      - --enable-auto-tool-choice
+      - --tool-call-parser=hermes
+      - --host=0.0.0.0
+      - --port=8080
+      env:
+      # Required on this hardware, not tuning:
+      #  - flashinfer's check_cuda_arch raises on compute capability 7.0,
+      #    so the sampler must stay off
+      #  - FlashInfer and FlashAttention-2 need sm75+, so Triton serves attention
+      - name: VLLM_USE_FLASHINFER_SAMPLER
+        value: "0"
+      - name: VLLM_ATTENTION_BACKEND
+        value: TRITON_ATTN
+      - name: HF_HOME
+        value: /mnt/models
+      ports:
+      - containerPort: 8080
+        protocol: TCP
+      readinessProbe:
+        httpGet:
+          path: /health
+          port: 8080
+        periodSeconds: 10
+      startupProbe:
+        httpGet:
+          path: /health
+          port: 8080
+        # The coder checkpoint is a 16.9 GB cold download on first start, then
+        # ~20 s of weight load and ~80 s of torch.compile. 80 x 15 s covers it.
+        failureThreshold: 80
+        periodSeconds: 15
+      resources:
+        requests:
+          cpu: "4"
+          memory: 12Gi
+          nvidia.com/gpu: "1"
+        limits:
+          cpu: "8"
+          memory: 36Gi
+          nvidia.com/gpu: "1"
+      volumeMounts:
+      - mountPath: /mnt/models
+        name: models
+      - mountPath: /dev/shm
+        name: shm
+    runtimeClassName: nvidia
+    volumes:
+    - name: models
+      persistentVolumeClaim:
+        claimName: llm-models
+    - name: shm
+      emptyDir:
+        medium: Memory
+        sizeLimit: 2Gi
@@ -10,8 +10,12 @@ kind: Kustomization
 # way after any edit here: a GPU predictor restart is a weights reload measured
 # in tens of seconds, not a rolling update.
 resources:
+  - coder.yaml
  - embeddings.yaml
  - ornith.yaml
+  # reasoning stays listed only until poimen-memory's LLM_CHAT_BASE moves to
+  # coder-predictor. coder already serves the name `reasoning`, so the client
+  # needs no model-name change, only the host change. Drop this line after that.
  - reasoning.yaml
  - reranker.yaml
  - qwen-cpu.yaml
@@ -66,6 +66,19 @@ spec:
          port: 80
        - protocol: TCP
          port: 8000
+    # Allow from poimen namespace (poimen-memory ingest + query).
+    # The client-side egress rule already exists and is not the gap:
+    # NetworkPolicy poimen/poimen-memory-llm-egress permits port 8080 to pods
+    # labelled llm-reasoning, llm-embeddings and llm-reranker. This policy had no
+    # matching ingress rule, so the packet was allowed out and dropped here.
+    # The client then sees a bare timeout and the predictor log stays empty.
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: poimen
+      ports:
+        - protocol: TCP
+          port: 8080
    # Allow intra-namespace (pod-to-pod within llm-serving)
    - from:
        - podSelector:
@@ -249,26 +249,3 @@ spec:
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
---
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: local-path-provisioner
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: "1"
-spec:
-  project: homelab
-  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    targetRevision: main
-    path: k8s/infra/local-path
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: local-path-storage
-  # No CreateNamespace: the Namespace object (with its privileged PSA label,
-  # which the helper Pods need for their hostPath) is part of the manifest.
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
@@ -1,36 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: azhi-riotpiao
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: '8'
-spec:
-  project: homelab
-  source:
-    # Private GitHub. The read-only credential is Secret azhi-infra-github in
-    # namespace argocd, created out-of-band.
-    repoURL: https://github.com/ToyoGlobal/azhi-infra.git
-    # The repository's default branch. It is not `main`: this repo's work
-    # happens on the staging slice branch, and `main` is behind it.
-    targetRevision: feat/azi-riotpiao-cnpg
-    # One directory, because ArgoCD reads a single path per source. That
-    # directory is a root kustomization over foundation, postgres and
-    # azhi-server.
-    path: k8s/overlays/azi-riotpiao
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: azi-riotpiao
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-    - CreateNamespace=true
-    # The two Secrets this namespace needs are credentials, so they are created
-    # by hand and NOT by this Application:
-    #   acr-pull                 — the private Alibaba ACR pull credential
-    #   azhi-riotpiao-app-env    — DB credentials, JWT secrets, AES key
-    # A missing imagePullSecret fails the pod outright and a missing app-env
-    # Secret leaves it in CreateContainerConfigError, which is the intended
-    # loud failure rather than a pod that runs against unknown values.
@@ -16,12 +16,6 @@ spec:
    - https://github.com/Riotpiaole/Poimen-memory.git
    - https://github.com/Riotpiaole/Poimen-workflows.git
    - https://github.com/Riotpiaole/poimen*.git
-    # ToyoGlobal (GitHub, private). The azhi-infra Application reads its
-    # manifests from here, so the URL must be listed or ArgoCD rejects the
-    # Application with "repository not permitted". A read-only credential for
-    # this repo lives in Secret azhi-infra-github in namespace argocd; it is
-    # created out-of-band and is not committed, because it is a credential.
-    - https://github.com/ToyoGlobal/azhi-infra.git
    # In-cluster Forgejo repos — explicit allowlist (no wildcard)
    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
@@ -1,27 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-    name: etcd-defrag-creds
-    namespace: kube-system
-    labels:
-        app.kubernetes.io/name: etcd-defrag
-type: Opaque
-stringData:
-    ca.crt: ENC[AES256_GCM,data:vso/Ouz5QRoRF2CFj9TsGYhoGkX8oGJIeiFCoS92AAQbyyGPTiZ/fPNFM03p+TpeKDb+Z+Fyo/HXuDeT5Fee5kjPnUHXf2hiUBmJLyHnPMp5i57dRzTC8X4VaNeV/1MmP2WpkWwieWOGBxJVIIrgeQhQexxmIzSlQ5Io9dVOH8F6iyakwdf7qfLFGdoer3rfrXadSOJtKRpLneFJDL2OmR1042LZ8KtNuRRXI/qrdpgr2bnyxFuCRc+tk5QY73mpwwWfKGn6FW/rVdjiEftkC+AvTUItx7RkYMrz0PlTYPUOLustEF2gIgWheYZhE5ebZDurVSG//sudjASliYJItwgSk3G5mZihOSDA7Jcus+U8p1iHQ9CQsey3CSAMmuSGRFmq42kI8135kjtBYrzVswHjX1/51qPMyA2TIeWlRJ7AJlHihG317gT1vPeM1ULl3PkuCxaBLCtjotZBzKLJ/6zXr1ig77Baz71YOVB0D0MgFvzIdao+mdxS8WLOZdiXOCwbI5Y2eo7wJHG4BnpKEdx1rjPkv7DI2NNEoXA7e5FvJWinyAxOy6YGH4Fv6dTlEVhI3Nl/1X+CeafL6Hk8YoqA04Hm+/dJCZi7SsF0r6zfPNNsZLivM8DN18h8KYM3eVY3CwtBldE7atz+oprQ6K/yGUbblHRQwuG3sH3AHrCKLPiUrMIi0jWc0IpdPZRZFL7SnMULfV05QgcvGySOJCuQU+/OYzrRs7umOjwpqvi9PyJYVXBliiYfJutvmfpFAhQj,iv:KnW9LYNzxZ6r3Q6LTeOrgSLLPgiJlibVp6YmTEKBkjE=,tag:KTL1nOWdNWDA4dLVaA4//g==,type:str]
-    client.crt: ENC[AES256_GCM,data:IEmZe505SwkW3U5vmRtFxR6PpuSFGWEHflUZwyhSODzYG1zLM8olVWDtH+yUVI54kLXxTtcpMkKrTBjpAxHG8zUTLPdTCVzPMq6U20jH4PYFMzoldnSgpRl/eiADUmE4I32CBbPdM5nkPvjVEYMmRafEGyJctjG6Shhh/sUO1yE9A+h1R7o3UCRPttWTozSKN4dPi75v1TSAOLyXqvQYl5Ea9iKW47ea7Q+XTcoKfMPXsTJp44kiZnVW0VT/7xNydR7AVWJJzYltY5380j0Ho+7PfNd+QqmdodkaZPS4b0PaoinMiZ40JnChGVhdS91j3IkUlU1g/+G2iVnPR9KB07xqRubqiASxIvpSaSCeYl3rR7+dXYX18VpWcWnJ6XoREoqxJiqmMQWYSROYDTd4G/zs4Usuh5r0wIodmCjud+g89mmXFtH0PbsMCVvZDUswjVfeT2PUBm036TXlxF/1aAy2GDqycbPgJgDLyw9tdiQUZe2iA+Z/Wx80qgQjmmqCaw77cDergLFj9KnByl3Tinwfm+IWFeQiDW+fwSzz+/vrpl55BkSVkVhw7r2Jd6zyg4VtLVXbxYhzdNz695N+fjkZHuPE5tCwr4WCzAg3vSS9hA7IHo8k93WO8s6+KlyjuwTdjPIYZghebO4ktwdqvhvuqxv8Nzwpc5J7Z2H2IChjXg==,iv:l8BwxxEfHfYrm+mM5UE6Yj42DeeU9y23jjIE1KjIiME=,tag:HwRT6/myIsxc84BYDj3K6w==,type:str]
-    client.key: ENC[AES256_GCM,data:QD/Glf3lFslBEdEoJ3CNrYzlw2HLCvGSeVuJLtxnFne2aJOuw0sk/e0h/kSAxvG4dZ+LIWJLLbqTjKHOAhkvS/ZOA7GSea4ubNkvTkw8+JBRuYwb2D2iB8kZhlER/CpiGkYJukjkgGP1CzHdvBY1M88pW9w6rH74MD3jY47qExIaNduQj5vkUQZ8ajqkd79l0nk98ZTawfPG4AFcEtAIKAPIuubT2GZ3qqNLjOI7agbtRv8xI44hhb2wj5J9xdl1GrbOWwG5EOo+hLNrcAAOCcWBpNTUCKBy7mrk2ID8jX3HsFWN76DDlQzkhcphMhlM7g==,iv:49HGJAEgllEkB3YSiZLmbWs/vlbFbY52kiKesdGVC/Y=,tag:g8YZ9at3V2l6RFRfvO2lAQ==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4Z0h0R2dhbDdRcFJydE9o
-            R0hWeCtwVzluZWZlR0VtOXcxdW9FbklOQXlzCmo3eXNyNlFjejNETkJiVVRVWFhr
-            a2pORnhIQTI0UUk2N3dWMDZJcEhYU1UKLS0tIEduaHlmMHQza1A0TGVZb0JrbTYz
-            MWJwQjcrdkkzQXhOTmhycERsM0FtNVEKG+gBRcvUsyfRzuW999EQNzyDh2/5fY6p
-            8ab2xvTOJPiMUBTBuQZsA+iDImytUO/02ROQsbEZkALijp1nqrsV/g==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    encrypted_regex: ^(data|stringData)$
-    lastmodified: "2026-10-07T05:37:26Z"
-    mac: ENC[AES256_GCM,data:vu6ltV0jYQNGV4eqIcqAp9p2TJ8ULBX5JBELUgm/nA9GX+I6Pv3fyjpcHRSWY0HbLvCwgXd22CG4IYcj4HJFtQGZR7OcQA5WM5plZH3RCPdvPNix/798vkETfpJxorQjlHoq4nlr3dEmUCbzdtE07qV2W1M6GJr4C9xKvB947bQ=,iv:aA9mjZCFAoLEhGqkl6NfyBZ3x3vWGfLYYAEe8jXuA4U=,tag:Drdiji33ckDd8ildpvSEXw==,type:str]
-    version: 3.13.2
@@ -37,4 +37,3 @@ files:
    - oauth2-credentials.enc.yaml
    - forgejo-registry-token.enc.yaml
    - pi-sandbox-secrets.enc.yaml
-    - etcd-defrag-creds.enc.yaml
@@ -8,19 +8,6 @@
 # depend on them existing first.
 installCRDs: true

-# Point the ACME DNS-01 self-check at the public resolvers directly.
-#
-# Without these flags cert-manager resolves through the pod resolver
-# (CoreDNS at 10.96.0.10). The self-check then never saw the TXT record and
-# reported "Waiting for DNS-01 challenge propagation" forever. The wildcard
-# certificate renewal stalled on 2026-09-23 and the cert expired on
-# 2026-10-23, which broke every Cloudflare Tunnel origin with
-# "x509: certificate has expired".
-#
-extraArgs:
-  - --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53
-  - --dns01-recursive-nameservers-only
-
 # Tolerate the control-plane taint (talos-cp-1 is the only schedulable node).
 tolerations:
  - key: node-role.kubernetes.io/control-plane
@@ -1,21 +1,13 @@
 # k8s/bootstrap/ingress/riotpiao-com-cert.yaml
 # Public wildcard cert for *.riotpiao.com, signed by Let's Encrypt (DNS-01 via
-# Cloudflare, see ../cert-manager/letsencrypt-issuer.yaml).
+# Cloudflare, see ../cert-manager/letsencrypt-issuer.yaml). Distinct from
+# homelab-tls (self-signed homelab-ca, internal *.riotpiao.com) —
+# reference riotpiao-com-tls explicitly on ingresses that expose riotpiao.com
+# hosts to the internet; it is not nginx's default-ssl-certificate.
 #
-# nginx serves this secret as its default-ssl-certificate
-# (../ingress/nginx-values.yaml), so every Ingress that omits a tls: block uses
-# it. Do not add per-host tls: blocks for *.riotpiao.com hosts: they create a
-# second, single-host certificate and split the wildcard coverage.
-#
-# Renewal is automatic and does not drop traffic:
-#   - cert-manager starts a new order 30 days before expiry (renewBefore).
-#   - cert-manager writes the renewed pair into the same secret.
-#   - ingress-nginx watches the secret and hot-reloads. No pod restart.
-#   - The old pair stays valid until the new one is written, so no gap exists.
-#
-# The DNS-01 self-check must reach the public resolvers directly. Without the
-# --dns01-recursive-nameservers flags in ../../cert-manager/cert-manager-values.yaml
-# the self-check never sees the TXT record and renewal stalls until expiry.
+# issuerRef starts on letsencrypt-staging. Switch to letsencrypt-prod once the
+# staging cert issues cleanly, then delete this Certificate's Secret to force
+# a re-issue against prod (staging certs aren't browser-trusted).
 apiVersion: cert-manager.io/v1
 kind: Certificate
 metadata:
@@ -107,9 +107,7 @@ ingress:
  enabled: true
  className: nginx
  annotations:
-    # No cert-manager annotation. forgejo.riotpiao.com is covered by the
-    # *.riotpiao.com wildcard, served as nginx's default-ssl-certificate.
-    #
+    cert-manager.io/cluster-issuer: letsencrypt-prod
    # This chart Ingress and the hand-written one in
    # k8s/bootstrap/ingress/ingress.yaml both claim forgejo.riotpiao.com.
    # ingress-nginx breaks the tie by oldest creationTimestamp, and the chart's
@@ -127,9 +125,10 @@ ingress:
      paths:
        - path: /
          pathType: Prefix
-  # tls: intentionally omitted. The chart would create a second, single-host
-  # certificate (forgejo-tls) that expired on 2026-10-23 and competed with the
-  # wildcard. The wildcard covers this host.
+  tls:
+    - secretName: forgejo-tls
+      hosts:
+        - forgejo.riotpiao.com

 # Resources
 resources:
@@ -1,177 +0,0 @@
-# etcd backend maintenance: compact, defrag, disarm NOSPACE.
-#
-# Why this exists
-# ---------------
-# etcd's backend file only grows. The apiserver compacts every 5m, but
-# compaction marks superseded revisions free *inside* the file; only `defrag`
-# returns that space to the filesystem. Nothing ran defrag, so the file grew
-# until it crossed the 2 GiB quota:
-#
-#   2026-10-07  backend 2,148,843,520 B (2.1 GB) on talos-cp-1 and talos-cp-3
-#               -> etcd raised NOSPACE on those two members
-#               -> etcd rejected ALL writes cluster-wide
-#               -> kubelets could not post node status: 3 nodes NotReady
-#               -> apiserver writes returned 500, the tunnel API path 502
-#
-# The live data was tiny: 15,943 keys, 69 MB after compaction. The rest was
-# dead pages. Recovery was `compact <current-revision>` then `defrag` per
-# member then `alarm disarm`. Compaction had also stalled while the alarm was
-# up, so `dbSizeInUse` read 100% of the file - defrag alone would NOT have
-# worked. Order matters: compact first, then defrag.
-#
-# The earlier revision of this file never ran, for two independent reasons:
-#   1. it existed only on branch fix/etcd-defrag-cron; the ArgoCD application
-#      `cluster-maintenance` tracks `main` (targetRevision: main), so the
-#      cluster never had the CronJob or the Secret.
-#   2. the image was quay.io/coreos/etcd, which has NO /bin/sh (verified:
-#      `exec: "/bin/sh": no such file or directory`). Any container that runs
-#      a shell script needs an image that ships one.
-# This revision fixes both: image bitnamilegacy/etcd (ships bash + etcdctl
-# 3.6.4, matching the 3.6.11 servers) pinned by digest.
-#
-# Cadence: every 15 days. Measured growth is ~29 MB/day, so a 15-day cycle
-# peaks near 500 MB - comfortably under the 2 GiB quota. The schedule is daily
-# and the script exits unless the UTC epoch day is a multiple of 15. That gives
-# an exact 15-day interval; `*/15` in the day-of-month field does not (it fires
-# on the 1st, 16th and 31st, which puts two runs one day apart).
-#
-# Safety
-# ------
-# 1. Compaction uses the MINIMUM revision across members. Compacting to a
-#    revision a member has not reached wedges that member - measured on
-#    2026-10-07, when talos-cp-2 froze ~1.2M entries behind the leader and
-#    needed a reboot to rejoin.
-# 2. A member whose raft applied index lags the leader by more than 100000 is
-#    skipped and reported. Defragging a frozen member reclaims nothing.
-# 3. Members are defragged one at a time. A defrag pauses the member's apply
-#    loop for the duration of the file copy.
-#
-# Credentials: Secret etcd-defrag-creds in kube-system, SOPS-managed in
-# k8s/argocd/secrets/etcd-defrag-creds.enc.yaml (ksops generator 00-secrets).
-# Its client certificate is signed by the cluster's etcd CA and expires
-# 2036-10; rotate it before then by re-running the mint and re-encrypting.
-#
-# Companion alerts (monitoring): etcd_mvcc_db_total_size_in_bytes /
-# etcd_server_quota_backend_bytes > 0.8, and an etcd NOSPACE alarm alert.
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: etcd-defrag
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: etcd-defrag
-    app.kubernetes.io/part-of: cluster-maintenance
-spec:
-  schedule: "0 3 * * *"            # daily; the script acts only every 15th day
-  concurrencyPolicy: Forbid
-  startingDeadlineSeconds: 600
-  successfulJobsHistoryLimit: 3
-  failedJobsHistoryLimit: 3
-  jobTemplate:
-    spec:
-      backoffLimit: 2
-      template:
-        spec:
-          restartPolicy: OnFailure
-          tolerations:
-            - key: node-role.kubernetes.io/control-plane
-              operator: Exists
-              effect: NoSchedule
-          containers:
-            - name: defrag
-              image: bitnamilegacy/etcd:3.6.4@sha256:c86c06f1ce6a08b6df21eb7a9655b1818778b64c85b5cbb8413ebf7534789730
-              env:
-                - name: ETCDCTL_API
-                  value: "3"
-                - name: ETCDCTL_CACERT
-                  value: /etc/etcd/ca.crt
-                - name: ETCDCTL_CERT
-                  value: /etc/etcd/client.crt
-                - name: ETCDCTL_KEY
-                  value: /etc/etcd/client.key
-                - name: EPS
-                  value: "https://192.168.1.166:2379 https://192.168.1.214:2379 https://192.168.1.162:2379"
-              command: ["/bin/bash", "-c"]
-              args:
-                - |
-                  set -u
-
-                  DAY=$(( $(date -u +%s) / 86400 ))
-                  if [ $(( DAY % 15 )) -ne 0 ]; then
-                    echo "skip: UTC epoch day $DAY is not a 15-day boundary"
-                    exit 0
-                  fi
-
-                  echo "== before =="
-                  for ep in $EPS; do
-                    etcdctl --endpoints="$ep" endpoint status -w table || true
-                  done
-
-                  MINREV=""
-                  MAXAPPLIED=0
-                  for ep in $EPS; do
-                    json=$(etcdctl --endpoints="$ep" endpoint status -w json 2>/dev/null) || continue
-                    rev=$(printf '%s' "$json" | sed -n 's/.*"revision":\([0-9]*\).*/\1/p' | head -1)
-                    app=$(printf '%s' "$json" | sed -n 's/.*"raftAppliedIndex":\([0-9]*\).*/\1/p' | head -1)
-                    echo "member $ep revision=${rev:-?} raftAppliedIndex=${app:-?}"
-                    if [ -n "$rev" ] && { [ -z "$MINREV" ] || [ "$rev" -lt "$MINREV" ]; }; then
-                      MINREV="$rev"
-                    fi
-                    if [ -n "$app" ] && [ "$app" -gt "$MAXAPPLIED" ]; then
-                      MAXAPPLIED="$app"
-                    fi
-                  done
-
-                  if [ -z "$MINREV" ]; then
-                    echo "FATAL: no member returned a revision; nothing attempted"
-                    exit 1
-                  fi
-
-                  echo "== compact to revision $MINREV =="
-                  etcdctl --endpoints="$(echo "$EPS" | awk '{print $1}')" compact "$MINREV" || echo "COMPACT-FAILED (continuing: defrag still reclaims free pages)"
-
-                  # Physical deletion of superseded keys runs in the background
-                  # after the compaction commit; give it a moment to drain.
-                  sleep 20
-
-                  echo "== defrag each member (sequential, one at a time) =="
-                  rc=0
-                  for ep in $EPS; do
-                    json=$(etcdctl --endpoints="$ep" endpoint status -w json 2>/dev/null) || { echo "SKIP $ep (unreachable)"; rc=1; continue; }
-                    app=$(printf '%s' "$json" | sed -n 's/.*"raftAppliedIndex":\([0-9]*\).*/\1/p' | head -1)
-                    if [ -n "$app" ] && [ "$app" -lt $(( MAXAPPLIED - 100000 )) ]; then
-                      echo "SKIP $ep (applied index $app lags leader $MAXAPPLIED; defrag would reclaim nothing)"
-                      rc=1
-                      continue
-                    fi
-                    echo "-- $ep"
-                    if ! etcdctl --endpoints="$ep" defrag; then
-                      echo "DEFRAG-FAILED $ep"
-                      rc=1
-                    fi
-                  done
-
-                  echo "== disarm NOSPACE =="
-                  for ep in $EPS; do
-                    etcdctl --endpoints="$ep" alarm list 2>/dev/null || true
-                    etcdctl --endpoints="$ep" alarm disarm 2>/dev/null || true
-                  done
-
-                  echo "== after =="
-                  for ep in $EPS; do
-                    etcdctl --endpoints="$ep" endpoint status -w table || true
-                  done
-                  etcdctl --endpoints="$(echo "$EPS" | awk '{print $1}')" alarm list || true
-                  exit $rc
-              resources:
-                requests: { cpu: 10m, memory: 32Mi }
-                limits:   { cpu: 200m, memory: 128Mi }
-              volumeMounts:
-                - name: creds
-                  mountPath: /etc/etcd
-                  readOnly: true
-          volumes:
-            - name: creds
-              secret:
-                secretName: etcd-defrag-creds
-                defaultMode: 0444
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-# No commonLabels on purpose: this is a vendored third-party manifest and
-# kustomize would also rewrite the provisioner Deployment's selector, which
-# is immutable on re-apply.
-resources:
-  - local-path-storage.yaml
@@ -1,200 +0,0 @@
-# Vendored from https://github.com/rancher/local-path-provisioner
-#   deploy/local-path-storage.yaml @ v0.0.37 (release 2026-08-05)
-#
-# Local modifications (the only three; re-apply them on any re-vendor):
-#   1. the namespace carries pod-security.kubernetes.io/enforce=privileged, because the
-#      provisioner's helper Pods mount a hostPath and this cluster defaults to the
-#      baseline policy, which forbids that;
-#   2. config.json nodePathMap is /var/lib/local-path-provisioner — Talos mounts the root
-#      filesystem read-only, so upstream's /opt path cannot be created;
-#   3. the helper Pod's busybox image is pinned to a tag instead of a floating name.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: local-path-storage
-  labels:
-    pod-security.kubernetes.io/enforce: privileged
-    pod-security.kubernetes.io/audit: privileged
-    pod-security.kubernetes.io/warn: privileged
-
---
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: local-path-provisioner-service-account
-  namespace: local-path-storage
-
---
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: local-path-provisioner-role
-  namespace: local-path-storage
-rules:
-  - apiGroups: [""]
-    resources: ["pods"]
-    verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
-
---
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: local-path-provisioner-role
-rules:
-  - apiGroups: [""]
-    resources: ["nodes", "persistentvolumeclaims", "configmaps", "pods", "pods/log"]
-    verbs: ["get", "list", "watch"]
-  - apiGroups: [""]
-    resources: ["persistentvolumes"]
-    verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
-  - apiGroups: [""]
-    resources: ["events"]
-    verbs: ["create", "patch"]
-  - apiGroups: ["storage.k8s.io"]
-    resources: ["storageclasses"]
-    verbs: ["get", "list", "watch"]
-
---
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: local-path-provisioner-bind
-  namespace: local-path-storage
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: local-path-provisioner-role
-subjects:
-  - kind: ServiceAccount
-    name: local-path-provisioner-service-account
-    namespace: local-path-storage
-
---
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: local-path-provisioner-bind
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: local-path-provisioner-role
-subjects:
-  - kind: ServiceAccount
-    name: local-path-provisioner-service-account
-    namespace: local-path-storage
-
---
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: local-path-provisioner
-  namespace: local-path-storage
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: local-path-provisioner
-  template:
-    metadata:
-      labels:
-        app: local-path-provisioner
-    spec:
-      serviceAccountName: local-path-provisioner-service-account
-      containers:
-        - name: local-path-provisioner
-          image: docker.io/rancher/local-path-provisioner:v0.0.37
-          imagePullPolicy: IfNotPresent
-          command:
-            - local-path-provisioner
-            - --debug
-            - start
-            - --config
-            - /etc/config/config.json
-          ports:
-            - name: health
-              containerPort: 8080
-              protocol: TCP
-          startupProbe:
-            httpGet:
-              path: /health
-              port: health
-            initialDelaySeconds: 5
-            periodSeconds: 5
-            failureThreshold: 12
-          livenessProbe:
-            httpGet:
-              path: /health
-              port: health
-            initialDelaySeconds: 10
-            periodSeconds: 10
-          readinessProbe:
-            httpGet:
-              path: /ready
-              port: health
-            initialDelaySeconds: 5
-            periodSeconds: 5
-          volumeMounts:
-            - name: config-volume
-              mountPath: /etc/config/
-          env:
-            - name: POD_NAMESPACE
-              valueFrom:
-                fieldRef:
-                  fieldPath: metadata.namespace
-            - name: CONFIG_MOUNT_PATH
-              value: /etc/config/
-            - name: HEALTH_PORT
-              value: "8080"
-      volumes:
-        - name: config-volume
-          configMap:
-            name: local-path-config
-
---
-apiVersion: storage.k8s.io/v1
-kind: StorageClass
-metadata:
-  name: local-path
-provisioner: rancher.io/local-path
-volumeBindingMode: WaitForFirstConsumer
-reclaimPolicy: Delete
-
---
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: local-path-config
-  namespace: local-path-storage
-data:
-  config.json: |-
-    {
-            "nodePathMap":[
-            {
-                    "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES",
-                    "paths":["/var/lib/local-path-provisioner"]
-            }
-            ]
-    }
-  setup: |-
-    #!/bin/sh
-    set -eu
-    mkdir -m 0777 -p "$VOL_DIR"
-  teardown: |-
-    #!/bin/sh
-    set -eu
-    rm -rf "$VOL_DIR"
-  helperPod.yaml: |-
-    apiVersion: v1
-    kind: Pod
-    metadata:
-      name: helper-pod
-    spec:
-      priorityClassName: system-node-critical
-      tolerations:
-        - key: node.kubernetes.io/disk-pressure
-          operator: Exists
-          effect: NoSchedule
-      containers:
-      - name: helper-pod
-        image: docker.io/library/busybox:1.36
-        imagePullPolicy: IfNotPresent
@@ -41,24 +41,13 @@ spec:
          # - /var/lib/longhorn-disk1 (sdb1, 600GB)
          # - /var/lib/longhorn-disk2 (sdc1, 500GB)
          # - /var/lib/longhorn-disk3 (sdd1, 160GB)
-          #
-          # /var/lib/longhorn-disk4 (sde1, the 2TB PERC virtual disk) was RETIRED
-          # on 2026-10-12 and must not be re-added. The device returned SCSI I/O
-          # errors at ~7,000/minute, the XFS filesystem on it hung, and every
-          # process that touched it blocked in unkillable D state. Those unkillable
-          # processes held Longhorn's per-node rebuild slots forever, so no new
-          # replica could start on talos-cp-2 and every new volume there failed to
-          # attach. It is removed from the Terraform disk list too, so a re-render
-          # does not put it back.
+          # - /var/lib/longhorn-disk4 (sde1, 2TB)
          
          echo "Checking talos-cp-2..."
          CURRENT_DISKS=$(kubectl -n longhorn-system get nodes.longhorn.io talos-cp-2 -o json | jq -r '.spec.disks | keys | length')
          echo "  Current disk count: $CURRENT_DISKS"
          
-          # 4 = default-disk-080400000000 + disk1..disk3. It was 5 while disk4
-          # existed; leaving the old threshold would re-apply the patch on every
-          # run once disk4 is gone.
-          if [ "$CURRENT_DISKS" -lt 4 ]; then
+          if [ "$CURRENT_DISKS" -lt 5 ]; then
            echo "  Adding extra disks to talos-cp-2..."
            kubectl -n longhorn-system patch nodes.longhorn.io talos-cp-2 --type merge -p '{
              "spec": {
@@ -86,6 +75,14 @@ spec:
                    "path": "/var/lib/longhorn-disk3",
                    "storageReserved": 5368709120,
                    "tags": []
+                  },
+                  "disk4": {
+                    "allowScheduling": true,
+                    "diskType": "filesystem",
+                    "evictionRequested": false,
+                    "path": "/var/lib/longhorn-disk4",
+                    "storageReserved": 21474836480,
+                    "tags": []
                  }
                }
              }
@@ -20,7 +20,6 @@ controlplane_configs = {
  "talos-cp-1" = {
    hostname     = "talos-cp-1"
    lan_ip       = "192.168.1.166"
-    public_ip    = "https://k8s-api.riotpiao.com"
    lan_subnet   = "192.168.1.0/24"
    lan_gateway  = "192.168.1.254"
    install_disk = "/dev/nvme0n1"
@@ -33,7 +32,6 @@ controlplane_configs = {
  "talos-cp-2" = {
    hostname     = "talos-cp-2"
    lan_ip       = "192.168.1.214"
-    public_ip    = "https://k8s-api.riotpiao.com"
    lan_subnet   = "192.168.1.0/24"
    lan_gateway  = "192.168.1.254"
    install_disk = "/dev/disk/by-id/wwn-0x644a842029bd3f002720989b07d7143d"
@@ -41,11 +39,7 @@ controlplane_configs = {
        { device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b60f4375d97062", mountpoint = "/var/lib/longhorn-disk1" },
        { device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b5ffeb8bb8b47f", mountpoint = "/var/lib/longhorn-disk2" },
        { device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b6000c8dabb266", mountpoint = "/var/lib/longhorn-disk3" },
-        # /var/lib/longhorn-disk4 (wwn-0x6b083fe0c5782700321370dc23fd765b) was
-        # retired 2026-10-12: the 2TB PERC virtual disk failed with continuous
-        # SCSI I/O errors, hung its XFS filesystem, and wedged Longhorn replica
-        # rebuilds on talos-cp-2 with unkillable D-state processes. Do not
-        # re-add it.
+        { device = "/dev/disk/by-id/wwn-0x6b083fe0c5782700321370dc23fd765b", mountpoint = "/var/lib/longhorn-disk4" },
      ]
    zone             = "az-b"
    allow_scheduling = true
@@ -55,7 +49,6 @@ controlplane_configs = {
  "talos-cp-3" = {
    hostname     = "talos-cp-3"
    lan_ip       = "192.168.1.162"
-    public_ip    = "https://k8s-api.riotpiao.com"
    lan_subnet   = "192.168.1.0/24"
    lan_gateway  = "192.168.1.254"
    install_disk = "/dev/nvme0n1"