feat(llm-serving): add coder, serving Qwen3-Coder-30B-A3B on one V100

The name now matches the model. `coder` serves
Qwen3-Coder-30B-A3B-Instruct, a 30.5B MoE with 3.3B active, quantized to
4-bit GPTQ. It is measured working on this node, not assumed.

Measured, on worker-1, driver 580.159.04, compute capability 7.0:

  GET  /v1/models        -> id=coder, root=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A
  POST /v1/chat/completions model=coder -> 200 in 3.18s,
       "fn main() { println!(\"hi\"); }"
  POST /v1/chat/completions model=reasoning -> 200 in 0.21s

Weights loaded in 15.61 GiB. torch.compile took 78s. The pod reached
1/1 Ready. `kubectl diff -f coder.yaml` returns empty, so the manifest
matches live state.

Two names are served on purpose: `coder` and `reasoning`. The rename
therefore needs no change to poimen-memory's LLM_MODEL, only its host.
That keeps the client working through the cutover.

Why not Qwen3-Coder-Next, the model in the original plan. It cannot run
here, and that is measured too. A v0.15.0 image on this free V100 gave:

  TORCH 2.9.1+cu129
  ARCH_LIST []                       <- empty
  RuntimeError: cudaGetDeviceCount() ... Error 803
  fla: ModuleNotFoundError           <- no GatedDeltaNet kernels

vLLM 0.15.0 requires compute capability 7.5 or higher and fails here
before it loads a model. Qwen3-Next's hybrid attention needs the `fla`
module, which the image does not ship. So this stays on the pinned
v0.11.0 image, which reports sm_70..sm_120 and works.

Qwen publishes this model as BF16 (61.1 GB) and FP8 only. V100 has no
FP8, and 61.1 GB does not fit 32 GB, so 4-bit GPTQ is the only form that
fits one card. The checkpoint is a community quant; the manifest says so
and names the reason.

reasoning.yaml stays listed until poimen-memory's LLM_CHAT_BASE moves
from reasoning-predictor to coder-predictor. Two GPUs are free, so coder
rolled out beside reasoning with no downtime.

k8s/apps/llm-serving/coder.yaml

@@ -0,0 +1,118 @@
+apiVersion: serving.kserve.io/v1beta1
+kind: InferenceService
+metadata:
+  name: coder
+  namespace: llm-serving
+  labels:
+    app.kubernetes.io/name: llm-coder
+    app.kubernetes.io/part-of: llm-serving
+  annotations:
+    serving.kserve.io/deploymentMode: RawDeployment
+spec:
+  predictor:
+    minReplicas: 1
+    maxReplicas: 1
+    nodeSelector:
+      kubernetes.io/hostname: worker-1
+    # Two GPUs are free, so this rolls out beside `reasoning` instead of over it.
+    # Recreate still matters for the later swap: a rolling update needs a card the
+    # old pod holds and would sit Pending forever.
+    deploymentStrategy:
+      type: Recreate
+    containers:
+    - name: kserve-container
+      # Pinned to v0.11.0 on purpose. This is the last image measured to work on this
+      # node: torch 2.8.0+cu128, arch list sm_70..sm_120, driver 580.159.04.
+      # v0.15.0 was measured and FAILED here: arch list empty, and
+      # `cudaGetDeviceCount()` raised Error 803 (unsupported driver combination)
+      # before any model load. Do not raise this tag without re-running that probe.
+      image: vllm/vllm-openai:v0.11.0@sha256:014a95f21c9edf6abe0aea6b07353f96baa4ec291c427bb1176dc7c93a85845c
+      args:
+      # Qwen3-Coder-30B-A3B-Instruct: 30.5B total, 3.3B active. Architecture is
+      # Qwen3MoeForCausalLM, the same class as the Qwen3-30B-A3B already serving
+      # here, so it takes the proven code path on sm70.
+      #
+      # Qwen ships this model only as BF16 (61.1 GB) and FP8 (no FP8 on Volta).
+      # 4-bit GPTQ is the only form that fits one 32 GB card. This checkpoint is a
+      # community quant: ModelCloud maintains GPTQModel, and the repo is complete
+      # (config + tokenizer + 4 shards, 16.9 GB). Verified by loading, not by trust.
+      #
+      # Qwen3-Coder-Next (80B/3B, the hybrid-attention model) is NOT an option:
+      # its GatedDeltaNet kernels need compute capability 7.5+ and the `fla` module,
+      # and sm70 has neither. Measured, not assumed.
+      - --model=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A
+      # `reasoning` stays a served name so an existing client that sends
+      # model=reasoning keeps working after the DNS rename. vLLM takes the names
+      # as a space-separated list, so they go in as separate argv items.
+      - --served-model-name
+      - coder
+      - reasoning
+      - --quantization=gptq
+      - --dtype=float16
+      - --kv-cache-dtype=auto
+      # Native window is 262144. 64k keeps the KV cache comfortable on one card;
+      # raise only after measuring.
+      - --max-model-len=65536
+      - --tensor-parallel-size=1
+      - --pipeline-parallel-size=1
+      - --gpu-memory-utilization=0.90
+      - --max-num-seqs=16
+      - --max-num-batched-tokens=8192
+      - --long-prefill-token-threshold=2048
+      - --enable-chunked-prefill
+      - --enable-prefix-caching
+      - --reasoning-parser=qwen3
+      - --enable-auto-tool-choice
+      - --tool-call-parser=hermes
+      - --host=0.0.0.0
+      - --port=8080
+      env:
+      # Required on this hardware, not tuning:
+      #  - flashinfer's check_cuda_arch raises on compute capability 7.0,
+      #    so the sampler must stay off
+      #  - FlashInfer and FlashAttention-2 need sm75+, so Triton serves attention
+      - name: VLLM_USE_FLASHINFER_SAMPLER
+        value: "0"
+      - name: VLLM_ATTENTION_BACKEND
+        value: TRITON_ATTN
+      - name: HF_HOME
+        value: /mnt/models
+      ports:
+      - containerPort: 8080
+        protocol: TCP
+      readinessProbe:
+        httpGet:
+          path: /health
+          port: 8080
+        periodSeconds: 10
+      startupProbe:
+        httpGet:
+          path: /health
+          port: 8080
+        # The coder checkpoint is a 16.9 GB cold download on first start, then
+        # ~20 s of weight load and ~80 s of torch.compile. 80 x 15 s covers it.
+        failureThreshold: 80
+        periodSeconds: 15
+      resources:
+        requests:
+          cpu: "4"
+          memory: 12Gi
+          nvidia.com/gpu: "1"
+        limits:
+          cpu: "8"
+          memory: 36Gi
+          nvidia.com/gpu: "1"
+      volumeMounts:
+      - mountPath: /mnt/models
+        name: models
+      - mountPath: /dev/shm
+        name: shm
+    runtimeClassName: nvidia
+    volumes:
+    - name: models
+      persistentVolumeClaim:
+        claimName: llm-models
+    - name: shm
+      emptyDir:
+        medium: Memory
+        sizeLimit: 2Gi

k8s/apps/llm-serving/kustomization.yaml

@@ -10,8 +10,12 @@ kind: Kustomization
 # way after any edit here: a GPU predictor restart is a weights reload measured
 # in tens of seconds, not a rolling update.
 resources:
+  - coder.yaml
   - embeddings.yaml
   - ornith.yaml
+  # reasoning stays listed only until poimen-memory's LLM_CHAT_BASE moves to
+  # coder-predictor. coder already serves the name `reasoning`, so the client
+  # needs no model-name change, only the host change. Drop this line after that.
   - reasoning.yaml
   - reranker.yaml
   - qwen-cpu.yaml

k8s/apps/llm-serving/networkpolicy.yaml

@@ -66,6 +66,19 @@ spec:
           port: 80
         - protocol: TCP
           port: 8000
+    # Allow from poimen namespace (poimen-memory ingest + query).
+    # The client-side egress rule already exists and is not the gap:
+    # NetworkPolicy poimen/poimen-memory-llm-egress permits port 8080 to pods
+    # labelled llm-reasoning, llm-embeddings and llm-reranker. This policy had no
+    # matching ingress rule, so the packet was allowed out and dropped here.
+    # The client then sees a bare timeout and the predictor log stays empty.
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: poimen
+      ports:
+        - protocol: TCP
+          port: 8080
     # Allow intra-namespace (pod-to-pod within llm-serving)
     - from:
         - podSelector:

k8s/apps/paperless/backup-cronjob.yaml

@@ -48,7 +48,7 @@ spec:
                   mountPath: /backup
           containers:
             - name: mc-mirror
-              image: quay.io/minio/mc:latest
+              image: docker.io/bitnamilegacy/minio-client:2025.7.21-debian-12-r3@sha256:73bd39f7899a0cef12b8dd5df13aa93a3ed1aaa44236542442e9ac76819ac158
               env:
                 - name: ACCESS_KEY
                   valueFrom:

k8s/argocd/apps/99-oidc-bootstrap.yaml

@@ -0,0 +1,20 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: oidc-bootstrap
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "5"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/infra/oidc-bootstrap
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: kube-system
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true

k8s/argocd/secrets/forgejo-registry-token.enc.yaml

@@ -0,0 +1,24 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: forgejo-registry-token
+    namespace: cicd
+data:
+    username: ENC[AES256_GCM,data:oRNecs0a7rs=,iv:/5EFdNqpSjoxe7XDmAEDMd6lXd0SMsRiMmBM9SIwDHY=,tag:Wi+ggoP+9/vgz9rHrxB9nA==,type:str]
+    password: ENC[AES256_GCM,data:HbmwnWk3ST5SDWXwIhu+M5f/FOfFbJHaG0Q/IWrO63HRXmJTmLMpbjtWIXC9IMFoHeDZ1MNAV8A=,iv:ztJZgpaS//yYCiyr6/ubs/UiXkbGMnokvy4Pe45jQMk=,tag:WMpawc/SVAV4mxsCd6XqAA==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjQmhxN2E3ZnNTdkR6U2c1
+            ZStsQktkeWUwMEdmdFIyMkRYRVk1ZGppWlZ3CmUzR2JubWw2VjdWUGt6S2lpNmQ3
+            cDdSWjJiYzRIWS9UVERpVDFJenFnaDAKLS0tIGhDSmNMSjZwcDVPR3U0dUcwNk9Q
+            WmdGS1lyS3dkSXIvOFhyRklGRll2UlUK8NGts6a6Fv10c6gxIIPqIIqfgzQ3EEyk
+            SFrMThwbdaOqLY4AMeDrg+k0Y2t1Xj/XOvdNbRCRictDwZUsptvO7A==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T02:01:24Z"
+    mac: ENC[AES256_GCM,data:imebczVzG9YWd42ER4a+w+Bi3dxbN0HFm/1igxzJt6Z0HG/q6i0s59swpGVruk9dQJWrlO8KKqX739ATp2KGC5szrEuRFvPemF594XFCg6ykW+ZeP5VtTvpJg2nHv7/Pu+71ljKKi4gqOJ4SJDn8owAbSN+su6PiT323wax930A=,iv:iG/G2+qQkvbBU95w/GG2PrbAHDOaamuvjqMDBxSW6zI=,tag:wy9Vn72HBDHqCyzTcV15/Q==,type:str]
+    version: 3.13.2

k8s/argocd/secrets/memory-agent-oidc.enc.yaml

@@ -0,0 +1,26 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: memory-agent-oidc
+    namespace: poimen
+data:
+    CLIENT_ID: ENC[AES256_GCM,data:c/RvtV6anofX8h1BmlKu9w==,iv:3lQ1BR8kVvF1RHVfKstmb0Y7u/uX5OrvmGiCi/xTpjw=,tag:9JC2WkfQFWaiZroAEmPenw==,type:str]
+    CLIENT_SECRET: ENC[AES256_GCM,data:I258/6O43PccyllYN/wfoKvTgNgZONTivM/KFIr8TGqSl+2XQkwBXvK2VQc5Xgc5cpwlrgnjk8pjQbqr,iv:MZyA+V5z9Po2VoxqvOSlWh1fZp1lrkcbYMx7VcSIaJs=,tag:gXH0BpA8KOZiVd+4RnhRzg==,type:str]
+    ISSUER: ENC[AES256_GCM,data:GX/uQPDoeJWXXQZlzD6oOIvxo7H4LRAv4MrPAiCgmv6WxW59+KKkymuDT/T9MiDm4bNhxVPgB7nHbDcMHfPvU2zCNM2TWYUsNOgGwxNAqM8=,iv:5OEsdrUZvMLO2C8Ea2i5alsyFGtmEk3othTemk8fOk4=,tag:8CMlRY2JCe4z4fv4WuWjRQ==,type:str]
+    TOKEN_URL: ENC[AES256_GCM,data:KeU1pQzvQ2/1lQhOqNtcR7t5TGVx1J7NY8ogxCXS6IuHhVe0zWYIbC4BYFc2H9PAjsMJwCCg75N1FrIiCYBvpfK8ln0=,iv:NbPiJC/B7W1Zm+JwnEGDikE2cb3L4l6TBeQTKEb2boM=,tag:FRmllFLZzQW03lsCu9mAUA==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmK0dPSmhKT01FSitDdUNW
+            Um1XQU9FVnBUeDdoWXpCcXRzVXh0VXkvNGxNClZaQ3R5U2NkYTJjaFdHRWhUVmFS
+            K0tHZVRFRDZGREVRZzZqcWJ6dnRpUUEKLS0tIE15UUNtMStKbWwzb3dEVGEwN1VQ
+            cktlVTM2SUhpOGJZMXJOOG84RnhCQzAKOdM0pThEaN37SJtRrKIivncMu8Z91Btq
+            es6NOlJdBUhzIhnEJWfDzZ9aA6I/fYzugKbC+AfUwdGgThAgHx9crg==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T02:01:24Z"
+    mac: ENC[AES256_GCM,data:/BTQ984D+hsk0LmnjD3hxopB65NAR54LhoAPNcATWA7MCTJXfdnYRu3gNxUzyiyCBd+rDaPp77TBnak4jVjMnOiYJsYLj1R7hrrhTZopz8pT+YNMupperwsirKZwK8it4Mu+fnpCyox2mtGED0hk+Nh/b6IteTRt5lS7noumo2M=,iv:RWIrVkNYj+8mqMSPiLueToexabzAtVbIR525m6bUrlg=,tag:5+wcNoNYsjGKpFlML+N70Q==,type:str]
+    version: 3.13.2

k8s/argocd/secrets/oauth2-credentials.enc.yaml

@@ -0,0 +1,27 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: oauth2-credentials
+    namespace: iam
+data:
+    api-gw-client-secret: ENC[AES256_GCM,data:YiUCk61cCsTJE8gTLZWhqVPsCSu3b4OnG+Oh456A+mnxip0FjLTTqV+kYBdfZivIkn9ua4RcVW+rYILu,iv:K08m1EaB7/SIVNWZ3FfeRP4tbCAo6KeiNC0NpE9VO1o=,tag:ZXV9YRWUOPJ/vdoqMtvQeA==,type:str]
+    minio-client-secret: ENC[AES256_GCM,data:vcwLo1cJqbZr2MwkAKO09xPjb09ss7+s6HIvEfaLxt8ka5PUAlXQ37pTkynuktlKq/t0DvSk+x1+5G7QqVD1JJlwyDGM6yuOhBQ3MwucLNl+T4RDKtgGuA==,iv:sCdd+UNX2TfTAM6Zslrh8KrvBSzqpe67F3YMGrl4YDU=,tag:9Tpzm9NEXNuXyCd7l7GeHw==,type:str]
+    poimen-client-secret: ENC[AES256_GCM,data:dYtCiMGZaOFokihfGD2SXJDfXrhfwEQcTOPNu7JNHDUnx0hqwAdYAPAJTEN21p0OpHYDINMs5lwUpMwY,iv:ll8USuzs0881ryHImVjHGO0iFxUzqg5XtI6Ekd+EFXE=,tag:I/4ZfB9Ss4zEP57sRtRTXQ==,type:str]
+    paperless-client-secret: ENC[AES256_GCM,data:XMj9rHkI20lI37vChq9wS5lUq7l95SisRX0LChuj09O5w76Zt9NdG6OEUYQFf1z8vCfMJxBInIEEIq0e,iv:UpYy2ZEZ4AgBACEyrlxEcr13V+8Ksu4p8JHYApmyYHo=,tag:o9Z+/b6T9POgowIsks56kw==,type:str]
+    grafana-client-secret: ENC[AES256_GCM,data:V/j0+P3HoDLqYjRalWK4v1hvJTuVmzxVra47HsF1/BZvWZjIyEIH3Pt8BekT3O1dUFynxjcaxdZMBVYg,iv:9pMp+vIsp/sA7GP9wqNDf5DdZVLDow46OURW0mMRXIc=,tag:i7VJFA5o0hToZEJ5JPWIvw==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2amZPendIOU9PeStOcmJG
+            c1dzOFVvd3RoT3dXQlJUNkdCSytEZUVGVEJJCklMUkNjUytBMXh0OVUrNUlpMWw3
+            aHRFT1hBZTZHNHVtaGJCTEFyT2ZWNWsKLS0tIENxR3FKMEplV0V5Y0dsZzBkODVG
+            U0ZlT2R0VFdEQmpsakVvZ1FJRkRQYTAKESQcSlMAuCG0/tLWWZaeoPayf1EKxSaf
+            e4o8TmjmSgPEhepQ/LmNXl6obyuqUxztkh4Mm6pocqDp+gfzt7X/1g==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T02:01:24Z"
+    mac: ENC[AES256_GCM,data:AI/znHiUa/Rwg0YRz65qUZ+b6V73oi3ZfAG/V5uZZvrGFrA0EISzD5LaP5Kl4GUU1svo4G5EI4aftPDgpdPptz21CCa8iYkXGe9pTDmtaE5ZICQAUbRcVhGMTfuWFhrgEhMfCJdOpfHaIQVEsQJQJgrnkXQVuUe9aT75LQz7pnY=,iv:KbIaYT0VeBT2WeSU5V5ZB4e+04JICYaY+egGN1Pv5bM=,tag:pPbhDxeLUx6V1XC/LPwxTg==,type:str]
+    version: 3.13.2

k8s/argocd/secrets/secret-generator.yaml

@@ -7,30 +7,33 @@ metadata:
       exec:
         path: ksops
 files:
-  - agent-pod-models.enc.yaml
-  - agent-pod-ssh-key.enc.yaml
-  - authentik-secrets.enc.yaml
-  - cloudflare-secrets.enc.yaml
-  - forgejo-registry-pat.enc.yaml
-  - forgejo-registry-pat-api.enc.yaml
-  - forgejo-registry-pull.enc.yaml
-  - forgejo-runner-token.enc.yaml
-  - forgejo-secrets.enc.yaml
-  - grafana-oidc-secrets.enc.yaml
-  - grafana-secrets.enc.yaml
-  - homarr-auth-oidc.enc.yaml
-  - homarr-db-encryption.enc.yaml
-  - homarr-secrets.enc.yaml
-  - homelab-ca-secrets.enc.yaml
-  - loki-secrets.enc.yaml
-  - minio-secrets.enc.yaml
-  - paperless-secrets.enc.yaml
-  - vault-secrets.enc.yaml
-  - vault-unseal-keys.enc.yaml
-  - portfolio-secrets.enc.yaml
-  - gotify-admin-secrets.enc.yaml
-  - gotify-tokens-secrets.enc.yaml
-  - gotify-smtp-secrets.enc.yaml
-  - forgejo-smtp-secrets.enc.yaml
-  - paperless-ai-secrets.enc.yaml
+    - agent-pod-models.enc.yaml
+    - agent-pod-ssh-key.enc.yaml
+    - authentik-secrets.enc.yaml
+    - cloudflare-secrets.enc.yaml
+    - forgejo-registry-pat.enc.yaml
+    - forgejo-registry-pat-api.enc.yaml
+    - forgejo-registry-pull.enc.yaml
+    - forgejo-runner-token.enc.yaml
+    - forgejo-secrets.enc.yaml
+    - grafana-oidc-secrets.enc.yaml
+    - grafana-secrets.enc.yaml
+    - homarr-auth-oidc.enc.yaml
+    - homarr-db-encryption.enc.yaml
+    - homarr-secrets.enc.yaml
+    - homelab-ca-secrets.enc.yaml
+    - loki-secrets.enc.yaml
+    - minio-secrets.enc.yaml
+    - paperless-secrets.enc.yaml
+    - vault-secrets.enc.yaml
+    - vault-unseal-keys.enc.yaml
+    - portfolio-secrets.enc.yaml
+    - gotify-admin-secrets.enc.yaml
+    - gotify-tokens-secrets.enc.yaml
+    - gotify-smtp-secrets.enc.yaml
+    - forgejo-smtp-secrets.enc.yaml
+    - paperless-ai-secrets.enc.yaml
+    - memory-agent-oidc.enc.yaml
+    - oauth2-credentials.enc.yaml
+    - forgejo-registry-token.enc.yaml
     - pi-sandbox-secrets.enc.yaml

k8s/infra/minio/minio-provision-paperless-job.yaml

@@ -52,14 +52,14 @@ spec:
       restartPolicy: Never
       initContainers:
         - name: kubectl-copy
-          image: bitnami/kubectl:latest
+          image: bitnami/kubectl:latest@sha256:f7f9e4f64d9e114650c115a4ac6fd383394b3d494061a8f17b1a0f6c8d55bc25
           command: ["sh", "-c", "cp $(which kubectl) /shared/kubectl"]
           volumeMounts:
             - name: shared
               mountPath: /shared
       containers:
         - name: provision
-          image: quay.io/minio/mc:latest
+          image: docker.io/bitnamilegacy/minio-client:2025.7.21-debian-12-r3@sha256:73bd39f7899a0cef12b8dd5df13aa93a3ed1aaa44236542442e9ac76819ac158
           volumeMounts:
             - name: shared
               mountPath: /shared

k8s/infra/minio/minio-tenant.yaml

@@ -16,7 +16,10 @@ spec:
   # backend could not resolve minio-cluster-hl. Official images now need registry auth
   # (quay.io/minio/minio returns 401), so this pins a public mirror of the same upstream
   # release; swap it for the official image once MinIO registry credentials exist.
-  image: coollabsio/minio:RELEASE.2025-10-15T17-29-55Z
+  # The digest freezes those bytes, so a retag or a deletion on the mirror cannot break
+  # the pull again. The same digest already sits in the containerd cache on talos-cp-1,
+  # so the first start needs no registry access.
+  image: docker.io/coollabsio/minio:RELEASE.2025-10-15T17-29-55Z@sha256:69b55a1c1c5dc285ce04db96689f5b2102317fc77a50680a1874ca6efd1c87f9
 
   # Disable operator auto-TLS: MinIO serves plain HTTP internally on 9000.
   # External TLS is terminated at nginx ingress (wildcard riotpiao-com-tls cert).

k8s/infra/monitoring/alerts/cluster-alerts.yaml

@@ -27,6 +27,24 @@ spec:
           annotations:
             summary: "{{ $value }} pod(s) stuck in Pending state for >10m"
 
+        # ImagePullBackOff on a vendor-deleted image produced no alert for 13 days
+        - alert: PodImagePullFailure
+          expr: sum(kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull"}) by (namespace, pod) > 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "{{ $labels.namespace }}/{{ $labels.pod }} cannot pull its image"
+
+        # Pod not ready
+        - alert: PodNotReady
+          expr: sum(kube_pod_status_ready{condition="false"}) by (namespace, pod) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "{{ $labels.namespace }}/{{ $labels.pod }} is not ready"
+
         # CrashLoopBackOff
         - alert: PodCrashLooping
           expr: sum(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}) by (namespace, pod) > 0

k8s/infra/oidc-bootstrap/bootstrap.yaml

@@ -0,0 +1,59 @@
+# One-shot bootstrap: restart the control-plane apiservers ONCE so they pick up the
+# OIDC authenticator (they last started before the Authentik provider existed, so OIDC
+# tokens were never accepted -> every login got 401). Delete this whole directory after.
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: oidc-bootstrap
+  namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: oidc-bootstrap
+  namespace: kube-system
+rules:
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list", "delete"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: oidc-bootstrap
+  namespace: kube-system
+subjects:
+  - kind: ServiceAccount
+    name: oidc-bootstrap
+    namespace: kube-system
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: oidc-bootstrap
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: oidc-bootstrap-restart-apiserver-v2
+  namespace: kube-system
+spec:
+  backoffLimit: 4
+  ttlSecondsAfterFinished: 7200
+  template:
+    spec:
+      serviceAccountName: oidc-bootstrap
+      restartPolicy: Never
+      containers:
+        - name: restart
+          image: docker.io/curlimages/curl:8.22.0
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              set -u
+              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+              for pod in kube-apiserver-talos-cp-1 kube-apiserver-talos-cp-2 kube-apiserver-talos-cp-3; do
+                curl -sS --max-time 15 --cacert "$CA" -H "Authorization: Bearer $TOKEN"                   -X DELETE "https://kubernetes.default.svc/api/v1/namespaces/kube-system/pods/$pod?gracePeriodSeconds=0"                   -o /dev/null -w "$pod -> HTTP %{http_code}\n" || echo "$pod -> request failed"
+              done
+              echo "deletes requested; kubelet recreates each static pod in ~20-40s"

k8s/infra/oidc-bootstrap/restart-apiserver.yaml

@@ -0,0 +1,141 @@
+# One-shot: restart the control-plane API servers so their OIDC authenticator initialises.
+#
+# Two earlier mechanisms failed and are documented so nobody repeats them:
+#   1. deleting the mirror pod does NOT restart a static pod (deletes returned 200, no restart);
+#   2. matching /proc/*/cmdline for 'kube-apiserver' also matched this script's own cmdline,
+#      so every pod SIGTERMed itself (exit 143) before reaching the real process.
+# This version matches /proc/<pid>/comm (the process name) and skips \$\$, so self-kill is
+# impossible. hostPID + privileged are permitted in the 'cicd' namespace. Delays stagger the
+# three control planes by 2 minutes each, so only one API server is down at a time.
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: oidc-restart-apiserver-cp1-v2
+  namespace: cicd
+spec:
+  backoffLimit: 0
+  ttlSecondsAfterFinished: 3600
+  template:
+    spec:
+      hostPID: true
+      restartPolicy: Never
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-1
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: kill
+          image: docker.io/library/busybox:1.36
+          securityContext:
+            privileged: true
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              echo "sleeping 10s before restarting the API server on talos-cp-1"
+              sleep 10
+              self=$$
+              killed=0
+              for d in /proc/[0-9]*; do
+                pid=${d#/proc/}
+                [ "$pid" = "$self" ] && continue
+                comm=$(cat "$d/comm" 2>/dev/null)
+                [ "$comm" = "kube-apiserver" ] || continue
+                echo "SIGTERM -> pid $pid"
+                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
+              done
+              echo "signalled $killed API server process(es); kubelet recreates the static pod"
+
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: oidc-restart-apiserver-cp2-v2
+  namespace: cicd
+spec:
+  backoffLimit: 0
+  ttlSecondsAfterFinished: 3600
+  template:
+    spec:
+      hostPID: true
+      restartPolicy: Never
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-2
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: kill
+          image: docker.io/library/busybox:1.36
+          securityContext:
+            privileged: true
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              echo "sleeping 120s before restarting the API server on talos-cp-2"
+              sleep 120
+              self=$$
+              killed=0
+              for d in /proc/[0-9]*; do
+                pid=${d#/proc/}
+                [ "$pid" = "$self" ] && continue
+                comm=$(cat "$d/comm" 2>/dev/null)
+                [ "$comm" = "kube-apiserver" ] || continue
+                echo "SIGTERM -> pid $pid"
+                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
+              done
+              echo "signalled $killed API server process(es); kubelet recreates the static pod"
+
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: oidc-restart-apiserver-cp3-v2
+  namespace: cicd
+spec:
+  backoffLimit: 0
+  ttlSecondsAfterFinished: 3600
+  template:
+    spec:
+      hostPID: true
+      restartPolicy: Never
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-3
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: kill
+          image: docker.io/library/busybox:1.36
+          securityContext:
+            privileged: true
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              echo "sleeping 240s before restarting the API server on talos-cp-3"
+              sleep 240
+              self=$$
+              killed=0
+              for d in /proc/[0-9]*; do
+                pid=${d#/proc/}
+                [ "$pid" = "$self" ] && continue
+                comm=$(cat "$d/comm" 2>/dev/null)
+                [ "$comm" = "kube-apiserver" ] || continue
+                echo "SIGTERM -> pid $pid"
+                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
+              done
+              echo "signalled $killed API server process(es); kubelet recreates the static pod"
+