fix(cert-manager): restore wildcard renewal and drop per-host TLS

The ACME DNS-01 self-check resolved through CoreDNS and never saw the TXT record. The wildcard renewal stalled on 2026-09-23 and riotpiao-com-tls expired on 2026-10-23. Every Cloudflare Tunnel origin then failed with "x509: certificate has expired or is not yet valid". Changes: - cert-manager: pin the self-check to the public resolvers with --dns01-recursive-nameservers and --dns01-recursive-nameservers-only. Verified against Let's Encrypt staging: the challenge went valid and the certificate issued. - comfyui and forgejo: drop the per-host tls blocks and the cert-manager.io/cluster-issuer annotations. The wildcard covers both hosts, and the per-host certificates (comfyui-tls, forgejo-tls) expired with the wildcard. - riotpiao-com-cert: document the wildcard as nginx's default certificate and the zero-downtime renewal path. Renewal now runs 30 days before expiry, writes the same secret, and nginx hot-reloads without a pod restart.
Merge pull request 'fix(longhorn): retire the failed 2TB PERC disk on talos-cp-2' (#62 ) from fix/retire-longhorn-disk4 into main
2026-10-24 02:30:48 -07:00 · 2026-10-13 00:10:01 +00:00 · 2026-10-12 13:41:38 +00:00 · 2026-10-12 21:24:47 +08:00 · 2026-10-12 12:54:43 +00:00 · 2026-10-12 20:54:00 +08:00
14 changed files with 405 additions and 43 deletions
@@ -4,7 +4,9 @@ metadata:
  name: comfyui
  namespace: comfyui
  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-prod
+    # No cert-manager annotation and no tls: block. This host is covered by
+    # the *.riotpiao.com wildcard, which nginx serves as its
+    # default-ssl-certificate (ingress-nginx/riotpiao-com-tls).
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
@@ -16,10 +18,6 @@ metadata:
      proxy_set_header Connection "upgrade";
 spec:
  ingressClassName: nginx
-  tls:
-    - secretName: comfyui-tls
-      hosts:
-        - comfyui.riotpiao.com
  rules:
    - host: comfyui.riotpiao.com
      http:
@@ -0,0 +1,36 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: azhi-riotpiao
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: '8'
+spec:
+  project: homelab
+  source:
+    # Private GitHub. The read-only credential is Secret azhi-infra-github in
+    # namespace argocd, created out-of-band.
+    repoURL: https://github.com/ToyoGlobal/azhi-infra.git
+    # The repository's default branch. It is not `main`: this repo's work
+    # happens on the staging slice branch, and `main` is behind it.
+    targetRevision: feat/azi-riotpiao-cnpg
+    # One directory, because ArgoCD reads a single path per source. That
+    # directory is a root kustomization over foundation, postgres and
+    # azhi-server.
+    path: k8s/overlays/azi-riotpiao
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: azi-riotpiao
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+    - CreateNamespace=true
+    # The two Secrets this namespace needs are credentials, so they are created
+    # by hand and NOT by this Application:
+    #   acr-pull                 — the private Alibaba ACR pull credential
+    #   azhi-riotpiao-app-env    — DB credentials, JWT secrets, AES key
+    # A missing imagePullSecret fails the pod outright and a missing app-env
+    # Secret leaves it in CreateContainerConfigError, which is the intended
+    # loud failure rather than a pod that runs against unknown values.
@@ -0,0 +1,20 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: oidc-bootstrap
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "5"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/infra/oidc-bootstrap
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: kube-system
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
@@ -16,6 +16,12 @@ spec:
    - https://github.com/Riotpiaole/Poimen-memory.git
    - https://github.com/Riotpiaole/Poimen-workflows.git
    - https://github.com/Riotpiaole/poimen*.git
+    # ToyoGlobal (GitHub, private). The azhi-infra Application reads its
+    # manifests from here, so the URL must be listed or ArgoCD rejects the
+    # Application with "repository not permitted". A read-only credential for
+    # this repo lives in Secret azhi-infra-github in namespace argocd; it is
+    # created out-of-band and is not committed, because it is a credential.
+    - https://github.com/ToyoGlobal/azhi-infra.git
    # In-cluster Forgejo repos — explicit allowlist (no wildcard)
    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
@@ -0,0 +1,24 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: forgejo-registry-token
+    namespace: cicd
+data:
+    username: ENC[AES256_GCM,data:oRNecs0a7rs=,iv:/5EFdNqpSjoxe7XDmAEDMd6lXd0SMsRiMmBM9SIwDHY=,tag:Wi+ggoP+9/vgz9rHrxB9nA==,type:str]
+    password: ENC[AES256_GCM,data:HbmwnWk3ST5SDWXwIhu+M5f/FOfFbJHaG0Q/IWrO63HRXmJTmLMpbjtWIXC9IMFoHeDZ1MNAV8A=,iv:ztJZgpaS//yYCiyr6/ubs/UiXkbGMnokvy4Pe45jQMk=,tag:WMpawc/SVAV4mxsCd6XqAA==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjQmhxN2E3ZnNTdkR6U2c1
+            ZStsQktkeWUwMEdmdFIyMkRYRVk1ZGppWlZ3CmUzR2JubWw2VjdWUGt6S2lpNmQ3
+            cDdSWjJiYzRIWS9UVERpVDFJenFnaDAKLS0tIGhDSmNMSjZwcDVPR3U0dUcwNk9Q
+            WmdGS1lyS3dkSXIvOFhyRklGRll2UlUK8NGts6a6Fv10c6gxIIPqIIqfgzQ3EEyk
+            SFrMThwbdaOqLY4AMeDrg+k0Y2t1Xj/XOvdNbRCRictDwZUsptvO7A==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T02:01:24Z"
+    mac: ENC[AES256_GCM,data:imebczVzG9YWd42ER4a+w+Bi3dxbN0HFm/1igxzJt6Z0HG/q6i0s59swpGVruk9dQJWrlO8KKqX739ATp2KGC5szrEuRFvPemF594XFCg6ykW+ZeP5VtTvpJg2nHv7/Pu+71ljKKi4gqOJ4SJDn8owAbSN+su6PiT323wax930A=,iv:iG/G2+qQkvbBU95w/GG2PrbAHDOaamuvjqMDBxSW6zI=,tag:wy9Vn72HBDHqCyzTcV15/Q==,type:str]
+    version: 3.13.2
@@ -0,0 +1,26 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: memory-agent-oidc
+    namespace: poimen
+data:
+    CLIENT_ID: ENC[AES256_GCM,data:c/RvtV6anofX8h1BmlKu9w==,iv:3lQ1BR8kVvF1RHVfKstmb0Y7u/uX5OrvmGiCi/xTpjw=,tag:9JC2WkfQFWaiZroAEmPenw==,type:str]
+    CLIENT_SECRET: ENC[AES256_GCM,data:I258/6O43PccyllYN/wfoKvTgNgZONTivM/KFIr8TGqSl+2XQkwBXvK2VQc5Xgc5cpwlrgnjk8pjQbqr,iv:MZyA+V5z9Po2VoxqvOSlWh1fZp1lrkcbYMx7VcSIaJs=,tag:gXH0BpA8KOZiVd+4RnhRzg==,type:str]
+    ISSUER: ENC[AES256_GCM,data:GX/uQPDoeJWXXQZlzD6oOIvxo7H4LRAv4MrPAiCgmv6WxW59+KKkymuDT/T9MiDm4bNhxVPgB7nHbDcMHfPvU2zCNM2TWYUsNOgGwxNAqM8=,iv:5OEsdrUZvMLO2C8Ea2i5alsyFGtmEk3othTemk8fOk4=,tag:8CMlRY2JCe4z4fv4WuWjRQ==,type:str]
+    TOKEN_URL: ENC[AES256_GCM,data:KeU1pQzvQ2/1lQhOqNtcR7t5TGVx1J7NY8ogxCXS6IuHhVe0zWYIbC4BYFc2H9PAjsMJwCCg75N1FrIiCYBvpfK8ln0=,iv:NbPiJC/B7W1Zm+JwnEGDikE2cb3L4l6TBeQTKEb2boM=,tag:FRmllFLZzQW03lsCu9mAUA==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmK0dPSmhKT01FSitDdUNW
+            Um1XQU9FVnBUeDdoWXpCcXRzVXh0VXkvNGxNClZaQ3R5U2NkYTJjaFdHRWhUVmFS
+            K0tHZVRFRDZGREVRZzZqcWJ6dnRpUUEKLS0tIE15UUNtMStKbWwzb3dEVGEwN1VQ
+            cktlVTM2SUhpOGJZMXJOOG84RnhCQzAKOdM0pThEaN37SJtRrKIivncMu8Z91Btq
+            es6NOlJdBUhzIhnEJWfDzZ9aA6I/fYzugKbC+AfUwdGgThAgHx9crg==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T02:01:24Z"
+    mac: ENC[AES256_GCM,data:/BTQ984D+hsk0LmnjD3hxopB65NAR54LhoAPNcATWA7MCTJXfdnYRu3gNxUzyiyCBd+rDaPp77TBnak4jVjMnOiYJsYLj1R7hrrhTZopz8pT+YNMupperwsirKZwK8it4Mu+fnpCyox2mtGED0hk+Nh/b6IteTRt5lS7noumo2M=,iv:RWIrVkNYj+8mqMSPiLueToexabzAtVbIR525m6bUrlg=,tag:5+wcNoNYsjGKpFlML+N70Q==,type:str]
+    version: 3.13.2
@@ -0,0 +1,27 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: oauth2-credentials
+    namespace: iam
+data:
+    api-gw-client-secret: ENC[AES256_GCM,data:YiUCk61cCsTJE8gTLZWhqVPsCSu3b4OnG+Oh456A+mnxip0FjLTTqV+kYBdfZivIkn9ua4RcVW+rYILu,iv:K08m1EaB7/SIVNWZ3FfeRP4tbCAo6KeiNC0NpE9VO1o=,tag:ZXV9YRWUOPJ/vdoqMtvQeA==,type:str]
+    minio-client-secret: ENC[AES256_GCM,data:vcwLo1cJqbZr2MwkAKO09xPjb09ss7+s6HIvEfaLxt8ka5PUAlXQ37pTkynuktlKq/t0DvSk+x1+5G7QqVD1JJlwyDGM6yuOhBQ3MwucLNl+T4RDKtgGuA==,iv:sCdd+UNX2TfTAM6Zslrh8KrvBSzqpe67F3YMGrl4YDU=,tag:9Tpzm9NEXNuXyCd7l7GeHw==,type:str]
+    poimen-client-secret: ENC[AES256_GCM,data:dYtCiMGZaOFokihfGD2SXJDfXrhfwEQcTOPNu7JNHDUnx0hqwAdYAPAJTEN21p0OpHYDINMs5lwUpMwY,iv:ll8USuzs0881ryHImVjHGO0iFxUzqg5XtI6Ekd+EFXE=,tag:I/4ZfB9Ss4zEP57sRtRTXQ==,type:str]
+    paperless-client-secret: ENC[AES256_GCM,data:XMj9rHkI20lI37vChq9wS5lUq7l95SisRX0LChuj09O5w76Zt9NdG6OEUYQFf1z8vCfMJxBInIEEIq0e,iv:UpYy2ZEZ4AgBACEyrlxEcr13V+8Ksu4p8JHYApmyYHo=,tag:o9Z+/b6T9POgowIsks56kw==,type:str]
+    grafana-client-secret: ENC[AES256_GCM,data:V/j0+P3HoDLqYjRalWK4v1hvJTuVmzxVra47HsF1/BZvWZjIyEIH3Pt8BekT3O1dUFynxjcaxdZMBVYg,iv:9pMp+vIsp/sA7GP9wqNDf5DdZVLDow46OURW0mMRXIc=,tag:i7VJFA5o0hToZEJ5JPWIvw==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2amZPendIOU9PeStOcmJG
+            c1dzOFVvd3RoT3dXQlJUNkdCSytEZUVGVEJJCklMUkNjUytBMXh0OVUrNUlpMWw3
+            aHRFT1hBZTZHNHVtaGJCTEFyT2ZWNWsKLS0tIENxR3FKMEplV0V5Y0dsZzBkODVG
+            U0ZlT2R0VFdEQmpsakVvZ1FJRkRQYTAKESQcSlMAuCG0/tLWWZaeoPayf1EKxSaf
+            e4o8TmjmSgPEhepQ/LmNXl6obyuqUxztkh4Mm6pocqDp+gfzt7X/1g==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T02:01:24Z"
+    mac: ENC[AES256_GCM,data:AI/znHiUa/Rwg0YRz65qUZ+b6V73oi3ZfAG/V5uZZvrGFrA0EISzD5LaP5Kl4GUU1svo4G5EI4aftPDgpdPptz21CCa8iYkXGe9pTDmtaE5ZICQAUbRcVhGMTfuWFhrgEhMfCJdOpfHaIQVEsQJQJgrnkXQVuUe9aT75LQz7pnY=,iv:KbIaYT0VeBT2WeSU5V5ZB4e+04JICYaY+egGN1Pv5bM=,tag:pPbhDxeLUx6V1XC/LPwxTg==,type:str]
+    version: 3.13.2
@@ -7,31 +7,34 @@ metadata:
      exec:
        path: ksops
 files:
-  - agent-pod-models.enc.yaml
-  - agent-pod-ssh-key.enc.yaml
-  - authentik-secrets.enc.yaml
-  - cloudflare-secrets.enc.yaml
-  - forgejo-registry-pat.enc.yaml
-  - forgejo-registry-pat-api.enc.yaml
-  - forgejo-registry-pull.enc.yaml
-  - forgejo-runner-token.enc.yaml
-  - forgejo-secrets.enc.yaml
-  - grafana-oidc-secrets.enc.yaml
-  - grafana-secrets.enc.yaml
-  - homarr-auth-oidc.enc.yaml
-  - homarr-db-encryption.enc.yaml
-  - homarr-secrets.enc.yaml
-  - homelab-ca-secrets.enc.yaml
-  - loki-secrets.enc.yaml
-  - minio-secrets.enc.yaml
-  - paperless-secrets.enc.yaml
-  - vault-secrets.enc.yaml
-  - vault-unseal-keys.enc.yaml
-  - portfolio-secrets.enc.yaml
-  - gotify-admin-secrets.enc.yaml
-  - gotify-tokens-secrets.enc.yaml
-  - gotify-smtp-secrets.enc.yaml
-  - forgejo-smtp-secrets.enc.yaml
-  - paperless-ai-secrets.enc.yaml
+    - agent-pod-models.enc.yaml
+    - agent-pod-ssh-key.enc.yaml
+    - authentik-secrets.enc.yaml
+    - cloudflare-secrets.enc.yaml
+    - forgejo-registry-pat.enc.yaml
+    - forgejo-registry-pat-api.enc.yaml
+    - forgejo-registry-pull.enc.yaml
+    - forgejo-runner-token.enc.yaml
+    - forgejo-secrets.enc.yaml
+    - grafana-oidc-secrets.enc.yaml
+    - grafana-secrets.enc.yaml
+    - homarr-auth-oidc.enc.yaml
+    - homarr-db-encryption.enc.yaml
+    - homarr-secrets.enc.yaml
+    - homelab-ca-secrets.enc.yaml
+    - loki-secrets.enc.yaml
+    - minio-secrets.enc.yaml
+    - paperless-secrets.enc.yaml
+    - vault-secrets.enc.yaml
+    - vault-unseal-keys.enc.yaml
+    - portfolio-secrets.enc.yaml
+    - gotify-admin-secrets.enc.yaml
+    - gotify-tokens-secrets.enc.yaml
+    - gotify-smtp-secrets.enc.yaml
+    - forgejo-smtp-secrets.enc.yaml
+    - paperless-ai-secrets.enc.yaml
+    - memory-agent-oidc.enc.yaml
+    - oauth2-credentials.enc.yaml
+    - forgejo-registry-token.enc.yaml
    - pi-sandbox-secrets.enc.yaml
    - etcd-defrag-creds.enc.yaml
@@ -8,6 +8,19 @@
 # depend on them existing first.
 installCRDs: true

+# Point the ACME DNS-01 self-check at the public resolvers directly.
+#
+# Without these flags cert-manager resolves through the pod resolver
+# (CoreDNS at 10.96.0.10). The self-check then never saw the TXT record and
+# reported "Waiting for DNS-01 challenge propagation" forever. The wildcard
+# certificate renewal stalled on 2026-09-23 and the cert expired on
+# 2026-10-23, which broke every Cloudflare Tunnel origin with
+# "x509: certificate has expired".
+#
+extraArgs:
+  - --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53
+  - --dns01-recursive-nameservers-only
+
 # Tolerate the control-plane taint (talos-cp-1 is the only schedulable node).
 tolerations:
  - key: node-role.kubernetes.io/control-plane
@@ -1,13 +1,21 @@
 # k8s/bootstrap/ingress/riotpiao-com-cert.yaml
 # Public wildcard cert for *.riotpiao.com, signed by Let's Encrypt (DNS-01 via
-# Cloudflare, see ../cert-manager/letsencrypt-issuer.yaml). Distinct from
-# homelab-tls (self-signed homelab-ca, internal *.riotpiao.com) —
-# reference riotpiao-com-tls explicitly on ingresses that expose riotpiao.com
-# hosts to the internet; it is not nginx's default-ssl-certificate.
+# Cloudflare, see ../cert-manager/letsencrypt-issuer.yaml).
 #
-# issuerRef starts on letsencrypt-staging. Switch to letsencrypt-prod once the
-# staging cert issues cleanly, then delete this Certificate's Secret to force
-# a re-issue against prod (staging certs aren't browser-trusted).
+# nginx serves this secret as its default-ssl-certificate
+# (../ingress/nginx-values.yaml), so every Ingress that omits a tls: block uses
+# it. Do not add per-host tls: blocks for *.riotpiao.com hosts: they create a
+# second, single-host certificate and split the wildcard coverage.
+#
+# Renewal is automatic and does not drop traffic:
+#   - cert-manager starts a new order 30 days before expiry (renewBefore).
+#   - cert-manager writes the renewed pair into the same secret.
+#   - ingress-nginx watches the secret and hot-reloads. No pod restart.
+#   - The old pair stays valid until the new one is written, so no gap exists.
+#
+# The DNS-01 self-check must reach the public resolvers directly. Without the
+# --dns01-recursive-nameservers flags in ../../cert-manager/cert-manager-values.yaml
+# the self-check never sees the TXT record and renewal stalls until expiry.
 apiVersion: cert-manager.io/v1
 kind: Certificate
 metadata:
@@ -107,7 +107,9 @@ ingress:
  enabled: true
  className: nginx
  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-prod
+    # No cert-manager annotation. forgejo.riotpiao.com is covered by the
+    # *.riotpiao.com wildcard, served as nginx's default-ssl-certificate.
+    #
    # This chart Ingress and the hand-written one in
    # k8s/bootstrap/ingress/ingress.yaml both claim forgejo.riotpiao.com.
    # ingress-nginx breaks the tie by oldest creationTimestamp, and the chart's
@@ -125,10 +127,9 @@ ingress:
      paths:
        - path: /
          pathType: Prefix
-  tls:
-    - secretName: forgejo-tls
-      hosts:
-        - forgejo.riotpiao.com
+  # tls: intentionally omitted. The chart would create a second, single-host
+  # certificate (forgejo-tls) that expired on 2026-10-23 and competed with the
+  # wildcard. The wildcard covers this host.

 # Resources
 resources:
@@ -0,0 +1,59 @@
+# One-shot bootstrap: restart the control-plane apiservers ONCE so they pick up the
+# OIDC authenticator (they last started before the Authentik provider existed, so OIDC
+# tokens were never accepted -> every login got 401). Delete this whole directory after.
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: oidc-bootstrap
+  namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: oidc-bootstrap
+  namespace: kube-system
+rules:
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list", "delete"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: oidc-bootstrap
+  namespace: kube-system
+subjects:
+  - kind: ServiceAccount
+    name: oidc-bootstrap
+    namespace: kube-system
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: oidc-bootstrap
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: oidc-bootstrap-restart-apiserver-v2
+  namespace: kube-system
+spec:
+  backoffLimit: 4
+  ttlSecondsAfterFinished: 7200
+  template:
+    spec:
+      serviceAccountName: oidc-bootstrap
+      restartPolicy: Never
+      containers:
+        - name: restart
+          image: docker.io/curlimages/curl:8.22.0
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              set -u
+              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+              for pod in kube-apiserver-talos-cp-1 kube-apiserver-talos-cp-2 kube-apiserver-talos-cp-3; do
+                curl -sS --max-time 15 --cacert "$CA" -H "Authorization: Bearer $TOKEN"                   -X DELETE "https://kubernetes.default.svc/api/v1/namespaces/kube-system/pods/$pod?gracePeriodSeconds=0"                   -o /dev/null -w "$pod -> HTTP %{http_code}\n" || echo "$pod -> request failed"
+              done
+              echo "deletes requested; kubelet recreates each static pod in ~20-40s"
@@ -0,0 +1,141 @@
+# One-shot: restart the control-plane API servers so their OIDC authenticator initialises.
+#
+# Two earlier mechanisms failed and are documented so nobody repeats them:
+#   1. deleting the mirror pod does NOT restart a static pod (deletes returned 200, no restart);
+#   2. matching /proc/*/cmdline for 'kube-apiserver' also matched this script's own cmdline,
+#      so every pod SIGTERMed itself (exit 143) before reaching the real process.
+# This version matches /proc/<pid>/comm (the process name) and skips \$\$, so self-kill is
+# impossible. hostPID + privileged are permitted in the 'cicd' namespace. Delays stagger the
+# three control planes by 2 minutes each, so only one API server is down at a time.
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: oidc-restart-apiserver-cp1-v2
+  namespace: cicd
+spec:
+  backoffLimit: 0
+  ttlSecondsAfterFinished: 3600
+  template:
+    spec:
+      hostPID: true
+      restartPolicy: Never
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-1
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: kill
+          image: docker.io/library/busybox:1.36
+          securityContext:
+            privileged: true
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              echo "sleeping 10s before restarting the API server on talos-cp-1"
+              sleep 10
+              self=$$
+              killed=0
+              for d in /proc/[0-9]*; do
+                pid=${d#/proc/}
+                [ "$pid" = "$self" ] && continue
+                comm=$(cat "$d/comm" 2>/dev/null)
+                [ "$comm" = "kube-apiserver" ] || continue
+                echo "SIGTERM -> pid $pid"
+                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
+              done
+              echo "signalled $killed API server process(es); kubelet recreates the static pod"
+
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: oidc-restart-apiserver-cp2-v2
+  namespace: cicd
+spec:
+  backoffLimit: 0
+  ttlSecondsAfterFinished: 3600
+  template:
+    spec:
+      hostPID: true
+      restartPolicy: Never
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-2
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: kill
+          image: docker.io/library/busybox:1.36
+          securityContext:
+            privileged: true
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              echo "sleeping 120s before restarting the API server on talos-cp-2"
+              sleep 120
+              self=$$
+              killed=0
+              for d in /proc/[0-9]*; do
+                pid=${d#/proc/}
+                [ "$pid" = "$self" ] && continue
+                comm=$(cat "$d/comm" 2>/dev/null)
+                [ "$comm" = "kube-apiserver" ] || continue
+                echo "SIGTERM -> pid $pid"
+                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
+              done
+              echo "signalled $killed API server process(es); kubelet recreates the static pod"
+
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: oidc-restart-apiserver-cp3-v2
+  namespace: cicd
+spec:
+  backoffLimit: 0
+  ttlSecondsAfterFinished: 3600
+  template:
+    spec:
+      hostPID: true
+      restartPolicy: Never
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-3
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: kill
+          image: docker.io/library/busybox:1.36
+          securityContext:
+            privileged: true
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              echo "sleeping 240s before restarting the API server on talos-cp-3"
+              sleep 240
+              self=$$
+              killed=0
+              for d in /proc/[0-9]*; do
+                pid=${d#/proc/}
+                [ "$pid" = "$self" ] && continue
+                comm=$(cat "$d/comm" 2>/dev/null)
+                [ "$comm" = "kube-apiserver" ] || continue
+                echo "SIGTERM -> pid $pid"
+                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
+              done
+              echo "signalled $killed API server process(es); kubelet recreates the static pod"
+
Author	SHA1	Message	Date
rock	e93d4ba86c	fix(cert-manager): restore wildcard renewal and drop per-host TLS The ACME DNS-01 self-check resolved through CoreDNS and never saw the TXT record. The wildcard renewal stalled on 2026-09-23 and riotpiao-com-tls expired on 2026-10-23. Every Cloudflare Tunnel origin then failed with "x509: certificate has expired or is not yet valid". Changes: - cert-manager: pin the self-check to the public resolvers with --dns01-recursive-nameservers and --dns01-recursive-nameservers-only. Verified against Let's Encrypt staging: the challenge went valid and the certificate issued. - comfyui and forgejo: drop the per-host tls blocks and the cert-manager.io/cluster-issuer annotations. The wildcard covers both hosts, and the per-host certificates (comfyui-tls, forgejo-tls) expired with the wildcard. - riotpiao-com-cert: document the wildcard as nginx's default certificate and the zero-downtime renewal path. Renewal now runs 30 days before expiry, writes the same secret, and nginx hot-reloads without a pod restart.	2026-10-24 02:30:48 -07:00
rock	7a1213d504	Merge pull request 'fix(longhorn): retire the failed 2TB PERC disk on talos-cp-2' (#62 ) from fix/retire-longhorn-disk4 into main	2026-10-13 00:10:01 +00:00
rock	40aa23dfdf	Merge pull request 'test(argocd): point azhi-riotpiao at the CNPG branch' (#61 ) from feat/azhi-cnpg-test into main	2026-10-12 13:41:38 +00:00
rock	71fce30506	test(argocd): point azhi-riotpiao at the CNPG branch The StatefulSet database could not attach its Longhorn volume, so the branch runs the database as a CNPG Cluster instead. This pins the Application to that branch for the test; it returns to the default branch once the branch merges.	2026-10-12 21:24:47 +08:00
rock	0666715a5e	Merge pull request 'feat(argocd): reconcile azhi-infra from its private GitHub repo' (#60 ) from feat/azhi-riotpiao-argocd into main	2026-10-12 12:54:43 +00:00
rock	949ae5cb38	feat(argocd): let ArgoCD reconcile azhi-infra from its private GitHub repo The homelab AppProject allowlist had no azhi repository, so an Application pointing at ToyoGlobal/azhi-infra fails admission with 'repository not permitted'. Adding the URL is the whole gate. The Application itself targets the repository default branch (harness/staging-slice), not main, and path k8s/overlays/azi-riotpiao. That overlay renders postgres and the API into namespace azi-riotpiao. The two Secrets the namespace needs stay out of Git: acr-pull carries the ACR pull credential and azhi-riotpiao-app-env carries the database credentials, the JWT secrets and the AES key.	2026-10-12 20:54:00 +08:00
rock	e150969070	Merge pull request #59 from fix/etcd-defrag-main	2026-10-12 12:27:28 +00:00
rock	abfdda5f57	fix(etcd): add a working 15-day defrag CronJob under the backend quota etcd's backend file only grows: the apiserver compacts every 5m, but compaction marks superseded revisions free inside the file, and only defrag returns that space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached 2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets could not post node status (3 nodes NotReady), apiserver writes returned 500 and the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after compaction - the rest was dead pages. Compaction had also stalled while the alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have reclaimed nothing. Compact must come first. An earlier revision of this CronJob never ran, for two independent reasons, both fixed here: - its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c] failed with exec: "/bin/sh": no such file or directory - it existed only on a side branch while the ArgoCD application cluster-maintenance tracks main, so the cluster never had it Contents: - CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the day-of-month field fires on the 1st, 16th and 31st, putting two runs one day apart. - Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4 against the 3.6.11 servers. - Compaction uses the MINIMUM revision across members, so a lagging member is never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat ~1.2M entries behind the leader with a frozen raft term and needed a reboot to rejoin. - Defrag skips any member whose applied index lags the leader by 100k+, and runs one member at a time since defrag pauses that member's apply loop. - Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a client certificate signed by the cluster's etcd CA, valid to 2036-10.	2026-10-07 17:12:39 +08:00
rock	2d9e3abee8	fix(oidc): match /proc/<pid>/comm instead of cmdline when signalling the API server Exit code 143 proved the previous job killed itself: the script's own cmdline contains the string 'kube-apiserver', so it matched its own process. Matching comm (== 'kube-apiserver') and skipping $$ removes that failure mode. Staggered 2 minutes apart.	2026-10-07 04:07:37 +00:00
rock	9fa8d9cf60	fix(oidc): restart the apiservers by signalling the process, not by deleting the mirror pod Static pods survive mirror-pod deletion, so the first bootstrap job had no effect. These jobs share the host PID namespace and SIGTERM the kube-apiserver process itself, staggered by 100s so only one control plane restarts at a time.	2026-10-07 03:59:22 +00:00
rock	2e4fad3bf7	fix(oidc): use curlimages/curl for the bootstrap job (bitnami/kubectl tag did not exist) The first job could not pull its image; this one deletes the apiserver pods via the in-cluster API with the mounted service-account token instead of a kubectl image.	2026-10-07 03:52:10 +00:00
rock	5d22691c3e	chore(oidc): one-shot bootstrap - restart apiservers to initialise the OIDC authenticator k8s/argocd/apps/99-oidc-bootstrap.yaml	2026-10-07 03:46:01 +00:00
rock	4eee161e0f	chore(oidc): one-shot bootstrap - restart apiservers to initialise the OIDC authenticator k8s/infra/oidc-bootstrap/bootstrap.yaml	2026-10-07 03:45:56 +00:00
rock	719fa77521	Merge pull request 'fix(secrets): populate the empty sops placeholders and drop the unused one' (#55 ) from fix/secrets-placeholders into main	2026-10-06 04:48:51 +00:00
rock	64cd1f2c4e	fix(minio): freeze the community image by digest and alert on pull failure (#57 ) Co-authored-by: rock <[email protected]>	2026-10-06 03:43:12 +00:00
rock	f780e31f9c	Merge origin/main into fix/secrets-placeholders (rebuild ksops file list, fix #54 indent)	2026-10-06 10:05:45 +08:00
rock	c93a24f786	fix(secrets): populate the empty sops placeholders and drop the unused one (cherry picked from commit `f6207ff2b0`)	2026-10-06 10:03:04 +08:00