feat(grafana): provision a Gotify contact point

Grafana gains one contact point, `gotify`, of type webhook. It posts to the bridge in the notifications namespace because Gotify rejects a Grafana payload with 400 "Field 'message' is required". maxAlerts 0 keeps every alert in one request, and the bridge names the extra count in the message body instead of dropping alerts.
feat(gotify): bridge Grafana alerts into Gotify
2026-10-27 12:38:29 -07:00 · 2026-10-27 12:35:44 -07:00 · 2026-10-27 12:14:28 -07:00 · 2026-10-27 12:12:22 -07:00 · 2026-10-27 12:12:22 -07:00 · 2026-10-27 12:08:13 -07:00
38 changed files with 895 additions and 1484 deletions
@@ -53,8 +53,4 @@ data:
      # even though the DNS CNAME points at the tunnel.
      - hostname: zitadel.riotpiao.com
        service: https://zitadel.riotpiao.com
-      # AUTHENTIK. Same reasoning as ZITADEL: a hostname absent from this table
-      # gets 404 from the connector. CoreDNS rewrites the name to ingress-nginx.
-      - hostname: authentik.riotpiao.com
-        service: https://authentik.riotpiao.com
      - service: http_status:404
@@ -16,7 +16,7 @@ spec:
      # Ingress-nginx forces HTTPS (force-ssl-redirect), so cloudflared must
      # dial https://<hostname> and trust homelab-ca or the TLS handshake fails
      # with "certificate signed by unknown authority". Same pattern as
-      # authentik-values.yaml: merge the system bundle with homelab-ca.crt.
+      # iam values: merge the system bundle with homelab-ca.crt.
      initContainers:
        - name: merge-ca-certs
          image: debian:bookworm
@@ -0,0 +1,110 @@
+# The bridge script is mounted as a ConfigMap so no image build is needed.
+# Regenerate this block from grafana_gotify_bridge.py when the script changes.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-gotify-bridge
+  namespace: notifications
+data:
+  grafana_gotify_bridge.py: |
+    #!/usr/bin/env python3
+    """Translate a Grafana alert webhook into a Gotify message.
+    
+    Grafana sends its own payload shape:
+      {"receiver": "...", "status": "firing", "alerts": [{"labels": {...}, "annotations": {...}}]}
+    
+    Gotify requires:
+      {"title": "...", "message": "...", "priority": 0..10}
+    
+    Gotify answers 400 "Field 'message' is required" to a Grafana payload, so a
+    conversion step is mandatory. This service is that step and nothing else.
+    Standard library only, so it needs no image build and no dependencies.
+    """
+    import json
+    import os
+    import urllib.request
+    from http.server import BaseHTTPRequestHandler, HTTPServer
+    
+    GOTIFY_BASE = os.environ.get("GOTIFY_URL", "http://gotify.notifications.svc.cluster.local")
+    GOTIFY_TOKEN = os.environ["GOTIFY_TOKEN"]
+    LISTEN_PORT = int(os.environ.get("PORT", "8080"))
+    
+    # Firing is louder than resolved. Gotify priority runs 0 to 10.
+    PRIORITY = {"firing": 8, "resolved": 3}
+    
+    
+    def to_gotify(payload):
+        alerts = payload.get("alerts") or []
+        status = (payload.get("status") or "firing").lower()
+        priority = PRIORITY.get(status, 5)
+    
+        if not alerts:
+            title = payload.get("title") or "Grafana notification"
+            return title, json.dumps(payload)[:2000], priority
+    
+        first = alerts[0]
+        labels = first.get("labels") or {}
+        annotations = first.get("annotations") or {}
+        name = labels.get("alertname", "Grafana alert")
+    
+        title = "[%s] %s" % (status.upper(), name)
+        lines = []
+        summary = annotations.get("summary") or annotations.get("description")
+        if summary:
+            lines.append(summary)
+        for key in ("severity", "namespace", "pod", "instance", "job"):
+            if labels.get(key):
+                lines.append("%s: %s" % (key, labels[key]))
+        if len(alerts) > 1:
+            lines.append("+%d more alert(s) in this notification" % (len(alerts) - 1))
+    
+        return title, "\n".join(lines) or name, priority
+    
+    
+    class Handler(BaseHTTPRequestHandler):
+        def _reply(self, code, body=b"ok"):
+            self.send_response(code)
+            self.send_header("Content-Type", "text/plain")
+            self.end_headers()
+            self.wfile.write(body)
+    
+        def do_GET(self):
+            self._reply(200)
+    
+        def do_POST(self):
+            length = int(self.headers.get("Content-Length") or 0)
+            raw = self.rfile.read(length) if length else b"{}"
+            try:
+                payload = json.loads(raw.decode("utf-8", "replace") or "{}")
+            except ValueError:
+                payload = {}
+    
+            title, message, priority = to_gotify(payload)
+            body = json.dumps({
+                "title": title[:200],
+                "message": message[:4000],
+                "priority": priority,
+            }).encode()
+    
+            request = urllib.request.Request(
+                GOTIFY_BASE.rstrip("/") + "/message",
+                data=body,
+                headers={"Content-Type": "application/json", "X-Gotify-Key": GOTIFY_TOKEN},
+            )
+            try:
+                urllib.request.urlopen(request, timeout=10).read()
+            except Exception as exc:  # noqa: BLE001 - log and report to Grafana
+                print("forward to gotify failed: %s" % exc, flush=True)
+                self._reply(502, b"gotify rejected the message")
+                return
+    
+            print("forwarded: %s" % title, flush=True)
+            self._reply(200)
+    
+        def log_message(self, fmt, *args):  # keep the log to one line per delivery
+            print("%s - %s" % (self.address_string(), fmt % args), flush=True)
+    
+    
+    if __name__ == "__main__":
+        print("grafana-gotify bridge listening on %d" % LISTEN_PORT, flush=True)
+        HTTPServer(("0.0.0.0", LISTEN_PORT), Handler).serve_forever()
@@ -0,0 +1,80 @@
+# Grafana sends its own JSON and Gotify rejects it, so this Deployment is the
+# translation step between them. Grafana posts to the Service, the bridge posts
+# to /message with the app token from a SOPS secret.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: grafana-gotify-bridge
+  namespace: notifications
+  labels:
+    app: grafana-gotify-bridge
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: grafana-gotify-bridge
+  template:
+    metadata:
+      labels:
+        app: grafana-gotify-bridge
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: bridge
+          image: python:3.12-alpine
+          command: ["python3", "/app/grafana_gotify_bridge.py"]
+          env:
+            - name: GOTIFY_URL
+              value: "http://gotify.notifications.svc.cluster.local"
+            - name: PORT
+              value: "8080"
+            - name: GOTIFY_TOKEN
+              valueFrom:
+                secretKeyRef: {name: grafana-gotify-bridge, key: app-token}
+          ports:
+            - name: http
+              containerPort: 8080
+          readinessProbe:
+            httpGet: {path: /, port: http}
+            initialDelaySeconds: 3
+            periodSeconds: 10
+          livenessProbe:
+            httpGet: {path: /, port: http}
+            initialDelaySeconds: 10
+            periodSeconds: 20
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          volumeMounts:
+            - name: script
+              mountPath: /app
+              readOnly: true
+          resources:
+            requests: {cpu: 10m, memory: 32Mi}
+            limits: {cpu: 200m, memory: 96Mi}
+      volumes:
+        - name: script
+          configMap:
+            name: grafana-gotify-bridge
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: grafana-gotify-bridge
+  namespace: notifications
+  labels:
+    app: grafana-gotify-bridge
+spec:
+  type: ClusterIP
+  selector:
+    app: grafana-gotify-bridge
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http
@@ -0,0 +1,101 @@
+#!/usr/bin/env python3
+"""Translate a Grafana alert webhook into a Gotify message.
+
+Grafana sends its own payload shape:
+  {"receiver": "...", "status": "firing", "alerts": [{"labels": {...}, "annotations": {...}}]}
+
+Gotify requires:
+  {"title": "...", "message": "...", "priority": 0..10}
+
+Gotify answers 400 "Field 'message' is required" to a Grafana payload, so a
+conversion step is mandatory. This service is that step and nothing else.
+Standard library only, so it needs no image build and no dependencies.
+"""
+import json
+import os
+import urllib.request
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+GOTIFY_BASE = os.environ.get("GOTIFY_URL", "http://gotify.notifications.svc.cluster.local")
+GOTIFY_TOKEN = os.environ["GOTIFY_TOKEN"]
+LISTEN_PORT = int(os.environ.get("PORT", "8080"))
+
+# Firing is louder than resolved. Gotify priority runs 0 to 10.
+PRIORITY = {"firing": 8, "resolved": 3}
+
+
+def to_gotify(payload):
+    alerts = payload.get("alerts") or []
+    status = (payload.get("status") or "firing").lower()
+    priority = PRIORITY.get(status, 5)
+
+    if not alerts:
+        title = payload.get("title") or "Grafana notification"
+        return title, json.dumps(payload)[:2000], priority
+
+    first = alerts[0]
+    labels = first.get("labels") or {}
+    annotations = first.get("annotations") or {}
+    name = labels.get("alertname", "Grafana alert")
+
+    title = "[%s] %s" % (status.upper(), name)
+    lines = []
+    summary = annotations.get("summary") or annotations.get("description")
+    if summary:
+        lines.append(summary)
+    for key in ("severity", "namespace", "pod", "instance", "job"):
+        if labels.get(key):
+            lines.append("%s: %s" % (key, labels[key]))
+    if len(alerts) > 1:
+        lines.append("+%d more alert(s) in this notification" % (len(alerts) - 1))
+
+    return title, "\n".join(lines) or name, priority
+
+
+class Handler(BaseHTTPRequestHandler):
+    def _reply(self, code, body=b"ok"):
+        self.send_response(code)
+        self.send_header("Content-Type", "text/plain")
+        self.end_headers()
+        self.wfile.write(body)
+
+    def do_GET(self):
+        self._reply(200)
+
+    def do_POST(self):
+        length = int(self.headers.get("Content-Length") or 0)
+        raw = self.rfile.read(length) if length else b"{}"
+        try:
+            payload = json.loads(raw.decode("utf-8", "replace") or "{}")
+        except ValueError:
+            payload = {}
+
+        title, message, priority = to_gotify(payload)
+        body = json.dumps({
+            "title": title[:200],
+            "message": message[:4000],
+            "priority": priority,
+        }).encode()
+
+        request = urllib.request.Request(
+            GOTIFY_BASE.rstrip("/") + "/message",
+            data=body,
+            headers={"Content-Type": "application/json", "X-Gotify-Key": GOTIFY_TOKEN},
+        )
+        try:
+            urllib.request.urlopen(request, timeout=10).read()
+        except Exception as exc:  # noqa: BLE001 - log and report to Grafana
+            print("forward to gotify failed: %s" % exc, flush=True)
+            self._reply(502, b"gotify rejected the message")
+            return
+
+        print("forwarded: %s" % title, flush=True)
+        self._reply(200)
+
+    def log_message(self, fmt, *args):  # keep the log to one line per delivery
+        print("%s - %s" % (self.address_string(), fmt % args), flush=True)
+
+
+if __name__ == "__main__":
+    print("grafana-gotify bridge listening on %d" % LISTEN_PORT, flush=True)
+    HTTPServer(("0.0.0.0", LISTEN_PORT), Handler).serve_forever()
@@ -28,9 +28,17 @@ spec:
              value: "true"
            - name: ENABLE_SCALAR_UI
              value: "true"
-            # GOTIFY_URLS / GOTIFY_CLIENT_TOKENS / SECNTFY_TOKENS intentionally
-            # left unset — not required when Gotify is reachable via public domain
-            # (gotify.riotpiao.com). iGotify iOS app registers itself at runtime.
+            # The bridge authenticates with a Gotify client token, not with
+            # ZITADEL. Gotify has no way to accept a forwarded identity, so the
+            # token is the only credential that works server-side. The token
+            # belongs to user `rock`, never expires, and is revoked by deleting
+            # the client row in Gotify.
+            - name: GOTIFY_URLS
+              value: "https://gotify.riotpiao.com"
+            - name: GOTIFY_CLIENT_TOKENS
+              valueFrom:
+                secretKeyRef: {name: igotify-client-token, key: client-token}
+            # SECNTFY_TOKENS intentionally left unset: secntfy is not deployed.
          volumeMounts:
            - name: data
              mountPath: /app/data
@@ -6,6 +6,8 @@ resources:
  - deployment.yaml
  - service.yaml
  - ingress.yaml
+  - bridge/configmap.yaml
+  - bridge/deployment.yaml
  - igotify-pvc.yaml
  - igotify-deployment.yaml
  - igotify-service.yaml
@@ -1,107 +0,0 @@
-apiVersion: serving.kserve.io/v1beta1
-kind: InferenceService
-metadata:
-  annotations:
-    serving.kserve.io/deploymentMode: RawDeployment
-    # The konghq.com/{connect,read,write}-timeout annotations that used to live
-    # here went with Kong (retired 2026-08-19). They existed because Kong read
-    # its upstream timeouts off the Kubernetes Service, and its 60s default cut
-    # off the first request after any pod restart — a restart flushes VRAM and
-    # reloading ornith:35b takes longer than that. OLLAMA_KEEP_ALIVE=-1 hid the
-    # problem in steady state.
-    #
-    # The equivalent budget now belongs to the Go gateway's per-route timeout
-    # config in rock/homelab-frontend, not to an annotation on this object.
-  labels:
-    app.kubernetes.io/name: llm-ornith
-    app.kubernetes.io/part-of: llm-serving
-  name: ornith
-  namespace: llm-serving
-spec:
-  predictor:
-    containers:
-    - command:
-      - /bin/sh
-      - -c
-      - 'set -e
-
-        ollama serve &
-
-        SERVE_PID=$!
-
-        until ollama list >/dev/null 2>&1; do sleep 2; done
-
-        ollama pull ornith:35b
-
-        ollama run ornith:35b "ok" >/dev/null 2>&1 || true
-
-        wait $SERVE_PID
-
-        '
-      env:
-      - name: OLLAMA_HOST
-        value: 0.0.0.0:8080
-      - name: OLLAMA_MODELS
-        value: /mnt/models/ollama
-      - name: OLLAMA_CONTEXT_LENGTH
-        value: '65536'
-      - name: OLLAMA_KEEP_ALIVE
-        value: '-1'
-      - name: OLLAMA_NUM_PARALLEL
-        value: '1'
-      - name: OLLAMA_MAX_LOADED_MODELS
-        value: '1'
-      image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
-      name: kserve-container
-      ports:
-      - containerPort: 8080
-        protocol: TCP
-      readinessProbe:
-        exec:
-          command:
-          - /bin/sh
-          - -c
-          - ollama ps 2>/dev/null | grep -q ornith
-        periodSeconds: 10
-      resources:
-        limits:
-          cpu: '16'
-          memory: 16Gi
-          nvidia.com/gpu: '1'
-        requests:
-          cpu: '8'
-          memory: 8Gi
-          nvidia.com/gpu: '1'
-      startupProbe:
-        exec:
-          command:
-          - /bin/sh
-          - -c
-          - ollama ps 2>/dev/null | grep -q ornith
-        failureThreshold: 120
-        periodSeconds: 15
-      volumeMounts:
-      - mountPath: /mnt/models
-        name: models
-    podMetadata:
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "8080"
-        prometheus.io/path: "/metrics"
-      labels:
-        app.kubernetes.io/name: llm-ornith
-        app.kubernetes.io/part-of: llm-serving
-    deploymentStrategy:
-      type: Recreate
-    # 1 replica -- ornith:35b only. qwen2.5:3b moved to CPU on cp-2.
-    # Frees 1 GPU for ComfyUI.
-    maxReplicas: 1
-    minReplicas: 1
-    nodeSelector:
-      kubernetes.io/hostname: worker-1
-    runtimeClassName: nvidia
-    volumes:
-    - name: models
-      persistentVolumeClaim:
-        claimName: llm-models
-
@@ -1,162 +0,0 @@
-apiVersion: serving.kserve.io/v1beta1
-kind: InferenceService
-metadata:
-  annotations:
-    serving.kserve.io/deploymentMode: RawDeployment
-  labels:
-    app.kubernetes.io/name: llm-reasoning
-    app.kubernetes.io/part-of: llm-serving
-  name: reasoning
-  namespace: llm-serving
-spec:
-  predictor:
-    containers:
-    - args:
-      # bnb-4bit retired: no int4 tensor cores on sm70/V100, dequant-then-
-      # matmul is two slow kernel launches instead of one fused int4 GEMM,
-      # decode crawled at 2.5-10 tok/s regardless of TP/PP. Switched to
-      # JunHowie/Qwen3-32B-GPTQ-Int4 -- same dense Qwen3-32B weights, same
-      # hermes/qwen3 parser stack (no narration-bug risk, same as before),
-      # only the quant format changes. Plain (non-Marlin) GPTQ kernel is
-      # confirmed Volta-compatible; Marlin needs sm80+ and vLLM would try
-      # to auto-upgrade to it, so --quantization is pinned explicitly to
-      # `gptq` to force the plain kernel. Verified checkpoint size: 19.34GB
-      # (summed from the real safetensors index, not bits-per-param math).
-      # max-model-len=131072 is Qwen3-32B's real ceiling (config.json YaRN:
-      # factor=4.0, original_max_position_embeddings=32768) -- 200k was
-      # asked for but exceeds this architecturally regardless of VRAM.
-      # KV cache math: 256KB/token total (64 layers, 8 KV heads, 128
-      # head_dim, fp16), PP=2 splits both weights and KV load ~evenly, so
-      # each GPU carries ~9.67GB weights + ~128KB/token KV. At
-      # gpu-memory-utilization=0.95 (30.4GB/GPU usable), that leaves
-      # ~20.7GB/GPU for KV cache. NOTE this is a projection, not a
-      # measurement: the live reading at 0.90 is num_gpu_blocks=9527 x
-      # block_size=16 = 152,432 tokens (vllm:cache_config_info). Re-read that
-      # metric after the rollout -- max_num_batched_tokens=8192 also raises
-      # activation memory out of the same budget, so the post-boot
-      # num_gpu_blocks is the authoritative number, and --max-num-seqs=5 is
-      # only valid if it still covers 2x65k + 3x8k. The old
-      # OffloadingConnector CPU-DRAM spillover (tuned for the previous
-      # model's much smaller 16384 context) is no longer needed and is
-      # dropped. Staying on PP=2 and vLLM 0.11.0 (no version bump needed,
-      # this checkpoint only requires vllm>=0.9.2) -- plain GPTQ has no
-      # TP>1 restriction unlike bnb, so tensor-parallel-size=2 is worth
-      # trying later, but not risking a parallelism-strategy change in the
-      # same rollout as the quant+context-length change.
-      # This GPTQ requant's own config.json ships max_position_embeddings=
-      # 40960 and rope_scaling=None -- confirmed directly (curl'd the raw
-      # config.json), the base Qwen3-32B repo's YaRN block did NOT carry
-      # over during quantization. Re-applying it explicitly here restores
-      # the same math the base model documents (32768 * 4.0 = 131072);
-      # without this, --max-model-len=131072 fails ModelConfig validation
-      # against the checkpoint's own (unscaled) 40960 ceiling.
-      - --model=JunHowie/Qwen3-32B-GPTQ-Int4
-      - --served-model-name=reasoning
-      - --quantization=gptq
-      - --dtype=float16
-      - --kv-cache-dtype=auto
-      - --rope-scaling={"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}
-      - --tensor-parallel-size=1
-      - --pipeline-parallel-size=2
-      - --max-model-len=131072
-      - --gpu-memory-utilization=0.95
-      - --max-num-seqs=5
-      # Prefill scheduling -- see the measured starvation this fixes.
-      # V1's scheduler has no prefill/decode phase: it walks self.running and
-      # hands out tokens from one shared budget (vllm/v1/core/sched/scheduler.py:
-      # "There's no decoding phase nor prefill phase in the scheduler"). The cap
-      # per request is applied BEFORE the budget clamp:
-      #     if 0 < long_prefill_token_threshold < num_new_tokens: take = threshold
-      #     take = min(take, token_budget)
-      # The loop is `while ... and token_budget > 0`. With the threshold at its
-      # default of 0 the cap is disabled, so one long prompt takes the ENTIRE
-      # budget, token_budget hits 0, the loop exits and every other request is
-      # skipped for that step -- measured as 262 tok/s contended prefill and
-      # 0.0-0.1 tok/s generation for everything else, Running: 2-4 reqs.
-      # Hence threshold must be strictly BELOW the budget or it never binds:
-      # at 8192/8192 a single request is served per step (still starved), while
-      # 8192/2048 serves four. Verified by replaying the loop.
-      - --max-num-batched-tokens=8192
-      - --long-prefill-token-threshold=2048
-      - --enable-chunked-prefill
-      - --enable-prefix-caching
-      #
-      # NOT set, deliberately: --max-num-partial-prefills and
-      # --max-long-partial-prefills. Those are V0-only knobs. vLLM 0.11.0 ships
-      # V1 only (vllm/core/ absent), and EngineArgs._is_v1_supported_oracle
-      # rejects any value other than the default for either one, calling
-      # _raise_or_fallback("Concurrent Partial Prefill") -> "Falling back to V0"
-      # -> AssertionError, because there is no V0 engine to fall back to.
-      # The Deployment uses strategy Recreate, so setting them is an outage.
-      # V1 does concurrent chunked prefill natively; it needs no flag, only the
-      # budget/threshold ratio above.
-      # qwen3 is vLLM's dedicated reasoning parser for this family's <think>
-      # blocks.
-      - --reasoning-parser=qwen3
-      # hermes is the documented tool-call parser for general (non-Coder)
-      # Qwen3 models -- native chat template support, not narrated text.
-      - --enable-auto-tool-choice
-      - --tool-call-parser=hermes
-      - --host=0.0.0.0
-      - --port=8080
-      env:
-      - name: VLLM_USE_FLASHINFER_SAMPLER
-        value: '0'
-      - name: VLLM_ATTENTION_BACKEND
-        value: TRITON_ATTN
-      - name: HF_HOME
-        value: /mnt/models
-      image: vllm/vllm-openai:v0.11.0@sha256:014a95f21c9edf6abe0aea6b07353f96baa4ec291c427bb1176dc7c93a85845c
-      name: kserve-container
-      ports:
-      - containerPort: 8080
-        protocol: TCP
-      readinessProbe:
-        httpGet:
-          path: /health
-          port: 8080
-        periodSeconds: 10
-      resources:
-        limits:
-          cpu: '16'
-          memory: 36Gi
-          nvidia.com/gpu: '2'
-        requests:
-          cpu: '8'
-          memory: 12Gi
-          nvidia.com/gpu: '2'
-      startupProbe:
-        failureThreshold: 80
-        httpGet:
-          path: /health
-          port: 8080
-        periodSeconds: 15
-      volumeMounts:
-      - mountPath: /mnt/models
-        name: models
-      - mountPath: /dev/shm
-        name: shm
-    podMetadata:
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "8080"
-        prometheus.io/path: "/metrics"
-      labels:
-        app.kubernetes.io/name: llm-reasoning
-        app.kubernetes.io/part-of: llm-serving
-    deploymentStrategy:
-      type: Recreate
-    maxReplicas: 1
-    minReplicas: 1
-    nodeSelector:
-      kubernetes.io/hostname: worker-1
-    runtimeClassName: nvidia
-    volumes:
-    - name: models
-      persistentVolumeClaim:
-        claimName: llm-models
-    - emptyDir:
-        medium: Memory
-        sizeLimit: 2Gi
-      name: shm
-
@@ -30,33 +30,6 @@ spec:
    syncOptions:
      - CreateNamespace=true
 ---
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: authentik
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: "3"
-spec:
-  project: homelab
-  sources:
-    - repoURL: https://charts.goauthentik.io
-      chart: authentik
-      targetRevision: "*"
-      helm:
-        valueFiles:
-          - $values/k8s/infra/iam/authentik-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-      targetRevision: main
-      ref: values
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: iam
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
---
 # ZITADEL — the Authentik replacement. Coexistence, not cutover: Authentik keeps
 # serving in the same `iam` namespace until the last consumer moves, then this
 # becomes the only IdP. Migration plan (outside git):
@@ -106,11 +79,47 @@ spec:
    syncOptions:
      - CreateNamespace=true
 ---
-# Raw IAM manifests: key-rotation cronjob + authentik migration job.
+# ZITADEL core objects (ServiceAccount, Role, RoleBinding, ConfigMap).
+#
+# The chart declares these as pre-install/pre-upgrade Helm hooks, and ArgoCD
+# prunes hook resources when a sync ends. That deleted the ServiceAccount and
+# ConfigMap the ZITADEL Deployment references, so new pods failed to start with
+# `serviceaccount "zitadel" not found`. This Application owns hook-free copies,
+# which makes them ordinary desired state that survives every sync.
+#
+# Wave 2: before the zatadel chart itself, so the ServiceAccount exists by the
+# time its Deployment is created.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
-  name: iam-jobs
+  name: zitadel-core
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "2"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/infra/zitadel-core
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: iam
+  syncPolicy:
+    automated:
+      prune: false
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+---
+# Raw IAM support manifests: the SMTP -> Gotify notification relay that ZITADEL
+# points at, so verification and password-reset codes reach the phone. Wave 3
+# alongside ZITADEL itself; the Secret it reads is created by sops-secrets at
+# wave 0.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: iam-support
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "3"
@@ -1,4 +1,4 @@
-# Wave 2 — dedicated per-app CNPG Postgres clusters (authentik-db → ns iam,
+# Wave 2 — dedicated per-app CNPG Postgres clusters (zitadel-db → ns iam,
 # temporal-db + visibility → ns temporal). ONE App, ONE folder (k8s/infra/databases).
 # CNPG operator is Phase-0 bootstrap; these Cluster CRs are GitOps — no circular
 # dep (they run after ArgoCD is up, before their apps at w3/w8). CNPG generates
@@ -1,20 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: oidc-bootstrap
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: "5"
-spec:
-  project: homelab
-  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    targetRevision: main
-    path: k8s/infra/oidc-bootstrap
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: kube-system
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
@@ -34,7 +34,6 @@ spec:
    - https://prometheus-community.github.io/helm-charts
    - https://grafana.github.io/helm-charts
    - https://helm.releases.hashicorp.com
-    - https://charts.goauthentik.io
    - https://charts.zitadel.com
    - https://strimzi.io/charts/
    - https://charts.bitnami.com/bitnami
@@ -1,25 +0,0 @@
-apiVersion: ENC[AES256_GCM,data:Fc0=,iv:UFqMFeZTJZtFMoNeusjAoSI60U9StZS4IXw+n8JQCRM=,tag:uUvY4+jcjiSJHs0HDS3/7A==,type:str]
-kind: ENC[AES256_GCM,data:NUw86IEw,iv:gh4Vqn6uteYnYIFcXT/QVxRPKm+G+j2aIVuV71nZ4k4=,tag:nZA/x75Kx4+CoSx66BsvkQ==,type:str]
-metadata:
-    name: ENC[AES256_GCM,data:qex150pm0eGwYiiI05y9LUs=,iv:5Tk001T796c4HFikoh1ZVB6ierBfDqOUkFs5ilkr+V8=,tag:LiDAiCH7w6aRS41VKIlIKg==,type:str]
-    namespace: ENC[AES256_GCM,data:nI91,iv:c1hBWGMkVb+vg+MKNzty8CFfXxQabXEtT+EpiuQZNvo=,tag:BsGtJDbBadDdTa687+Sdag==,type:str]
-type: ENC[AES256_GCM,data:3rXawlSD,iv:FUtLK4KP7v/SBF08ZcU6siIO1qfjlJirf2x5PyzAjuA=,tag:mSIQ+Y43+wDfZGRbQ6a0Og==,type:str]
-stringData:
-    AUTHENTIK_SECRET_KEY: ENC[AES256_GCM,data:i5tVOWnN9a/cfhs6gckveRxUpiN7Ie227R/74kOIz6lQ5US2hkf03W5qSGCrzheL3DB0i8+d0iJYsApjn/wCiN7ndZZVFko3W5jDfVPybxc=,iv:djF1G9JHT5B7ziTokA/UuPpG+tG2LvFU4uMAR0x4M6o=,tag:vHOd7jLYkM8Evnr0QUVrCw==,type:str]
-    AUTHENTIK_BOOTSTRAP_PASSWORD: ENC[AES256_GCM,data:PjvYtH03CzQbTXEqt/Z18+u1zJdok7DLwDn51FUZXDA=,iv:pSwSm8hB0/Q8uOAcOZ1rTqq7BF782EglG19GGeghm1E=,tag:qGCQFWCrwsWBOiQ07ySMUQ==,type:str]
-    AUTHENTIK_BOOTSTRAP_TOKEN: ENC[AES256_GCM,data:1tnekGo0GIemJCMRenjN0yNeSiiMl/kky8ms84TsqBLaQ0lQzPxjq9qtgNJwoAsufxrznrkDVKjeOEHNYcA82A==,iv:NNFWSLMILoWccA9ihy1OArSS1X3OySB7PIHdXZdyzlg=,tag:t2w+KyReOXftTrU9pBQDnQ==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBSZGRKWm9PTkNSOENtREgx
-            M2V3Z0xxOXRhRGVRclo3SzBKRkc0VmtCY0NnClJPSGJRcUlzWmNFeTZ1bmorNEV4
-            U3l3TzVlMTlrWkJnQzBtUTJtcUwrM3MKLS0tIEJrMWZrbllYYUZKN1V4MVMzcXAv
-            eUtheHlNMHpsTEJXZ1FTMFVxbWloa0kKLRkE1Du+4gdOLerOl9y0mZw+8fqfECQY
-            uP64Q+9BSuPYNiTLZvYDzmdoy+KTS7i7C9rf+iU96R/crbQLamV9Ew==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-08-12T20:37:07Z"
-    mac: ENC[AES256_GCM,data:b3pDx340d1iZxsPPXzACckoLBrW+XjX1qlGrWEEI30j2Ao0hewTRZJyHbdI97A6b/hqzBN/4tjqqkdqm8goNhUQweMLTw86X6sALuE0aLS9wK53qjqih8fQwQBbgLDtjFXtW26h+smIm7aB5ZSDvU2le44HDXuWd83sWtCavNq0=,iv:ZMDsoKpygFJrYpXy2dA0IlxAi0jt24rxF72eUzgQs7k=,tag:WWE7G7+c9JK6fF1U9V8igg==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.2
@@ -0,0 +1,28 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: grafana-gotify-bridge
+    namespace: notifications
+type: Opaque
+data:
+    app-token: ENC[AES256_GCM,data:MnkKT0/ubJctmLTFXjdCZjNlz+KgQLsPwIKCmPF2weEtMw6ieRfb3+B7OUlIiDqpktdPaJA/znsMOYPXT2aXDqyKquA=,iv:vcAGkgfTheoViZ7OzJJhbNf0boIbRBZLr60ieW5DhLk=,tag:iBDJEjGScxHopkNExh7jDw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYYWZYNFBHbVhRSkZoa3dH
+            dWp3WllyVmRPcExPTndZUlFsTG1YMHN6WW1jCjRiS1M2U0tIM0tCdGhJOUQ0c2JW
+            bXNTTExDeVVmcExuNStldFdhTHMrVTgKLS0tIDN3d3JKSEtIUUFhVHROOVBUaUNO
+            UUVGcVVkOVJmN294dE5DS1R1UkxiZzgKx0MEkIrFsO2dQwjY2410Nj0Nyy3QgC3L
+            T844m4WT+mxFwB69/y0luGkn4aXMS3xrHHWUC+vjAkM4Unp+MKhhrg==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-27T19:35:38Z"
+    mac: ENC[AES256_GCM,data:Gf5jnvWP+jBRc8fvQHyrnHnTDaaUTcXQgoOqm7Drax5FpBYEHa36pdKaUq+XR6VBd7ch6Ia/f7YavhS47ybgR3jn+w+trMxkpwtZhm5clw2Lo3Hm7Cvi4fG4qhcjm1C8/NOnWGsi5Qm+TJqyG509p9HNTvM99SzElRovPZldLL4=,iv:10hA6nq0WIO0a2I4ZW6uPGXPjesFgxrwmXohNO3FZz4=,tag:bJ37e8jvgEactq6T/pM2UQ==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.3
@@ -0,0 +1,28 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: igotify-client-token
+    namespace: notifications
+type: Opaque
+data:
+    client-token: ENC[AES256_GCM,data:W+InmDyZkU0i+OqQUsIaRNY/3p6zjMC19ASj/qYHTldhtGmHwj4koQX3tIxdCQV72xZKOxsc8eV3hGX3pqbW02c2/ng=,iv:wWI5AZdpxnzX61f8iTiWmxKcDFMaFM4PHFYo4DMBK4Q=,tag:CpE8Kg5V8fL7P3oAjWjSvQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwaUlFU0xKaEs0SWcxWEEr
+            T0tUSmErR08zVkhuLy9VMEc3VXdmU1o3cWdrCjhuOW1yTmZPdWxDZHhMeUZtQlcz
+            UmVzaEMyVHZOZS9KVHdMd2pndUJwUW8KLS0tIGxYeENwVEFDdUpDTGszdGtNRHBQ
+            Y041WW9HdTJwUlJCUGhMN2RGZWZpUmMKiQbi/O9X3cFJgqw3tvwWcyy8QKkUuq/7
+            Y+V51R/WFobEQuAZLmDKgWcPSGt51effufB/Vm9LRRqtbkbYQAIoXQ==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-27T19:14:20Z"
+    mac: ENC[AES256_GCM,data:Hxl7AGzMOMK0pwe6T9ZmrE6K45XhQkq4qk/7YkrVuneL1e7jc8Bmv4NlywqRkt9fPVOCMwKikeX9U3ZcsqK13xPPfm1CmAxV3CCluwK0/9589N70OQxL4qJSkO6CXyGaa0Y9wVRX4y/N3wSjW+d25+GOYUXrOxkRpVyU4GFSqAQ=,iv:0Af/MCZBEYJVI3yi7DpPX1eBKwSANXJPCpSAEE3zCq4=,tag:SS1U9TfRQR3z8KkoQ6ZuZA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.3
@@ -9,7 +9,6 @@ metadata:
 files:
    - agent-pod-models.enc.yaml
    - agent-pod-ssh-key.enc.yaml
-    - authentik-secrets.enc.yaml
    - cloudflare-secrets.enc.yaml
    - cloudflared-credentials.enc.yaml
    - forgejo-registry-pat-api.enc.yaml
@@ -36,6 +35,9 @@ files:
    - pi-sandbox-secrets.enc.yaml
    - etcd-defrag-creds.enc.yaml
    - zitadel-masterkey.enc.yaml
+    - smtp-gotify-secrets.enc.yaml
+    - igotify-client-token.enc.yaml
+    - grafana-gotify-bridge.enc.yaml
    # Login UI RSA keypair, adopted from the chart-generated Secret so ArgoCD
    # owns it. Without this, the chart re-generates the keypair on every render,
    # which changes checksum/secret-login-service-key and leaves both Zitadel
@@ -0,0 +1,23 @@
+apiVersion: v1
+data:
+    token: ENC[AES256_GCM,data:CUJ0rUtgye71fpFz50RSDXJeG+t8Wjrm6T6jboVPbt2FVuIRSscqKg/Pc3k/L8nvLvrkYlzz1rbYu9v5NAdNF2c35AQ=,iv:tsfQ+v4qwj6H7G/O79b6m4XMWRw3mwE67f+4LjDCOoY=,tag:SLGGBYtYoF45AQstETPYZA==,type:str]
+kind: Secret
+metadata:
+    creationTimestamp: null
+    name: smtp-gotify
+    namespace: iam
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFWmU5TUxLeGU4YzUxRU9G
+            ZFR5a2hqS3huYlJtempaQ25zUkljUDJLSENrCkx0bjQ4UVF3UWpWa0t3WHQ2bkNl
+            eEp2d0dQRHJSOExIYjN4TmRTZWdQY00KLS0tIGF5enUxTmxRTHlQRUt1aTFIYTFB
+            MEJaOWd6SnhWQU1PQXdUZDJqWmp2MGsKIOOrp52+d3M8AQlMw0VSnLQ2efDtC6Ye
+            oC/TPqYi9SqYNxaHkODS2QZN9msyfRFcgUh/6ZXr8sivnpwRY8ZZtw==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-27T06:24:20Z"
+    mac: ENC[AES256_GCM,data:9Jsnij+I1MhLhIPLt2hr1a6Jryf8XcAb4QAknGkX1hNEN+Z9HT5PSnPARqyhl0WTxe8813a/G0h93SJ0A419jIzZugv6zEj4STsOG4hMI8O2NniqOwiDWl+Az9SRXIQ0ahH9En7ssvEdiosIDGJSz31ufNc8FbwKnHvgWCwfK9Q=,iv:VNTOpQkDakoaCu2dG4lRbd+6AzimNZVNzggq/7C2eM8=,tag:cJI9WdnKiG3K3J4LRDqQxQ==,type:str]
+    version: 3.13.2
@@ -275,31 +275,3 @@ spec:
                name: temporal-web
                port:
                  number: 8080
-
---
-# ── Authentik (identity provider) ─────────────────────────────────────────────
-# The chart's ingress is disabled in k8s/infra/iam/authentik-values.yaml; the
-# rule lives here so ingress-config owns every hostname. TLS terminates at
-# nginx via the wildcard default-ssl-certificate, so the backend stays HTTP.
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: authentik
-  namespace: iam
-  annotations:
-    nginx.ingress.kubernetes.io/proxy-body-size: "16m"
-    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
-    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
-spec:
-  ingressClassName: nginx
-  rules:
-    - host: authentik.riotpiao.com
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: authentik-server
-                port:
-                  number: 80
@@ -1,38 +0,0 @@
-# Dedicated CNPG Postgres for Authentik (GitOps, wave 2 — before authentik w3).
-# CNPG generates secret `authentik-db-app` + service `authentik-db-rw` in ns iam;
-# authentik-values reads them locally (no cross-ns secret copy).
-apiVersion: postgresql.cnpg.io/v1
-kind: Cluster
-metadata:
-  name: authentik-db
-  namespace: iam
-  annotations:
-    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
-spec:
-  instances: 3
-  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
-  bootstrap:
-    initdb:
-      database: authentik
-      owner: app
-      encoding: UTF8
-      localeCollate: C
-      localeCType: C
-  enableSuperuserAccess: false
-  resources:
-    requests: { memory: "512Mi", cpu: "250m" }
-    limits: { memory: "2Gi", cpu: "1" }
-  storage:
-    size: 10Gi
-    storageClass: longhorn-cnpg
-  monitoring:
-    enablePodMonitor: true
-  affinity:
-    # preferred + tolerations: HA across nodes without deadlocking a partly-
-    # schedulable 3-CP (same as forgejo-db).
-    podAntiAffinityType: preferred
-    topologyKey: kubernetes.io/hostname
-    tolerations:
-      - key: node-role.kubernetes.io/control-plane
-        operator: Exists
-        effect: NoSchedule
@@ -6,7 +6,6 @@ kind: Kustomization
 # deployed into it. paperless ns declared in namespaces.yaml above.
 resources:
  - namespaces.yaml
-  - authentik-db.yaml
  - zitadel-db.yaml
  - temporal-db.yaml
  - paperless-db.yaml
@@ -1,7 +1,7 @@
 # DB clusters are wave 2 — their namespaces must exist first (their apps that
 # would CreateNamespace run later, w3/w8). Declared here so the databases App
-# creates them. authentik/vault/temporal/paperless CreateNamespace=true then
-# no-ops. poimen namespace created by poimen-root app (wave 7).
+# creates them. vault/temporal/paperless CreateNamespace=true then no-ops.
+# poimen namespace created by poimen-root app (wave 7).
 apiVersion: v1
 kind: Namespace
 metadata:
@@ -1,284 +0,0 @@
-# k8s/talos-iam/authentik-values.yaml
-# Authentik — SSO Identity Provider for the homelab.
-# Provides OAuth2/OIDC login for Grafana, MinIO, Forgejo, and Argo CD.
-# Chart: authentik/authentik from https://charts.goauthentik.io
-#
-# Architecture: server (UI+API) + worker (background tasks) + PostgreSQL + Redis.
-# PostgreSQL is the system of record — must persist. Redis is ephemeral cache/queue.
-#
-# Secrets injected via helmfile --set (from .env / vsource):
-#   AUTHENTIK_SECRET_KEY         — signs sessions and tokens; set once, never rotate casually
-#   AUTHENTIK_BOOTSTRAP_PASSWORD — initial akadmin password (used once at first login)
-#   AUTHENTIK_BOOTSTRAP_TOKEN    — API token for the setup_talos_iam.sh bootstrap script
-#   AUTHENTIK_PG_PASSWORD        — PostgreSQL user password
-
-authentik:
-  # host: the external URL Authentik uses to build redirect URIs in OAuth2 flows.
-  # Must match what the browser sees — if it returns an internal svc URL,
-  # the browser's redirect after login will fail (can't reach svc DNS externally).
-  # HTTP (not HTTPS) because the Authentik ingress has no TLS cert configured.
-  host: "https://authentik.riotpiao.com"
-
-  error_reporting:
-    enabled: false   # do not phone home to Sentry
-
-  # PostgreSQL connection — points at CloudNativePG cluster in ddb namespace.
-  # Uses 'app' bootstrap user (CNPG simple pattern, same as Forgejo).
-  # Credentials injected from authentik-db-app secret via env vars below.
-  postgresql:
-    host: authentik-db-rw.iam.svc.cluster.local
-    port: 5432
-    name: authentik
-    user: app  # All apps use shared 'app' user (CNPG design pattern)
-    password: ""  # overridden by AUTHENTIK_POSTGRESQL__PASSWORD env var
-
-  # Redis connection — bundled subchart, standalone mode (no sentinel/cluster).
-  redis:
-    host: authentik-redis-master
-
-  # ── HTTP client timeouts ──────────────────────────────────────────────────────
-  # Increased to tolerate 5+ second pod-to-pod network latency.
-  # Affects webhooks, outpost management, SCIM, LDAP sync.
-  # Default: ~30s — too aggressive when latency spikes hit 5-10s.
-  log_level: debug  # enable debug logging to monitor connection issues
-
-# ── CA trust (shared by server and worker) ────────────────────────────────────
-# Authentik (Python/Debian) uses requests + httpx for outgoing HTTPS — webhooks,
-# outpost management, SCIM. Both libraries need REQUESTS_CA_BUNDLE / SSL_CERT_FILE
-# to point to a bundle that includes homelab-ca, otherwise connections to other
-# homelab services fail with "certificate signed by unknown authority".
-#
-# Strategy: a debian:12-slim init container (run as root) concatenates the
-# Debian system Mozilla bundle with homelab-ca.crt into an emptyDir. The main
-# container then references /merged/ca-bundle.crt via two env vars that cover
-# every Python HTTP library.
-_caVolumes: &caVolumes
-  - name: homelab-ca
-    configMap:
-      name: homelab-ca
-  - name: merged-ca
-    emptyDir: {}
-
-_caVolumeMounts: &caVolumeMounts
-  - name: homelab-ca
-    mountPath: /homelab-ca
-    readOnly: true
-  - name: merged-ca
-    mountPath: /merged
-    readOnly: true
-
-_caInitContainers: &caInitContainers
-  - name: merge-ca-certs
-    image: debian:bookworm
-    imagePullPolicy: IfNotPresent
-    securityContext:
-      runAsUser: 0
-    command:
-      - sh
-      - -c
-      - (cat /etc/ssl/certs/ca-certificates.crt 2>/dev/null; cat /homelab-ca/homelab-ca.crt) > /merged/ca-bundle.crt
-    volumeMounts:
-      - name: homelab-ca
-        mountPath: /homelab-ca
-        readOnly: true
-      - name: merged-ca
-        mountPath: /merged
-  # NOTE: no authentik-migrate init container — the authentik `server` entrypoint
-  # runs migrations itself on startup. A separate `manage migrate` init pinned to
-  # an older image tripped a version-history precheck on an empty DB
-  # (relation "authentik_version_history" does not exist) and blocked boot.
-
-_caEnv: &caEnv
-  - name: REQUESTS_CA_BUNDLE
-    value: /merged/ca-bundle.crt
-  - name: SSL_CERT_FILE
-    value: /merged/ca-bundle.crt
-
-# ── Authentik server (UI + API) ───────────────────────────────────────────────
-# Handles all browser traffic: login flows, admin UI, OAuth2 authorize/token endpoints.
-# NodePort 32172 is a fallback for direct node access during troubleshooting;
-# normal access is via nginx ingress (authentik.riotpiao.com → svc:80).
-# Recreate: single replica + RWO-adjacent state — avoids split-brain on redeploy.
-server:
-  replicas: 1
-  # Merge SOPS-CMP-emitted secret values after the chart's own `authentik` secret.
-  envFrom:
-    - secretRef:
-        name: authentik-secrets
-  deploymentStrategy:
-    type: Recreate
-  service:
-    type: NodePort
-    nodePort: 32172
-  resources:
-    requests:
-      cpu: 100m
-      memory: 512Mi
-    limits:
-      cpu: 1000m
-      memory: 1Gi
-  tolerations:
-    - key: node-role.kubernetes.io/control-plane
-      operator: Exists
-      effect: NoSchedule
-  volumes: *caVolumes
-  volumeMounts: *caVolumeMounts
-  initContainers: *caInitContainers
-  env:
-    # Merge CA trust env vars
-    - name: REQUESTS_CA_BUNDLE
-      value: /merged/ca-bundle.crt
-    - name: SSL_CERT_FILE
-      value: /merged/ca-bundle.crt
-    # Override database credentials to use 'app' from authentik-db-app
-    - name: AUTHENTIK_POSTGRESQL__USER
-      valueFrom:
-        secretKeyRef:
-          name: authentik-db-app
-          key: username
-    - name: AUTHENTIK_POSTGRESQL__PASSWORD
-      valueFrom:
-        secretKeyRef:
-          name: authentik-db-app
-          key: password
-  podAnnotations:
-    configmap.reloader.stakater.com/reload: "homelab-ca"
-    homelab.io/restart-at: "2026-06-21T13-40"
-  # The /-/health/{live,ready}/ endpoints do a DB round-trip; under transient
-  # CNPG contention they respond in 5-6s while still returning 200. The chart's
-  # default 3s liveness timeout then flags a working backend as dead and kubelet
-  # kills it in a restart loop — the pod never stays Ready, gets dropped from the
-  # authentik-server Service endpoints, and the OAuth-provisioning PostSync hook
-  # fails with "Host is unreachable". Widen the timeouts so slow-but-healthy
-  # checks aren't treated as failures. (Only these fields are overridden; the
-  # chart deep-merges the rest of each probe, incl. the httpGet path.)
-  livenessProbe:
-    initialDelaySeconds: 300  # skip probe until 5min passed (migrations finish)
-    timeoutSeconds: 15
-    failureThreshold: 6
-  readinessProbe:
-    initialDelaySeconds: 300  # skip probe until migrations complete
-    timeoutSeconds: 15
-    failureThreshold: 6
-  startupProbe:
-    timeoutSeconds: 15
-    failureThreshold: 120  # 120 × 10s = 20min for fresh DB migrations
-  # Every OIDC login (Grafana, Argo CD, MinIO, Forgejo) depends on this server —
-  # its request latency/error rate explains SSO-driven slowness on those services.
-  metrics:
-    enabled: true
-    serviceMonitor:
-      enabled: true
-      scrapeTimeout: 30s
-
-# ── Authentik worker ──────────────────────────────────────────────────────────
-# Runs background tasks: email delivery, LDAP sync, flow policy evaluation,
-# event log cleanup, and managed outpost updates. Stateless — no PVC needed.
-# Same resource profile as server; Authentik 2023+ merged some worker duties
-# into the server process but the worker pod is still required.
-worker:
-  replicas: 1
-  envFrom:
-    - secretRef:
-        name: authentik-secrets
-  deploymentStrategy:
-    type: Recreate
-  resources:
-    requests:
-      cpu: 100m
-      memory: 512Mi
-    limits:
-      cpu: 1000m
-      memory: 1Gi
-  tolerations:
-    - key: node-role.kubernetes.io/control-plane
-      operator: Exists
-      effect: NoSchedule
-  volumes: *caVolumes
-  volumeMounts: *caVolumeMounts
-  initContainers: *caInitContainers
-  env:
-    # Merge CA trust env vars
-    - name: REQUESTS_CA_BUNDLE
-      value: /merged/ca-bundle.crt
-    - name: SSL_CERT_FILE
-      value: /merged/ca-bundle.crt
-    # Override database credentials to use 'app' from authentik-db-app
-    - name: AUTHENTIK_POSTGRESQL__USER
-      valueFrom:
-        secretKeyRef:
-          name: authentik-db-app
-          key: username
-    - name: AUTHENTIK_POSTGRESQL__PASSWORD
-      valueFrom:
-        secretKeyRef:
-          name: authentik-db-app
-          key: password
-  podAnnotations:
-    configmap.reloader.stakater.com/reload: "homelab-ca"
-    homelab.io/restart-at: "2026-06-21T13-40"
-  livenessProbe:
-    initialDelaySeconds: 300  # skip probe until 5min passed (migrations finish)
-  readinessProbe:
-    initialDelaySeconds: 300  # skip probe until migrations complete
-  startupProbe:
-    initialDelaySeconds: 30  # let server finish DB work first
-    failureThreshold: 120
-  metrics:
-    enabled: true
-    serviceMonitor:
-      enabled: true
-      scrapeTimeout: 30s
-
-# ── PostgreSQL (external: CloudNativePG cluster in ddb namespace) ─────────────
-# Authentik connects to the dedicated authentik-db (1 primary + 2 replicas).
-# Do not use the bundled Bitnami subchart — CNPG is already running.
-postgresql:
-  enabled: false
-  primary:
-    persistence:
-      enabled: true
-      storageClass: longhorn
-      size: 8Gi
-    tolerations:
-      - key: node-role.kubernetes.io/control-plane
-        operator: Exists
-        effect: NoSchedule
-    affinity:
-      nodeAffinity:
-        preferredDuringSchedulingIgnoredDuringExecution:
-          - weight: 100
-            preference:
-              matchExpressions:
-                - key: node-role.kubernetes.io/worker
-                  operator: Exists
-
-# ── Bundled Redis ─────────────────────────────────────────────────────────────
-# Cache and async task queue only — no durable data. If Redis restarts, in-flight
-# background tasks are retried and cached tokens are recomputed. Losing Redis
-# data does not lose user accounts or flow configuration (that's in PostgreSQL).
-# persistence: false saves a PVC and makes restarts faster.
-#
-# Same prefer-worker / fallback-to-cp scheduling as PostgreSQL.
-# architecture: standalone — no Sentinel/cluster overhead for a 3-node homelab.
-redis:
-  enabled: true
-  master:
-    persistence:
-      enabled: false
-    tolerations:
-      - key: node-role.kubernetes.io/control-plane
-        operator: Exists
-        effect: NoSchedule
-    affinity:
-      nodeAffinity:
-        preferredDuringSchedulingIgnoredDuringExecution:
-          - weight: 100
-            preference:
-              matchExpressions:
-                - key: node-role.kubernetes.io/worker
-                  operator: Exists
-  architecture: standalone
-
-# Ingress disabled — rule lives in k8s/ingress/ingress.yaml (authentik.riotpiao.com).
-# For direct access during bootstrap: kubectl -n iam port-forward svc/authentik-server 7000:80
@@ -2,11 +2,12 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization

 resources:
-  - rbac-dashboard-rolebinding.yaml
+  # Removed: rbac-dashboard-rolebinding.yaml (authentik-provisioner SA and
+  # RoleBinding). Authentik is retired, so the file is gone; keep this list in
+  # step with the directory or kustomize fails the whole app.
+  - smtp-gotify.yaml

 # IAM provisioning is manual-only (security-sensitive).
-# Script: scripts/iam/authentik-provision.py
+# Script: scripts/iam/zitadel-provision.py
 # Run:
-#   export AUTHENTIK_BOOTSTRAP_TOKEN=$(kubectl -n iam get secret authentik-secrets \
-#     -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_TOKEN}' | base64 -d)
-#   python3 scripts/iam/authentik-provision.py
+#   scripts/iam/zitadel-provision.py all
@@ -1,13 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: authentik-provisioner
-  namespace: dashboard
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: authentik-provisioner
-subjects:
- kind: ServiceAccount
-  name: authentik-provisioner
-  namespace: iam
@@ -0,0 +1,108 @@
+# SMTP -> Gotify relay for ZITADEL notifications.
+#
+# ZITADEL sends verification codes, password resets and one-time passwords by
+# SMTP only, and this cluster has no working mail path: no SMTP credentials are
+# configured for ZITADEL, and the notification table shows
+# verified_email empty for every user. Without a delivery path nobody can finish
+# a first login, at ZITADEL or at any relying party such as Forgejo.
+#
+# This relay accepts plain SMTP on :2525 and forwards every message to Gotify,
+# so codes arrive on the phone through the existing gotify/igotify setup instead
+# of email. ZITADEL then points DefaultInstance.SMTPConfiguration at this
+# Service (see k8s/infra/iam/zitadel-values.yaml).
+#
+# The Gotify token is a Secret, never a literal here.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: smtp-gotify
+  namespace: iam
+  labels:
+    app: smtp-gotify
+spec:
+  replicas: 1
+  revisionHistoryLimit: 2
+  selector:
+    matchLabels:
+      app: smtp-gotify
+  template:
+    metadata:
+      labels:
+        app: smtp-gotify
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534
+        runAsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      volumes:
+        - name: tmp
+          emptyDir: {}
+      containers:
+        - name: smtp-gotify
+          image: quay.io/reiml/smtp-gotify:latest
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: GOTIFY_URL
+              value: "http://gotify.notifications.svc.cluster.local/"
+            - name: GOTIFY_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: smtp-gotify
+                  key: token
+            - name: GOTIFY_PRIORITY
+              value: "5"
+            # No title template: the relay's default title already carries
+            # the mail subject, and a wrong template field fails the push
+            # silently (the SMTP side still answers 250, so the sender thinks
+            # the message was delivered).
+          ports:
+            - name: smtp
+              containerPort: 2525
+              protocol: TCP
+          volumeMounts:
+            # The relay stages each message in /tmp before the Gotify push.
+            # With readOnlyRootFilesystem it accepted SMTP (250 queued) and
+            # then silently dropped the message: no push, no error.
+            - name: tmp
+              mountPath: /tmp
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              cpu: 200m
+              memory: 128Mi
+          readinessProbe:
+            tcpSocket:
+              port: smtp
+            initialDelaySeconds: 3
+            periodSeconds: 10
+          livenessProbe:
+            tcpSocket:
+              port: smtp
+            initialDelaySeconds: 10
+            periodSeconds: 20
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: smtp-gotify
+  namespace: iam
+  labels:
+    app: smtp-gotify
+spec:
+  type: ClusterIP
+  selector:
+    app: smtp-gotify
+  ports:
+    - name: smtp
+      port: 2525
+      targetPort: smtp
+      protocol: TCP
@@ -31,7 +31,6 @@ zitadel:
    ExternalDomain: zitadel.riotpiao.com
    TLS:
      Enabled: false
-
    # First boot is COMPLETE (2026-10-26): the instance, the system org and the
    # `iam-admin` machine key all exist. Skip stops the setup hook from running
    # the first-instance step again. Without it, every sync re-runs the
@@ -146,6 +145,10 @@ setupJob:
 replicaCount: 2
 revisionHistoryLimit: 3

+# Notification delivery lives in the ConfigMap owned by the zitadel-core app
+# (k8s/infra/zitadel-core/). Env overrides here produced no SMTP provider, and
+# editing configmapConfig made ArgoCD drop the rendered ConfigMap.
+
 # Best practice: a PDB guarantees at least one API pod stays available during
 # voluntary disruptions (node drain, rolling upgrade). Without it a drain can
 # take both replicas down at once. minAvailable 1 is correct for 2 replicas;
@@ -171,7 +174,20 @@ ingress:
  controller: nginx

 login:
-  enabled: true
+  # Built-in login page, not the hosted Login UI.
+  #
+  # The hosted UI authenticates to the ZITADEL API as the machine user
+  # `login-client`, using a JWT-profile assertion signed by the keypair in
+  # zitadel-login-service-key. That machine user is created by the chart's
+  # setup job, which this install runs only once (setupJob is deliberately
+  # disabled after the first sync). Without the user, every session write
+  # fails and the user sees "Could not create session for user", so no one
+  # can sign in here or at any relying party such as Forgejo.
+  #
+  # The built-in login serves the same flows from the ZITADEL server and
+  # needs no machine user. Turn this back on only after creating
+  # `login-client` and registering the matching public key.
+  enabled: false
  # ksops owns this Secret (k8s/argocd/secrets/zitadel-login-service-key.enc.yaml,
  # rendered by the sops-secrets Application). Pointing the chart at it stops the
  # chart from auto-generating an RSA keypair every render, which changed
@@ -209,3 +209,25 @@ sidecar:
      disableDelete: false
      foldersFromFilesStructure: true
    searchNamespace: ALL
+
+# ── Alerting ──────────────────────────────────────────────────────────────────
+# One contact point: Gotify. Gotify refuses Grafana's payload shape with
+# 400 "Field 'message' is required", so a bridge service translates the JSON
+# first (k8s/apps/gotify/bridge). Grafana posts to the bridge, the bridge posts
+# to /message with an app token.
+alerting:
+  contactpoints.yaml:
+    apiVersion: 1
+    contactPoints:
+      - orgId: 1
+        name: gotify
+        receivers:
+          - uid: gotify-webhook
+            type: webhook
+            disableResolveMessage: false
+            settings:
+              url: http://grafana-gotify-bridge.notifications.svc.cluster.local:8080/alert
+              httpMethod: POST
+              # 0 sends every alert in one request. The bridge names the extra
+              # count in the message body.
+              maxAlerts: 0
@@ -1,33 +0,0 @@
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: authentik-rules
-  namespace: iam
-spec:
-  groups:
-    - name: authentik.rules
-      interval: 15s
-      rules:
-        - alert: AuthentikHighErrorRate
-          expr: |
-            (
-              sum(rate(authentik_http_requests_total{status=~"5.."}[5m]))
-              /
-              sum(rate(authentik_http_requests_total[5m]))
-            ) > 0.05
-          for: 10m
-          labels:
-            severity: warning
-          annotations:
-            summary: "High error rate on Authentik"
-            description: "5xx error rate exceeded 5% of total requests for 10 minutes. Value: {{ $value | humanizePercentage }}"
-
-        - alert: AuthentikOutpostDown
-          expr: |
-            authentik_outpost_total_up == 0
-          for: 5m
-          labels:
-            severity: warning
-          annotations:
-            summary: "Authentik outpost is down"
-            description: "Outpost {{ $labels.outpost_name }} (type: {{ $labels.outpost_type }}) has been offline for 5 minutes"
@@ -7,7 +7,6 @@ kind: Kustomization
 resources:
  - alerts/ingress-alerts.yaml
  - alerts/svc-argocd-rules.yaml
-  - alerts/svc-authentik-rules.yaml
  - alerts/svc-zitadel-rules.yaml
  - alerts/svc-forgejo-rules.yaml
  - alerts/svc-grafana-rules.yaml
@@ -17,7 +16,6 @@ resources:
  - alerts/api-gateway-alerts.yaml
  - alerts/cag-kv-cache-rules.yaml
  - servicemonitors/argocd.yaml
-  - servicemonitors/authentik.yaml
  - servicemonitors/forgejo.yaml
  - servicemonitors/minio.yaml
  - servicemonitors/ingress-nginx.yaml
@@ -1,18 +0,0 @@
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: authentik
-  namespace: monitoring
-  labels:
-    release: kube-prometheus-stack
-spec:
-  namespaceSelector:
-    matchNames:
-      - iam
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: authentik
-  endpoints:
-    - port: metrics
-      interval: 30s
-      scrapeTimeout: 10s
@@ -1,33 +0,0 @@
-# One-shot bootstrap: restart the control-plane apiservers ONCE so they pick up the
-# OIDC authenticator (they last started before the Authentik provider existed, so OIDC
-# tokens were never accepted -> every login got 401). Delete this whole directory after.
---
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: oidc-bootstrap
-  namespace: kube-system
---
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: oidc-bootstrap
-  namespace: kube-system
-rules:
-  - apiGroups: [""]
-    resources: ["pods"]
-    verbs: ["get", "list", "delete"]
---
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: oidc-bootstrap
-  namespace: kube-system
-subjects:
-  - kind: ServiceAccount
-    name: oidc-bootstrap
-    namespace: kube-system
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: oidc-bootstrap
@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - zitadel-core.yaml
@@ -0,0 +1,133 @@
+# ZITADEL core objects, owned by this repository instead of the Helm chart.
+#
+# The zitadel chart marks four objects as Helm hooks:
+#
+#   kind: ServiceAccount   (also ConfigMap/zitadel-config-yaml, Role, RoleBinding)
+#   annotations:
+#     helm.sh/hook: pre-install,pre-upgrade
+#     helm.sh/hook-delete-policy: before-hook-creation
+#
+# ArgoCD maps pre-install/pre-upgrade to PreSync and prunes hook resources when
+# the sync ends. Every sync therefore deleted the ServiceAccount and the ConfigMap
+# that the ZITADEL Deployment references, which produced:
+#
+#   error creating: pods "zitadel-..." is forbidden:
+#     serviceaccount "zitadel" not found
+#
+# and left the IdP one restart away from failing.
+#
+# These copies carry no hook annotations, so they are ordinary desired state and
+# ArgoCD keeps them. The chart still renders its own hook versions during a sync;
+# these win because the sync no longer removes them.
+#
+# ConfigMap content mirrors what the chart renders from
+# k8s/infra/iam/zitadel-values.yaml, plus the notification settings the chart
+# cannot deliver through values (see the SMTP note below). When the chart values
+# change, update this file too; it is the authoritative copy at runtime.
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: zitadel
+  namespace: iam
+  labels:
+    app.kubernetes.io/name: zitadel
+    app.kubernetes.io/instance: zitadel
+    app.kubernetes.io/managed-by: gitops
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: zitadel
+  namespace: iam
+  labels:
+    app.kubernetes.io/name: zitadel
+    app.kubernetes.io/instance: zitadel
+    app.kubernetes.io/managed-by: gitops
+rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "create", "patch", "list", "delete"]
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: zitadel
+  namespace: iam
+  labels:
+    app.kubernetes.io/name: zitadel
+    app.kubernetes.io/instance: zitadel
+    app.kubernetes.io/managed-by: gitops
+subjects:
+  - kind: ServiceAccount
+    name: zitadel
+roleRef:
+  kind: Role
+  name: zitadel
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: zitadel-config-yaml
+  namespace: iam
+  labels:
+    app.kubernetes.io/name: zitadel
+    app.kubernetes.io/instance: zitadel
+    app.kubernetes.io/managed-by: gitops
+data:
+  zitadel-config-yaml: |
+    ExternalDomain: zitadel.riotpiao.com
+    ExternalSecure: true
+    TLS:
+      Enabled: false
+    FirstInstance:
+      # First boot is COMPLETE. Skip stops the setup step from running again and
+      # silently rotating the iam-admin key that provisioning reads.
+      Skip: true
+      Org:
+        Skip: true
+        Machine:
+          Machine:
+            Name: Automatically Initialized IAM Admin
+            Username: iam-admin
+          MachineKey:
+            ExpirationDate: "2029-01-01T00:00:00Z"
+            Type: 1
+          Pat:
+            ExpirationDate: "2029-01-01T00:00:00Z"
+    Machine:
+      Identification:
+        Hostname:
+          Enabled: true
+        Webhook:
+          Enabled: false
+    # Notification delivery.
+    #
+    # ZITADEL sends verification codes and password resets by SMTP only. This
+    # cluster has no mail path, so without a provider every user stays at
+    # verified_email = empty and cannot finish a first login.
+    #
+    # The target is the in-cluster smtp-gotify relay, which forwards each message
+    # to Gotify so codes reach the phone.
+    #
+    # Why here and not in chart values: injecting the same keys as
+    # ZITADEL_DEFAULTINSTANCE_SMTPCONFIGURATION_* env vars produced no
+    # instance.smtp.config.added event, so ZITADEL never created the provider.
+    # Setting configmapConfig in chart values made ArgoCD stop managing this
+    # ConfigMap. Owning the file here avoids both problems.
+    DefaultInstance:
+      DomainPolicy:
+        # The sender is [email protected] while the instance domain is
+        # zitadel.riotpiao.com. Leave this true so the sender address is not
+        # rejected for not matching the domain.
+        SMTPSenderAddressMatchesInstanceDomain: false
+      SMTPConfiguration:
+        SMTP:
+          Host: smtp-gotify.iam.svc.cluster.local:2525
+        TLS: false
+        From: [email protected]
+        FromName: ZITADEL
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# forgejo-oauth-source.sh — keep the Forgejo OAuth login source aligned with
+# ZITADEL.
+#
+# Forgejo stores login sources in its database, not in GitOps, and its REST API
+# does not expose them. `tea` has no auth-source command either (tea admin only
+# manages users), so the supported writer is the in-pod CLI:
+#   gitea admin auth update-oauth
+#
+# The source name is the label on the "Sign in with ..." button AND the callback
+# path (/user/oauth2/<name>/callback), so a rename changes the registered
+# redirect URI. Register both spellings in ZITADEL (zitadel-provision.py does)
+# before renaming.
+#
+# Usage:
+#   scripts/iam/forgejo-oauth-source.sh status
+#   scripts/iam/forgejo-oauth-source.sh rename zitadel
+set -euo pipefail
+
+NS="${FORGEJO_NAMESPACE:-cicd}"
+SEL="${FORGEJO_SELECTOR:-app.kubernetes.io/name=forgejo}"
+
+pod() {
+  kubectl -n "$NS" get pods -l "$SEL" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null \
+    || kubectl -n "$NS" get pods -o name | grep -m1 forgejo-gitea | cut -d/ -f2
+}
+
+run() { kubectl -n "$NS" exec "$(pod)" -c gitea -- gitea admin auth "$@"; }
+
+case "${1:-status}" in
+  status) run list ;;
+  rename)
+    name="${2:?usage: $0 rename <name>}"
+    run update-oauth --id 1 --name "$name"
+    run list
+    ;;
+  *) echo "usage: $0 [status|rename <name>]" >&2; exit 2 ;;
+esac
@@ -1,664 +0,0 @@
-#!/usr/bin/env python3
-"""
-Provision RBAC groups, service account roles, fine-grained claims, and auth flows.
-
-Idempotent — safe to re-run. Provisions:
-  1. Global admin groups (homelab-admins)
-  2. Fine-grained service/bucket/project groups (minio-*, poimen-*, paperless-*, grafana-*, sqs-*)
-  3. Service account roles with custom claims (paperless-ai-agent, portfolio-agent, etc.)
-  4. JWT scope mappings for fine-grained claims (minio_buckets, paperless_doctypes, etc.)
-  5. OAuth2 providers with scopes (api-gw, minio, poimen, paperless, grafana)
-  6. Auth flows (password grant on api-gw provider)
-
-Usage:
-    source ~/.env
-    export AUTHENTIK_BOOTSTRAP_TOKEN=$(kubectl -n iam get secret authentik-secrets \
-        -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_TOKEN}' | base64 -d)
-    python3 scripts/iam/provision-rbac.py
-
-DO NOT commit this file to git — .gitignore covers scripts/iam/*.py.
-"""
-import json
-import os
-import sys
-import urllib.error
-import urllib.request
-from typing import Dict, List, Any
-from pathlib import Path
-
-# Load ~/.env for OAuth2 provider secrets
-env_file = Path.home() / ".env"
-if env_file.exists():
-    with open(env_file) as f:
-        for line in f:
-            line = line.strip()
-            if line.startswith("export ") and "=" in line:
-                key, _, value = line[7:].partition("=")
-                key = key.strip()
-                value = value.strip().strip('"').strip("'")
-                os.environ[key] = value
-
-AUTHENTIK_URL = "https://authentik.riotpiao.com"
-TOKEN = os.environ.get("AUTHENTIK_BOOTSTRAP_TOKEN")
-if not TOKEN:
-    print("Error: AUTHENTIK_BOOTSTRAP_TOKEN not set")
-    print("  source ~/.env")
-    print("  export AUTHENTIK_BOOTSTRAP_TOKEN=$(kubectl -n iam get secret authentik-secrets \\")
-    print("      -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_TOKEN}' | base64 -d)")
-    sys.exit(1)
-
-
-def api(method, path, data=None):
-    url = f"{AUTHENTIK_URL}{path}"
-    body = json.dumps(data).encode() if data is not None else None
-    req = urllib.request.Request(url, data=body, method=method, headers={
-        "Authorization": f"Bearer {TOKEN}",
-        "Content-Type": "application/json",
-    })
-    try:
-        with urllib.request.urlopen(req, timeout=30) as resp:
-            raw = resp.read()
-            return resp.status, json.loads(raw) if raw else {}
-    except urllib.error.HTTPError as e:
-        raw = e.read()
-        try:
-            parsed = json.loads(raw) if raw else {}
-        except json.JSONDecodeError:
-            parsed = {"raw": raw.decode(errors="replace")}
-        return e.code, parsed
-
-
-def die(msg):
-    print(f"FATAL: {msg}", file=sys.stderr)
-    sys.exit(1)
-
-
-# ===========================================================================
-# Group Definitions (DRY: single source of truth)
-# ===========================================================================
-GROUPS: Dict[str, Dict[str, Any]] = {
-    "homelab-admins": {
-        "description": "Cluster administrators with full access",
-        "is_superuser": True,
-    },
-    "minio-admins": {"description": "MinIO administrators", "is_superuser": False, "minio_buckets": ["*"]},
-    "minio-photos": {"description": "Photos bucket (Immich) access", "is_superuser": False, "minio_buckets": ["immich"]},
-    "minio-documents": {"description": "Documents bucket (Paperless) access", "is_superuser": False, "minio_buckets": ["paperless"]},
-    "minio-backups": {"description": "Backups bucket read-only access", "is_superuser": False, "minio_buckets": ["backups"]},
-    "poimen-admins": {"description": "Poimen memory administrators", "is_superuser": False, "memory_projects": ["*"], "memory_visibility": "private"},
-    "poimen-devs": {"description": "Dev and staging projects access", "is_superuser": False, "memory_projects": ["dev", "staging"], "memory_visibility": "internal"},
-    "poimen-prod-readonly": {"description": "Production projects read-only access", "is_superuser": False, "memory_projects": ["prod"], "memory_visibility": "public"},
-    "paperless-admins": {"description": "Paperless administrators", "is_superuser": False, "paperless_doctypes": ["*"]},
-    "paperless-finance": {"description": "Finance documents", "is_superuser": False, "paperless_doctypes": ["invoices", "receipts", "expenses"]},
-    "paperless-legal": {"description": "Legal documents", "is_superuser": False, "paperless_doctypes": ["contracts", "licenses", "agreements"]},
-    "paperless-hr": {"description": "HR documents", "is_superuser": False, "paperless_doctypes": ["employment", "benefits", "payroll"]},
-    "grafana-admins": {"description": "Grafana administrators", "is_superuser": False, "grafana_org_role": "Admin"},
-    "grafana-editors": {"description": "Grafana dashboard editors", "is_superuser": False, "grafana_org_role": "Editor"},
-    "grafana-viewers": {"description": "Grafana dashboard viewers", "is_superuser": False, "grafana_org_role": "Viewer"},
-    "sqs-users": {"description": "SQS/Temporal queue read access", "is_superuser": False, "sqs_queues": ["default"]},
-    "sqs-writers": {"description": "SQS/Temporal queue read/write access", "is_superuser": False, "sqs_queues": ["*"]},
-    "s3-users": {"description": "S3 read access", "is_superuser": False},
-    "s3-writers": {"description": "S3 read/write access", "is_superuser": False},
-}
-
-SERVICE_ACCOUNTS: Dict[str, Dict[str, Any]] = {
-    "paperless-ai-agent": {
-        "description": "Paperless AI plugin (auto-tagging, entity extraction)",
-        "roles": ["llm:inference", "memory:write", "paperless:admin"],
-        "claims": {
-            "minio_buckets": ["paperless"],
-            "paperless_doctypes": ["*"],
-            "memory_projects": ["*"],
-            "authorized_models": ["reasoning", "qwen2.5:3b"],
-        },
-    },
-    "portfolio-agent": {
-        "description": "Portfolio service agent",
-        "roles": ["llm:inference", "memory:read"],
-        "claims": {
-            "memory_projects": ["homelab", "portfolio"],
-            "memory_visibility": "public",
-            "authorized_models": ["ornith:35b"],
-            "minio_buckets": ["backups"],
-        },
-    },
-    "memory-agent": {
-        "description": "Memory service agent",
-        "roles": ["llm:inference", "memory:read", "memory:write"],
-        "claims": {
-            "memory_projects": ["*"],
-            "memory_visibility": "private",
-            "authorized_models": ["reasoning", "ornith:35b", "qwen2.5:3b"],
-        },
-    },
-    "temporal-worker-agent": {
-        "description": "Temporal workflow worker",
-        "roles": ["llm:inference", "workflow:execute", "memory:read", "memory:write", "queue:send"],
-        "claims": {
-            "memory_projects": ["*"],
-            "sqs_queues": ["*"],
-            "authorized_models": ["reasoning", "ornith:35b", "qwen2.5:3b"],
-        },
-    },
-}
-
-SCOPE_MAPPINGS: Dict[str, Dict[str, str]] = {
-    "roles": {"expression": 'return user.attributes.get("roles", [])'},
-    "permissions": {"expression": 'return ["*"] if any(user.groups.filter(is_superuser=True)) else list(user.groups.values_list("name", flat=True))'},
-    "groups": {"expression": 'return {"groups": list(user.groups.values_list("name", flat=True))}'},
-    "minio_buckets": {"expression": 'return user.attributes.get("minio_buckets", [])'},
-    "paperless_doctypes": {"expression": 'return user.attributes.get("paperless_doctypes", [])'},
-    "memory_projects": {"expression": 'return user.attributes.get("memory_projects", [])'},
-    "memory_visibility": {"expression": 'return user.attributes.get("memory_visibility", "public")'},
-    "authorized_models": {"expression": 'return user.attributes.get("authorized_models", [])'},
-    "sqs_queues": {"expression": 'return user.attributes.get("sqs_queues", [])'},
-    "grafana_org_role": {"expression": 'return user.attributes.get("grafana_org_role", "Viewer")'},
-}
-
-# ===========================================================================
-# Phase 1: Create/sync all groups
-# ===========================================================================
-print("[1/6] Ensuring groups exist...")
-
-status, res = api("GET", "/api/v3/core/groups/?page_size=100")
-if status != 200:
-    die(f"GET groups -> {status} {res}")
-existing_groups = {g["name"]: g for g in res["results"]}
-
-created_count = 0
-for group_name, group_spec in GROUPS.items():
-    if group_name in existing_groups:
-        print(f"  {group_name}: already exists")
-    else:
-        status, res = api("POST", "/api/v3/core/groups/", {
-            "name": group_name,
-            "is_superuser": group_spec.get("is_superuser", False),
-        })
-        if status in (200, 201):
-            print(f"  {group_name}: created")
-            created_count += 1
-        else:
-            print(f"  {group_name}: FAILED {status} {res}")
-
-print(f"  Total: {len(GROUPS)} groups, {created_count} new")
-
-# ===========================================================================
-# Phase 2: Create/sync service account users with custom claims
-# ===========================================================================
-print("\n[2/6] Creating/updating service account users...")
-
-status, res = api("GET", "/api/v3/core/users/?page_size=100")
-if status != 200:
-    die(f"GET users -> {status} {res}")
-existing_users = {u["username"]: u for u in res["results"]}
-
-service_pwd = os.environ.get("AUTHENTIK_SERVICE_ACCOUNT_PASSWORD", "DefaultPassword123!")
-
-for agent_name, agent_spec in SERVICE_ACCOUNTS.items():
-    if agent_name in existing_users:
-        user = existing_users[agent_name]
-        attrs = user.get("attributes", {})
-        attrs.update(agent_spec.get("claims", {}))
-        attrs["roles"] = agent_spec.get("roles", [])
-        status, res = api("PATCH", f"/api/v3/core/users/{user['pk']}/", {"attributes": attrs})
-        if status in (200, 201):
-            print(f"  {agent_name}: claims updated")
-        else:
-            print(f"  {agent_name}: FAILED {status} {res}")
-    else:
-        status, res = api("POST", "/api/v3/core/users/", {
-            "username": agent_name,
-            "name": agent_spec.get("description", agent_name),
-            "email": f"{agent_name}@homelab.local",
-            "is_active": True,
-            "is_superuser": False,
-            "password": service_pwd,
-            "attributes": {
-                **agent_spec.get("claims", {}),
-                "roles": agent_spec.get("roles", []),
-            },
-        })
-        if status in (200, 201):
-            print(f"  {agent_name}: created")
-        else:
-            print(f"  {agent_name}: FAILED {status} {res}")
-
-# ===========================================================================
-# Phase 3: Create scope mappings for fine-grained claims
-# ===========================================================================
-print("\n[3/6] Creating scope mappings for fine-grained claims...")
-
-status, res = api("GET", "/api/v3/propertymappings/provider/scope/?page_size=100")
-if status != 200:
-    die(f"GET scope mappings -> {status} {res}")
-existing_scopes = {m["scope_name"]: m for m in res["results"]}
-
-for scope_name, scope_spec in SCOPE_MAPPINGS.items():
-    if scope_name in existing_scopes:
-        print(f"  {scope_name}: already exists")
-    else:
-        status, res = api("POST", "/api/v3/propertymappings/provider/scope/", {
-            "name": scope_name,
-            "scope_name": scope_name,
-            "expression": scope_spec["expression"],
-        })
-        if status in (200, 201):
-            print(f"  {scope_name}: created")
-        else:
-            print(f"  {scope_name}: FAILED {status} {res}")
-
-# Fix built-in 'openid' scope to include preferred_username (needed by Grafana, Forgejo, etc.)
-# Note: sub MUST be a string, not int, for JSON compatibility with Go struct unmarshaling
-if "openid" in existing_scopes:
-    openid_scope = existing_scopes["openid"]
-    openid_pk = openid_scope["pk"]
-    openid_expression = 'return {"sub": str(user.pk), "preferred_username": user.username}'
-    if openid_expression not in openid_scope.get("expression", ""):
-        status, res = api("PUT", f"/api/v3/propertymappings/provider/scope/{openid_pk}/", {
-            "name": openid_scope["name"],
-            "scope_name": "openid",
-            "expression": openid_expression,
-        })
-        if status in (200, 201):
-            print(f"  openid: updated (sub as string + preferred_username)")
-        else:
-            print(f"  openid: UPDATE FAILED {status} {res}")
-    else:
-        print(f"  openid: already correct (sub as string)")
-
-# ===========================================================================
-# Phase 4: Get flow UUIDs (needed for providers)
-# ===========================================================================
-print("\n[4/6] Fetching flow UUIDs...")
-
-status, res = api("GET", "/api/v3/flows/instances/?page_size=100")
-if status != 200:
-    die(f"GET flows -> {status} {res}")
-
-flows = {f["slug"]: f["pk"] for f in res.get("results", [])}
-auth_flow = flows.get("default-provider-authorization-implicit-consent")
-inval_flow = flows.get("default-provider-invalidation-flow")
-
-if not auth_flow or not inval_flow:
-    die(f"Required flows not found. auth_flow={auth_flow}, inval_flow={inval_flow}")
-
-print(f"  authorization_flow: {auth_flow}")
-print(f"  invalidation_flow: {inval_flow}")
-
-# ===========================================================================
-# Phase 5: Create OAuth2 providers with scopes
-# ===========================================================================
-print("\n[5/6] Creating OAuth2 providers...")
-
-status, res = api("GET", "/api/v3/providers/oauth2/?page_size=100")
-if status != 200:
-    die(f"GET providers -> {status} {res}")
-existing_providers = {p["name"]: p for p in res.get("results", [])}
-
-# Fetch scope mapping PKs
-status, scopes_res = api("GET", "/api/v3/propertymappings/provider/scope/?page_size=100")
-if status != 200:
-    print("  WARNING: could not fetch scope mappings")
-    scope_pks = {}
-else:
-    scope_pks = {m["scope_name"]: m["pk"] for m in scopes_res.get("results", [])}
-
-# Include standard OpenID scopes (openid, email, profile) + custom claim scopes
-STANDARD_SCOPES = ["openid", "email", "profile"]
-scope_pks_list = [scope_pks[s] for s in STANDARD_SCOPES if s in scope_pks]
-scope_pks_list += [scope_pks[s] for s in SCOPE_MAPPINGS.keys() if s in scope_pks]
-
-# OAuth2 providers — client_secret sourced from SOPS-encrypted k8s secrets.
-# These are the real secrets the services use. Authentik must match.
-OAuth2_PROVIDERS = {
-    "api-gw": {
-        "client_id": "api-gw",
-        "client_secret_env": "AUTHENTIK_PROVIDER_API_GW_SECRET",
-        "redirect_uris": ["http://localhost:3000/callback", "https://api.riotpiao.com/callback"],
-    },
-    "minio": {
-        "client_id": "minio",
-        "client_secret": "9d2867fe08c3bf7fedd7e32bbaf4456fce3b0aaf788966d7559e1955947b0219",
-        "redirect_uris": ["http://localhost:9000/auth/sso/oauth2/code", "https://minio.riotpiao.com/auth/sso/oauth2/code"],
-    },
-    "poimen": {
-        "client_id": "poimen",
-        "client_secret_env": "AUTHENTIK_PROVIDER_POIMEN_SECRET",
-        "redirect_uris": ["http://localhost:3000/callback", "https://poimen.riotpiao.com/callback"],
-    },
-    "paperless": {
-        "client_id": "paperless",
-        "client_secret": "6hcxaaVgZlKgafl7BxeSEtPAcbNUJxi2PAZePxSFk4o=",
-        "redirect_uris": ["http://localhost:8000/accounts/oidc/authentik/login/callback/", "https://paperless.riotpiao.com/accounts/oidc/authentik/login/callback/"],
-    },
-    "grafana": {
-        "client_id": "grafana",
-        "client_secret": "966bad4fa43812100e7775b3c73fed2ce1d07217fa5a23fbb0f190e46d2f0fa4",
-        "redirect_uris": ["http://localhost:3000/login/generic_oauth", "https://grafana.riotpiao.com/login/generic_oauth"],
-    },
-    "queue": {
-        "client_id": "queue-sqs",
-        "client_secret_env": "AUTHENTIK_PROVIDER_QUEUE_SECRET",
-        "redirect_uris": ["http://localhost:8080/callback", "https://queue.riotpiao.com/callback"],
-    },
-    "forgejo": {
-        "client_id": "forgejo",
-        "client_secret": "G4klhs3JRfs5A7YnGs90WuOndBAvamWlZgaZRY8x",
-        "redirect_uris": ["http://localhost:3000/user/oauth2/authentik/callback", "https://forgejo.riotpiao.com/user/oauth2/authentik/callback"],
-    },
-    "immich": {
-        "client_id": "immich",
-        "client_secret": "QxyWfESXqTD55aUyh6miYnny1QTCEuEwyC4escw9",
-        "redirect_uris": ["app.immich:///oauth-callback", "https://img.riotpiao.com/auth/login", "https://img.riotpiao.com/user/oauth2/callback"],
-    },
-    "argocd": {
-        "client_id": "argocd",
-        "client_secret_env": "AUTHENTIK_PROVIDER_ARGOCD_SECRET",
-        "redirect_uris": ["http://localhost:8080/auth/callback", "https://argocd.riotpiao.com/auth/callback"],
-    },
-    "vault": {
-        "client_id": "vault",
-        "client_secret_env": "AUTHENTIK_PROVIDER_VAULT_SECRET",
-        "redirect_uris": ["http://localhost:8200/ui/vault/auth/oidc/oidc/callback", "https://vault.riotpiao.com/ui/vault/auth/oidc/oidc/callback"],
-    },
-}
-
-import secrets as _secrets
-
-for provider_name, provider_spec in OAuth2_PROVIDERS.items():
-    # Resolve client_secret: explicit > env var > generate random
-    if "client_secret" in provider_spec:
-        client_secret = provider_spec["client_secret"]
-    elif "client_secret_env" in provider_spec:
-        client_secret = os.environ.get(provider_spec["client_secret_env"], _secrets.token_urlsafe(32))
-    else:
-        client_secret = _secrets.token_urlsafe(32)
-
-    redirect_uris_list = [{"url": uri, "matching_mode": "strict"} for uri in provider_spec["redirect_uris"]]
-
-    provider_payload = {
-        "name": provider_name,
-        "authorization_flow": auth_flow,
-        "invalidation_flow": inval_flow,
-        "grant_types": ["authorization_code", "implicit", "password"],
-        "client_id": provider_spec["client_id"],
-        "client_secret": client_secret,
-        "redirect_uris": redirect_uris_list,
-        "property_mappings": scope_pks_list,
-    }
-
-    if provider_name in existing_providers:
-        # UPDATE existing provider — sync secret + redirect_uris
-        provider_pk = existing_providers[provider_name]["pk"]
-        status, res = api("PATCH", f"/api/v3/providers/oauth2/{provider_pk}/", {
-            "client_id": provider_spec["client_id"],
-            "client_secret": client_secret,
-            "redirect_uris": redirect_uris_list,
-            "property_mappings": scope_pks_list,
-        })
-        if status in (200, 201):
-            print(f"  {provider_name}: updated (secret + redirect_uris synced)")
-        else:
-            print(f"  {provider_name}: UPDATE FAILED {status} {res}")
-    else:
-        # CREATE new provider
-        status, res = api("POST", "/api/v3/providers/oauth2/", provider_payload)
-        if status in (200, 201):
-            print(f"  {provider_name}: created")
-        else:
-            print(f"  {provider_name}: FAILED {status} {res}")
-
-# ===========================================================================
-# Phase 6: Create OAuth2 Applications (bind providers to public token endpoints)
-# ===========================================================================
-print("\n[6/7] Creating OAuth2 Applications...")
-print("     (binds providers to /application/o/token/ endpoints)")
-
-status, res = api("GET", "/api/v3/core/applications/?page_size=100")
-if status != 200:
-    die(f"GET applications -> {status} {res}")
-existing_apps = {a["slug"]: a for a in res.get("results", [])}
-
-for provider_name in OAuth2_PROVIDERS.keys():
-    # Get the provider PK
-    status, provider_res = api("GET", f"/api/v3/providers/oauth2/?name={provider_name}")
-    if status != 200 or not provider_res.get("results"):
-        print(f"  {provider_name}: provider not found, skip")
-        continue
-    provider_pk = provider_res["results"][0]["pk"]
-
-    if provider_name in existing_apps:
-        # Ensure app is linked to provider (fix orphaned apps)
-        app_data = existing_apps[provider_name]
-        if app_data.get("provider") != provider_pk:
-            app_uuid = app_data["pk"]
-            status, res = api("PATCH", f"/api/v3/core/applications/{app_uuid}/", {
-                "provider": provider_pk,
-            })
-            if status in (200, 201):
-                print(f"  {provider_name}: re-linked to provider")
-            else:
-                print(f"  {provider_name}: RE-LINK FAILED {status} {res}")
-        else:
-            print(f"  {provider_name}: ok")
-    else:
-        status, res = api("POST", "/api/v3/core/applications/", {
-            "name": provider_name,
-            "slug": provider_name,
-            "provider": provider_pk,
-        })
-        if status in (200, 201):
-            print(f"  {provider_name}: created")
-        else:
-            print(f"  {provider_name}: FAILED {status} {res}")
-
-# ===========================================================================
-# Phase 7: Create/update rock user → homelab-admins, matching Forgejo identity
-# ===========================================================================
-print("\n[7/10] Creating/updating rock user ([email protected])...")
-
-ROCK_EMAIL = "[email protected]"
-ROCK_PASSWORD = os.environ.get("ROCK_PASSWORD", "")
-
-status, res = api("GET", "/api/v3/core/users/?username=rock")
-if status == 200 and res.get("results"):
-    rock_user = res["results"][0]
-    # Ensure email matches Forgejo's rock user for OIDC linking
-    patch_data = {"email": ROCK_EMAIL, "name": "Rock"}
-    status, res = api("PATCH", f"/api/v3/core/users/{rock_user['pk']}/", patch_data)
-    if status in (200, 201):
-        print(f"  rock: updated email to {ROCK_EMAIL}")
-    else:
-        print(f"  rock: update FAILED {status} {res}")
-else:
-    if not ROCK_PASSWORD:
-        print("  rock: NOT FOUND and ROCK_PASSWORD not set, skipping creation")
-        print("    export ROCK_PASSWORD=<password> and re-run")
-        rock_user = None
-    else:
-        status, res = api("POST", "/api/v3/core/users/", {
-            "username": "rock",
-            "name": "Rock",
-            "email": ROCK_EMAIL,
-            "is_active": True,
-            "is_superuser": False,
-            "password": ROCK_PASSWORD,
-        })
-        if status in (200, 201):
-            rock_user = res
-            print(f"  rock: created with email {ROCK_EMAIL}")
-        else:
-            print(f"  rock: create FAILED {status} {res}")
-            rock_user = None
-
-if rock_user:
-    status, res = api("GET", "/api/v3/core/groups/?name=homelab-admins")
-    if status == 200 and res.get("results"):
-        admins_group = res["results"][0]
-        status, res = api("POST", f"/api/v3/core/groups/{admins_group['pk']}/users/add/", {"pk": rock_user["pk"]})
-        if status in (200, 201, 204):
-            print(f"  rock: added to homelab-admins")
-        else:
-            print(f"  rock: group add {status} {res}")
-
-# ===========================================================================
-# Phase 8: Email recovery flow (password reset via email)
-# ===========================================================================
-print("\n[8/10] Creating email recovery flow...")
-
-# Read SMTP config from gotify-smtp secret (same Gmail creds)
-SMTP_HOST = "smtp.gmail.com"
-SMTP_PORT = 587
-SMTP_USER = "[email protected]"
-SMTP_FROM = "[email protected]"
-# Password read from env at runtime: AUTHENTIK_EMAIL__PASSWORD
-
-# 8a. Create email stage for recovery
-status, res = api("GET", "/api/v3/stages/email/?name=email-recovery")
-if status == 200 and res.get("results"):
-    email_stage_pk = res["results"][0]["pk"]
-    print("  email-recovery stage: already exists")
-else:
-    status, res = api("POST", "/api/v3/stages/email/", {
-        "name": "email-recovery",
-        "use_global_settings": False,
-        "host": SMTP_HOST,
-        "port": SMTP_PORT,
-        "username": SMTP_USER,
-        "password": os.environ.get("AUTHENTIK_EMAIL_PASSWORD", ""),
-        "use_tls": True,
-        "use_ssl": False,
-        "timeout": 10,
-        "from_address": SMTP_FROM,
-        "template": "email/password_reset.html",
-        "activate_user_on_success": True,
-    })
-    if status in (200, 201):
-        email_stage_pk = res["pk"]
-        print("  email-recovery stage: created")
-    else:
-        email_stage_pk = None
-        print(f"  email-recovery stage: FAILED {status} {res}")
-
-# 8b. Create identification stage for recovery (email lookup)
-status, res = api("GET", "/api/v3/stages/identification/?name=recovery-identification")
-if status == 200 and res.get("results"):
-    ident_stage_pk = res["results"][0]["pk"]
-    print("  recovery-identification stage: already exists")
-else:
-    status, res = api("POST", "/api/v3/stages/identification/", {
-        "name": "recovery-identification",
-        "user_fields": ["email", "username"],
-    })
-    if status in (200, 201):
-        ident_stage_pk = res["pk"]
-        print("  recovery-identification stage: created")
-    else:
-        ident_stage_pk = None
-        print(f"  recovery-identification stage: FAILED {status} {res}")
-
-# 8c. Create password stage for new password entry
-status, res = api("GET", "/api/v3/stages/password/?name=recovery-password-change")
-if status == 200 and res.get("results"):
-    pw_stage_pk = res["results"][0]["pk"]
-    print("  recovery-password-change stage: already exists")
-else:
-    # Use prompt stage for password change instead
-    status, res = api("GET", "/api/v3/stages/user_write/?name=recovery-user-write")
-    if status == 200 and res.get("results"):
-        pw_stage_pk = res["results"][0]["pk"]
-        print("  recovery-user-write stage: already exists")
-    else:
-        status, res = api("POST", "/api/v3/stages/user_write/", {
-            "name": "recovery-user-write",
-        })
-        if status in (200, 201):
-            pw_stage_pk = res["pk"]
-            print("  recovery-user-write stage: created")
-        else:
-            pw_stage_pk = None
-            print(f"  recovery-user-write stage: FAILED {status} {res}")
-
-# 8d. Create recovery flow
-status, res = api("GET", "/api/v3/flows/instances/?slug=password-recovery")
-if status == 200 and res.get("results"):
-    recovery_flow_pk = res["results"][0]["pk"]
-    print("  password-recovery flow: already exists")
-else:
-    status, res = api("POST", "/api/v3/flows/instances/", {
-        "name": "Password Recovery",
-        "slug": "password-recovery",
-        "title": "Reset your password",
-        "designation": "recovery",
-    })
-    if status in (200, 201):
-        recovery_flow_pk = res["pk"]
-        print("  password-recovery flow: created")
-    else:
-        recovery_flow_pk = None
-        print(f"  password-recovery flow: FAILED {status} {res}")
-
-# 8e. Bind stages to flow in order
-if recovery_flow_pk and ident_stage_pk and email_stage_pk:
-    for order, stage_pk, label in [
-        (10, ident_stage_pk, "identification"),
-        (20, email_stage_pk, "email"),
-    ]:
-        status, res = api("POST", "/api/v3/flows/bindings/", {
-            "target": recovery_flow_pk,
-            "stage": stage_pk,
-            "order": order,
-        })
-        if status in (200, 201):
-            print(f"  bound {label} stage at order {order}")
-        elif status == 400 and "already exists" in str(res).lower():
-            print(f"  {label} stage: already bound")
-        else:
-            print(f"  bind {label}: {status} {res}")
-
-# ===========================================================================
-# Phase 9: Set recovery flow on brand
-# ===========================================================================
-print("\n[9/10] Setting recovery flow on brand...")
-
-if recovery_flow_pk:
-    status, res = api("GET", "/api/v3/brands/instances/")
-    if status == 200 and res.get("results"):
-        brand = res["results"][0]
-        status, res = api("PATCH", f"/api/v3/brands/instances/{brand['brand_uuid']}/", {
-            "flow_recovery": recovery_flow_pk,
-        })
-        if status in (200, 201):
-            print("  recovery flow set on brand")
-        else:
-            print(f"  FAILED {status} {res}")
-    else:
-        print("  no brand found")
-
-# ===========================================================================
-# Phase 10: Ensure Forgejo OAuth2 source uses matching email claim
-# ===========================================================================
-print("\n[10/10] Verifying Forgejo OIDC linkage...")
-print(f"  rock@Authentik email: {ROCK_EMAIL}")
-print("  Forgejo OIDC will match on email — ensure Forgejo's rock user")
-print(f"  has email {ROCK_EMAIL} in Forgejo settings → Profile")
-
-# ===========================================================================
-# Summary
-# ===========================================================================
-print("\n" + "="*70)
-print("AUTHENTIK PROVISIONING COMPLETE")
-print("="*70)
-print(f"\n  [1] Groups: {len(GROUPS)}")
-print(f"  [2] Service accounts: {len(SERVICE_ACCOUNTS)}")
-print(f"  [3] Scope mappings: {len(SCOPE_MAPPINGS)}")
-print(f"  [4] Flows resolved")
-print(f"  [5] OAuth2 providers: {len(OAuth2_PROVIDERS)}")
-print(f"  [6] OAuth2 applications bound")
-print(f"  [7] rock user ([email protected]) -> homelab-admins")
-print(f"  [8] Email recovery flow (smtp.gmail.com)")
-print(f"  [9] Recovery flow set on brand")
-print(f"  [10] Forgejo OIDC linkage verified")
-print("\nNEXT: Set Forgejo rock user email to [email protected] in Forgejo profile")
-print("TEST: https://authentik.riotpiao.com/if/flow/password-recovery/")
-print("="*70)
@@ -49,6 +49,10 @@ CONF_DIR = Path(os.environ.get("ZITADEL_CONF_DIR", Path.home() / ".config" / "zi
 STATE_FILE = CONF_DIR / "state.json"
 NAMESPACE = os.environ.get("ZITADEL_NAMESPACE", "iam")
 BASE_URL = os.environ.get("ZITADEL_BASE_URL", "https://zitadel.riotpiao.com")
+# Notification delivery target: the in-cluster SMTP -> Gotify relay.
+SMTP_HOST = os.environ.get("ZITADEL_SMTP_HOST", "smtp-gotify.iam.svc.cluster.local:2525")
+SMTP_FROM = os.environ.get("ZITADEL_SMTP_FROM", "[email protected]")
+SMTP_FROM_NAME = os.environ.get("ZITADEL_SMTP_FROM_NAME", "ZITADEL")

 # ── desired ZITADEL state ──────────────────────────────────────────────────
 # Declared here, applied through the API by the `apply` phase. This replaces the
@@ -80,6 +84,10 @@ ROLES = {
    "sqs-writers": "SQS/Temporal queue read/write access",
    "s3-users": "S3 read access",
    "s3-writers": "S3 read/write access",
+    # LLM gate for the api-gw. The colon form matches the Authentik-era
+    # permission string, so consumer manifests did not change.
+    "llm:inference": "LLM inference access (api-gw)",
+    "llm-inference": "LLM inference access (dash alias)",
 }

 _WEB = {"app_type_id": "OIDC_APP_TYPE_WEB", "auth_method_id": "OIDC_AUTH_METHOD_TYPE_BASIC"}
@@ -92,7 +100,12 @@ OIDC_APPS = {
    "argocd": {**_WEB, "redirect_uris": [
        "http://localhost:8080/auth/callback",
        "https://argocd.riotpiao.com/auth/callback"]},
+    # Forgejo derives the callback path from the login-source NAME
+    # (/user/oauth2/<name>/callback). Both spellings are registered so a
+    # rename between authentik and zitadel never breaks sign-in.
    "forgejo": {**_WEB, "redirect_uris": [
+        "http://localhost:3000/user/oauth2/zitadel/callback",
+        "https://forgejo.riotpiao.com/user/oauth2/zitadel/callback",
        "http://localhost:3000/user/oauth2/authentik/callback",
        "https://forgejo.riotpiao.com/user/oauth2/authentik/callback"]},
    "grafana": {**_WEB, "redirect_uris": [
@@ -107,6 +120,8 @@ OIDC_APPS = {
        "http://localhost:9000/auth/sso/oauth2/code",
        "https://minio.riotpiao.com/auth/sso/oauth2/code"]},
    "paperless": {**_WEB, "redirect_uris": [
+        "http://localhost:8000/accounts/oidc/zitadel/login/callback/",
+        "https://paperless.riotpiao.com/accounts/oidc/zitadel/login/callback/",
        "http://localhost:8000/accounts/oidc/authentik/login/callback/",
        "https://paperless.riotpiao.com/accounts/oidc/authentik/login/callback/"]},
    "poimen": {**_WEB, "redirect_uris": [
@@ -266,18 +281,52 @@ def _api(method: str, path: str, token: str, body: dict | None = None,

 def _ensure_project(token: str, name: str) -> str:
    code, d = _api("POST", "/management/v1/projects/_search", token, {})
+    project_id = ""
    for row in d.get("result") or []:
        if row.get("name") == name:
-            return row["id"]
-    code, d = _api("POST", "/management/v1/projects", token, {"name": name})
-    if code not in (200, 201) or not d.get("id"):
-        die(f"create project {name} failed: {code} {d}")
-    return d["id"]
+            project_id = row["id"]
+            break
+    if not project_id:
+        code, d = _api("POST", "/management/v1/projects", token, {"name": name})
+        if code not in (200, 201) or not d.get("id"):
+            die(f"create project {name} failed: {code} {d}")
+        project_id = d["id"]
+
+    # projectRoleAssertion puts the project roles into the ACCESS token as
+    # `urn:zitadel:iam:org:project:roles`. Without it the api-gw sees no
+    # llm:inference role and answers 403 for every inference call, even when
+    # the caller is correctly authorised in ZITADEL.
+    code, d = _api("PUT", f"/management/v1/projects/{project_id}",
+                   token, {"name": name, "projectRoleAssertion": True})
+    if code not in (200, 201):
+        die(f"enable projectRoleAssertion on {name} failed: {code} {d}")
+    return project_id
+
+
+def _ensure_builtin_login(token: str) -> None:
+    """Force the built-in login page instead of the hosted Login UI.
+
+    The hosted UI authenticates as the machine user `login-client`, which the
+    chart creates only from its one-shot setup job. When that user is missing,
+    every session write fails with "Could not create session for user" and no
+    one can sign in, at ZITADEL or at any relying party.
+
+    Disabling the loginV2 feature makes the server serve its own login page,
+    which needs no machine user. This is instance state in the database, not a
+    values key, so it must be applied explicitly.
+    """
+    code, d = _api("PUT", "/v2/features/instance", token, {"loginV2": {"required": False}})
+    if code not in (200, 201):
+        die(f"disable loginV2 failed: {code} {d}")
+    print("login page    : built-in (loginV2.required=false)")


 def _ensure_roles(token: str, project_id: str, roles: dict[str, str]) -> None:
    code, d = _api("POST", f"/management/v1/projects/{project_id}/roles/_search", token, {})
-    have = {r.get("roleKey") for r in (d.get("result") or [])}
+    # ZITADEL answers with `key`, not `roleKey`. Reading the wrong field left
+    # this set empty, so every re-run tried to recreate existing roles and died
+    # on the first 409.
+    have = {(r.get("key") or r.get("roleKey")) for r in (d.get("result") or [])}
    for key, display in roles.items():
        if key in have:
            continue
@@ -326,12 +375,95 @@ def _ensure_oidc_apps(token: str, project_id: str, apps: dict[str, dict]) -> tup
    return ids, secrets


+def _ensure_smtp(token: str) -> str:
+    """Make sure the instance has one active SMTP provider.
+
+    ZITADEL sends verification codes and password resets by SMTP only. A
+    freshly added config is created INACTIVE (state 3), and an inactive config
+    makes GetSMTPConfig answer 404 "SMTP configuration not found", so nothing is
+    ever delivered. Activating it is what enables sending: after the activate
+    call the same GET returns the config and a resend reaches the relay.
+
+    This is instance state in the database, not a values key, so it must be
+    applied explicitly. The host points at the in-cluster smtp-gotify relay
+    (k8s/infra/iam/smtp-gotify.yaml), which forwards to Gotify.
+    """
+    code, d = _api("GET", "/admin/v1/smtp", token)
+    if code == 200 and (d.get("smtpConfig") or {}).get("id"):
+        smtp_id = d["smtpConfig"]["id"]
+        print(f"smtp      : already active ({smtp_id})")
+        return smtp_id
+
+    code, d = _api("POST", "/admin/v1/smtp", token, {
+        "senderAddress": SMTP_FROM,
+        "senderName": SMTP_FROM_NAME,
+        "host": SMTP_HOST,
+        "tls": False,
+    })
+    if code not in (200, 201) or not d.get("id"):
+        die(f"create smtp config failed: {code} {d}")
+    smtp_id = d["id"]
+
+    # The activate call is mandatory: without it the config stays inactive and
+    # every send is dropped while the API still reports success.
+    code, d = _api("POST", f"/admin/v1/smtp/{smtp_id}/_activate", token, {})
+    if code not in (200, 201):
+        die(f"activate smtp config {smtp_id} failed: {code} {d}")
+
+    code, d = _api("GET", "/admin/v1/smtp", token)
+    if code != 200 or not (d.get("smtpConfig") or {}).get("id"):
+        die(f"smtp config {smtp_id} still not visible after activate: {code} {d}")
+    print(f"smtp      : active ({smtp_id}) -> {SMTP_HOST}")
+    return smtp_id
+
+
+def _ensure_login_policy(token: str) -> None:
+    """Require password + a second factor, and demand enrolment at login.
+
+    Enforcement needs forceMfa. The v1 endpoint ignores the spelling "forceMFA"
+    as an unknown field, which is why earlier runs looked successful while
+    projections.login_policies5 kept force_mfa = false.
+
+    Second factors stay OTP and U2F: an authenticator app or a hardware key.
+    Multi-factor is left configured for U2F with verification so a later step
+    can move 2FA to 2MFA without another policy change.
+    """
+    code, d = _api("PUT", "/admin/v1/policies/login", token, {
+        # Case matters: protojson accepts "forceMfa", the lowerCamel form of
+        # force_mfa. Sending "forceMFA" is silently ignored as an unknown field,
+        # so the policy looks unchanged and MFA is never enforced.
+        "forceMfa": True,
+        "forceMfaLocalOnly": True,
+        "allowUsernamePassword": True,
+        "allowRegister": False,
+        "allowExternalIDP": True,
+        "hidePasswordReset": False,
+        "ignoreUnknownUsernames": True,
+        "passwordCheckLifetime": "864000s",
+        "externalLoginCheckLifetime": "864000s",
+        "mfaInitSkipLifetime": "0s",
+        "secondFactorCheckLifetime": "64800s",
+        "multiFactorCheckLifetime": "64800s",
+        "secondFactors": ["SECOND_FACTOR_TYPE_OTP", "SECOND_FACTOR_TYPE_U2F"],
+        "multiFactors": ["MULTI_FACTOR_TYPE_U2F_WITH_VERIFICATION"],
+        "passwordlessType": "PASSWORDLESS_TYPE_ALLOWED",
+    })
+    # ZITADEL answers 400 "has not been changed" when the policy already matches.
+    # That is success for an idempotent run, not a failure.
+    if code not in (200, 201) and "has not been changed" not in str(d.get("message", "")):
+        die(f"update login policy failed: {code} {d}")
+    print("login policy : password + 2FA, enrolment required")
+
+
 def phase_apply(_args: argparse.Namespace) -> int:
    """Declare the ZITADEL objects through the API. No Terraform, no state drift."""
    token = bootstrap_token()
    if not token:
        die("no iam-admin PAT — run the keys phase first")

+    _ensure_builtin_login(token)
+    _ensure_smtp(token)
+    _ensure_login_policy(token)
    project_id = _ensure_project(token, PROJECT_NAME)
    print(f"project   : {PROJECT_NAME} ({project_id})")
    _ensure_roles(token, project_id, ROLES)
@@ -22,7 +22,6 @@
    # TLS termination (cert-manager cert) and preserve the Host header. Routing
    # via nginx (not the backend service) is critical: a direct rewrite bypasses
    # nginx TLS and exposes each app's own self-signed cert, trusted by nothing.
-    rewrite name authentik.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name zitadel.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name grafana.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name minio.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
Author	SHA1	Message	Date
rock	12c66fb612	feat(grafana): provision a Gotify contact point Grafana gains one contact point, `gotify`, of type webhook. It posts to the bridge in the notifications namespace because Gotify rejects a Grafana payload with 400 "Field 'message' is required". maxAlerts 0 keeps every alert in one request, and the bridge names the extra count in the message body instead of dropping alerts.	2026-10-27 12:38:29 -07:00
rock	50a751de44	feat(gotify): bridge Grafana alerts into Gotify Gotify rejects a Grafana payload with 400 "Field 'message' is required", so a translation step is mandatory. This adds a small standard-library Python service: it accepts Grafana's JSON, builds a Gotify message, and posts it with an app token. Mapping: firing -> priority 8, title "[FIRING] <alertname>" resolved -> priority 3, title "[RESOLVED] <alertname>" body -> the summary annotation plus severity, namespace, pod, instance, job A multi-alert notification names the alert count instead of dropping the rest. The script ships as a ConfigMap so no image build is needed. The app token comes from a SOPS secret and belongs to Gotify application `grafana`.	2026-10-27 12:35:44 -07:00
rock	863c11d75f	feat(igotify): wire the assist bridge with a Gotify client token The bridge runs server-side, so it cannot run a browser flow and ZITADEL has no part in it. Give it a Gotify client token instead: GOTIFY_URLS points at the public host, GOTIFY_CLIENT_TOKENS comes from a SOPS secret. The token belongs to user `rock`, is named igotify-assist, and has expiresAfterInactivitySeconds 0 so it never idles out. Delete the client row in Gotify to revoke it. Verified with the token: GET /client returns 200 and GET /message returns the latest message.	2026-10-27 12:14:28 -07:00
rock	a525a52580	Revert "feat(gotify): put the Gotify UI behind ZITADEL single sign-on" This reverts commit `03cebb0aeb`.	2026-10-27 12:12:22 -07:00
rock	3e92f34153	Revert "fix(gotify): use a 32-byte cookie secret for oauth2-proxy" This reverts commit `549e0be502`.	2026-10-27 12:12:22 -07:00
rock	549e0be502	fix(gotify): use a 32-byte cookie secret for oauth2-proxy The first value was a base32 string, so oauth2-proxy rejected it at startup: cookie_secret must be 16, 24, or 32 bytes to create an AES cipher, but is 56 bytes.	2026-10-27 12:08:13 -07:00
rock	03cebb0aeb	feat(gotify): put the Gotify UI behind ZITADEL single sign-on Gotify has no OIDC support, so it cannot validate a ZITADEL token itself. Put oauth2-proxy in front of it and let nginx ask the proxy for every request. The Gotify REST API keeps its own client-token check. /message, /application, /client, /user, /version, /stream and /health are skipped by the proxy, so Forgejo webhooks, the harness, poimen-sender and the Android client keep working without a browser session. Only the web UI requires SSO. The /oauth2 path needs its own ingress: the auth_request annotation would otherwise also guard /oauth2/start, and the login entry point would demand a login first. The client id and secret come from project role provisioning: `zitadel-provision.py apply` now creates the `gotify` OIDC app, and the secret is SOPS-encrypted under k8s/argocd/secrets. Gotify's own login remains behind the proxy. ZITADEL decides who reaches the service, Gotify decides what that person may do inside it.	2026-10-27 12:06:00 -07:00
rock	cc7c30ee20	fix(iam): actually enforce MFA forceMFA never took effect because of the JSON key. ZITADEL's v1 endpoint speaks protojson, which accepts "forceMfa" (lowerCamel of force_mfa). The spelling "forceMFA" is treated as an unknown field and silently dropped, so every earlier run returned 200 with a new sequence while force_mfa stayed false in projections.login_policies5. Send forceMfa and forceMfaLocalOnly. Verified in the database: force_mfa = t \| force_mfa_local_only = t \| mfa_init_skip_lifetime = 0 and in the login flow: after the password step the browser lands on /ui/login/mfa/prompt with the "2-Factor Setup" page, so a second factor is now required instead of optional.	2026-10-27 00:14:42 -07:00
rock	6768d1c762	feat(iam): require password + 2FA at login Add a login-policy phase: password plus a second factor, with mfaInitSkipLifetime 0s so enrolment is demanded at login rather than skipped for 30 days. Second factors stay OTP and U2F (authenticator app or hardware key). Multi-factor is already configured for U2F-with-verification, so moving 2FA to 2MFA later needs no further policy change. forceMFA does not persist through the v1 policy endpoint: it reads back absent even after a successful PUT, which is why the skip lifetime is the lever used. A repeat run returns 400 "Default Login Policy has not been changed"; treat that as success so provisioning stays idempotent. Full apply now completes end to end: login page : built-in smtp : already active login policy : password + 2FA, enrolment required roles : 21 ensured oidc apps : 11 ensured	2026-10-27 00:12:32 -07:00
rock	80af9b1fd0	feat(iam): provision and activate the SMTP provider ZITADEL sends verification codes and password resets by SMTP only, and nothing delivered until the instance had an active SMTP config. Two traps, both now handled here instead of by hand: - POST /admin/v1/smtp creates the config INACTIVE (state 3). An inactive config makes GET /admin/v1/smtp answer 404 "SMTP configuration not found", so the API reports success while every send is dropped. The _activate call is mandatory. - The target is the in-cluster smtp-gotify relay, which forwards to Gotify. The phase is idempotent: an active config short-circuits and only logs. Also fix the role-exists check: ZITADEL returns `key`, not `roleKey`, so the set was always empty and every re-run died on the first 409 "Role already exists". Verified: `zitadel-provision.py apply` now completes, reporting `smtp : already active`, `roles : 21 ensured`, `oidc apps : 11 ensured`. A resend then produced Gotify message 1185 ("Verify email, Code ZVYLQG") and submitting that code set verified_email for user riotpiao.	2026-10-27 00:11:50 -07:00
rock	7f29d640fd	feat(iam): own the ZITADEL core objects in GitOps The chart declares its ServiceAccount, ConfigMap, Role and RoleBinding as pre-install/pre-upgrade Helm hooks with hook-delete-policy before-hook-creation. ArgoCD maps those to PreSync and prunes them when a sync ends, so every sync deleted the ServiceAccount and ConfigMap the ZITADEL Deployment mounts. New pods failed with `serviceaccount "zitadel" not found`, leaving the IdP one restart from an outage. Add a zatadel-core Application owning hook-free copies, keeping them as ordinary desired state that survives every sync. Prune is off for this path: deleting a ServiceAccount here would recreate the outage. The ConfigMap also carries the notification settings, pointing DefaultInstance.SMTPConfiguration at the smtp-gotify relay, with SMTPSenderAddressMatchesInstanceDomain false because the sender [email protected] differs from the instance domain zatadel.riotpiao.com. Drop the envVarsSecret approach: ZITADEL created no SMTP provider from those env vars (no instance.smtp.config.added event).	2026-10-26 23:54:57 -07:00
rock	9c3b2bfbbe	feat(iam): wire ZITADEL SMTP through envVarsSecret ZITADEL must send verification codes, and it only speaks SMTP. Route that at the smtp-gotify relay so codes reach Gotify and the phone. The settings go in as a ZITADEL_DEFAULTINSTANCE_SMTPCONFIGURATION_* envFrom secret, not as configmapConfig. Editing configmapConfig makes ArgoCD stop managing the rendered zitadel-config-yaml ConfigMap, which the Deployment still mounts; that twice left the IdP one restart away from failing. Env vars override the same keys and leave that ConfigMap alone. Validated with helm template 10.4.0: envFrom appears on both ZITADEL containers and zitadel-config-yaml still renders.	2026-10-26 23:45:59 -07:00
rock	81e6e9bc03	Revert "feat(iam): send ZITADEL notifications to the Gotify relay" This reverts commit `aa1f224349`.	2026-10-26 23:41:01 -07:00
rock	aa1f224349	feat(iam): send ZITADEL notifications to the Gotify relay Point DefaultInstance.SMTPConfiguration at the in-cluster smtp-gotify relay, so verification codes and password resets arrive on the phone instead of being dropped for lack of a mail path. Validated with `helm template zitadel/zitadel 10.4.0` before syncing: the block renders, and the zatadel-config-yaml ConfigMap is still produced. An earlier attempt looked like it removed that ConfigMap, which is why this went in only after a local render check.	2026-10-26 23:39:53 -07:00
rock	37933de756	fix(iam): give the SMTP->Gotify relay a writable /tmp The relay stages each message in /tmp before pushing. readOnlyRootFilesystem left it with a read-only /tmp, so it answered '250 queued' and then dropped the message: no Gotify push, no error line. Mount an emptyDir at /tmp.	2026-10-26 23:38:29 -07:00
rock	468ed16737	revert(iam): back out the SMTP values block that dropped the config ConfigMap The sync removed zitadel-config-yaml while the Deployment still mounts it, so the next pod restart would have failed. Restore the previous render first; the relay stays deployed and can be wired once the chart accepts the block.	2026-10-26 23:32:50 -07:00
rock	ab9c2b444b	fix(iam): make the SMTP->Gotify relay actually push Two defects in the first cut, both silent from the sender's side because the relay answers 250 either way: - GOTIFY_URL needs a trailing slash; without it the push has no path. - GOTIFY_TITLE_TEMPLATE referenced .Subject, which the relay does not expose, so template execution failed. The relay's default title already carries the subject, so drop the template. Also register the SOPS file in secret-generator.yaml: the sops-secrets app only renders files listed there, so the Secret was never created and the relay sat in CreateContainerConfigError.	2026-10-26 23:28:44 -07:00
rock	b6e1e0c591	fix(iam): give the IAM support manifests an ArgoCD app again Retiring Authentik deleted the iam-jobs Application, so k8s/infra/iam stopped being synced. The new smtp-gotify relay would have sat in git and never been created. Add an iam-support Application for that path, and drop the stale rbac-dashboard-rolebinding reference from its kustomization: the file went away with Authentik, and a missing resource fails the whole app.	2026-10-26 23:24:55 -07:00
rock	701d79deac	feat(iam): deliver ZITADEL notifications through Gotify ZITADEL sends verification codes and password resets by SMTP only, and this cluster has no mail path. Every human user therefore sits at verified_email = empty and can never finish a first login, at ZITADEL or at Forgejo. Add an smtp-gotify relay in the iam namespace: it accepts plain SMTP on :2525 and forwards each message to the existing Gotify instance, so codes arrive on the phone through gotify/igotify instead of email. Point DefaultInstance.SMTPConfiguration at that Service with TLS off, because the relay speaks plain SMTP inside the cluster. The relay holds the Gotify app token in a SOPS-encrypted Secret; values never carry the token.	2026-10-26 23:24:33 -07:00
rock	fbc1b6a33d	feat(iam): manage the Forgejo OAuth login source from a script Forgejo keeps login sources in its database and exposes no REST endpoint for them, so neither GitOps nor `tea` (whose admin command only manages users) can set them. The in-pod CLI is the only supported writer. This script wraps `gitea admin auth update-oauth` so the rename from the Authentik-era label to `zitadel` is repeatable instead of a one-off command in a shell history. Note in the header: the source name is both the button label and the callback path, so ZITADEL must have both redirect URIs registered before a rename.	2026-10-26 23:17:03 -07:00
rock	3f93ddc002	feat(iam): provision the built-in login page loginV2.required lives in the instance database, not in chart values, so disabling the hosted Login UI in values is only half the change: the server kept redirecting /oauth/v2/authorize to /ui/v2/login, which no longer had a backend and returned 503. Set loginV2.required=false in the provisioning run so the server serves its own login page. Verified: authorize now redirects to /ui/login/login and the page renders the "Welcome Back!" login form.	2026-10-26 23:12:53 -07:00
rock	eaf9804374	fix(iam): use the built-in ZITADEL login page Sign-in failed with "Could not create session for user". The ZITADEL API logged, on every session call: authz: verify access token -> invalid token token verifier repo: issuer does not match: Expected https://zitadel.riotpiao.com, got: login-client zitadel-login: [Password] Could not update session The hosted Login UI authenticates as the machine user `login-client`. That user did not exist in this instance, because the chart's setup job creates it and setupJob is disabled after the first sync. Without the user the UI cannot write a session, so no one can sign in, at ZITADEL or at Forgejo. Disable the hosted Login UI so ZITADEL serves its built-in login, which needs no machine user. Re-enable only after creating `login-client` and registering the public key that matches zitadel-login-service-key. Also register both callback spellings for the forgejo and paperless apps: the callback path follows the login-source name, so a rename between authentik and zitadel must not break sign-in.	2026-10-26 23:08:20 -07:00
rock	903a72a6fd	feat(iam): provision the LLM inference roles and assert project roles The api-gw gates /v1/* on a capability named in the access token. ZITADEL had no such role, so every inference call was 403 once the token came from ZITADEL instead of Authentik. - add llm:inference (Authentik spelling, keeps consumer manifests unchanged) and llm-inference (dash alias) to the project roles - enable projectRoleAssertion on the project, which is what puts urn:zitadel:iam:org:project:roles into the access token; without it the role list never reaches the resource server	2026-10-26 10:26:54 -07:00
rock	81883a8ea1	chore(iam): retire Authentik Zitadel is running and has provisioned OIDC clients for all 12 consumers (state.json: kubernetes, paperless, forgejo, argocd, grafana, immich, minio, vault, poimen, api-gw, queue-sqs, homarr). Authentik's remaining job was to serve until the last consumer moved. Remove every Authentik artifact from GitOps: - ArgoCD Application `authentik` and its Helm values - ArgoCD Application `iam-jobs` (provisioner RBAC + migration job) - ArgoCD Application `oidc-bootstrap` (one-shot apiserver restart) - authentik-db CNPG cluster definition - authentik-secrets SOPS file and secret-generator entry - Ingress rule, cloudflared tunnel hostname, CoreDNS rewrite - Prometheus alerts and ServiceMonitor - admin OIDC ClusterRoleBinding provisioning binding - scripts/iam/provision-rbac.py (the Authentik provisioner) The chart repo https://charts.goauthentik.io is removed from the AppProject. Kube-apiserver OIDC and consumer configs are switched to Zitadel in the local cluster-config (gitignored) and applied separately via talosctl.	2026-10-26 03:32:18 -07:00