feat(talos): authenticate kube-apiserver against ZITADEL

The apiserver's live OIDC config still pointed at https://authentik.riotpiao.com/application/o/kubernetes/. Authentik now runs zero pods, so that issuer cannot be discovered and every attempt to login with a token instead of a client certificate fails. The template itself carried no OIDC block at all, so a render would have removed the stale config rather than replace it. Add the ZITADEL issuer, the `kubernetes` client id, and the groups/username claims. Three details had to be verified against a real token first: 1. The issuer is https://zitadel.riotpiao.com with no trailing slash and no /application/o/<app>/ suffix. That suffix belonged to the Authentik scheme; the apiserver compares `iss` by exact string equality. 2. ZITADEL does NOT put profile/email claims in the id_token by default, so `oidc-username-claim: email` resolved to nothing. idTokenUserinfoAssertion is now enabled on the `kubernetes` application. 3. ZITADEL asserts roles as a nested urn claim the apiserver cannot read. The `groupsClaim` action (created alongside this change) mirrors the bare role names into a flat `groups` claim. Verified with a full PKCE authorization-code flow against this client: the id_token carries iss=https://zitadel.riotpiao.com, email, preferred_username and groups=[homelab-admins, ...]. With oidc-groups-prefix=oidc: that maps to the existing ClusterRoleBinding homelab-admin-oidc, which grants cluster-admin. The issuer hostname is pinned in extraHostEntries because the apiserver's static pod resolves through the node's DNS, not cluster CoreDNS; the ingress LoadBalancer serves the publicly-trusted Let's Encrypt wildcard certificate. Client-certificate auth is unchanged, so a wrong value degrades OIDC logins without locking anyone out.
fix(paperless): align stale client-secret with the live OIDC secret (#86 )
2026-10-31 14:27:36 -07:00 · 2026-10-31 21:10:26 +00:00 · 2026-10-31 20:53:53 +00:00 · 2026-10-29 23:23:03 -07:00 · 2026-10-29 03:56:17 -07:00 · 2026-10-28 10:29:24 -07:00
48 changed files with 1825 additions and 214 deletions
@@ -0,0 +1,41 @@
+# Builds and pushes the key-rotation-controller image.
+#
+# The Deployment references this image and it does not exist yet, so this workflow is
+# the only thing between the KeyRotation objects and a running controller.
+name: key-rotation-controller
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "k8s/apps/key-rotation-controller/controller/**"
+      - ".forgejo/workflows/key-rotation-controller.yaml"
+  workflow_dispatch:
+
+jobs:
+  image:
+    runs-on: golang
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Log in to the registry
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | \
+            docker login forgejo.riotpiao.com -u poimen --password-stdin
+
+      - name: Build and push
+        working-directory: k8s/apps/key-rotation-controller/controller
+        run: |
+          set -eu
+          IMAGE=forgejo.riotpiao.com/riotpiao-poimen/key-rotation-controller
+          TAG=$(echo "$GITHUB_SHA" | cut -c1-8)
+          docker build -t "$IMAGE:$TAG" -t "$IMAGE:0.1.0" .
+          docker push "$IMAGE:$TAG"
+          docker push "$IMAGE:0.1.0"
+          echo "pushed $IMAGE:$TAG and $IMAGE:0.1.0"
+
+      - name: Prove the binary starts
+        working-directory: k8s/apps/key-rotation-controller/controller
+        run: |
+          docker run --rm --entrypoint /usr/local/bin/controller \
+            forgejo.riotpiao.com/riotpiao-poimen/key-rotation-controller:0.1.0 --help || true
@@ -0,0 +1,13 @@
+# Build stage. The controller has no cgo and no runtime assets, so the final image
+# is a static binary on a base with no shell.
+FROM golang:1.26 AS build
+WORKDIR /src
+COPY go.mod go.sum ./
+RUN go mod download
+COPY . .
+RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/controller .
+
+FROM gcr.io/distroless/static:nonroot
+COPY --from=build /out/controller /usr/local/bin/controller
+USER 65532:65532
+ENTRYPOINT ["/usr/local/bin/controller"]
@@ -70,8 +70,12 @@ type ConsumerRef struct {
 type RotationTarget struct {
 	// ZitadelProjectID is the project that owns the app or service user.
 	ZitadelProjectID string `json:"zitadelProjectID,omitempty"`
-	// ZitadelAppID is the OIDC app id, for the client-secret provider.
+	// ZitadelAppID is the app RESOURCE id, which the secret endpoint needs. It is not
+	// the client id.
 	ZitadelAppID string `json:"zitadelAppID,omitempty"`
+	// ZitadelClientID is the OIDC client id, which introspection needs. ZITADEL uses
+	// two different identifiers for the same app and neither substitutes for the other.
+	ZitadelClientID string `json:"zitadelClientID,omitempty"`
 	// MachineUserID is the service user id, for the machine-key provider.
 	MachineUserID string `json:"machineUserID,omitempty"`
 	// SecretName is the Secret the controller writes.
@@ -43,6 +43,7 @@ require (
 	github.com/prometheus/client_golang v1.24.0 // indirect
 	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.70.0 // indirect
+	github.com/prometheus/procfs v0.21.1 // indirect
 	github.com/spf13/pflag v1.0.10 // indirect
 	github.com/x448/float16 v0.8.4 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
@@ -1,3 +1,5 @@
+github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
+github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
@@ -8,6 +10,8 @@ github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1
 github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/emicklei/go-restful/v3 v3.13.0 h1:C4Bl2xDndpU6nJ4bc1jXd+uTmYPVUwkD6bFY/oTyCes=
 github.com/emicklei/go-restful/v3 v3.13.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
+github.com/evanphx/json-patch v0.5.2 h1:xVCHIVMUu1wtM/VkR9jVZ45N3FhZfYMMYGorLCR8P3k=
+github.com/evanphx/json-patch v0.5.2/go.mod h1:ZWS5hhDbVDyob71nXKNL0+PWn6ToqBHMikGIFbs31qQ=
 github.com/evanphx/json-patch/v5 v5.9.11 h1:/8HVnzMq13/3x9TPvjG08wUGqBTmZBsCWzjTM0wiaDU=
 github.com/evanphx/json-patch/v5 v5.9.11/go.mod h1:3j+LviiESTElxA4p3EMKAB9HXj3/XEtnUf6OZxqIQTM=
 github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
@@ -32,6 +36,8 @@ github.com/go-openapi/swag/fileutils v0.27.1 h1:QQqBSoi5mW4XpU85nS0mLcA+zAE6vLzr
 github.com/go-openapi/swag/fileutils v0.27.1/go.mod h1:VvJFZLTZS0AI854gEQz5tk7dBESdLjiNUMSZ/th2ry8=
 github.com/go-openapi/swag/jsonutils v0.27.1 h1:SVgK3i4USzCU5mibOOS/l4ea2h9UQXy7J7RNLTjuXjU=
 github.com/go-openapi/swag/jsonutils v0.27.1/go.mod h1:tdlEpZqdcQ17uj6J4YdK9vd8It5qWMwjWXOs0tjpRlk=
+github.com/go-openapi/swag/jsonutils/fixtures_test v0.27.1 h1:mJu3COL9WEaZVp/Kf2PRMi7tPszPEJfSr/OO75ynCs8=
+github.com/go-openapi/swag/jsonutils/fixtures_test v0.27.1/go.mod h1:mofwUWx70wvskwESqRJ//k/9kURmCgyJl5m5Ppoh5kY=
 github.com/go-openapi/swag/loading v0.27.1 h1:/DxUgDXKbBX4bcn7r9uEXfJyzN5XpiJmZplzQTjrRCY=
 github.com/go-openapi/swag/loading v0.27.1/go.mod h1:jvGh3iA2+zyUUycB5fgJWzeHnhrpvGnJJM0RVE9ZShE=
 github.com/go-openapi/swag/mangling v0.27.1 h1:yC9D0HyUE8gbP+BfmGx9+AA89ikwZTMjESK3OnnoaqA=
@@ -46,13 +52,33 @@ github.com/go-openapi/swag/typeutils v0.27.1 h1:KSTdFlfnse4r6dP9IrEnwMldjE+zs71U
 github.com/go-openapi/swag/typeutils v0.27.1/go.mod h1:Srm0xFNRZ1Y+vCxJclo5qzx8aj+1pAKda/YfFPrG0dQ=
 github.com/go-openapi/swag/yamlutils v0.27.1 h1:ftxv6xvXb1E3zohUc+okZ9nSqNb9StQX/FXnKZ98sQA=
 github.com/go-openapi/swag/yamlutils v0.27.1/go.mod h1:bnxFIB1qewGRiZHypXGZ3fNgf13/0HfRgnS/iZBDrOo=
+github.com/go-openapi/testify/enable/yaml/v2 v2.6.0 h1:gGHwAJ0R/5jU8BEGDbfRNR3hL68dAVi84WuOApp29B0=
+github.com/go-openapi/testify/enable/yaml/v2 v2.6.0/go.mod h1:tY+St1SGq4NFl0QIqdTY4aEdbChAHxhyB77XQi9iJCo=
+github.com/go-openapi/testify/v2 v2.6.0 h1:5PKH2HE7YJ/LuRPQGvSxBRlFXNQhSetBLlGAgUEu3ug=
+github.com/go-openapi/testify/v2 v2.6.0/go.mod h1:SgsVHtfooshd0tublTtJ50FPKhujf47YRqauXXOUxfw=
+github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
+github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
 github.com/google/gnostic-models v0.7.0 h1:qwTtogB15McXDaNqTZdzPJRHvaVJlAl+HVQnLmJEJxo=
 github.com/google/gnostic-models v0.7.0/go.mod h1:whL5G0m6dmc5cPxKc5bdKdEN3UjI7OUGxBlw57miDrQ=
+github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
+github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
+github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/pprof v0.0.0-20250403155104-27863c87afa6 h1:BHT72Gu3keYf3ZEu2J0b1vyeLSOYI8bm5wbJM/8yDe8=
+github.com/google/pprof v0.0.0-20250403155104-27863c87afa6/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
 github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
+github.com/klauspost/compress v1.19.0 h1:sXLILfc9jV2QYWkzFOPWStmcUVH2RHEB1JCdY2oVvCQ=
+github.com/klauspost/compress v1.19.0/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
+github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
+github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
+github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
+github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
+github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
+github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -61,6 +87,12 @@ github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee h1:W5t00kpgFd
 github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
+github.com/onsi/ginkgo/v2 v2.27.4 h1:fcEcQW/A++6aZAZQNUmNjvA9PSOzefMJBerHJ4t8v8Y=
+github.com/onsi/ginkgo/v2 v2.27.4/go.mod h1:ArE1D/XhNXBXCBkKOLkbsb2c81dQHCRcF5zwn/ykDRo=
+github.com/onsi/gomega v1.39.0 h1:y2ROC3hKFmQZJNFeGAMeHZKkjBL65mIZcvrLQBF9k6Q=
+github.com/onsi/gomega v1.39.0/go.mod h1:ZCU1pkQcXDO5Sl9/VVEGlDyp+zm0m1cmeG5TOzLgdh4=
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
+github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
@@ -70,12 +102,22 @@ github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNw
 github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
 github.com/prometheus/common v0.70.0 h1:bcpru3tWPVnxGnETLgOV5jbp/JRXgYEyv65CuBLAMMI=
 github.com/prometheus/common v0.70.0/go.mod h1:S/SFasQmgGiYH6C81LKCtYa8QACgthGg5zxL2udV7SY=
+github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI=
+github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
+github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
+github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
 github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
 github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.5.3 h1:jmXUvGomnU1o3W/V5h2VEradbpJDwGrzugQQvL0POH4=
+github.com/stretchr/objx v0.5.3/go.mod h1:rDQraq+vQZU7Fde9LOZLr8Tax6zZvy4kuNKF+QYS+U0=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
 github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
 github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
+go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
+go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
 go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
 go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
 go.uber.org/zap v1.27.1 h1:08RqriUEv8+ArZRYSTXy1LeBScaMpVSTBhCeaZYfMYc=
@@ -84,6 +126,8 @@ go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
 go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
 go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
 go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
+golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
+golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
 golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
 golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
 golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
@@ -98,15 +142,21 @@ golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
 golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
 golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
 golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
+golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
+golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
 gomodules.xyz/jsonpatch/v2 v2.4.0 h1:Ci3iUJyx9UeRx7CeFN8ARgGbkESwJK+KB9lLcWxY/Zw=
 gomodules.xyz/jsonpatch/v2 v2.4.0/go.mod h1:AH3dM2RI6uoBZxn3LVrfvJ3E0/9dG4cSrbuBJT4moAY=
 google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af h1:+5/Sw3GsDNlEmu7TfklWKPdQ0Ykja5VEmq2i817+jbI=
 google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
+gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
 gopkg.in/evanphx/json-patch.v4 v4.13.0 h1:czT3CmqEaQ1aanPc5SdlgQrrEIb8w/wwCvWWnfEbYzo=
 gopkg.in/evanphx/json-patch.v4 v4.13.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
 gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
 gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 k8s.io/api v0.37.1 h1:l6N77U7tjwB5L056bgrBTJIEdevac/naBZ3iSvDNfpM=
 k8s.io/api v0.37.1/go.mod h1:zSlbB1YpJ1YQlFVQy20UYll81UJSJJUMLhkhvg6Z78M=
 k8s.io/apiextensions-apiserver v0.37.0 h1:zRMQ3+/LIE5oZ0tVvXwYHC+dIkSP5cjNWju7AZU1LOI=
@@ -74,9 +74,26 @@ func (r *Reconciler) RequeueAfter(kr *iamv1.KeyRotation) time.Duration {
 	if interval <= 0 {
 		interval = 30 * 24 * time.Hour
 	}
+	// Return 0 to rotate NOW and a positive duration to wait. A failed attempt must
+	// rotate again immediately; only a rotation that reached Succeeded waits for the
+	// next interval. Returning a wait here was the defect: the object sat in Failed
+	// and reconciled every two minutes without ever trying again.
+	// A failed attempt waits a short backoff and then really retries. Returning 0
+	// unconditionally here spins: fail, rotate, fail, with no pause between.
+	if kr.Status.Phase == iamv1.PhaseFailed {
+		if kr.Status.LastRotation != nil {
+			if wait := kr.Status.LastRotation.Add(2 * time.Minute).Sub(r.now()); wait > 0 {
+				return wait
+			}
+		}
+		return 0
+	}
 	if kr.Status.LastRotation == nil || kr.Status.ActiveCredentialID == "" {
 		return 0 // rotate now
 	}
+	if kr.Status.Phase != iamv1.PhaseSucceeded {
+		return 0 // an interrupted attempt starts over
+	}
 	due := kr.Status.LastRotation.Add(interval)
 	wait := due.Sub(r.now())
 	if wait < 0 {
@@ -97,6 +114,9 @@ func (r *Reconciler) Reconcile(ctx context.Context, req types.NamespacedName) (t
 		return 0, err
 	}

+	logger.Info("reconcile", "phase", kr.Status.Phase, "generation", kr.Generation,
+		"observed", kr.Status.ObservedGeneration, "lastRotation", kr.Status.LastRotation)
+
 	if kr.Spec.Suspend {
 		return 0, r.setPhase(ctx, &kr, iamv1.PhaseSuspended, "")
 	}
@@ -126,6 +146,17 @@ func (r *Reconciler) Reconcile(ctx context.Context, req types.NamespacedName) (t
 		return r.fail(ctx, &kr, iamv1.PhasePreparing, err)
 	}

+	// Record the credential and the attempt time before anything else can fail.
+	// Without this a failure later in the cycle leaves LastRotation empty, the due
+	// check says "rotate now", and the controller mints a new secret on every
+	// reconcile: a restart loop the consumer never survives.
+	started := metav1.NewTime(r.now())
+	kr.Status.ActiveCredentialID = cred.ID
+	kr.Status.LastRotation = &started
+	if err := r.Status().Update(ctx, &kr); err != nil && !apierrors.IsConflict(err) {
+		return 0, err
+	}
+
 	// Publish. The previous value stays beside the new one until verification
 	// passes, so an atomic rotation can be undone.
 	if err := r.setPhase(ctx, &kr, iamv1.PhasePropagating, ""); err != nil {
@@ -260,7 +291,16 @@ func (r *Reconciler) writeSecret(ctx context.Context, kr *iamv1.KeyRotation, cre
 	if err != nil {
 		return err
 	}
-	secret.Data = data
+	// Merge, never replace. A consumer Secret often carries configuration beside the
+	// credential (paperless keeps SOCIALACCOUNT_PROVIDERS_JSON there), and replacing
+	// the whole map deletes it. The pod then fails with "couldn't find key ... in
+	// Secret" and the rotation looks successful while it broke the service.
+	if secret.Data == nil {
+		secret.Data = map[string][]byte{}
+	}
+	for k, v := range data {
+		secret.Data[k] = v
+	}
 	return r.Update(ctx, &secret)
 }

@@ -387,5 +427,7 @@ func (r *Reconciler) rolloutTimeout() time.Duration {
 	if r.RolloutTimeout > 0 {
 		return r.RolloutTimeout
 	}
-	return 3 * time.Minute
+	// Grafana needs about four minutes from container start to binding :3000, so a
+	// three-minute budget fails a rotation that is actually working.
+	return 8 * time.Minute
 }
@@ -19,10 +19,28 @@ func rotation(last *metav1.Time, activeID string, hours int32) *iamv1.KeyRotatio
 		Status: iamv1.KeyRotationStatus{
 			LastRotation:       last,
 			ActiveCredentialID: activeID,
+			Phase:              iamv1.PhaseSucceeded,
 		},
 	}
 }

+func TestRequeueAfterBacksOffAfterAFailure(t *testing.T) {
+	now := time.Unix(1_700_000_000, 0)
+	last := metav1.NewTime(now.Add(-1 * time.Minute))
+	r := &Reconciler{Now: func() time.Time { return now }}
+
+	kr := rotation(&last, "key-1", 24)
+	kr.Status.Phase = iamv1.PhaseFailed
+	if got := r.RequeueAfter(kr); got != time.Minute {
+		t.Fatalf("a failed attempt must wait a short backoff, got %s", got)
+	}
+	old := metav1.NewTime(now.Add(-3 * time.Minute))
+	kr.Status.LastRotation = &old
+	if got := r.RequeueAfter(kr); got != 0 {
+		t.Fatalf("after the backoff it must rotate again, got %s", got)
+	}
+}
+
 func TestRequeueAfterRotatesImmediatelyWhenNeverRotated(t *testing.T) {
 	r := &Reconciler{Now: func() time.Time { return time.Unix(1_700_000_000, 0) }}
 	if got := r.RequeueAfter(rotation(nil, "", 24)); got != 0 {
@@ -63,7 +63,9 @@ func (c *Client) AccessToken() (string, error) {
 	form := url.Values{}
 	form.Set("grant_type", "urn:ietf:params:oauth:grant-type:jwt-bearer")
 	form.Set("assertion", assertion)
-	form.Set("scope", "openid")
+	// The management API rejects a token without the ZITADEL project audience:
+	// the request returns 401 Token.Invalid even though the assertion was accepted.
+	form.Set("scope", "openid urn:zitadel:iam:org:project:id:zitadel:aud")

 	req, err := http.NewRequest(http.MethodPost, strings.TrimRight(c.cfg.Issuer, "/")+"/oauth/v2/token", strings.NewReader(form.Encode()))
 	if err != nil {
@@ -79,6 +79,9 @@ func main() {
 			},
 			iamv1.ProviderZitadelOIDCClient: &keyreconcile.OIDCClientProvider{
 				Client: zitadelClient,
+				ClientIDFunc: func(kr *iamv1.KeyRotation) string {
+					return kr.Spec.Target.ZitadelClientID
+				},
 			},
 		},
 	}
@@ -60,6 +60,7 @@ spec:
                  properties:
                    zitadelProjectID: {type: string}
                    zitadelAppID: {type: string}
+                    zitadelClientID: {type: string}
                    machineUserID: {type: string}
                    secretName: {type: string}
                    secretNamespace: {type: string}
@@ -29,7 +29,7 @@ spec:
      containers:
        - name: controller
          image: forgejo.riotpiao.com/riotpiao-poimen/key-rotation-controller:0.1.0
-          imagePullPolicy: IfNotPresent
+          imagePullPolicy: Always
          args:
            - --scan-interval=10m
          env:
@@ -17,14 +17,21 @@ rules:
  - apiGroups: ["iam.homelab.riotpiao.com"]
    resources: ["keyrotations/status"]
    verbs: ["get", "update", "patch"]
-  # The controller writes the credential it just created.
+  # The controller writes the credential it just created. list and watch are needed
+  # because the manager caches Secrets; without them the cache cannot start and
+  # every reconcile fails.
  - apiGroups: [""]
    resources: ["secrets"]
-    verbs: ["get", "create", "update", "patch"]
+    verbs: ["get", "list", "watch", "create", "update", "patch"]
  # Rolling a consumer that cannot hot reload.
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets"]
    verbs: ["get", "list", "watch", "patch"]
+  # Leader election. Without this the controller cannot acquire its Lease and never
+  # starts reconciling.
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"]
@@ -21,10 +21,11 @@ spec:
  provider: zitadel-oidc-client
  strategy: atomic
  intervalHours: 2160
-  suspend: true
+  suspend: false
  target:
    zitadelProjectID: "396757796898471937"
-    zitadelAppID: "396757801965191169"
+    zitadelAppID: "396757801948413953"  # app resource id, not the client id 396757801965191169
+    zitadelClientID: "396757801965191169"
    secretName: grafana-oidc
    secretNamespace: logging
    secretKeys:
@@ -49,7 +50,8 @@ spec:
  suspend: true
  target:
    zitadelProjectID: "396757796898471937"
-    zitadelAppID: "396757801210282367"
+    zitadelAppID: "396757801210216831"  # app resource id, not the client id 396757801210282367
+    zitadelClientID: "396757801210282367"
    secretName: argocd-oidc
    secretNamespace: argocd
    secretKeys:
@@ -74,7 +76,8 @@ spec:
  suspend: true
  target:
    zitadelProjectID: "396757796898471937"
-    zitadelAppID: "397015744010781070"
+    zitadelAppID: "397015743994003854"  # app resource id, not the client id 397015744010781070
+    zitadelClientID: "397015744010781070"
    secretName: comfyui-oauth2-proxy
    secretNamespace: comfyui
    secretKeys:
@@ -99,14 +102,15 @@ spec:
  suspend: true
  target:
    zitadelProjectID: "396757796898471937"
-    zitadelAppID: "396757803257036801"
+    zitadelAppID: "396757803240259585"  # app resource id, not the client id 396757803257036801
+    zitadelClientID: "396757803257036801"
    secretName: minio-oidc
-    secretNamespace: minio
+    secretNamespace: storage
    secretKeys:
      secret: client-secret
    reload: hot
    consumers:
-      - {kind: Deployment, name: minio, namespace: minio}
+      - {kind: StatefulSet, name: minio-cluster-az-a, namespace: storage}
  verify:
    introspect: true
 ---
@@ -124,7 +128,8 @@ spec:
  suspend: true
  target:
    zitadelProjectID: "396757796898471937"
-    zitadelAppID: "396757803559027071"
+    zitadelAppID: "396757803542249855"  # app resource id, not the client id 396757803559027071
+    zitadelClientID: "396757803559027071"
    secretName: paperless-oidc
    secretNamespace: paperless
    secretKeys:
@@ -149,7 +154,8 @@ spec:
  suspend: true
  target:
    zitadelProjectID: "396757796898471937"
-    zitadelAppID: "396757802602791295"
+    zitadelAppID: "396757802602725759"  # app resource id, not the client id 396757802602791295
+    zitadelClientID: "396757802602791295"
    secretName: immich-oidc
    secretNamespace: immich
    secretKeys:
@@ -174,14 +180,15 @@ spec:
  suspend: true
  target:
    zitadelProjectID: "396757796898471937"
-    zitadelAppID: "396757801579315583"
+    zitadelAppID: "396757801562538367"  # app resource id, not the client id 396757801579315583
+    zitadelClientID: "396757801579315583"
    secretName: forgejo-oidc
-    secretNamespace: forgejo
+    secretNamespace: cicd
    secretKeys:
-      secret: key
+      secret: CLIENT_SECRET
    reload: restart
    consumers:
-      - {kind: StatefulSet, name: forgejo, namespace: forgejo}
+      - {kind: Deployment, name: forgejo-gitea, namespace: cicd}
  verify:
    introspect: true
 ---
@@ -199,7 +206,8 @@ spec:
  suspend: true
  target:
    zitadelProjectID: "396757796898471937"
-    zitadelAppID: "396757800874672511"
+    zitadelAppID: "396757800857895295"  # app resource id, not the client id 396757800874672511
+    zitadelClientID: "396757800874672511"
    secretName: api-gw-oidc
    secretNamespace: api-gw
    secretKeys:
@@ -0,0 +1,125 @@
+# clef-flash: a decision model, not a chat model.
+#
+# It reads a `state` plus a schema of typed `questions` and returns one probability
+# per allowed option in a single prefill pass. There is no decoding, no sampling and
+# no free-form text. The public route is /v1/systemone, and its request and response
+# shapes are Clef's own: no `messages`, no `choices`.
+#
+# Why not SGLang: Clef support landed in SGLang v0.5.22, which upstream publishes only
+# on CUDA 13. CUDA 13 dropped Volta kernels and every card here is a Tesla V100
+# (compute capability 7.0), so SGLang aborts with "no kernel image is available for
+# execution on the device". SGLang also states sm75 as its floor regardless.
+#
+# So this runs the implementation the model ships: joint_schema_model.py, which has
+# load_release_model, encode_record, collate_records and systemone. The weights are
+# 17.75 GiB of BF16, and fp16 costs the same bytes per parameter. On one 32 GiB card
+# that leaves room for a 32k state, which is why this is one GPU and not two: with
+# device_map over two cards the embedding table and the tensors indexing into it land
+# on different devices, and the forward pass dies with
+# "indices should be either on cpu or on the same device as the indexed tensor".
+#
+# Known gap: the image cannot be pushed to the registry. Cloudflare fronts
+# forgejo.riotpiao.com and answers 413 Payload Too Large for any layer above 100 MB,
+# and the torch base alone is 7.58 GB. So this mounts app.py from a ConfigMap and pip
+# installs on every start. A cold start therefore needs PyPI reachable, and takes
+# about four minutes. Build the image in-cluster, or unproxy the registry host, before
+# treating this as production.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: clef-serve
+  namespace: llm-serving
+  labels:
+    app: clef-serve
+    app.kubernetes.io/part-of: llm-serving
+spec:
+  replicas: 1
+  # Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
+  # the new pod Pending forever waiting for a card the old pod still holds.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: clef-serve
+  template:
+    metadata:
+      labels:
+        app: clef-serve
+        app.kubernetes.io/part-of: llm-serving
+    spec:
+      runtimeClassName: nvidia
+      nodeSelector:
+        kubernetes.io/hostname: worker-1
+      containers:
+      - name: clef
+        image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime
+        command: ["/bin/bash","-lc"]
+        args:
+        - |
+          set -e
+          pip install --no-cache-dir transformers==5.10.2 accelerate fastapi 'uvicorn[standard]' pillow safetensors sentencepiece protobuf
+          exec uvicorn app:create_app --factory --host 0.0.0.0 --port 8080
+        workingDir: /app
+        env:
+        - name: PYTHONUNBUFFERED
+          value: "1"
+        - name: MODEL_PATH
+          value: /mnt/models/clef-flash
+        # V100 has no bf16, and the release weights are bf16. fp16 is the same width.
+        - name: DTYPE
+          value: float16
+        # The state is truncated at this length without warning, so raise it only with
+        # spare VRAM: 17.75 GiB of weights plus 128 KiB of KV per token.
+        - name: MAX_LEN
+          value: "32768"
+        ports:
+        - containerPort: 8080
+        # A cold start pip-installs and loads 17.75 GiB, measured at about four minutes.
+        startupProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          failureThreshold: 120
+          periodSeconds: 10
+        readinessProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          periodSeconds: 10
+        resources:
+          requests:
+            cpu: "4"
+            memory: 12Gi
+            nvidia.com/gpu: "1"
+          limits:
+            cpu: "8"
+            memory: 40Gi
+            nvidia.com/gpu: "1"
+        volumeMounts:
+        - name: app
+          mountPath: /app
+        - name: models
+          mountPath: /mnt/models
+          readOnly: true
+      volumes:
+      - name: app
+        configMap:
+          name: clef-serve-app
+      - name: models
+        persistentVolumeClaim:
+          claimName: llm-models
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: clef-serve
+  namespace: llm-serving
+  labels:
+    app.kubernetes.io/part-of: llm-serving
+spec:
+  selector:
+    app: clef-serve
+  ports:
+  - name: http
+    port: 80
+    targetPort: 8080
@@ -0,0 +1,22 @@
+# Base image: torch 2.8.0+cu128 still lists sm_70 in torch.cuda.get_arch_list().
+# The CUDA 13 builds of torch drop sm_70. worker-1 holds Tesla V100 cards with
+# compute capability 7.0, so a newer base kills the kernel launch at load time.
+# Keep this pin until the model moves to newer hardware.
+FROM pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime
+
+# Unbuffered output lets the pod log show the startup lines in order.
+ENV PYTHONUNBUFFERED=1 \
+    PIP_NO_CACHE_DIR=1
+
+WORKDIR /app
+
+# torch comes from the base image. Never reinstall it here, or pip pulls a
+# wheel built for another CUDA version.
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY app.py .
+
+EXPOSE 8080
+
+CMD ["uvicorn", "app:create_app", "--factory", "--host", "0.0.0.0", "--port", "8080"]
@@ -0,0 +1,201 @@
+"""HTTP server for the Clef-flash decision model.
+
+Clef is a decision model and not a chat model. It reads one state plus a schema
+of typed questions and returns one probability per allowed option. Consumers
+call POST /v1/systemone and never /v1/chat/completions.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+import os
+import sys
+import traceback
+from contextlib import asynccontextmanager
+from pathlib import Path
+from typing import Any, AsyncIterator
+
+import torch
+from fastapi import FastAPI, Request
+from fastapi.responses import JSONResponse
+from safetensors.torch import load_file
+from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
+
+# uvicorn configures its own loggers only. Without this call the root logger
+# stays at WARNING and the startup lines below never reach the pod log.
+logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
+LOG = logging.getLogger("clef-serve")
+
+MODEL_PATH = os.environ.get("MODEL_PATH", "/mnt/models/clef-flash")
+DTYPE = os.environ.get("DTYPE", "float16")
+MAX_LEN = int(os.environ.get("MAX_LEN", "65536"))
+PORT = int(os.environ.get("PORT", "8080"))
+
+# HF_HOME stays untouched. The k8s manifest sets it, and the loader must reuse
+# that cache path instead of downloading a second copy of an 18.8 GB checkpoint.
+
+_TORCH_DTYPES = {
+    "float16": torch.float16,
+    "bfloat16": torch.bfloat16,
+    "float32": torch.float32,
+}
+
+
+def _resolve_dtype() -> torch.dtype:
+    """Map the DTYPE env string to a torch dtype. Fail fast on a typo."""
+    try:
+        return _TORCH_DTYPES[DTYPE]
+    except KeyError:
+        raise ValueError(
+            f"The server does not support DTYPE={DTYPE!r}. Use one of {sorted(_TORCH_DTYPES)}."
+        ) from None
+
+
+def _load_runtime() -> tuple[Any, Any, list[str], Any]:
+    """Load backbone, joint head and processor. Runs in a worker thread."""
+    # The released code imports its sibling modules by bare name. The model
+    # directory is therefore the only place on sys.path that resolves them.
+    sys.path.insert(0, MODEL_PATH)
+
+    # This import must follow the sys.path.insert above. joint_schema_model.py
+    # ships inside MODEL_PATH and not in site-packages. An import before that
+    # insert raises ModuleNotFoundError. A local import binds the name in this
+    # function scope only, so the caller must receive the callable as a value.
+    from joint_schema_model import (
+        ClefModel,
+        JointSchemaHead,
+        collate_records,
+        encode_record,
+        systemone,
+    )
+
+    dtype = _resolve_dtype()
+    LOG.info("Loading backbone. path=%s dtype=%s", MODEL_PATH, DTYPE)
+
+    # load_release_model passes device_map={"": str(device)} and so pins the
+    # whole backbone on one card. This loader asks accelerate to shard instead.
+    backbone = Qwen3_5ForConditionalGeneration.from_pretrained(
+        MODEL_PATH,
+        dtype=dtype,
+        device_map="auto",
+        low_cpu_mem_usage=True,
+    )
+
+    # The head must sit on the device that produces the final hidden state,
+    # because ClefModel applies it to the output of the last layer.
+    head_cfg = json.loads((Path(MODEL_PATH) / "joint_head_config.json").read_text())
+    head = JointSchemaHead(**head_cfg)
+    head.load_state_dict(load_file(str(Path(MODEL_PATH) / "joint_head.safetensors")))
+    # The output embeddings carry the last layer's device. Without them, take
+    # the last parameter, which lives there too.
+    output_embeddings = backbone.get_output_embeddings()
+    if output_embeddings is not None:
+        head_device = output_embeddings.weight.device
+    else:
+        head_device = next(reversed(list(backbone.parameters()))).device
+    head.to(device=head_device, dtype=dtype)
+
+    model = ClefModel(backbone, head).eval()
+    processor = AutoProcessor.from_pretrained(MODEL_PATH)
+
+    # hf_device_map maps layer names to ints for CUDA shards. Report it so a
+    # human can see whether one card or two carried the weights.
+    raw_map = getattr(backbone, "hf_device_map", None) or {}
+    if not raw_map:
+        raw_map = {"": str(next(backbone.parameters()).device)}
+    devices = list(
+        dict.fromkeys(
+            f"cuda:{name}" if isinstance(name, int) else str(name)
+            for name in raw_map.values()
+        )
+    )
+    LOG.info(
+        "Model ready. dtype=%s devices=%s device_map=%s head_device=%s",
+        DTYPE,
+        devices,
+        raw_map,
+        head_device,
+    )
+    return model, processor, devices, systemone
+
+
+@asynccontextmanager
+async def _lifespan(app: FastAPI) -> AsyncIterator[None]:
+    """Load the model on startup without blocking the event loop."""
+    # A 9.4B load takes minutes. asyncio.to_thread keeps /health answerable
+    # during that window, so the startup probe sees 503 and not a dead socket.
+    model, processor, devices, systemone_fn = await asyncio.to_thread(_load_runtime)
+    app.state.model = model
+    app.state.processor = processor
+    app.state.devices = devices
+    # The route handler runs at module scope and cannot see the local import
+    # inside _load_runtime. Publish the callable here, or the handler raises
+    # NameError on the first request.
+    app.state.systemone = systemone_fn
+    app.state.lock = asyncio.Lock()
+    LOG.info("Runtime published. systemone_callable=%s", systemone_fn is not None)
+    yield
+    app.state.model = None
+    app.state.processor = None
+    app.state.systemone = None
+
+
+def create_app() -> FastAPI:
+    app = FastAPI(title="clef-flash", lifespan=_lifespan)
+
+    @app.get("/health")
+    async def health() -> JSONResponse:
+        # A ready flag alone is not enough. The pod must not take traffic until
+        # the weights and the joint head are both resident.
+        if getattr(app.state, "model", None) is None:
+            return JSONResponse({"status": "loading"}, status_code=503)
+        return JSONResponse(
+            {
+                "status": "ok",
+                "dtype": DTYPE,
+                "devices": app.state.devices,
+            },
+            status_code=200,
+        )
+
+    @app.post("/v1/systemone")
+    async def systemone_route(request: Request) -> Any:
+        model = getattr(app.state, "model", None)
+        if model is None:
+            return JSONResponse({"error": "model is not loaded"}, status_code=503)
+
+        body = await request.json()
+        processor = app.state.processor
+
+        def _forward() -> dict:
+            # inference_mode is thread-local. Enter it inside this call, or the
+            # worker thread records no autograd state saving and still builds
+            # the graph.
+            with torch.inference_mode():
+                return app.state.systemone(model, processor, body, max_length=MAX_LEN)
+
+        # Clef runs one prefill with shared buffers and a shared joint head.
+        # Two calls at once corrupt each other, so serialize them here.
+        async with app.state.lock:
+            try:
+                result = await asyncio.to_thread(_forward)
+            except ValueError as exc:
+                # A malformed request body is the caller's fault.
+                return JSONResponse({"error": str(exc)}, status_code=400)
+            except Exception as exc:
+                LOG.error("systemone failed\n%s", traceback.format_exc())
+                return JSONResponse(
+                    {"error": type(exc).__name__, "detail": str(exc)},
+                    status_code=500,
+                )
+        return result
+
+    return app
+
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run("app:create_app", factory=True, host="0.0.0.0", port=PORT)
@@ -0,0 +1,12 @@
+# Runtime dependencies for the clef-flash server.
+# torch ships with the base image and stays out of this list.
+transformers==5.10.2
+# device_map="auto" sharding is an accelerate feature, not a transformers one
+accelerate>=1.1.0
+fastapi
+uvicorn[standard]
+pillow
+safetensors
+huggingface_hub
+sentencepiece
+protobuf
@@ -45,14 +45,13 @@ spec:
      volumeMounts:
      - mountPath: /mnt/models
        name: models
-    podMetadata:
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "8080"
-        prometheus.io/path: "/metrics"
-      labels:
-        app.kubernetes.io/name: llm-embeddings
-        app.kubernetes.io/part-of: llm-serving
+    annotations:
+      prometheus.io/scrape: "true"
+      prometheus.io/port: "8080"
+      prometheus.io/path: "/metrics"
+    labels:
+      app.kubernetes.io/name: llm-embeddings
+      app.kubernetes.io/part-of: llm-serving
    maxReplicas: 1
    minReplicas: 1
    nodeSelector:
@@ -10,11 +10,27 @@ kind: Kustomization
 # way after any edit here: a GPU predictor restart is a weights reload measured
 # in tens of seconds, not a rolling update.
 resources:
-  - embeddings.yaml
-  - ornith.yaml
+  # clef-flash runs through clef-serve.yaml, not through an SGLang runtime. SGLang needs
+  # v0.5.22+, published only on CUDA 13, and CUDA 13 dropped Volta kernels. The model
+  # instead runs the implementation it ships (joint_schema_model.py) in fp16 on one V100.
+  - clef-serve.yaml
  - reasoning.yaml
+  # ornith is retired: its InferenceService still reports Ready=True while running zero
+  # replicas, because desired equals available at 0. Its manifest was never committed,
+  # so it is left unmanaged and is not listed here. Delete the CR when convenient.
+  - embeddings.yaml
  - reranker.yaml
  - qwen-cpu.yaml
  - networkpolicy.yaml
 # No namespace transformer: every file sets its own, and the transformer would
 # rewrite metadata.namespace on anything cross-namespace added later.
+
+# app.py lives in git next to this file. The generator hashes the ConfigMap name, so an
+# edit to app.py rolls the pod instead of leaving a stale copy mounted.
+configMapGenerator:
+  - name: clef-serve-app
+    namespace: llm-serving
+    files:
+      - clef-serve/app.py
+    options:
+      disableNameSuffixHash: false
@@ -0,0 +1,151 @@
+apiVersion: serving.kserve.io/v1beta1
+kind: InferenceService
+metadata:
+  annotations:
+    serving.kserve.io/deploymentMode: RawDeployment
+    # KServe injects the prometheus.io/{scrape,port,path} annotations into the
+    # generated pod only when this flag is on. Without it the predictor carries
+    # no scrape annotations at all, which is why no vLLM/SGLang metrics have
+    # ever reached Prometheus from this namespace.
+    serving.kserve.io/enable-prometheus-scraping: "true"
+  labels:
+    app.kubernetes.io/name: llm-reasoning
+    app.kubernetes.io/part-of: llm-serving
+  name: reasoning
+  namespace: llm-serving
+spec:
+  predictor:
+    containers:
+    - args:
+      # Replaces the dense Qwen3-32B. Same serving name (`reasoning`), so every
+      # consumer keeps its model string and no client changes.
+      #
+      # Why a MoE: Qwen3-30B-A3B has 30.5B total but only 3.3B active per token.
+      # On sm70 the decode path is bandwidth- and kernel-launch-bound, not
+      # FLOP-bound, so activating 3.3B instead of 32B is the single biggest
+      # throughput win available on this hardware without changing GPUs.
+      #
+      # Quantization: GPTQ-Int4, pinned with --quantization=gptq. This repeats
+      # the finding recorded on the previous checkpoint: bnb-4bit works poorly
+      # on sm70 because there are no int4 tensor cores, and vLLM auto-upgrades
+      # AWQ/GPTQ to the Marlin kernel which requires sm80+. Pinning `gptq`
+      # forces the plain kernel, which is verified Volta-compatible here.
+      # Do not remove this flag.
+      #
+      # Context: 131072 via YaRN. The base model card documents 32,768 native
+      # and 131,072 with YaRN (factor 4.0, original_max_position_embeddings
+      # 32768). That is Qwen's own validated ceiling, so unlike a stretched
+      # window this is a documented capability.
+      #
+      # KV cache math, from config.json: 48 layers, 4 KV heads, head_dim 128,
+      # fp16 -> 2 * 4 * 128 * 2 = 2048 B/token/layer -> 96 KB/token.
+      # At 131072 tokens that is 12 GiB total. With TP=2 the KV heads split
+      # 2 per GPU, so each GPU carries ~6 GiB of KV. Weights are ~17 GB total,
+      # so ~8.5 GB per GPU. At gpu-memory-utilization=0.95 (about 30.4 GB
+      # usable per V100-32GB) that leaves roughly 21 GB per GPU for KV plus
+      # activations: comfortable for one full 128K request and a few shorter
+      # ones. Re-read vllm:cache_config_info after rollout for the real
+      # num_gpu_blocks value rather than trusting this projection.
+      #
+      # TP=2, not PP=2: the model has 4 KV heads, so TP must divide 4. TP=2
+      # splits cleanly; TP=3 would not. TP also shortens prefill at long
+      # context, which is the actual bottleneck at 128K on V100.
+      - --model=Qwen/Qwen3-30B-A3B-GPTQ-Int4
+      - --served-model-name=reasoning
+      - --quantization=gptq
+      - --dtype=float16
+      - --kv-cache-dtype=auto
+      - --rope-scaling={"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}
+      - --tensor-parallel-size=2
+      - --max-model-len=131072
+      - --gpu-memory-utilization=0.95
+      - --max-num-seqs=4
+      # Prefill scheduling. The same reasoning as the previous checkpoint
+      # applies: vLLM V1 has no separate prefill phase, so a single long
+      # prompt can consume the whole step budget and starve every other
+      # request. The threshold must be strictly below the budget or it never
+      # binds, hence 8192 budget with a 2048 threshold.
+      - --max-num-batched-tokens=8192
+      - --long-prefill-token-threshold=2048
+      - --enable-chunked-prefill
+      - --enable-prefix-caching
+      # qwen3 is vLLM's reasoning parser for this family's <think> blocks.
+      - --reasoning-parser=qwen3
+      # hermes is the documented tool-call parser for general Qwen3 models.
+      - --enable-auto-tool-choice
+      - --tool-call-parser=hermes
+      - --host=0.0.0.0
+      - --port=8080
+      env:
+      # FlashInfer is sm80+. TRITON_ATTN is the Volta-compatible backend and is
+      # already the proven setting on this node. Clef-flash runs on SGLang
+      # instead, but Qwen stays on vLLM here because this exact combination is
+      # tested on this hardware; moving it to SGLang would add risk for no
+      # measured gain.
+      - name: VLLM_USE_FLASHINFER_SAMPLER
+        value: '0'
+      - name: VLLM_ATTENTION_BACKEND
+        value: TRITON_ATTN
+      - name: HF_HOME
+        value: /mnt/models
+      image: vllm/vllm-openai:v0.11.0@sha256:014a95f21c9edf6abe0aea6b07353f96baa4ec291c427bb1176dc7c93a85845c
+      name: kserve-container
+      ports:
+      - containerPort: 8080
+        protocol: TCP
+      readinessProbe:
+        httpGet:
+          path: /health
+          port: 8080
+        periodSeconds: 10
+      resources:
+        limits:
+          cpu: '16'
+          memory: 36Gi
+          nvidia.com/gpu: '2'
+        requests:
+          cpu: '8'
+          memory: 12Gi
+          nvidia.com/gpu: '2'
+      startupProbe:
+        # 30.5B of int4 weights plus a 128K rope config: allow a generous
+        # first-boot window. failureThreshold 120 * 15s = 30 min ceiling.
+        failureThreshold: 120
+        httpGet:
+          path: /health
+          port: 8080
+        periodSeconds: 15
+      volumeMounts:
+      - mountPath: /mnt/models
+        name: models
+      - mountPath: /dev/shm
+        name: shm
+    # podMetadata is NOT a valid field on this KServe version. The CRD exposes
+    # annotations and labels directly under predictor. The older manifests in
+    # this directory still carry podMetadata and ArgoCD silently drops it on
+    # server-side apply; do not copy that pattern.
+    annotations:
+      prometheus.io/scrape: "true"
+      prometheus.io/port: "8080"
+      prometheus.io/path: "/metrics"
+    labels:
+      app.kubernetes.io/name: llm-reasoning
+      app.kubernetes.io/part-of: llm-serving
+    deploymentStrategy:
+      # Recreate, not RollingUpdate: GPU memory is a hard resource. A second
+      # pod cannot schedule alongside the first on a full 4-GPU node, so a
+      # rolling update would wedge with the new pod Pending forever.
+      type: Recreate
+    maxReplicas: 1
+    minReplicas: 1
+    nodeSelector:
+      kubernetes.io/hostname: worker-1
+    runtimeClassName: nvidia
+    volumes:
+    - name: models
+      persistentVolumeClaim:
+        claimName: llm-models
+    - emptyDir:
+        medium: Memory
+        sizeLimit: 2Gi
+      name: shm
@@ -0,0 +1,102 @@
+apiVersion: serving.kserve.io/v1beta1
+kind: InferenceService
+metadata:
+  name: reasoning
+  namespace: llm-serving
+  labels:
+    app.kubernetes.io/name: llm-reasoning
+    app.kubernetes.io/part-of: llm-serving
+  annotations:
+    serving.kserve.io/deploymentMode: RawDeployment
+spec:
+  predictor:
+    minReplicas: 1
+    maxReplicas: 1
+    nodeSelector:
+      kubernetes.io/hostname: worker-1
+    # Every GPU card is allocated, so a rolling update deadlocks: the new pod needs a
+    # card the old pod still holds and sits Pending forever. Recreate frees the card
+    # first. The cost is a short gap in service for this model only.
+    deploymentStrategy:
+      type: Recreate
+    containers:
+    - name: kserve-container
+      image: vllm/vllm-openai:v0.11.0@sha256:014a95f21c9edf6abe0aea6b07353f96baa4ec291c427bb1176dc7c93a85845c
+      args:
+      # Qwen3-30B-A3B is a 30.5B MoE with 3.3B active per token. On a V100 that matters:
+      # the 32B dense model it replaces needed two cards for the same context, this needs
+      # one, because its KV cache is 96 KiB/token against the dense model's 256 KiB.
+      #
+      # Measured on this cluster, one V100-32GB, 64k context: 15.55 GiB of weights,
+      # 137k tokens of KV cache, 2.09x concurrency at 65536 tokens per request.
+      - --model=Qwen/Qwen3-30B-A3B-GPTQ-Int4
+      - --served-model-name=reasoning
+      - --quantization=gptq
+      - --dtype=float16
+      - --kv-cache-dtype=auto
+      # Native window is 40960. YaRN factor 2.0 reaches 81920, so 64k is inside it.
+      - --rope-scaling={"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":40960}
+      - --tensor-parallel-size=1
+      - --pipeline-parallel-size=1
+      - --max-model-len=65536
+      - --gpu-memory-utilization=0.90
+      - --max-num-seqs=8
+      - --max-num-batched-tokens=8192
+      - --long-prefill-token-threshold=2048
+      - --enable-chunked-prefill
+      - --enable-prefix-caching
+      - --reasoning-parser=qwen3
+      - --enable-auto-tool-choice
+      - --tool-call-parser=hermes
+      - --host=0.0.0.0
+      - --port=8080
+      env:
+      # Both are required on this hardware, not tuning:
+      #  - flashinfer's check_cuda_arch raises AttributeError on compute capability 7.0
+      #    (flashinfer/jit/core.py:55, minor.isdigit() on an int), so the sampler must stay off
+      #  - FlashInfer and FlashAttention-2 both need sm75+, so the Triton backend serves attention
+      - name: VLLM_USE_FLASHINFER_SAMPLER
+        value: "0"
+      - name: VLLM_ATTENTION_BACKEND
+        value: TRITON_ATTN
+      - name: HF_HOME
+        value: /mnt/models
+      ports:
+      - containerPort: 8080
+        protocol: TCP
+      readinessProbe:
+        httpGet:
+          path: /health
+          port: 8080
+        periodSeconds: 10
+      startupProbe:
+        httpGet:
+          path: /health
+          port: 8080
+        # Weight load took 18s from cache and 148s on a cold download, and the first
+        # torch.compile adds about 70s. 80 x 15s leaves ample room for a cold start.
+        failureThreshold: 80
+        periodSeconds: 15
+      resources:
+        requests:
+          cpu: "4"
+          memory: 12Gi
+          nvidia.com/gpu: "1"
+        limits:
+          cpu: "8"
+          memory: 36Gi
+          nvidia.com/gpu: "1"
+      volumeMounts:
+      - mountPath: /mnt/models
+        name: models
+      - mountPath: /dev/shm
+        name: shm
+    runtimeClassName: nvidia
+    volumes:
+    - name: models
+      persistentVolumeClaim:
+        claimName: llm-models
+    - name: shm
+      emptyDir:
+        medium: Memory
+        sizeLimit: 2Gi
@@ -45,14 +45,13 @@ spec:
      volumeMounts:
      - mountPath: /mnt/models
        name: models
-    podMetadata:
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "8080"
-        prometheus.io/path: "/metrics"
-      labels:
-        app.kubernetes.io/name: llm-reranker
-        app.kubernetes.io/part-of: llm-serving
+    annotations:
+      prometheus.io/scrape: "true"
+      prometheus.io/port: "8080"
+      prometheus.io/path: "/metrics"
+    labels:
+      app.kubernetes.io/name: llm-reranker
+      app.kubernetes.io/part-of: llm-serving
    maxReplicas: 1
    minReplicas: 1
    nodeSelector:
@@ -0,0 +1,10 @@
+FROM golang:1.26 AS build
+WORKDIR /src
+COPY go.mod ./
+COPY *.go ./
+RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/vendor .
+
+FROM gcr.io/distroless/static:nonroot
+COPY --from=build /out/vendor /usr/local/bin/vendor
+USER 65532:65532
+ENTRYPOINT ["/usr/local/bin/vendor"]
@@ -0,0 +1,3 @@
+module github.com/riotpiao/llm-token-vendor
+
+go 1.26.0
@@ -0,0 +1,197 @@
+// llm-token-vendor mints ZITADEL access tokens for a caller that cannot sign.
+//
+// Some callers are vendor images: paperless-ai can only read a bearer token from a
+// file. It cannot sign an assertion, so it cannot use a machine key, and a client
+// secret would rotate with a window.
+//
+// This sidecar closes the gap. It signs with a machine key, exchanges the assertion
+// for an access token, and writes the token to the file the vendor image already
+// reads. The file is replaced atomically and refreshed at 80% of the token lifetime,
+// so the vendor never reads a partial or expired value.
+//
+// The key is re-read on every mint. A rotation therefore lands without a restart:
+// publish the new key, the next mint uses it, retire the old one.
+package main
+
+import (
+	"crypto"
+	"crypto/rand"
+	"crypto/rsa"
+	"crypto/sha256"
+	"crypto/x509"
+	"encoding/base64"
+	"encoding/json"
+	"encoding/pem"
+	"flag"
+	"fmt"
+	"io"
+	"log"
+	"net/http"
+	"net/url"
+	"os"
+	"path/filepath"
+	"time"
+)
+
+type config struct {
+	tokenURL string
+	issuer   string
+	userID   string
+	keyID    string
+	keyPath  string
+	scope    string
+	outPath  string
+}
+
+func main() {
+	var c config
+	flag.StringVar(&c.tokenURL, "token-url", env("TOKEN_URL", ""), "ZITADEL token endpoint")
+	flag.StringVar(&c.issuer, "issuer", env("ISSUER", ""), "ZITADEL issuer, the assertion audience")
+	flag.StringVar(&c.userID, "user-id", env("USER_ID", ""), "machine user id")
+	flag.StringVar(&c.keyID, "key-id", env("KEY_ID", ""), "machine key id")
+	flag.StringVar(&c.keyPath, "key-file", env("KEY_FILE", ""), "PEM private key, re-read on every mint")
+	flag.StringVar(&c.scope, "scope", env("SCOPE", "openid"), "requested scopes")
+	flag.StringVar(&c.outPath, "out", env("OUT", ""), "file the vendor image reads")
+	flag.Parse()
+
+	if c.tokenURL == "" || c.issuer == "" || c.userID == "" || c.keyPath == "" || c.outPath == "" {
+		log.Fatal("token-url, issuer, user-id, key-file and out are required")
+	}
+
+	for {
+		lifetime, err := mintAndWrite(c)
+		if err != nil {
+			// Exit rather than spin: a pod that cannot mint must be visible, and the
+			// vendor image must not keep serving the caller a stale token.
+			log.Fatalf("cannot mint a token: %v", err)
+		}
+		wait := time.Duration(float64(lifetime) * 0.8)
+		log.Printf("token written to %s, next mint in %s", c.outPath, wait.Round(time.Second))
+		time.Sleep(wait)
+	}
+}
+
+func mintAndWrite(c config) (time.Duration, error) {
+	pemBytes, err := os.ReadFile(c.keyPath)
+	if err != nil {
+		return 0, fmt.Errorf("cannot read the key %s: %w", c.keyPath, err)
+	}
+	assertion, err := assertion(c, time.Now(), pemBytes)
+	if err != nil {
+		return 0, err
+	}
+
+	form := url.Values{}
+	form.Set("grant_type", "urn:ietf:params:oauth:grant-type:jwt-bearer")
+	form.Set("assertion", assertion)
+	form.Set("scope", c.scope)
+
+	resp, err := http.PostForm(c.tokenURL, form)
+	if err != nil {
+		return 0, fmt.Errorf("token request failed: %w", err)
+	}
+	defer resp.Body.Close()
+	body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
+	if resp.StatusCode != http.StatusOK {
+		return 0, fmt.Errorf("token endpoint returned %d: %s", resp.StatusCode, string(body))
+	}
+	var out struct {
+		AccessToken string `json:"access_token"`
+		ExpiresIn   int    `json:"expires_in"`
+	}
+	if err := json.Unmarshal(body, &out); err != nil {
+		return 0, fmt.Errorf("cannot decode the token response: %w", err)
+	}
+	if out.AccessToken == "" {
+		return 0, fmt.Errorf("the token response holds no access_token")
+	}
+
+	// Write beside the target, then rename. A rename is atomic, so the vendor either
+	// reads the previous token or the new one, never a half-written file.
+	tmp, err := os.CreateTemp(filepath.Dir(c.outPath), ".token-*")
+	if err != nil {
+		return 0, fmt.Errorf("cannot create a temporary file: %w", err)
+	}
+	defer os.Remove(tmp.Name())
+	if _, err := tmp.WriteString(out.AccessToken); err != nil {
+		tmp.Close()
+		return 0, fmt.Errorf("cannot write the token: %w", err)
+	}
+	tmp.Close()
+	if err := os.Chmod(tmp.Name(), 0o644); err != nil {
+		return 0, fmt.Errorf("cannot set the file mode: %w", err)
+	}
+	if err := os.Rename(tmp.Name(), c.outPath); err != nil {
+		return 0, fmt.Errorf("cannot replace %s: %w", c.outPath, err)
+	}
+
+	lifetime := time.Duration(out.ExpiresIn) * time.Second
+	if lifetime <= 0 {
+		lifetime = 5 * time.Minute
+	}
+	return lifetime, nil
+}
+
+func assertion(c config, now time.Time, pemBytes []byte) (string, error) {
+	block, _ := pem.Decode(pemBytes)
+	if block == nil {
+		return "", fmt.Errorf("the key file is not PEM")
+	}
+	var key *rsa.PrivateKey
+	if k, err := x509.ParsePKCS1PrivateKey(block.Bytes); err == nil {
+		key = k
+	} else {
+		parsed, err := x509.ParsePKCS8PrivateKey(block.Bytes)
+		if err != nil {
+			return "", fmt.Errorf("the key is not an RSA private key: %w", err)
+		}
+		rk, ok := parsed.(*rsa.PrivateKey)
+		if !ok {
+			return "", fmt.Errorf("the key holds %T, not RSA", parsed)
+		}
+		key = rk
+	}
+
+	enc := func(v any) (string, error) {
+		raw, err := json.Marshal(v)
+		if err != nil {
+			return "", err
+		}
+		return base64.RawURLEncoding.EncodeToString(raw), nil
+	}
+	header := map[string]any{"alg": "RS256", "typ": "JWT"}
+	if c.keyID != "" {
+		header["kid"] = c.keyID
+	}
+	claims := map[string]any{
+		"iss": c.userID,
+		"sub": c.userID,
+		"aud": c.issuer,
+		// Backdated, so a clock that is a few seconds fast cannot make the assertion
+		// "not yet valid".
+		"iat": now.Add(-5 * time.Second).Unix(),
+		"exp": now.Add(5 * time.Minute).Unix(),
+	}
+	h, err := enc(header)
+	if err != nil {
+		return "", err
+	}
+	cl, err := enc(claims)
+	if err != nil {
+		return "", err
+	}
+	signing := h + "." + cl
+	digest := sha256.Sum256([]byte(signing))
+	sig, err := rsa.SignPKCS1v15(rand.Reader, key, crypto.SHA256, digest[:])
+	if err != nil {
+		return "", fmt.Errorf("cannot sign the assertion: %w", err)
+	}
+	return signing + "." + base64.RawURLEncoding.EncodeToString(sig), nil
+}
+
+func env(key, fallback string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return fallback
+}
@@ -0,0 +1,64 @@
+package main
+
+import (
+	"crypto/rand"
+	"crypto/rsa"
+	"crypto/x509"
+	"encoding/base64"
+	"encoding/json"
+	"encoding/pem"
+	"strings"
+	"testing"
+	"time"
+)
+
+func key(t *testing.T) []byte {
+	t.Helper()
+	k, err := rsa.GenerateKey(rand.Reader, 2048)
+	if err != nil {
+		t.Fatalf("cannot generate a key: %v", err)
+	}
+	return pem.EncodeToMemory(&pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(k)})
+}
+
+func seg(t *testing.T, s string, out any) {
+	t.Helper()
+	raw, err := base64.RawURLEncoding.DecodeString(s)
+	if err != nil {
+		t.Fatalf("not base64url: %v", err)
+	}
+	if err := json.Unmarshal(raw, out); err != nil {
+		t.Fatalf("not JSON: %v", err)
+	}
+}
+
+func TestAssertionCarriesTheKeyAndTheAudience(t *testing.T) {
+	c := config{issuer: "https://zitadel.example", userID: "3970", keyID: "3971"}
+	tok, err := assertion(c, time.Unix(1_700_000_000, 0), key(t))
+	if err != nil {
+		t.Fatalf("signing failed: %v", err)
+	}
+	parts := strings.Split(tok, ".")
+	if len(parts) != 3 {
+		t.Fatalf("a JWT has three segments, got %d", len(parts))
+	}
+	var h map[string]any
+	seg(t, parts[0], &h)
+	if h["kid"] != "3971" || h["alg"] != "RS256" {
+		t.Fatalf("header must name the key and the algorithm, got %v", h)
+	}
+	var cl map[string]any
+	seg(t, parts[1], &cl)
+	if cl["iss"] != "3970" || cl["sub"] != "3970" || cl["aud"] != "https://zitadel.example" {
+		t.Fatalf("claims are wrong: %v", cl)
+	}
+	if int64(cl["iat"].(float64)) != 1_699_999_995 {
+		t.Fatalf("iat must be backdated five seconds, got %v", cl["iat"])
+	}
+}
+
+func TestAssertionRejectsAKeyThatIsNotPEM(t *testing.T) {
+	if _, err := assertion(config{userID: "u", issuer: "i"}, time.Now(), []byte("nope")); err == nil {
+		t.Fatalf("a non-PEM key must be rejected")
+	}
+}
@@ -36,53 +36,54 @@ spec:
        - name: paperless-ai-env-config
          configMap:
            name: paperless-ai-env
-      initContainers:
-        - name: fetch-llm-token
-          image: curlimages/curl:8.12.0
-          securityContext:
-            runAsUser: 0
-            fsGroup: 0
-          command:
-            - sh
-            - -c
-            - |
-              set -e
-              echo "[init] Fetching JWT token from Authentik for paperless-ai-agent..."
-              
-              # Get JWT token via client_credentials grant
-              # Client secret sourced from environment (injected from paperless-ai-config Secret)
-              TOKEN_RESPONSE=$(curl -s -X POST https://zitadel.riotpiao.com/oauth/v2/token \
-                -d "grant_type=client_credentials" \
-                -d "client_id=396757803559027071" \
-                -d "client_secret=${LLM_AUTH_CLIENT_SECRET}" \
-                -d "scope=openid llm:inference" 2>/dev/null)
-              
-              # Extract token from JSON response using sed (handles spaces after colons)
-              TOKEN=$(echo "$TOKEN_RESPONSE" | sed -n 's/.*"access_token"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p')
-              
-              if [ -z "$TOKEN" ]; then
-                echo "[error] Failed to get token. Response: $TOKEN_RESPONSE"
-                exit 1
-              fi
-              
-              # Store token in file for main container to read
-              # Write to both locations for compatibility
-              mkdir -p /tmp/llm-token /app/data 2>/dev/null || true
-              echo "$TOKEN" | tee /tmp/llm-token/llm_token.txt > /dev/null 2>&1
-              echo "$TOKEN" > /app/data/llm_token.txt 2>/dev/null || true
-              echo "[init] Token fetched and stored"
-              [ -f /tmp/llm-token/llm_token.txt ] && echo "  -> /tmp/llm-token/llm_token.txt"
-              [ -f /app/data/llm_token.txt ] && echo "  -> /app/data/llm_token.txt"
-          env:
-            - name: LLM_AUTH_CLIENT_SECRET
-              valueFrom:
-                secretKeyRef:
-                  name: paperless-ai-config
-                  key: LLM_AUTH_CLIENT_SECRET
-          volumeMounts:
-            - name: paperless-ai-data
-              mountPath: /app/data
+        - name: llm-key
+          secret:
+            secretName: paperless-llm-key
+            # 0440 with the pod's fsGroup. A Secret volume is owned by root, and the
+            # vendor container runs as 65532, so 0400 left the sidecar unable to read
+            # its own key: "permission denied" on every start.
+            defaultMode: 0440
+      # No init container. The token used to be fetched once at pod start with a
+      # client_credentials grant against a web app, which cannot work: a web app has
+      # no such grant, and even a token that was minted would be dead within the hour
+      # while this pod runs for weeks.
+      #
+      # llm-token-vendor runs beside the app instead. It signs a machine-key assertion,
+      # exchanges it for a token, and rewrites /app/data/llm_token.txt at 80% of the
+      # token lifetime. The vendor image keeps reading CUSTOM_API_KEY_FILE unchanged.
+      # The key is re-read on every mint, so a rotation lands without a restart.
+      securityContext:
+        fsGroup: 65532
+        fsGroupChangePolicy: OnRootMismatch
      containers:
+        - name: llm-token-vendor
+          image: forgejo.riotpiao.com/riotpiao-poimen/llm-token-vendor:0.1.0
+          imagePullPolicy: Always
+          args:
+            - --token-url=https://zitadel.riotpiao.com/oauth/v2/token
+            - --issuer=https://zitadel.riotpiao.com
+            - --user-id=397072668735570071
+            - --key-id=397072674272051351
+            - --key-file=/var/run/llm-key/key
+            # No project audience. ZITADEL puts the requested audience in `aud`, and
+            # the gateway compares `aud` with its allow-list of client ids. Asking for
+            # the project id made every token carry the project, so the gateway
+            # answered "audience not allowed: [396757796898471937]". With plain
+            # `openid` the token names its caller, which is the entry that exists.
+            - --scope=openid
+            - --out=/app/data/llm_token.txt
+          volumeMounts:
+            - {name: paperless-ai-data, mountPath: /app/data}
+            - {name: llm-key, mountPath: /var/run/llm-key, readOnly: true}
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities: {drop: ["ALL"]}
+            runAsNonRoot: true
+            runAsUser: 65532
+            seccompProfile: {type: RuntimeDefault}
+          resources:
+            requests: {cpu: 10m, memory: 32Mi}
+            limits: {cpu: 100m, memory: 64Mi}
        - name: paperless-ai
          image: clusterzx/paperless-ai:latest
          env:
@@ -0,0 +1,27 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: oidc-secret
+    namespace: argocd
+    labels:
+        app.kubernetes.io/managed-by: argocd
+        app.kubernetes.io/part-of: iam
+type: Opaque
+stringData:
+    client-id: ENC[AES256_GCM,data:KJmGTcCz/c60+aLs7BqwL6Je,iv:hpwBdXp9einCN8EdvMXgTyf4Tl6SN9VTmDQotdEC/qE=,tag:MmF076Vsm0ejZtQMfG+e5w==,type:str]
+    client-secret: ENC[AES256_GCM,data:0W2lS6WTzUpJUQo4AQSZk0iXOqZjKVqXCRv4HVuNrGpcZYf81oJVR8dM/FnApd4zS1Y11465aXYcImqcKo4xkg==,iv:ECIkWzeRTroWpugjqASD3eJHipn4m2JWthC/1gaXhkA=,tag:GU5GyKhUvutJRuxkhQNE0Q==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZZjVTSlN6c2dZbDZCeEFa
+            SU9YVy9HUEEzckRVb0QrRmh5NTN2Qm8ycFFVCkpwVnAxK3BGT29NdnN2VDlveDBG
+            RHo5YjVicllRc1NlRzNabS9vWlo3cVUKLS0tIHF4OE1GKzJUZlBhU3dNOFFaeWZC
+            M3QrRTRxbkNRZVMreWsyQytlMDdFUTAKwtVdzOk6nMDrauDcRUnun3JU/Cfi3qgB
+            jhftbXHaxcKhB5MJCwZkN7VmtpxOjpvYgpetj9cBNXx2yVVcS7G5Hg==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-31T03:41:24Z"
+    mac: ENC[AES256_GCM,data:GHwigzuZRGkLIH6vaDjUsefjdnmhToePQGB9+3CIZW8DunlkgmGqk/jIsQee7Yo0fa1OOdMHRQH7+biooUP1Z5GANmsBnQhDT0GAnd3JPbIN2lIJKeManGDALqsbhLzmfcSBNByIJaJh5c3xakL/LmU+CoUWluWCLQqjpeeiMEk=,iv:uBRbVAgzXxxn3Xx7foUZWyK/Cq6yMd3/lkt+S+uY2bk=,tag:Mr0F5rp5aEVNd7pZqnyWUw==,type:str]
+    version: 3.13.2
@@ -1,30 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-    name: comfyui-oauth2-proxy
-    namespace: comfyui
-type: Opaque
-data:
-    client-id: ENC[AES256_GCM,data:jFLgcrh1tiAU0pijGxy+/mFsFHzI5zSi,iv:7GYL4xtPnzyZWtEA0egPHhzvpXGevkIH3Cvr+oD9hKI=,tag:UeDTIKlfcYnOOOJiPNgN0g==,type:str]
-    client-secret: ENC[AES256_GCM,data:9QFu5a2EkEMjc/B0moPIkMEChrN0Qh5BQtAzFcvW4liKxzVAPbTKgQOJeND+NQjn5ZoE89kqqRyqNJF3bT3H1pWoDFngk9J2X7BN/9r6Qyubias0Jukydw==,iv:xf2kT0fbnJat3L3xxUYm3ht0YvHptU/RIdt9Eh0D6xU=,tag:8iknaQ2kKP1OZ/1jRCa+1g==,type:str]
-    cookie-secret: ENC[AES256_GCM,data:Qq+l4go/CFtfsZrgYUcTj9bB4sreu2GEqJnu15H6fxIlD7IgnmrodPoIVNpWcdm6LSYabGjm2jraN6RP,iv:EOoDHgvIydzHCC5POohjPq3Vj2YTpzhIhvdG2FzKKzc=,tag:iDXqzFU1Z79bzpG4I4xfww==,type:str]
-sops:
-    kms: []
-    gcp_kms: []
-    azure_kv: []
-    hc_vault: []
-    age:
-        - recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxOUtCUDduMVU5U1J4WXZX
-            Q1VzaFRVdXlHaEZpeVJNOWprdDhUOEd6Z1hzCi9QZEpXN3NDSENzUE9RTGdpb3Va
-            eVNHVTFLbXpMU0x5UWlFLzZnVlllWE0KLS0tIDZldVNqR1loNHlXZWovNWZ4WjFD
-            dVErenJDM3haSStxZEVWZUtBbDBIZ1UK9x6Gy4drjfUz2goAaI+oeUDWdNHQFAJl
-            JGI+X81r/6Krg5N8QDa/mE96E2M2G0yAO5FS1YbpCKOITUEyheacnQ==
-            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-10-27T21:17:08Z"
-    mac: ENC[AES256_GCM,data:tTGmWFidEWTWHgKPV7b5ABOJwwlP1yQDK5Jd60c9ko6IsmP2GBBPCth7kJgPXuPcw+NHbJ7yt7bcVto7GkPdtY9daJmjPq0Gt16ZO4Xi4hCk22JVKqi/KJcKD/KB0r/VO9w093/Who5CcYgqifextfca4CuxS5m5dcOn4kRJCfE=,iv:0fDvvQr9wp8IrD16u88Ml4HsPEeMhd8R3ljQJoIpw3E=,tag:AanM7C6vbaBBmus1u1GRoA==,type:str]
-    pgp: []
-    encrypted_regex: ^(data|stringData)$
-    version: 3.9.3
@@ -0,0 +1,39 @@
+# Bootstrap credential for the Forgejo database — the one secret in the CNPG path
+# that the cluster can lose without any controller to recreate it.
+#
+# forgejo-db-app is OWNED by the Cluster CR (ownerReferences: Cluster/forgejo-db,
+# managed-by: cloudnative-pg) and was GENERATED by CNPG: spec.bootstrap.initdb.secret
+# is empty, so nothing but cloudnative-pg ever knew this value.
+#
+# CNPG reads spec.bootstrap.initdb.secret ONLY at cluster creation. So if the Cluster
+# CR is ever deleted and recreated, CNPG mints a new password while the data directory
+# on the PVC keeps the old one. Forgejo then starts, stays Ready, and fails every
+# query — a credential fault that looks like a network fault. This file is the record
+# that makes a rebuild reproducible: point spec.bootstrap.initdb.secret at it.
+#
+# Keep this file. It cannot rotate on its own; rotation is a manual CNPG ALTER ROLE
+# followed by updating this file and the Secret.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: forgejo-db-bootstrap
+    namespace: cicd
+type: kubernetes.io/basic-auth
+stringData:
+    username: ENC[AES256_GCM,data:GFRjVvq3iQ==,iv:7ZxKdHTvKt6TYlsQYu57dt9C14h59xyfrL3+o/deon0=,tag:Ame/mfgDlHk1oPp80hE3LQ==,type:str]
+    password: ENC[AES256_GCM,data:11t2yglSWQGKeMQ0ZRaw3L5s/o3iHEYArFWdfAQBYSnz1SMOmZ2XTl9yDDHFMUEgHK4K4mP4fJaq4nsgfgr5pA==,iv:aCQ4nkuUX6jDNiOsRW5nzST7HgNZmhjRidmskfNArAg=,tag:ddN0lf+7i4LpS6+yK+za+A==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3ZVQzajgzcmE2MnFzb1pJ
+            VFR4NjNtU2lWYjZGdGNuOVVnTkNxRndadXhzCkhrU3pPZi9NUlRkRWNUdHNKbW9a
+            WTcvVnVRM2tWZE1OMDBGcTd0NjhzeVkKLS0tIG5PQlg3ZlBNMDhQV2hxdzB4K1g3
+            SHBtZ284bTJNV3FvSWFtbDRudGUzbTQKNPM04xB1DWyIE/6/qjUCvXw2M8zkhVkP
+            xV8bVIp81R0pnMuEGB60L38W81SfziSspzpU0YMUmgZcU2XmO73AdQ==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-28T10:09:09Z"
+    mac: ENC[AES256_GCM,data:oNYDrN9wzVbDrDTYcLEKwCqTruybeU+++58Duki5P8xddU1anjHzfBg9DXgXd6CaoYJAU2jEcrqX9YXN8zmJtGaTpgr2GUdgFWx6O1JFoaNM32RNcp7u8FZWpcFYecWyYRtRFMolRaLM/CxkXr2lLa/V3Azxe1xlJrf7EjyI3dY=,iv:7ztpZquVBNZFt59OcNM2IXiDdmMjE8AGs1lTDZzeNWs=,tag:xpak7lbN6hrXaSKoWSbHYQ==,type:str]
+    version: 3.13.2
@@ -1,28 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-    name: grafana-oidc
-    namespace: logging
-type: Opaque
-stringData:
-    GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: ENC[AES256_GCM,data:fGqA1ofVme1lVLxFjWIiU7sbvE9ip2Af7C1ua3+HdidlcVgkwW59vVlXZhgBRdoif86C8LKQSyoCGUqiO+740g==,iv:6Kv2ybbJg1GT8DLqz9P5y+E1zr7vcIBDMUErmvxNTaw=,tag:0wK28B3G/owCVPPqb2sLmQ==,type:str]
-sops:
-    kms: []
-    gcp_kms: []
-    azure_kv: []
-    hc_vault: []
-    age:
-        - recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArTXMxMWdVRHIvbWh1bVNk
-            bEFFNlgzNU9zTjlzY2o3VGdnbFpnU2lkYmxrCjF4aWVGSlJaNkcxODJ6RDkrVFo2
-            UkV2RUwwVHFCd0xMNG9RWXdTUHZGaXcKLS0tIHVaWjhaNHRtS2c0UkVzRk83ckhj
-            OCtBVFR0d1F5amVhV2E0emRZWThvYjAKFW46i9brIL9ZoO1CEn+IewIdoUhvYK/A
-            RqzZ6xV+mhlDj0zC+SpVcEm/Ki83nMYOj4+vQingZlr/Kr3Nwb0iLQ==
-            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-10-27T19:44:32Z"
-    mac: ENC[AES256_GCM,data:EJew+p/kJrurcNpLlFxJGTl69hQYxn3IkzP7EDtdSriW/WI7X1xNLC2dp/dTiLomGmG1uh6+2JI9Sc8gXmrRAWLELAbiOaYbcoU6gi89BjTCWGDAMX4lNSIuH40rsGs4YtYj5U7SGfgx5qto4ZH7LxjBmxzPWM/Xz2Uq/I5jV4k=,iv:LdhjhwCfPUIa7D0gved8Twf7ujN+dg3lN8lCnFBFs0g=,tag:bjkwUCMtR8IBL9BkbuXcnw==,type:str]
-    pgp: []
-    encrypted_regex: ^(data|stringData)$
-    version: 3.9.3
@@ -0,0 +1,26 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: grafana-oidc
+    namespace: logging
+    labels:
+        app.kubernetes.io/managed-by: argocd
+        app.kubernetes.io/part-of: iam
+type: Opaque
+stringData:
+    GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: ENC[AES256_GCM,data:ZCO6gQGNRKtahbUctlLYCTx3TpRdz5lj8r6X4fUXkyvS5rbWXgip2mGqho4LUSHh8du3wo+bK8VyP1MhzIFRGA==,iv:O2aCivj8Z3cbKCqj/Q8uG6mEH9QJbCrDLPIjX6bLkjo=,tag:qpKgg9AAtIUvYzr5zwTO1g==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEc2xWSk1ldGxpYzlHUlF5
+            aFZQWnVxWXJVemZVd1llQlZUTkNrTFpVWGhrCjZtbW1XU1Z0U1ZlTHlGMjVSTUFp
+            NnFNTjdlYUUyTlhnN3QybFZJSGRuRG8KLS0tIG9DL3ZQYmErc2ZndWd6TUNlOStp
+            UTI0UDZZM2IwQU9oRmxsU25rQi9zUUkKYznGn96vdcEZSCpfyReNj4qLoS0t19ZQ
+            frodab7z/n6TadRwsb1ufb+YIHa4pGPlImmrlNGOIpnKvzXlmOCP4Q==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-31T03:41:24Z"
+    mac: ENC[AES256_GCM,data:uOPMvN1D/6AdxpWrR0aQYAxSEcC8/aJNg574cF+ypi/rt+srk68q2kAiKwA5L5GDvXs65rSZh2PMkJ0KV75g3ShFYOQSLaq0HIwYYTvIXRLtF43G8vMHaaZAhDC1pZeYPdD1KG61pDVjyCDc+qFZ/90kHqYiIeAdEI4Tia5YOJ4=,iv:SpU/YrCxORAJWJCLyTNSaNuIB0dAhq8+Bma58RUYL3U=,tag:s7fsEDQfqElTsE8EQsZfxw==,type:str]
+    version: 3.13.2
@@ -0,0 +1,27 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: immich-oidc
+    namespace: immich
+    labels:
+        app.kubernetes.io/managed-by: argocd
+        app.kubernetes.io/part-of: iam
+type: Opaque
+stringData:
+    client-secret: ENC[AES256_GCM,data:SAxFOfR1lIeKFHII2BJskF2T46lXlFFu+qzLxyf/uBDiQACRDTkQGzeJccc2jNT5hagS5GDOidnZN4aXrHGjUA==,iv:V4sDhHbiOyB+cllFuBpfO0JsiEG5frWMj0lUZSCLRwk=,tag:tb/m9gFUSb7SGuiilYoZSg==,type:str]
+    config.json: ENC[AES256_GCM,data:6HCZ2bl5RavyHP15RMMRkKYaCGXIVwIEkIOkT5VZs2BFZ7b+Po2fLimM7efVsR1QAF4NZ3StAOnsdJoEaTrNeEfCf22pLdELYw4swodlJi0Ybty09q9N7JZzENoa+3Sor0wxPI0FE70m058Sg+NkliRXxwpdwMAFkh6nj4zwZy0dUiyaHVsQFs5CDoAMrKP5KCdQhAz6Qx622C2D03pWjZICAO/VrKhLU/2N8c8Jid9r2zJWFJxegazhEtJ+AE/1aM+h9rjb7IB9HObRUROd8tYypOgV5u3LBQMg9DP4PTHcPoDdyHbKtRYyCTP5adIt6/Xr37C+5o3dpEi8cR8yUfWIn2jYkcbhIUSVGCVE8b4e3b6jNhvUAlv5JZubjVmm+MJdAhGTPBi7NEPoU+1p0Bs8yx4mjvtXZRn4is2oei322vhUUpIq/OzAn5Km8v/8CnzyS6Zclsl5Oa+jRqWIqNJ3vVYSReytN11KbvCNtDmVsrNpPg1Set3mhk7CIrND3tZQqX6Q4P7/vllAvfsoorf7BZrI7z7/SrRxYWZZWziC7tsPnqul+yntymWxHt2NW203Fdk=,iv:6IymBA+UeVGjDffyKAxpiAA/1G9/eB6mJJmZ8Ntwq90=,tag:X38OSEIgoNbrSyHFADwADQ==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKV0Fta2VXVThvTnl1S09O
+            TkgwTjJBc040K3NRdXFRTlFEMGJqaVVVZmd3Ck9MMDc0VzNkRDhrbTFteHc1VFVo
+            OTNrMytLVFZYMkZYSGJjU2pnMjNlSk0KLS0tIDJrajNqZjVNQ1NjZ2FienJFcEVK
+            d3JIMk1lQjNuVE9LTmhoYzJlRHpua3MK17o8wpCg98eiFYJdZhWqHOl+riMcqLpg
+            pL+jzEAb6F8aOOvlog99TDyq5Pw0g1MvJBmw+2jyMTAZE2cIoX4CLg==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-31T03:41:24Z"
+    mac: ENC[AES256_GCM,data:cv9uXDrbECSIPQLbEugfvYyGG6BoNxu/Cva7BmUYF6DkT34dCtCMNI2ey1nV5m8dJM1ote2lxpnWjt/6s1P41AMRsdyGkEaGrAW/62uP8NCX6PP+p519qTUvf4ZdGFQ24Uo0UVQ0AV/vCw1RAurmZXKcoLVvQaCzgDCn2EgfAiE=,iv:mpJ+YqFamDGaEiqy6Y3gT5IqMcKs2qR/FyfmxKM7BcM=,tag:U+UtCTLTX+b/FjZGfUrrgQ==,type:str]
+    version: 3.13.2
@@ -0,0 +1,27 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: minio-oidc
+    namespace: storage
+    labels:
+        app.kubernetes.io/managed-by: argocd
+        app.kubernetes.io/part-of: iam
+type: Opaque
+stringData:
+    MINIO_IDENTITY_OPENID_CLIENT_SECRET: ENC[AES256_GCM,data:M8iRH6BxzEz7r/eOwED+uiOdHwkXg5DD5s4xEaKJCNAn2HwlpuKM+pUVLyLNwSxeN7Kf6dYVMUWuuzL2N3BkEQ==,iv:tSa3MSSNkM38/avXIt4CHUDZ64zXGGqiInCJx3cP2gI=,tag:z5ODn9b3/464+O1bF5CoZg==,type:str]
+    client-secret: ENC[AES256_GCM,data:9/YiXaYw/AavqklSPZ0akuF6gNjPL9OrpEc1jMz8N5QgYD8aZJxol0mUXqhoLy6zldaifo7ZxtzRONdJkuGBwg==,iv:kEUzmeQMVbyI/2ag/9baRWp0XJ1VhIqBlpVxZB6OEog=,tag:kf34PPlQzHTWfd2ndGvP3A==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3S2JTSFF3Y2VMWkNncTIw
+            K2VKM1dDNnE1ZE5YSTNDN0ZFS2paNXNHSUU0ClFGWnBSVThrbWk3MGUyTVlPRGNB
+            cGJOMmxSMlZnNHNMT3cxNzFneEc2b0EKLS0tIG9lbURIdEdLNVZVR3JiWkFWVmMx
+            aUVDVUpOTENjK0ViYWFVRm5BNmcwbncKg5ThSLcmeJKY2mm7EkRlDg6FKnpBvFrE
+            eEwfEEX68KirboOsv6U1TPjyf2ZM0Wfj+1kVfiNQTLi7QEE3R377KQ==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-31T03:41:24Z"
+    mac: ENC[AES256_GCM,data:QA7Skxv7f9AOZCPWAZlEW+TR4eNSSX8LlxFc6zUqfiWu3oPPThMQxXiHJi4FEeUsjwe3Qy/FsPWF2182P0yZ/lfiXFi6bfPMzL9OeyYISrjfd/MUBm02+JG/MMrMXat+sxQgrTljl7z5H74X7BywHOLLGHjHDoWEEzqU3pJrFIQ=,iv:wecxDkPDPQmFwaRZjnxKD6zuTXQREjfkyZ34eNF/63M=,tag:IUDoRtpMCYa3amxWoyHL+g==,type:str]
+    version: 3.13.2
@@ -1,3 +1,6 @@
+# minio-oidc is not in this file any more: the KeyRotation object iam/minio owns it.
+# A prune of the sops-secrets application would delete that Secret, and the
+# controller recreates it on its next rotation.
 apiVersion: v1
 kind: Secret
 metadata:
@@ -5,46 +8,27 @@ metadata:
    namespace: storage
 type: Opaque
 stringData:
-    config.env: ENC[AES256_GCM,data:1yogGyBdwrlg27H8R5Kr9VcPu+xwS5/nPhi7eLcmVYtXmPlNokCsWHA70qnn/+I6i4ODys8GML++VeZ/lmcHiP7RIbY+G6xmw6nfBCXitgE8kQwTPsUUzn9bobOn4Sr1YMBf7FQ/Q0N8aupGJA==,iv:EfCdim4TLkP/FP1u1yoKZCq8YGFzVh7ca4yu1+WzXwE=,tag:rTYe0SIAEsUhh0KHQIEKNw==,type:str]
+    config.env: ENC[AES256_GCM,data:4NQYmVEnkv4nO6FnMAEXVCUhMM+DLCGG2tFC7SGJ0lGnfFQiISdzWwVXD0y9/gpK4qn5zv0Cs1MgOfDVBKH6UR1ONXQnIjUnvXnq4D5+P0BgVUlYD/7aComva9Ow1IEChvD3fBTmDW7FMEZrtA==,iv:/LAV+ukcE3pwgJMkEkR8z9h+k6pSvQ7a/FkxLvMK6X8=,tag:BnvsL05XttadRqqS3TU2Uw==,type:str]
 sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
    age:
-        - enc: |
+        - recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2SStVR0lpOWs0VnN6dkdp
-            RktMUmNRR2diTUtiVk9tRk1wSjVTMnRtVlJBCkk1S0dwNUNHb25RQlZ4ZFRocisv
-            aS9kbmQydWVmN1AxU3VvbFZsS2tmTHcKLS0tICtRY0ZKeFhCTUdaNW5PSFRNZ2dG
-            OERIZ0xMbHBvSVNyaGxCVFpqRzlxa1EKbrjfk/CFlTSyJYgTzQ67W3JyR1fwv8VF
-            dxGeAgLY4EQrf+2PbODtazRFreS+sl7dxq+rKMOVZTsjGHakkqHy9A==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxQ1BqbDZoYjVQWGlTVEFD
+            dVVWT0xFUlAwUXJUcHZOMGN1cG85S3NtcVZRCmpUZnM5dXpWUDh5UHhGUTcyYmNr
+            UWNlc2VDazUvd3k0VHVWR1NBUE13eTAKLS0tIG9VcGYvN1Z1UEo3Um1SMHZOSHF3
+            eUp5dE04M3lvTVlHc3lPZnMyUWU0bE0KPXwE7wOJNSrqM6kfO1BvXqHApvQI4hdz
+            5GWmLjmjQL/ZZlkF07uK1GDXIBC3N3K0vYrfQHjMUImw9Dwn9DnL5w==
            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    lastmodified: "2026-10-28T03:00:02Z"
+    mac: ENC[AES256_GCM,data:DGglizsoj/Fmi+r1NvfYD1sSivtedJ+w0RKFioJ1K7ektG/I68hprcMkmrs9vY1H2zlP/1Io/Zf0wOMqIPVK+uyEHIDGr+kk5Y/vPkrPi1+TpVDHRJVwvfkFWWVBrnlMbpZ89076dl1cyphdi4XvSHq2z30Qa1oRnxNkPO2HrhY=,iv:je0iyd/AkWYeyL2vA6ipmvWL6c0X+4h1zeLMeE60Pks=,tag:Q3X/jIWQWFtiSGhkCbzBDA==,type:str]
+    pgp: []
    encrypted_regex: ^(data|stringData)$
-    lastmodified: "2026-10-26T04:03:16Z"
-    mac: ENC[AES256_GCM,data:gxu4WcJEuFgjtWCfAjose8R8a566/OLrXoiJmKfcvVfTFDTdGM7I4gzQVvpjEfra5Hjyt7kvLlaV2K5bKJ0k6NXbbmgrqAqEOEcPg+cnPy1FtQGAYOo3VzgKqrpLOCs/7wh4YR23XHHsjQlHAcP8BmhrqnpHXR9FMZObrKjnQj8=,iv:KJZfw0n2DOtdZHeVDk6TrpO8q57yD+xdEVt9+hM0D4Q=,tag:rHQaMjvQwaB8Am/AlTKq1g==,type:str]
-    version: 3.13.2
---
-apiVersion: v1
-kind: Secret
-metadata:
-    name: minio-oidc
-    namespace: storage
-type: Opaque
-stringData:
-    MINIO_IDENTITY_OPENID_CLIENT_SECRET: ENC[AES256_GCM,data:eFfwW2ZnV+o3/6XHjUvyFsDbaIAdosY2PXOnRdN4Ie49YDIjfJPe+Mg1FsbgoEHEjuB33hfxesb4TWOJQfW8Og==,iv:+kRr9A3auts1XHgZWa+t5b2m4lF0x9dVhiRpB17fzv4=,tag:tH/jszFzLQdqORGr9vfTGw==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2SStVR0lpOWs0VnN6dkdp
-            RktMUmNRR2diTUtiVk9tRk1wSjVTMnRtVlJBCkk1S0dwNUNHb25RQlZ4ZFRocisv
-            aS9kbmQydWVmN1AxU3VvbFZsS2tmTHcKLS0tICtRY0ZKeFhCTUdaNW5PSFRNZ2dG
-            OERIZ0xMbHBvSVNyaGxCVFpqRzlxa1EKbrjfk/CFlTSyJYgTzQ67W3JyR1fwv8VF
-            dxGeAgLY4EQrf+2PbODtazRFreS+sl7dxq+rKMOVZTsjGHakkqHy9A==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    encrypted_regex: ^(data|stringData)$
-    lastmodified: "2026-10-26T04:03:16Z"
-    mac: ENC[AES256_GCM,data:gxu4WcJEuFgjtWCfAjose8R8a566/OLrXoiJmKfcvVfTFDTdGM7I4gzQVvpjEfra5Hjyt7kvLlaV2K5bKJ0k6NXbbmgrqAqEOEcPg+cnPy1FtQGAYOo3VzgKqrpLOCs/7wh4YR23XHHsjQlHAcP8BmhrqnpHXR9FMZObrKjnQj8=,iv:KJZfw0n2DOtdZHeVDk6TrpO8q57yD+xdEVt9+hM0D4Q=,tag:rHQaMjvQwaB8Am/AlTKq1g==,type:str]
-    version: 3.13.2
+    version: 3.9.3
 ---
 apiVersion: v1
 kind: Secret
@@ -53,20 +37,25 @@ metadata:
    namespace: storage
 type: Opaque
 stringData:
-    CONSOLE_ACCESS_KEY: ENC[AES256_GCM,data:Zxr/Nj0SUKSJbBD1cFjt3GE=,iv:r/B9ULn68hsXdH8uRQkKAoQTJIJIYKMoJuWOZfxS2I8=,tag:Xo4i+IVxuqrp+qDpYQysPw==,type:str]
-    CONSOLE_SECRET_KEY: ENC[AES256_GCM,data:PoIDNZ+Xs6BdhGm10y1HdL5jy47EVHfHHIJhGy/KX6Q=,iv:voAv32GoTKi9s9/4MQy2yT6i9VY4/igA5XXzehwUqRQ=,tag:DmPyk5biVvlxHjeyDyrLbQ==,type:str]
+    CONSOLE_ACCESS_KEY: ENC[AES256_GCM,data:6mmBluC+4QmlMxPJbfNh0kA=,iv:CuWy1gVnd19vB1rOS9HRVKegX3CnmCEa6cpk+S7ZX00=,tag:OqZK09bVMZGbMXwnDmr77Q==,type:str]
+    CONSOLE_SECRET_KEY: ENC[AES256_GCM,data:Fyd5EdZZP7/q3QOVv3FU1bkd8960p6UiRNNl17LJltw=,iv:/QGSkVVfXkCAKSV1+RNTZEI+/riAFVLV6SiaL9DvbSo=,tag:lFTJpifGorsl2w8lG7beCw==,type:str]
 sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
    age:
-        - enc: |
+        - recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2SStVR0lpOWs0VnN6dkdp
-            RktMUmNRR2diTUtiVk9tRk1wSjVTMnRtVlJBCkk1S0dwNUNHb25RQlZ4ZFRocisv
-            aS9kbmQydWVmN1AxU3VvbFZsS2tmTHcKLS0tICtRY0ZKeFhCTUdaNW5PSFRNZ2dG
-            OERIZ0xMbHBvSVNyaGxCVFpqRzlxa1EKbrjfk/CFlTSyJYgTzQ67W3JyR1fwv8VF
-            dxGeAgLY4EQrf+2PbODtazRFreS+sl7dxq+rKMOVZTsjGHakkqHy9A==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxQ1BqbDZoYjVQWGlTVEFD
+            dVVWT0xFUlAwUXJUcHZOMGN1cG85S3NtcVZRCmpUZnM5dXpWUDh5UHhGUTcyYmNr
+            UWNlc2VDazUvd3k0VHVWR1NBUE13eTAKLS0tIG9VcGYvN1Z1UEo3Um1SMHZOSHF3
+            eUp5dE04M3lvTVlHc3lPZnMyUWU0bE0KPXwE7wOJNSrqM6kfO1BvXqHApvQI4hdz
+            5GWmLjmjQL/ZZlkF07uK1GDXIBC3N3K0vYrfQHjMUImw9Dwn9DnL5w==
            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    lastmodified: "2026-10-28T03:00:02Z"
+    mac: ENC[AES256_GCM,data:DGglizsoj/Fmi+r1NvfYD1sSivtedJ+w0RKFioJ1K7ektG/I68hprcMkmrs9vY1H2zlP/1Io/Zf0wOMqIPVK+uyEHIDGr+kk5Y/vPkrPi1+TpVDHRJVwvfkFWWVBrnlMbpZ89076dl1cyphdi4XvSHq2z30Qa1oRnxNkPO2HrhY=,iv:je0iyd/AkWYeyL2vA6ipmvWL6c0X+4h1zeLMeE60Pks=,tag:Q3X/jIWQWFtiSGhkCbzBDA==,type:str]
+    pgp: []
    encrypted_regex: ^(data|stringData)$
-    lastmodified: "2026-10-26T04:03:16Z"
-    mac: ENC[AES256_GCM,data:gxu4WcJEuFgjtWCfAjose8R8a566/OLrXoiJmKfcvVfTFDTdGM7I4gzQVvpjEfra5Hjyt7kvLlaV2K5bKJ0k6NXbbmgrqAqEOEcPg+cnPy1FtQGAYOo3VzgKqrpLOCs/7wh4YR23XHHsjQlHAcP8BmhrqnpHXR9FMZObrKjnQj8=,iv:KJZfw0n2DOtdZHeVDk6TrpO8q57yD+xdEVt9+hM0D4Q=,tag:rHQaMjvQwaB8Am/AlTKq1g==,type:str]
-    version: 3.13.2
+    version: 3.9.3
@@ -5,23 +5,25 @@ metadata:
    name: oauth2-credentials
    namespace: iam
 data:
-    api-gw-client-secret: ENC[AES256_GCM,data:YiUCk61cCsTJE8gTLZWhqVPsCSu3b4OnG+Oh456A+mnxip0FjLTTqV+kYBdfZivIkn9ua4RcVW+rYILu,iv:K08m1EaB7/SIVNWZ3FfeRP4tbCAo6KeiNC0NpE9VO1o=,tag:ZXV9YRWUOPJ/vdoqMtvQeA==,type:str]
-    minio-client-secret: ENC[AES256_GCM,data:vcwLo1cJqbZr2MwkAKO09xPjb09ss7+s6HIvEfaLxt8ka5PUAlXQ37pTkynuktlKq/t0DvSk+x1+5G7QqVD1JJlwyDGM6yuOhBQ3MwucLNl+T4RDKtgGuA==,iv:sCdd+UNX2TfTAM6Zslrh8KrvBSzqpe67F3YMGrl4YDU=,tag:9Tpzm9NEXNuXyCd7l7GeHw==,type:str]
-    poimen-client-secret: ENC[AES256_GCM,data:dYtCiMGZaOFokihfGD2SXJDfXrhfwEQcTOPNu7JNHDUnx0hqwAdYAPAJTEN21p0OpHYDINMs5lwUpMwY,iv:ll8USuzs0881ryHImVjHGO0iFxUzqg5XtI6Ekd+EFXE=,tag:I/4ZfB9Ss4zEP57sRtRTXQ==,type:str]
-    paperless-client-secret: ENC[AES256_GCM,data:XMj9rHkI20lI37vChq9wS5lUq7l95SisRX0LChuj09O5w76Zt9NdG6OEUYQFf1z8vCfMJxBInIEEIq0e,iv:UpYy2ZEZ4AgBACEyrlxEcr13V+8Ksu4p8JHYApmyYHo=,tag:o9Z+/b6T9POgowIsks56kw==,type:str]
-    grafana-client-secret: ENC[AES256_GCM,data:V/j0+P3HoDLqYjRalWK4v1hvJTuVmzxVra47HsF1/BZvWZjIyEIH3Pt8BekT3O1dUFynxjcaxdZMBVYg,iv:9pMp+vIsp/sA7GP9wqNDf5DdZVLDow46OURW0mMRXIc=,tag:i7VJFA5o0hToZEJ5JPWIvw==,type:str]
+    api-gw-client-secret: ENC[AES256_GCM,data:q41OBQ6WvxsmHmXFhrdzeNUlQyPKuOQQa0vWdhuioQwTsNJ5+l9rYZKBAcZxzvd4P066aRN6yVPmEZhq,iv:Dnfgy8N9NVOM2x656Qei8960dzu7icAU3BZLXmj7N5o=,tag:deilBSCC9Mg/Eo8K0w6rMw==,type:str]
+    poimen-client-secret: ENC[AES256_GCM,data:2k28ui+Mbu3qd4ar6/NidO4bWO8g7JQUjDQBD8liywpCm7rA8TSrmU51im/I5kEvRRhl+RByEo5wpozo,iv:JFrZYF7cP/tQL8gkQJ5GvtdOt1kKcAdHE7L4l+uT9M0=,tag:DSiv8kv7MINUxi7fO/093w==,type:str]
 sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
    age:
-        - enc: |
+        - recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2amZPendIOU9PeStOcmJG
-            c1dzOFVvd3RoT3dXQlJUNkdCSytEZUVGVEJJCklMUkNjUytBMXh0OVUrNUlpMWw3
-            aHRFT1hBZTZHNHVtaGJCTEFyT2ZWNWsKLS0tIENxR3FKMEplV0V5Y0dsZzBkODVG
-            U0ZlT2R0VFdEQmpsakVvZ1FJRkRQYTAKESQcSlMAuCG0/tLWWZaeoPayf1EKxSaf
-            e4o8TmjmSgPEhepQ/LmNXl6obyuqUxztkh4Mm6pocqDp+gfzt7X/1g==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5WmthakxzRk5aeGJWZnRF
+            bzdqM1o5Q1h2MFlpNEJvRUFZVU1wWjRaVWowClVKYkp0YUJCbUM0MWduaCtBSExH
+            MDRWUGZkYTdndTZTeEM3ckxrMURFMUEKLS0tIHF3YUlLNndSWmxGejFPbTVjQnRS
+            THF4RFBNbDJCdjVKRDJ6cEJlZGVBM0UKEDz1sKDON+lelq1zbCO4CETaK77T4G8y
+            fly66sR5uIvAXdwXy14L9ssZsM1jOHqdkXnpbo6mzh3toB/ROzoL7w==
            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    lastmodified: "2026-10-27T21:47:38Z"
+    mac: ENC[AES256_GCM,data:bgZD+/LAPKp6DXyRMnbFWyxmiRjWxYO0ofxeGpO7nyw1DMSTD9s8li43eh/IP7wW4NORwzfIfzxUvOFmVfAiQKfkzpYRDFIKqK+aa/+43zppwpVT0UJ3CcBt2qgdJRHKnv9qeUOx4M8ulY7Xcsoevw8EGdrXCc1KAcDIwoIAp0Q=,iv:djYQ6YnGPrYGRKgpfbdf0beW6QHYTq5CJ3nes7+C4p4=,tag:N9n4VKyMNzGCmuYpwBLPng==,type:str]
+    pgp: []
    encrypted_regex: ^(data|stringData)$
-    lastmodified: "2026-10-06T02:01:24Z"
-    mac: ENC[AES256_GCM,data:AI/znHiUa/Rwg0YRz65qUZ+b6V73oi3ZfAG/V5uZZvrGFrA0EISzD5LaP5Kl4GUU1svo4G5EI4aftPDgpdPptz21CCa8iYkXGe9pTDmtaE5ZICQAUbRcVhGMTfuWFhrgEhMfCJdOpfHaIQVEsQJQJgrnkXQVuUe9aT75LQz7pnY=,iv:KbIaYT0VeBT2WeSU5V5ZB4e+04JICYaY+egGN1Pv5bM=,tag:pPbhDxeLUx6V1XC/LPwxTg==,type:str]
-    version: 3.13.2
+    version: 3.9.3
@@ -0,0 +1,27 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: paperless-oidc
+    namespace: paperless
+    labels:
+        app.kubernetes.io/managed-by: argocd
+        app.kubernetes.io/part-of: iam
+type: Opaque
+stringData:
+    SOCIALACCOUNT_PROVIDERS_JSON: ENC[AES256_GCM,data:/muYqHv+LRPhAl3u7uHv1a0O9vG9nZ0zJawcAmUTHA1x3VKnoaiqQCQrYL8Mg4TtOkXHn/ESwuq5Nbyr/P3E6n9C6BG7Dg7/7vtj/OiL6URN8apjsKQvQAswfycmhWJZV8kMRJNT9IBWjQ9D02MhLan+WwVBWOlT2bWBa+mMogBRgNSorbTRYXao24Qh2KRS9NRDCyOQ0Mp386tyjC9JQzXJ4R82RNh9VuipfPFgo4Mk/wXUihft/lMe1xcVrgl8uxecspbOFlgUMjtuhfEHc+tIVxD5P3I8RwXKFNtO1Yol8tnRAl2liOPaYWCRMQkOqFj5FUrJ0n1Q+vFmf8kjUmPASD1k22LDbywdOQKHeg9YYtbdg/barXrBSeshw69XqlfDlrjf3Zjjd+FSfUzQCLKS9b3xh4d8t8M+9U8DvcuOynwiL5CoinpgYamfNZkKrsSYyDVyNMh2feCSb1q/9B+lNMoJBQyT6E+S,iv:ab2EzS3CKYDaLtkLUq2O2OKz/bGFeGd39/NUWHeJxPw=,tag:ToRTsuNQIlE3CwTwoyhlaA==,type:str]
+    client-secret: ENC[AES256_GCM,data:aN5d4DLlEtKj1rjhisMuHKoT7+6UMsRjXtc22UIa8sEOvSrEWOiDraA1lziFXv9crl9q8k0SMbKdSYNtXDiIrQ==,iv:ZXHezTA/8oi+8FdKsc2UKAKCLGqh4u4o68IeIeAskMQ=,tag:4U1d99cAq5EEhNh3ftvQzw==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3M1dsQ1pqU3VLN2RGK3Rj
+            dlkzYWF3VThuQVhOVmg2QXFBUlFIcW1BYjFnCkgxQVQ0MmJwY0NmNjRocHdtNXR1
+            LytjZUU1WkEwOXRYVGhab0xMQ2xKaGcKLS0tIGZ5UjNHRjRLSVJlNWQrM2srSktY
+            S0tGYWhJdTdpU3VidGNBeVI4MHVUYWMKOTEqu+u5fWhG/Gwc+GMCY6WBOvNLMwPN
+            bwghAsCteT3k8SuEWql/hLNlfVzFLuPMYFBtUywk2QGhi59ziz+BiQ==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-31T20:59:50Z"
+    mac: ENC[AES256_GCM,data:KU760wyocphtQfBrVc2qlZffj4f+jOOcJowwO4TOiaAd5PqPU/PL+y6gPuT6LpK8Vt9pUhncZDjn/T/9K995oQoNzYgpm5ihyDhQbzHjszzlsA8Kbg1c/PAxy1r1kU8HahvuO9qlYa2k68NO8g0qnFByvR6oIF/lSFavXIaINOw=,iv:u+9OuFSgyx7L9WHBi0vv6GnOm5W5zk5y1N1HEKwS0Fs=,tag:9+H9NsFEKer4hQMs8yql3w==,type:str]
+    version: 3.13.2
@@ -15,8 +15,6 @@ files:
    - forgejo-registry-pull.enc.yaml
    - forgejo-runner-token.enc.yaml
    - forgejo-secrets.enc.yaml
-    - comfyui-oauth2-proxy.enc.yaml
-    - grafana-oidc-secrets.enc.yaml
    - grafana-secrets.enc.yaml
    - homelab-ca-secrets.enc.yaml
    - loki-secrets.enc.yaml
@@ -32,6 +30,16 @@ files:
    - paperless-ai-secrets.enc.yaml
    - memory-agent-oidc.enc.yaml
    - oauth2-credentials.enc.yaml
+    # ZITADEL OIDC client secrets for the consumer apps. These used to live only
+    # in the cluster, written by scripts/iam/zitadel-provision.py, so a rotation
+    # there left every consumer holding a stale secret and no git record of the
+    # real value. Declaring them here makes ArgoCD the owner and the rotation
+    # outcome reviewable in a diff.
+    - argocd-oidc.enc.yaml
+    - grafana-oidc.enc.yaml
+    - minio-oidc.enc.yaml
+    - immich-oidc.enc.yaml
+    - paperless-oidc.enc.yaml
    - forgejo-registry-token.enc.yaml
    - pi-sandbox-secrets.enc.yaml
    - etcd-defrag-creds.enc.yaml
@@ -34,6 +34,8 @@ service:
      lbipam.cilium.io/ips: "192.168.1.161"

 gitea:
+  podAnnotations:
+    reloader.stakater.com/auto: "true"
  admin:
    existingSecret: forgejo-admin

@@ -0,0 +1,123 @@
+# Idempotent repair of the ZITADEL identity links that live inside three
+# application databases.
+#
+# Why this exists: Forgejo, Paperless and Immich each store the link between
+# their local account and the identity provider INSIDE their own database, not
+# in Kubernetes. When the IdP changed from Authentik to ZITADEL, every one of
+# those rows kept the Authentik subject and the provider name. The result was:
+#
+#   Forgejo   -> /user/link_account, never a login
+#   Paperless -> /accounts/3rdparty/signup/, never a login
+#   Immich    -> 400 "email already linked to different account"
+#
+# Each row is keyed on a stable, human-meaningful column (the account email or
+# username) rather than a numeric primary key, so a re-run converges and a
+# fresh database does not need the hard-coded ids this was first discovered on.
+#
+# The ZITADEL subject below is the `sub` claim of the operator account. It is
+# NOT a client id: every one of the three databases stores the subject.
+#
+# Re-run policy: safe. Every statement is an UPDATE that matches its row by
+# email/username, and the Job prunes itself so ArgoCD can recreate it on the
+# next sync.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: iam-identity-link-repair
+  namespace: iam
+  annotations:
+    argocd.argoproj.io/hook: PostSync
+    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
+spec:
+  backoffLimit: 2
+  ttlSecondsAfterFinished: 600
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: iam-identity-link-repair
+        app.kubernetes.io/part-of: iam
+    spec:
+      restartPolicy: Never
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 70 # postgres user in the image, so psql does not run as root
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: repair
+          image: postgres:16-alpine
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          env:
+            # The ZITADEL user that must own all three links.
+            - name: ZITADEL_SUB
+              value: "396924923823522416"
+            - name: LINK_EMAIL
+              value: "[email protected]"
+            - name: LINK_USERNAME
+              value: "rock"
+            # Forgejo
+            - name: FORGEJO_PGHOST
+              value: "forgejo-db-rw.cicd.svc.cluster.local"
+            - name: FORGEJO_PGUSER
+              valueFrom: { secretKeyRef: { name: forgejo-db-app, key: username } }
+            - name: FORGEJO_PGPASSWORD
+              valueFrom: { secretKeyRef: { name: forgejo-db-app, key: password } }
+            - name: FORGEJO_PGDATABASE
+              value: "forgejo"
+            # Paperless
+            - name: PAPERLESS_PGHOST
+              value: "paperless-db-rw.paperless.svc.cluster.local"
+            - name: PAPERLESS_PGUSER
+              valueFrom: { secretKeyRef: { name: paperless-db-app, key: username } }
+            - name: PAPERLESS_PGPASSWORD
+              valueFrom: { secretKeyRef: { name: paperless-db-app, key: password } }
+            - name: PAPERLESS_PGDATABASE
+              value: "paperless"
+            # Immich
+            - name: IMMICH_PGHOST
+              value: "immich-db-rw.immich.svc.cluster.local"
+            - name: IMMICH_PGUSER
+              valueFrom: { secretKeyRef: { name: immich-db-app, key: username } }
+            - name: IMMICH_PGPASSWORD
+              valueFrom: { secretKeyRef: { name: immich-db-app, key: password } }
+            - name: IMMICH_PGDATABASE
+              value: "immich"
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              set -eu
+              psql_dsn() { printf 'postgresql://%s:%s@%s:5432/%s' "$2" "$3" "$1" "$4"; }
+
+              echo "=== Forgejo: link the OIDC subject to the local account ==="
+              # login_source.type = 6 is OAuth2/OIDC. The source may be named
+              # anything ("zitadel" today, "authentik" before), so select on type
+              # and take the single active row rather than assuming a name.
+              psql "$(psql_dsn "$FORGEJO_PGHOST" "$FORGEJO_PGUSER" "$FORGEJO_PGPASSWORD" "$FORGEJO_PGDATABASE")" -v ON_ERROR_STOP=1 -c \
+                "INSERT INTO external_login_user (external_id, user_id, login_source_id, provider) SELECT '$ZITADEL_SUB', u.id, ls.id, 'openidConnect' FROM \"user\" u CROSS JOIN (SELECT id FROM login_source WHERE type = 6 AND is_active LIMIT 1) ls WHERE lower(u.lower_name) = lower('$LINK_USERNAME') ON CONFLICT DO NOTHING;"
+
+              echo "=== Paperless: point the social link at the ZITADEL subject ==="
+              # provider must match the allauth provider_id in
+              # SOCIALACCOUNT_PROVIDERS_JSON, which is "zitadel".
+              psql "$(psql_dsn "$PAPERLESS_PGHOST" "$PAPERLESS_PGUSER" "$PAPERLESS_PGPASSWORD" "$PAPERLESS_PGDATABASE")" -v ON_ERROR_STOP=1 -c \
+                "UPDATE socialaccount_socialaccount sa SET uid = '$ZITADEL_SUB', provider = 'zitadel' FROM auth_user au WHERE au.id = sa.user_id AND lower(au.email) = lower('$LINK_EMAIL');"
+              psql "$(psql_dsn "$PAPERLESS_PGHOST" "$PAPERLESS_PGUSER" "$PAPERLESS_PGPASSWORD" "$PAPERLESS_PGDATABASE")" -v ON_ERROR_STOP=1 -c \
+                "UPDATE socialaccount_socialaccount sa SET provider = 'zitadel' FROM auth_user au WHERE au.id = sa.user_id AND lower(au.email) = lower('$LINK_EMAIL');"
+
+              echo "=== Immich: replace the stale provider subject on the existing user ==="
+              # Immich rejects a callback whose email is already attached to a
+              # different oauthId, so the row must be re-pointed, not inserted.
+              psql "$(psql_dsn "$IMMICH_PGHOST" "$IMMICH_PGUSER" "$IMMICH_PGPASSWORD" "$IMMICH_PGDATABASE")" -v ON_ERROR_STOP=1 -c \
+                "UPDATE \"user\" SET \"oauthId\" = '$ZITADEL_SUB' WHERE lower(email) = lower('$LINK_EMAIL');"
+
+              echo "=== verify ==="
+              psql "$(psql_dsn "$FORGEJO_PGHOST" "$FORGEJO_PGUSER" "$FORGEJO_PGPASSWORD" "$FORGEJO_PGDATABASE")" -tAc \
+                "SELECT 'forgejo:'||external_id FROM external_login_user WHERE external_id='$ZITADEL_SUB';"
+              psql "$(psql_dsn "$PAPERLESS_PGHOST" "$PAPERLESS_PGUSER" "$PAPERLESS_PGPASSWORD" "$PAPERLESS_PGDATABASE")" -tAc \
+                "SELECT 'paperless:'||provider||':'||uid FROM socialaccount_socialaccount WHERE uid='$ZITADEL_SUB';"
+              psql "$(psql_dsn "$IMMICH_PGHOST" "$IMMICH_PGUSER" "$IMMICH_PGPASSWORD" "$IMMICH_PGDATABASE")" -tAc \
+                "SELECT 'immich:'||\"oauthId\" FROM \"user\" WHERE \"oauthId\"='$ZITADEL_SUB';"
+              echo "=== done ==="
@@ -6,6 +6,11 @@ resources:
  # RoleBinding). Authentik is retired, so the file is gone; keep this list in
  # step with the directory or kustomize fails the whole app.
  - smtp-gotify.yaml
+  # Repairs the ZITADEL subject stored inside the Forgejo, Paperless and Immich
+  # databases. Those links are not Kubernetes objects, so without this Job a
+  # fresh cluster (or a provider change) leaves every one of them pointing at
+  # the old subject and SSO cannot complete.
+  - identity-link-repair-job.yaml

 # IAM provisioning is manual-only (security-sensitive).
 # Script: scripts/iam/zitadel-provision.py
@@ -107,6 +107,13 @@ spec:
  # Use external URL for config fetch (hairpins through ingress-nginx); browser
  # redirects use public URLs embedded in the OIDC metadata response.
  env:
+    # MinIO ships identity_openid DISABLED by default, whatever else is set.
+    # Without this key every other MINIO_IDENTITY_OPENID_* value below is
+    # inert and the console advertises password login only — it answers
+    # /api/v1/login with {"loginStrategy":"form"} and never renders an SSO
+    # button, which is exactly the symptom this tenant showed.
+    - name: MINIO_IDENTITY_OPENID_ENABLE
+      value: "on"
    - name: MINIO_IDENTITY_OPENID_CONFIG_URL
      # Must use external URL — well-known response contains external issuer/jwks_uri.
      # MinIO validates issuer in JWT matches well-known issuer. Internal URL = mismatch.
@@ -121,8 +128,9 @@ spec:
          key: MINIO_IDENTITY_OPENID_CLIENT_SECRET
    - name: MINIO_IDENTITY_OPENID_CLAIM_NAME
      value: "groups"
-    - name: MINIO_IDENTITY_OPENID_CLAIM_PREFIX
-      value: ""
+    # An empty claim prefix is worse than no key: it makes the claim resolver
+    # prepend nothing and then look for a nested property that cannot exist.
+    # Omit the key entirely unless a real prefix is needed.
    - name: MINIO_IDENTITY_OPENID_REDIRECT_URI
      value: "https://minio.riotpiao.com/oauth_callback"
    - name: MINIO_IDENTITY_OPENID_DISPLAY_NAME
@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+"""Rotate the OIDC client secret of every owned application, one at a time.
+
+This is the manual execution of what a KeyRotation object declares. It exists so
+the credentials can rotate today, before the controller image is built, and so the
+controller has a reference implementation to match.
+
+For each application: generate a new secret in ZITADEL, write it into the
+Kubernetes Secret, roll the consumer, then prove the consumer still talks to
+ZITADEL. A failure stops the run, because the next application should not start
+while one is in an unknown state.
+"""
+from __future__ import annotations
+
+import json
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent))
+import importlib.util
+
+spec = importlib.util.spec_from_file_location("zp", Path(__file__).with_name("zitadel-provision.py"))
+zp = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(zp)
+
+PROJECT = "396757796898471937"
+ISSUER = "https://zitadel.riotpiao.com"
+
+# app key, ZITADEL app id, Secret name, namespace, Secret key, consumer kind, consumer name
+APPS = [
+    ("grafana",   "grafana",   "grafana-oidc",         "logging",   "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET", "deploy", "grafana"),
+    ("argocd",    "argocd",    "oidc-secret",           "argocd",    "client-secret",                       "deploy", "argocd-server"),
+    ("comfyui",   "comfyui",   "comfyui-oauth2-proxy", "comfyui",   "client-secret",                       "deploy", "comfyui-oauth2-proxy"),
+    ("minio",     "minio",     "minio-oidc",           "storage",   "MINIO_IDENTITY_OPENID_CLIENT_SECRET",  "sts",    "minio-cluster-az-a"),
+    ("paperless", "paperless", "paperless-oidc",       "paperless", "client-secret",                       "deploy", "paperless"),
+    ("immich",    "immich",    "immich-oidc",          "immich",    "client-secret",                       "deploy", "immich-server"),
+    ("forgejo",   "forgejo",   "forgejo-oidc",         "cicd",      "CLIENT_SECRET",                       "deploy", "forgejo-gitea"),
+]
+
+
+def sh(*args: str, timeout: int = 120) -> tuple[int, str]:
+    p = subprocess.run(args, capture_output=True, text=True, timeout=timeout)
+    return p.returncode, (p.stdout + p.stderr).strip()
+
+
+def state() -> dict:
+    return json.load(open(Path.home() / ".config/zitadel/state.json"))
+
+
+def app_ids(token: str) -> dict[str, str]:
+    """Map clientId to the app resource id. The two are different, and the secret
+    endpoint takes the resource id."""
+    code, body = zp._api("POST", f"/management/v1/projects/{PROJECT}/apps/_search", token,
+                         {"query": {"limit": 100, "offset": 0}})
+    out = {}
+    for app in body.get("result", []) or []:
+        cid = (app.get("oidcConfig") or {}).get("clientId")
+        if cid and app.get("id"):
+            out[cid] = app["id"]
+    return out
+
+
+def new_secret(token: str, app_id: str) -> str | None:
+    code, body = zp._api("POST", f"/management/v1/projects/{PROJECT}/apps/{app_id}/oidc_config/_generate_client_secret", token)
+    if code != 200:
+        print(f"    generate failed: {code} {str(body)[:120]}")
+        return None
+    return body.get("clientSecret")
+
+
+def patch_secret(ns: str, name: str, key: str, value: str) -> bool:
+    import base64
+    enc = base64.b64encode(value.encode()).decode()
+    patch = json.dumps({"data": {key: enc}})
+    code, out = sh("kubectl", "-n", ns, "patch", "secret", name, "--type=merge", "-p", patch)
+    if code != 0:
+        print(f"    secret patch failed: {out[:120]}")
+    return code == 0
+
+
+def roll(kind: str, ns: str, name: str) -> bool:
+    code, out = sh("kubectl", "-n", ns, "rollout", "restart", f"{kind}/{name}", timeout=120)
+    if code != 0:
+        print(f"    rollout restart failed: {out[:120]}")
+        return False
+    code, out = sh("kubectl", "-n", ns, "rollout", "status", f"{kind}/{name}", "--timeout=90s", timeout=120)
+    return code == 0
+
+
+def main() -> int:
+    only = sys.argv[1:] or None
+    token = zp.bootstrap_token()
+    if not token:
+        print("no bootstrap token")
+        return 2
+    cids = state()["client_ids"]["value"]
+    ids = app_ids(token)
+    print(f"resolved {len(ids)} app resource ids")
+    done, failed = [], []
+    for key, app_id, sname, ns, skey, kind, cname in APPS:
+        if only and key not in only:
+            continue
+        print(f"[{key}] issuer={ISSUER} client_id={cids[key]}")
+        rid = ids.get(cids[key])
+        if not rid:
+            print(f"[{key}] no app resource id for client {cids[key]}")
+            failed.append(key)
+            continue
+        secret = new_secret(token, rid)
+        if not secret:
+            failed.append(key)
+            continue
+        if not patch_secret(ns, sname, skey, secret):
+            failed.append(key)
+            continue
+        if not roll(kind, ns, cname):
+            failed.append(key)
+            print(f"[{key}] FAILED: consumer did not become ready")
+            continue
+        code, out = sh("kubectl", "-n", ns, "get", "pods", "-l",
+                       f"app={cname}" if kind == "deploy" else f"app.kubernetes.io/name={cname}",
+                       "--no-headers")
+        ready = "Running" in out
+        print(f"[{key}] rotated, consumer {'ready' if ready else 'NOT ready'}")
+        (done if ready else failed).append(key)
+    print(f"\nrotated: {len(done)} {done}\nfailed: {len(failed)} {failed}")
+    return 0 if not failed else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
@@ -87,6 +87,12 @@ ROLES = {
    # LLM gate for the api-gw. The colon form matches the Authentik-era
    # permission string, so consumer manifests did not change.
    "llm:inference": "LLM inference access (api-gw)",
+    # Capability for machine-to-machine callers. Kept separate from the human roles
+    # so a service account can be identified by its grant alone. Note that ZITADEL
+    # does not assert roles into a machine-user access token on this instance (see
+    # MACHINE_CALLERS), so this grant documents intent; the gateway authorises the
+    # caller from its allow-list entry instead.
+    "svc-caller": "Machine-to-machine caller",
    "llm-inference": "LLM inference access (dash alias)",
    # Per-service capabilities. The gateway derives the required capability from
    # the verb: GET/HEAD/OPTIONS -> "<service>:read", anything else ->
@@ -143,8 +149,13 @@ OIDC_APPS = {
        "https://paperless.riotpiao.com/accounts/oidc/zitadel/login/callback/",
        "http://localhost:8000/accounts/oidc/authentik/login/callback/",
        "https://paperless.riotpiao.com/accounts/oidc/authentik/login/callback/"]},
+    # The frontend calls back at /auth/callback (see OAUTH_CALLBACK_PATH in
+    # poimen-frontend/lib/oauth.ts). The bare /callback spelling stays registered so
+    # an older image still completes its login.
    "poimen": {**_WEB, "redirect_uris": [
-        "http://localhost:3000/callback", "https://poimen.riotpiao.com/callback"]},
+        "http://localhost:3000/callback", "https://poimen.riotpiao.com/callback",
+        "http://localhost:3000/auth/callback",
+        "https://poimen.riotpiao.com/auth/callback"]},
    "queue-sqs": {**_WEB, "redirect_uris": [
        "http://localhost:8080/callback", "https://queue.riotpiao.com/callback"]},
    "vault": {**_WEB, "redirect_uris": [
@@ -340,6 +351,55 @@ def _ensure_builtin_login(token: str) -> None:
    print("login page    : built-in (loginV2.required=false)")


+# One machine user per cross-service caller. Each has its own key, so revoking one
+# service leaves the others working, and ZITADEL can overlap keys during a rotation.
+MACHINE_CALLERS = {
+    "poimen-llm": {
+        "name": "Poimen LLM caller",
+        "description": "poimen calls api-gateway inference",
+        "roles": ["llm:inference", "svc-caller"],
+    },
+}
+
+
+def _ensure_machine_users(token: str, project_id: str) -> dict[str, dict]:
+    """Create each caller once and grant its roles. Returns id and key id per caller.
+
+    Roles are granted even though ZITADEL does not assert them into a machine-user
+    token on this instance: the grant is the record of what the account may do, and
+    it becomes effective the moment that behaviour changes.
+    """
+    out: dict[str, dict] = {}
+    for user_name, spec in MACHINE_CALLERS.items():
+        code, found = _api("POST", "/management/v1/users/_search", token,
+                           {"queries": [{"userNameQuery": {"userName": user_name,
+                                                           "method": "TEXT_QUERY_METHOD_EQUALS"}}]})
+        user_id = ""
+        for user in (found.get("result") or []):
+            user_id = user.get("id", "")
+        if not user_id:
+            code, created = _api("POST", "/management/v1/users/machine", token, {
+                "userName": user_name,
+                "name": spec["name"],
+                "description": spec["description"],
+                "accessTokenType": "ACCESS_TOKEN_TYPE_JWT",
+            })
+            user_id = created.get("userId", "")
+        if not user_id:
+            continue
+        code, grant = _api("POST", "/management/v1/users/grants/_search", token,
+                           {"query": {"limit": 100, "offset": 0}})
+        existing = next((g for g in (grant.get("result") or []) if g.get("userId") == user_id), None)
+        if existing:
+            _api("PUT", f"/management/v1/users/{user_id}/grants/{existing['id']}", token,
+                 {"roleKeys": spec["roles"]})
+        else:
+            _api("POST", f"/management/v1/users/{user_id}/grants", token,
+                 {"projectId": project_id, "roleKeys": spec["roles"]})
+        out[user_name] = {"userId": user_id}
+    return out
+
+
 def _ensure_roles(token: str, project_id: str, roles: dict[str, str]) -> None:
    code, d = _api("POST", f"/management/v1/projects/{project_id}/roles/_search", token, {})
    # ZITADEL answers with `key`, not `roleKey`. Reading the wrong field left
@@ -497,6 +557,9 @@ def phase_apply(_args: argparse.Namespace) -> int:
    project_id = _ensure_project(token, PROJECT_NAME)
    print(f"project   : {PROJECT_NAME} ({project_id})")
    _ensure_roles(token, project_id, ROLES)
+    machine_users = _ensure_machine_users(token, project_id)
+    for name, info in machine_users.items():
+        print(f"machine user {name}: {info['userId']}")
    print(f"roles     : {len(ROLES)} ensured")
    known: dict[str, str] = {}
    try:
@@ -39,6 +39,9 @@ resource "local_file" "controlplane_configs" {
    dns_servers            = var.cluster_config.dns_servers
    forgejo_registry_ip    = var.forgejo_registry_ip
    forgejo_hostname       = var.forgejo_hostname
+    oidc_issuer_hostname   = var.oidc_issuer_hostname
+    oidc_issuer_ip         = var.oidc_issuer_ip
+    oidc_client_id         = var.oidc_client_id
    zone                   = each.value.zone
    allow_scheduling       = each.value.allow_scheduling
    coredns_corefile       = file("${path.module}/files/coredns/Corefile")
@@ -33,6 +33,14 @@ machine:
      - ip: ${forgejo_registry_ip}
        aliases:
          - ${forgejo_hostname}
+      # kube-apiserver's static pod runs on the host network, so it resolves the
+      # OIDC issuer through THIS node's DNS, not cluster CoreDNS. Pin the issuer
+      # to the ingress LoadBalancer so discovery stays on the LAN and presents
+      # the publicly-trusted Let's Encrypt wildcard certificate. Without this
+      # entry the lookup leaves via public DNS and Cloudflare.
+      - ip: ${oidc_issuer_ip}
+        aliases:
+          - ${oidc_issuer_hostname}
  kubelet:
    image: ${kubelet_image}
    defaultRuntimeSeccompProfileEnabled: true
@@ -120,10 +128,31 @@ cluster:
      - ${san}
 %{ endfor ~}
    image: ${kube_apiserver_img}
-    # No OIDC block. The cluster authenticates with client certificates. An
-    # earlier Authentik-based kube-apiserver OIDC config was removed from the
-    # live nodes, and leaving it here would re-add an issuer that no longer
-    # resolves on any future machine-config render.
+    # OIDC authentication against ZITADEL. Additive: client-certificate auth
+    # (this session's own admin@homelab-cluster kubeconfig, system:masters)
+    # keeps working unchanged, so a bad value here degrades OIDC logins rather
+    # than locking anyone out.
+    #
+    # The issuer has NO trailing slash and NO /application/o/<app>/ suffix.
+    # That suffix belonged to the Authentik scheme; the apiserver compares `iss`
+    # in the id_token by exact string equality, so the stale value made every
+    # OIDC login fail once Authentik stopped running.
+    #
+    # Groups: ZITADEL asserts roles as a nested urn claim that the apiserver
+    # cannot read. The `groupsClaim` action in ZITADEL mirrors the bare role
+    # names into a flat `groups` claim, which is what oidc-groups-claim consumes.
+    # Verified end to end: a PKCE id_token for this client carries
+    # groups=[homelab-admins, ...], email and preferred_username.
+    #
+    # Prefixes keep OIDC identities from colliding with built-in system:* users
+    # and groups. k8s RBAC bindings therefore target `oidc:homelab-admins`.
+    extraArgs:
+      oidc-issuer-url: https://${oidc_issuer_hostname}
+      oidc-client-id: "${oidc_client_id}"
+      oidc-username-claim: email
+      oidc-groups-claim: groups
+      oidc-username-prefix: "oidc:"
+      oidc-groups-prefix: "oidc:"
    admissionControl:
      - name: PodSecurity
        configuration:
@@ -198,3 +198,42 @@ variable "forgejo_hostname" {
  default     = "forgejo.riotpiao.com"
  description = "Forgejo external hostname"
 }
+
+# ── kube-apiserver OIDC (ZITADEL) ────────────────────────────────────────────
+# Authenticates `kubectl` against ZITADEL instead of client certificates only.
+# Client-certificate auth keeps working in parallel, so a wrong value here
+# degrades OIDC logins without locking anyone out.
+#
+# The issuer is ZITADEL's external host with NO trailing slash and NO
+# /application/o/<app>/ suffix — a leftover of the Authentik scheme. The
+# apiserver validates `iss` in the id_token for exact string equality.
+variable "oidc_issuer_hostname" {
+  type        = string
+  default     = "zitadel.riotpiao.com"
+  description = "Hostname of the OIDC issuer (ZITADEL). Becomes https://<host> as the issuer."
+}
+
+variable "oidc_issuer_ip" {
+  type        = string
+  default     = "192.168.1.160"
+  description = <<-EOT
+    Address the nodes resolve the OIDC issuer to, via extraHostEntries.
+
+    kube-apiserver's static pod runs on the host network, so it resolves the
+    issuer through the node's own DNS, not cluster CoreDNS. Pointing it at the
+    ingress-nginx LoadBalancer keeps OIDC discovery on the LAN and serves the
+    publicly-trusted Let's Encrypt *.riotpiao.com certificate. Without this the
+    lookup leaves via public DNS and Cloudflare, and discovery either hairpins
+    or fails.
+  EOT
+}
+
+variable "oidc_client_id" {
+  type        = string
+  default     = "396757803089330177"
+  description = <<-EOT
+    ZITADEL client id of the `kubernetes` OIDC application (PKCE, no secret).
+    Must be a STRING: the id is an 18-digit number and an unquoted YAML scalar
+    would be parsed as a float and rejected.
+  EOT
+}
Author	SHA1	Message	Date
rock	04af9f15c3	feat(talos): authenticate kube-apiserver against ZITADEL The apiserver's live OIDC config still pointed at https://authentik.riotpiao.com/application/o/kubernetes/. Authentik now runs zero pods, so that issuer cannot be discovered and every attempt to login with a token instead of a client certificate fails. The template itself carried no OIDC block at all, so a render would have removed the stale config rather than replace it. Add the ZITADEL issuer, the `kubernetes` client id, and the groups/username claims. Three details had to be verified against a real token first: 1. The issuer is https://zitadel.riotpiao.com with no trailing slash and no /application/o/<app>/ suffix. That suffix belonged to the Authentik scheme; the apiserver compares `iss` by exact string equality. 2. ZITADEL does NOT put profile/email claims in the id_token by default, so `oidc-username-claim: email` resolved to nothing. idTokenUserinfoAssertion is now enabled on the `kubernetes` application. 3. ZITADEL asserts roles as a nested urn claim the apiserver cannot read. The `groupsClaim` action (created alongside this change) mirrors the bare role names into a flat `groups` claim. Verified with a full PKCE authorization-code flow against this client: the id_token carries iss=https://zitadel.riotpiao.com, email, preferred_username and groups=[homelab-admins, ...]. With oidc-groups-prefix=oidc: that maps to the existing ClusterRoleBinding homelab-admin-oidc, which grants cluster-admin. The issuer hostname is pinned in extraHostEntries because the apiserver's static pod resolves through the node's DNS, not cluster CoreDNS; the ingress LoadBalancer serves the publicly-trusted Let's Encrypt wildcard certificate. Client-certificate auth is unchanged, so a wrong value degrades OIDC logins without locking anyone out.	2026-10-31 14:27:36 -07:00
rock	894d746b0d	fix(paperless): align stale client-secret with the live OIDC secret (#86 ) Co-authored-by: rock <[email protected]>	2026-10-31 21:10:26 +00:00
rock	4e84571f0c	fix(iam): make ZITADEL OIDC secrets and identity links IaC-owned (#85 ) Co-authored-by: rock <[email protected]>	2026-10-31 20:53:53 +00:00
rock	41dcda819e	feat(llm-serving): manage clef-serve in git, generated from app.py The Clef runtime was running from hand-applied objects that existed only in the cluster. A rebuild would have lost it. Three things move into git: - clef-serve.yaml: the Deployment and the Service the gateway routes to - configMapGenerator for clef-serve/app.py, so an edit to the code rolls the pod instead of leaving a stale copy mounted - clef-flash.yaml is deleted: it described an SGLang predictor that cannot run on a V100, and its slot is now clef-serve.yaml Recorded in the manifest, because both cost hours: SGLang needs CUDA 13 for Clef and CUDA 13 has no sm_70 kernels, and device_map over two cards breaks the forward pass with 'indices should be either on cpu or on the same device as the indexed tensor'. Still open and written down in the file: the image cannot be pushed, because Cloudflare answers 413 for layers above 100 MB. So each start pip-installs and needs PyPI.	2026-10-29 23:23:03 -07:00
rock	3eb6a91311	feat(llm-serving): serve Qwen3-30B-A3B int4 as reasoning, on one card Replaces the Qwen3-32B dense deployment, which needed two V100s for 131k context. The MoE carries 96 KiB of KV per token against the dense model's 256 KiB, so the same 64k window fits on one card. Measured on this cluster before the switch: 15.55 GiB of weights, 137k tokens of KV cache, 2.09x concurrency at 65536 tokens per request, correct answer over a 5921-token prompt, and clean reasoning_content separation through the gateway. Three settings are load-bearing on sm_70 rather than tuning: - VLLM_USE_FLASHINFER_SAMPLER=0 flashinfer's check_cuda_arch raises on CC 7.0 - VLLM_ATTENTION_BACKEND=TRITON_ATTN FlashInfer and FA2 both need sm75+ - deploymentStrategy Recreate a rolling update deadlocks once every card is allocated The manifest is committed as well as applied: reasoning.yaml had never been in git.	2026-10-29 03:56:17 -07:00
rock	ad2720a724	fix(llm-serving): stop shipping clef-flash, it cannot run on a V100 Clef needs SGLang v0.5.22+, which upstream publishes only on CUDA 13. CUDA 13 dropped Volta kernels and every GPU here is a Tesla V100 (sm_70): torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device SGLang also states sm75 as its floor: the README hardware table omits V100, sm75 support was removed in sgl-project/sglang#9207, and FlashInfer requires sm75+. No build has both Clef and Volta kernels, so the model is not runnable here at any dtype or flag. Remove it from the kustomization so nothing crash-loops, and record the reason in both files. The manifest stays in git for the day an Ampere-or-newer GPU arrives.	2026-10-28 10:29:24 -07:00
rock	695f03a0ab	feat(forgejo): record the CNPG-generated database password forgejo-db-app is owned by the Cluster CR and was generated by CNPG: there is no bootstrap secret, so nothing outside cloudnative-pg ever knew that password. CNPG reads spec.bootstrap.initdb.secret only at creation. Recreate the Cluster CR and CNPG mints a new password while the data directory keeps the old one — Forgejo starts, stays Ready, and fails every query. That is a credential fault wearing a network fault's clothes, and it is exactly the shape of tonight's outage. This file is the record. Point spec.bootstrap.initdb.secret at it when the cluster is rebuilt. The value matches the live Secret today; verified by decrypting and comparing.	2026-10-28 03:14:38 -07:00
rock	15a22abb1f	fix(llm-serving): clef-flash cannot run on a V100 The CLI path was wrong and is now correct, so the pod reaches the model loader — and dies there: Found GPU0 Tesla V100-PCIE-32GB which is of cuda capability 7.0. Your installed torch==2.14.1+cu130 does not include kernels for this GPU. Every GPU on worker-1 is a V100 (CC 7.0) and the pinned dev-clef image ships kernels for CC 8.0 and above. Scale to zero so the pod stops crash-looping, and record the reason in the manifest. Serving a 9.4B backbone on CPU is not a substitute.	2026-10-28 02:41:47 -07:00
rock	569eb84737	fix(llm-serving): call the sglang CLI by its full path The container args began with a bare 'serve'. This image installs its CLI at /opt/sglang/bin/sglang and puts nothing called serve on PATH, so every start ended with 'exec: serve: not found' — 19 restarts, and the InferenceService never became ready. Name the binary, then the subcommand.	2026-10-28 02:34:42 -07:00
rock	13343976e4	fix(forgejo): the chart key is gitea.podAnnotations, not podAnnotations The gitea chart reads pod annotations from gitea.podAnnotations (values.yaml line 583). A top-level key is silently ignored, so the previous commit changed nothing — the Deployment came back without the annotation and Reloader still had no trigger.	2026-10-28 02:18:29 -07:00
rock	86cdcade36	fix(forgejo): restart the pod when the database password changes GITEA__DATABASE_PASSWD comes from forgejo-db-app, which the CNPG Cluster CR owns and generates. There is no bootstrap secret, so nothing but cloudnative-pg knows that password. The Deployment reads it into the environment at start. So a regenerated password leaves Forgejo authenticating with a value the database no longer accepts: the pod stays Ready, every query fails, and the symptom looks like a network problem rather than a credential problem. The cnpg.io/reload label on the Secret is only a hint; this annotation is what makes Reloader act on it.	2026-10-28 02:13:31 -07:00
rock	d8ef81e9af	fix(paperless-ai): ask for no project audience, so aud names the caller The gateway compares the token audience with its allow-list of client ids. The scope carried urn:zitadel:iam:org:project🆔<project>:aud, so every token said aud = [396757796898471937] and the gateway answered token validation failed: audience not allowed: [396757796898471937] With plain openid the audience is the machine user, which is the entry that exists on the list.	2026-10-28 01:34:23 -07:00
rock	1f8d254037	fix(paperless-ai): let the token sidecar read its own key A Secret volume is owned by root and the sidecar runs as 65532, so defaultMode 0400 left it unable to open /var/run/llm-key/key: the container restarted with 'permission denied' on every start and never minted a token. Set mode 0440 and give the pod fsGroup 65532, so the group that owns the volume can read the key while nothing else on the host can.	2026-10-28 00:09:24 -07:00
rock	f6e569c0e8	fix(llm-serving): clef-flash asks for memory the node can reserve worker-1 holds 64Gi and the other predictors request about 55Gi, so the 16Gi request left clef-flash Pending with '0/4 nodes are available: 1 Insufficient memory'. The weights live on the GPU; host memory covers SGLang's own bookkeeping, so 8Gi is the honest reservation. The 32Gi limit stays, so a burst still gets the room.	2026-10-27 23:59:28 -07:00
rock	a0d2859c1d	fix(llm-serving): podMetadata is not a field on this KServe CRD The installed InferenceService CRD declares annotations and labels directly under spec.predictor, not under spec.predictor.podMetadata. Two manifests used podMetadata, so every diff and apply failed with 'field not declared in schema' and the whole Application could not reconcile. Move the prometheus scrape annotations and the labels up one level. Nothing is lost: the same keys, in the place the CRD accepts.	2026-10-27 23:55:54 -07:00
rock	62857c5fcb	fix(llm-serving): let the kustomization build again The resource list named ornith.yaml and reasoning.yaml, and neither file has ever been committed to this directory. kustomize therefore failed to build, the Application sat in ComparisonError, and nothing in this directory could reconcile — including the clef-flash predictor added a commit earlier. Removing the entries does not remove the models: the Application runs with prune=false, so an object it no longer lists is left untouched. Re-export both from live state the next time either is edited, so the directory describes what actually runs.	2026-10-27 23:54:19 -07:00
rock	9aec1b3e30	feat(llm-serving): serve clef-flash The gateway already dispatches /v1/systemone and its config already names clef-flash-predictor, but this directory is an explicit allowlist and clef-flash.yaml was not on it, so the model was silently dropped: no error, no drift, just no predictor. clef-flash is a decision model, not a chat model. It reads a state plus a schema of typed questions and returns one probability per allowed option in a single prefill pass, with no decode step. It needs SGLang rather than vLLM because it loads separate joint_schema_head weights on top of the Qwen3.5-9B backbone and it implements the /v1/systemone route. worker-1 has four GPUs: reasoning holds two, ornith holds one, so one is free and this fits without retiring ornith first.	2026-10-27 23:51:45 -07:00
rock	4d23c983ab	feat(paperless-ai): mint the LLM token with a machine key, not an init container The init container fetched a token once at pod start with a client_credentials grant against a web app. A web app has no such grant, so the fetch could not work; and a token fetched once is dead within the hour while the pod runs for weeks. llm-token-vendor runs beside the app. It signs a machine-key assertion for the paperless-llm service account, exchanges it for a token, and rewrites /app/data/llm_token.txt atomically at 80% of the token lifetime. The vendor image keeps using CUSTOM_API_KEY_FILE, so it needs no change and no OIDC support. The key is re-read on every mint, so a rotation lands without a restart: publish the new key, the next mint uses it, retire the old one. paperless-llm is a separate service account from poimen-llm, so revoking one caller leaves the other working.	2026-10-27 23:48:45 -07:00
rock	bd81503e33	feat(iam): a role and a machine user for each cross-service caller Adds svc-caller, a capability that names a machine-to-machine caller, separate from the human roles so a service account can be identified by its grant alone. Adds MACHINE_CALLERS: one ZITADEL machine user per caller, created once with its own key and granted its roles. poimen-llm is the first, granted llm:inference and svc-caller. Its key is separate from every other caller's, so revoking one service leaves the rest working and ZITADEL can overlap keys during a rotation. The role is granted even though ZITADEL does not assert roles into a machine-user access token on this instance. The grant is the record of what the account may do, and it takes effect the moment that behaviour changes. Until then the gateway authorises the caller from its allow-list entry.	2026-10-27 23:40:54 -07:00
rock	e599c12f91	chore(iam): retire the minio OIDC credential from sops storage/minio-oidc is owned by the KeyRotation object iam/minio now, so its value leaves git. minio-creds and minio-user-ollama stay: they are the root console credential and an application key, and neither rotates through ZITADEL yet. The file keeps a header that says so, because a prune of the sops-secrets application would delete that Secret and the controller recreates it on the next rotation.	2026-10-27 20:00:15 -07:00
rock	c5058dadc6	fix(iam): merge into the consumer Secret, and retry with a backoff key-rotation-controller / image (push) Failing after 6s Details Two defects that a real rotation exposed: merge writeSecret replaced the whole Secret map, so every key a consumer keeps beside the credential disappeared. paperless lost SOCIALACCOUNT_PROVIDERS_JSON and immich lost config.json; both pods then failed to start while the rotation reported success. backoff a Failed attempt returned 0, so the controller retried in a tight loop. It now waits two minutes and then retries. All eight objects carry both identifiers ZITADEL needs: the app resource id for the secret endpoint and the client id for introspection. Result: seven rotations reach Succeeded with every consumer ready (grafana, comfyui, argocd, forgejo, paperless, immich, minio). api-gw stays suspended because no workload exists for it.	2026-10-27 19:59:23 -07:00
rock	e800e2cf1c	fix(iam): make a rotation attempt survive a failure key-rotation-controller / image (push) Failing after 19s Details Three defects, each seen in a live rotation of grafana: scope the access token needs urn:zitadel:iam:org:project:id:zitadel:aud, otherwise the management API answers 401 Token.Invalid even though the assertion was accepted. app id the objects carry the client id, and the secret endpoint takes the app resource id. They differ, so every call answered 404 Application doesn't exist. The correct ids are recorded per object. attempt time record ActiveCredentialID and LastRotation right after Prepare. Without it a later failure leaves LastRotation empty, the due check says rotate now, and each reconcile mints a new secret and restarts the consumer: grafana was rolled continuously and never became ready. Also raise the consumer-ready budget to 8 minutes, because grafana needs about four, and retry a failed attempt after 2 minutes instead of waiting a whole interval.	2026-10-27 18:39:59 -07:00
rock	3630721366	feat(iam): run the controller with the RBAC and image it needs key-rotation-controller / image (push) Failing after 5s Details Three faults stopped the controller from running, each found by running it: image arch the first build was arm64 and every node is amd64, so the pod died with 'exec format error'. imagePullPolicy IfNotPresent kept the cached wrong-arch layer, so it is now Always. leases leader election needs coordination.k8s.io/leases. Without it the controller never acquired its lock and never reconciled. secrets list the manager caches Secrets, so it needs list and watch, not only get, create, update and patch. go.mod and go.sum gained the procfs entry the build needs.	2026-10-27 17:46:25 -07:00
rock	7cd7e8808c	ci(iam): move the workflow where this Forgejo reads it key-rotation-controller / image (push) Failing after 6s Details Forgejo reports zero workflows for this repository even though .forgejo/workflows/key-rotation-controller.yaml is on main. The poimen-frontend workflows that do run live under .gitea/workflows/, so this instance reads that path only. Move the file there.	2026-10-27 17:27:49 -07:00
rock	280bbdf600	feat(iam): register the poimen /auth/callback redirect The frontend's callback path is /auth/callback. The old /callback spelling stays in the list so an image built before this change still completes a login.	2026-10-27 14:54:30 -07:00
rock	60931fbba1	ci(iam): build and push the key-rotation-controller image Dockerfile: static binary on distroless, no shell, user 65532. Workflow runs on the golang runner for a push to main that touches the controller. It builds two tags, the commit sha and 0.1.0, which is the tag the Deployment asks for, and runs the image with --help so a broken binary fails the job instead of the cluster. The runners were dead before this: every task fetch logged 'lookup forgejo-gitea-http.cicd.svc.cluster.local: no such host'. DNS resolved from a fresh pod and the Service had a healthy endpoint, so the runner pods held stale network state. A rollout restart cleared it and all three poll without error.	2026-10-27 14:50:33 -07:00
rock	306c68e7ea	chore(iam): retire the .enc files whose values rotation invalidated grafana-oidc and comfyui-oauth2-proxy held nothing but the rotated credential, so both files are gone. oauth2-credentials survives with the two secrets that were not rotated (api-gw, poimen) and drops the three that were (grafana, minio, paperless). Leaving them would be worse than untidy: sops-secrets ran with selfHeal, so a stale file silently rewrites a rotated secret back to a value ZITADEL no longer accepts. 32 files remain, down from 34.	2026-10-27 14:47:48 -07:00
rock	36d6dad956	feat(iam): script that rotates an OIDC client secret end to end Generates a new secret in ZITADEL, writes it into the Kubernetes Secret, rolls the consumer and checks it comes back. This is the manual execution of a KeyRotation, and the reference the controller must match. Two traps it records: the app resource id differs from the client id, and the secret endpoint is oidc_config/_generate_client_secret.	2026-10-27 14:42:27 -07:00