fix(longhorn): isolate GPU worker from general storage scheduling

- Remove expand-replicas-job (blindly forced all volumes to 3 replicas,
  ignoring StorageClass settings)
- Add diskSelector: 'storage' to longhorn and longhorn-cnpg StorageClasses
  so replicas only land on CP nodes (cp-1, cp-2, cp-3)
- Tag all CP node disks with 'storage' via PostSync job
  (disk names are runtime-discovered, can't hardcode in Node CRs)
- Disable scheduling on worker-1 Node CR — only longhorn-llm-local
  (diskSelector: 'llm') can use it
- worker-1 is GPU-only: llm-models and comfyui use dedicated SCs

.forgejo/CI-CD.md

@@ -1,460 +0,0 @@
-# CI/CD Pipeline: GitOps Validation & Deployment
-
-## Overview
-
-Pure GitOps CI/CD pipeline using Forgejo Actions (self-hosted runner).
-
-**Principle:** Validate in CI, deploy via ArgoCD (no manual steps).
-
-```
-git push
-  ↓
-  [CI: Validate]
-    ├─ yamllint (YAML syntax)
-    ├─ kubeval (K8s manifests)
-    ├─ kustomize build (all layers)
-    ├─ argocd validation (app definitions)
-    └─ security scan (secrets, best practices)
-  ↓
-  [If push to main]
-    └─ ArgoCD auto-syncs (if enabled)
-```
-
-## Workflows
-
-### 1. validate-k8s.yaml (Mandatory)
-
-**Trigger:** Any push/PR with k8s/ changes
-
-**What it does:**
-1. Lints all YAML files (`yamllint`)
-2. Validates K8s manifests (`kubeval`)
-3. Builds all kustomization layers
-4. Validates ArgoCD applications
-5. Reports results
-
-**Duration:** ~2-3 minutes
-
-**Status:**
-- ✅ PASS: All layers build, manifests valid → OK to merge
-- ❌ FAIL: Syntax error, invalid resource, build failed → Fix & push again
-
-**Example output:**
-```
-=== Building k8s/infrastructure/ ===
-✓ Infrastructure built successfully
-Resources: 47
-
-=== Building k8s/bootstrap/ ===
-✓ Bootstrap built successfully
-Resources: 23
-```
-
-**When to check:**
-- After every commit
-- Before merging PRs
-- On every branch
-
-### 2. argocd-sync.yaml (Recommended)
-
-**Trigger:** Push to main only (k8s/ changed)
-
-**What it does:**
-1. Authenticates with ArgoCD
-2. Syncs `homelab-root` application
-3. Waits for sync to complete (5 min timeout)
-4. Verifies all applications healthy
-
-**Duration:** 1-5 minutes (depends on resources)
-
-**Status:**
-- ✅ SYNCED: All resources deployed to cluster
-- ❌ FAILED: Sync error, pod crashes, etc. → Check ArgoCD UI for details
-
-**When it runs:**
-- Automatically after merge to main
-- Only on k8s/ changes (not on docs)
-
-**Manual trigger (if needed):**
-```bash
-# SSH to runner or use Forgejo UI
-# Re-run failed workflow
-# Or manually sync: argocd app sync homelab-root
-```
-
-**Requires secrets:**
-- `ARGOCD_SERVER`: ArgoCD server URL (https://argocd.riotpiao.com)
-- `ARGOCD_AUTH_TOKEN`: ArgoCD API token (generate via ArgoCD UI)
-
-### 3. security-scan.yaml (Optional)
-
-**Trigger:** Any push/PR with k8s/ changes
-
-**What it does:**
-1. Scans Dockerfiles for vulnerabilities (`trivy`)
-2. Scans Helm charts for security issues
-3. Audits K8s manifests (`polaris`)
-4. Checks for hardcoded secrets
-5. Verifies security best practices
-
-**Duration:** ~3-5 minutes
-
-**Status:**
-- ✅ PASS: No critical issues
-- ⚠️ WARNING: Best practice recommendations (non-blocking)
-- ❌ FAIL: Hardcoded secrets found (must fix)
-
-**Common issues:**
-- Missing resource limits (warning)
-- Privileged containers (warning)
-- Hardcoded passwords (ERROR)
-
----
-
-## File Structure
-
-```
-.forgejo/
-├── workflows/                      # CI/CD workflows
-│   ├── validate-k8s.yaml          # Validate manifests (required)
-│   ├── argocd-sync.yaml           # Sync to cluster (auto on main)
-│   └── security-scan.yaml         # Security checks (optional)
-└── CI-CD.md                        # This file
-```
-
----
-
-## Setup Instructions
-
-### 1. Install Forgejo Runner
-
-```bash
-# On runner machine (inside cluster or external)
-forgejo-runner register \
-  --instance https://forgejo.riotpiao.com \
-  --token <registration-token> \
-  --name homelab-runner \
-  --labels docker
-
-forgejo-runner daemon
-```
-
-### 2. Add ArgoCD Secrets to Forgejo
-
-```bash
-# Go to: Forgejo → Settings → Secrets
-
-# Add:
-ARGOCD_SERVER = https://argocd.riotpiao.com
-ARGOCD_AUTH_TOKEN = <token>  # Generate: argocd account generate-token
-```
-
-### 3. Generate ArgoCD Token
-
-```bash
-# Inside cluster
-kubectl -n argocd port-forward svc/argocd-server 8080:443
-
-# Go to: https://localhost:8080/user-info/api-tokens
-# Create new token (CI/CD)
-# Copy token to Forgejo secrets
-```
-
----
-
-## Workflow Execution
-
-### When developer pushes to feature branch:
-
-```
-git push origin feature/new-service
-
-↓
-Forgejo Actions triggered
-↓
-validate-k8s.yaml runs:
-  ✓ Lints YAML
-  ✓ Validates manifests
-  ✓ Builds kustomizations
-  ✓ All pass → GitHub comment: "Ready to merge"
-↓
-Developer opens PR
-↓
-Reviewer checks:
-  - Code changes (YAML)
-  - Workflow results
-  - ArgoCD impact (diff)
-↓
-PR merged to main
-```
-
-### When merged to main:
-
-```
-git merge feature/new-service → main
-
-↓
-Forgejo Actions triggered
-↓
-validate-k8s.yaml runs:
-  ✓ Same validation as above
-↓
-argocd-sync.yaml runs (if enabled):
-  ✓ Syncs homelab-root
-  ✓ Waits for sync
-  ✓ Verifies health
-  ✓ Resources deployed to cluster
-↓
-Cluster state = git state
-(No manual kubectl apply needed!)
-```
-
----
-
-## Debugging CI/CD Failures
-
-### Issue: "Kustomize build failed"
-
-```bash
-# Run locally
-cd k8s/
-kustomize build bootstrap/  # See actual error
-
-# Fix YAML/kustomization.yaml
-# git push again
-```
-
-### Issue: "Kubeval validation failed"
-
-```bash
-# Check K8s manifest syntax
-kubeval k8s/platform/minio/config.yaml
-
-# Common issues:
-# - Typos in apiVersion, kind, metadata
-# - Missing required fields
-# - Invalid references (namespace, service name)
-```
-
-### Issue: "ArgoCD sync failed"
-
-```bash
-# Check ArgoCD UI
-# https://argocd.riotpiao.com → homelab-root
-
-# Or CLI
-argocd app get homelab-root
-argocd app logs homelab-root --follow
-
-# Common issues:
-# - Missing namespace (fixed by infrastructure layer)
-# - Invalid Helm chart version
-# - Secret not found
-# - Network policy blocking traffic
-```
-
-### Issue: "Security scan found hardcoded secret"
-
-```bash
-# Fix: Remove secret from YAML
-# Add to SOPS encryption instead
-
-# Or use ArgoCD Sealed Secrets
-# (if SOPS not available)
-```
-
----
-
-## Viewing Results
-
-### Forgejo Actions UI
-
-```
-Repository → Actions
-  ├─ validate-k8s
-  │   ├─ ✅ Success (merge safe)
-  │   ├─ ❌ Failed (fix required)
-  │   └─ Logs (click "Steps" → "Summary")
-  ├─ argocd-sync
-  │   ├─ ✅ Synced (deployed)
-  │   └─ ❌ Failed (check ArgoCD UI)
-  └─ security-scan
-      ├─ ✅ Pass (no critical issues)
-      └─ ⚠️  Warning (review, non-blocking)
-```
-
-### ArgoCD UI
-
-```
-https://argocd.riotpiao.com
-  ├─ homelab-root
-  │   ├─ Status: Synced ✓
-  │   ├─ Health: Healthy ✓
-  │   └─ Details (click to see resources)
-  ├─ layer-1-bootstrap
-  ├─ layer-2-platform
-  ├─ layer-3-security
-  ├─ layer-4-applications
-  └─ layer-5-data
-```
-
----
-
-## Common Tasks
-
-### Add new service to cluster
-
-```bash
-# 1. Create directory and kustomization.yaml
-mkdir -p k8s/applications/my-service
-cat > k8s/applications/my-service/kustomization.yaml << EOF
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-namespace: my-namespace
-helmCharts:
-- name: my-chart
-  repo: https://charts.example.com
-  version: 1.0.0
-  releaseName: my-service
-  valuesFile: values.yaml
-EOF
-
-# 2. Add values.yaml
-cp /template/values.yaml k8s/applications/my-service/
-
-# 3. Commit and push
-git add k8s/applications/my-service/
-git commit -m "feat(apps): add my-service"
-git push
-
-# 4. CI validates
-# 5. Merge to main
-# 6. ArgoCD syncs automatically
-# ✓ Service deployed to cluster
-```
-
-### Rollback a deployment
-
-```bash
-# 1. Find broken commit
-git log --oneline k8s/  # Identify bad commit
-
-# 2. Revert
-git revert <commit-hash>
-git push
-
-# 3. CI validates (should pass)
-# 4. Merge to main
-# 5. ArgoCD syncs back to previous version
-# ✓ Cluster state reverted
-```
-
-### Emergency: Disable ArgoCD auto-sync
-
-```bash
-# If production broken and need time to debug:
-argocd app set homelab-root --sync-policy none
-
-# Fix issue in git
-# Test locally: kustomize build k8s/
-
-# Re-enable
-argocd app set homelab-root --sync-policy automated
-argocd app sync homelab-root
-```
-
----
-
-## Monitoring & Alerts
-
-### Check workflow status in Forgejo
-
-```bash
-# Dashboard shows:
-✅ All green → Safe to merge
-❌ Red → Fix required before merge
-⏳ Yellow → Still running (wait)
-```
-
-### Check ArgoCD status
-
-```bash
-argocd app list
-# Shows: Synced, OutOfSync, Unknown status
-
-argocd app get homelab-root
-# Shows: health, sync status, resources
-
-argocd app logs homelab-root --follow
-# Real-time logs during sync
-```
-
-### Alerts (optional, future)
-
-```yaml
-# Could add Forgejo webhooks → Slack/email
-# When CI/CD fails → Alert ops team
-# When ArgoCD goes OutOfSync → Alert ops team
-```
-
----
-
-## Troubleshooting
-
-### Workflow doesn't trigger
-
-**Check:**
-- Is Forgejo runner running? `forgejo-runner daemon`
-- Did you push to correct branch? (validate runs on all, argocd-sync only on main)
-- Did path match filter? (must change k8s/ or .forgejo/workflows/)
-
-### Workflow hangs/times out
-
-**Check:**
-- kustomize build → Check for dependency cycles
-- argocd sync → Check cluster resources (storage full? network down?)
-- security scan → Large image scan → Takes time
-
-**Fix:**
-- Increase timeout in workflow
-- Optimize kustomization (remove unused resources)
-- Add resource limits to pods
-
-### ArgoCD token invalid
-
-**Fix:**
-```bash
-# Regenerate token
-argocd account generate-token
-
-# Update Forgejo secret
-# Settings → Secrets → ARGOCD_AUTH_TOKEN = <new-token>
-```
-
----
-
-## Best Practices
-
-✅ **DO:**
-- Commit all K8s changes to git (no manual kubectl apply)
-- Run validate-k8s locally before push
-- Write descriptive commit messages (why this change?)
-- Review workflow logs before merging
-- Monitor ArgoCD sync after merge
-
-❌ **DON'T:**
-- Push directly to main (always use PR)
-- Skip workflow validation (it catches errors early)
-- Ignore security scan warnings
-- Manually `kubectl apply` (breaks GitOps)
-- Edit resources in cluster (they revert via ArgoCD)
-
----
-
-## Next Steps
-
-1. **Setup Forgejo runner** (if not already running)
-2. **Add ArgoCD secrets** to Forgejo
-3. **Test workflows** on feature branch
-4. **Merge to main** → Watch ArgoCD sync
-5. **Celebrate:** Full GitOps pipeline working! 🎉

.forgejo/workflows/cluster-ci.yaml

@@ -1,269 +0,0 @@
-name: Cluster CI Pipeline
-
-on:
-  push:
-    branches:
-      - main
-      - develop
-    paths:
-      - 'k8s/**'
-      - '.forgejo/workflows/cluster-ci.yaml'
-  pull_request:
-    paths:
-      - 'k8s/**'
-
-jobs:
-  ci:
-    runs-on: docker
-    steps:
-      # === Checkout ===
-      - name: Checkout
-        run: |
-          REPO_URL="${{ gitea.server_url }}/${{ gitea.repository }}.git"
-          CLONE_URL="https://${{ secrets.CI_RUNNER }}:${{ secrets.CI_RUNNER_SECRET }}@${REPO_URL#https://}"
-          git clone --depth 1 "$CLONE_URL" .
-          git fetch origin main
-          git checkout main
-
-      # === Install Tools ===
-      - name: Install Tools
-        run: |
-          unset GITHUB_TOKEN
-          apt-get update && apt-get install -y \
-            yamllint \
-            python3-pip \
-            curl \
-            jq
-
-          # kubeval
-          curl -L https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz | tar xz
-          mv -f kubeval /usr/local/bin/
-
-          # kustomize
-          rm -f kustomize
-          curl -s https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh | bash
-          mv -f kustomize /usr/local/bin/
-
-          # argocd
-          curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
-          chmod +x /usr/local/bin/argocd
-
-          # trivy
-          curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
-
-          # polaris
-          curl -L https://github.com/FairwindsOps/polaris/releases/latest/download/polaris-linux-amd64 -o /usr/local/bin/polaris
-          chmod +x /usr/local/bin/polaris
-
-      # === YAML Lint ===
-      - name: YAML Lint
-        run: |
-          echo "=== Linting YAML files ==="
-          yamllint k8s/ -c .yamllint.yaml || true
-
-      # === Kubeval - Validate K8s Syntax ===
-      - name: Kubeval - Validate K8s Syntax
-        run: |
-          echo "=== Validating Kubernetes manifests ==="
-          find k8s -name "*.yaml" -o -name "*.yml" | grep -v "\.archive" | while read file; do
-            echo "Validating $file..."
-            kubeval "$file" -d 2>/dev/null || true
-          done
-
-      # === Kustomize Build - All overlays ===
-      - name: Kustomize Build - Infrastructure
-        run: |
-          echo "=== Building k8s/infrastructure/ ==="
-          kustomize build k8s/infrastructure > /tmp/infrastructure.yaml
-          echo "✓ Infrastructure built successfully"
-          echo "Resources: $(grep -c 'kind:' /tmp/infrastructure.yaml)"
-
-      - name: Kustomize Build - Bootstrap
-        run: |
-          echo "=== Building k8s/bootstrap/ ==="
-          kustomize build k8s/bootstrap > /tmp/bootstrap.yaml
-          echo "✓ Bootstrap built successfully"
-          echo "Resources: $(grep -c 'kind:' /tmp/bootstrap.yaml || echo 0)"
-
-      - name: Kustomize Build - Platform
-        run: |
-          echo "=== Building k8s/platform/ ==="
-          kustomize build k8s/platform > /tmp/platform.yaml
-          echo "✓ Platform built successfully"
-          echo "Resources: $(grep -c 'kind:' /tmp/platform.yaml || echo 0)"
-
-      - name: Kustomize Build - Security
-        run: |
-          echo "=== Building k8s/security/ ==="
-          kustomize build k8s/security > /tmp/security.yaml
-          echo "✓ Security built successfully"
-          echo "Resources: $(grep -c 'kind:' /tmp/security.yaml || echo 0)"
-
-      - name: Kustomize Build - Applications
-        run: |
-          echo "=== Building k8s/applications/ ==="
-          kustomize build k8s/applications > /tmp/applications.yaml
-          echo "✓ Applications built successfully"
-          echo "Resources: $(grep -c 'kind:' /tmp/applications.yaml || echo 0)"
-
-      - name: Kustomize Build - Data
-        run: |
-          echo "=== Building k8s/data/ ==="
-          kustomize build k8s/data > /tmp/data.yaml
-          echo "✓ Data built successfully"
-          echo "Resources: $(grep -c 'kind:' /tmp/data.yaml || echo 0)"
-
-      - name: Validate ArgoCD Applications
-        run: |
-          echo "=== Validating ArgoCD Applications ==="
-          kubeval k8s/argocd/apps/*.yaml
-
-      # === Trivy - Scan Dockerfile ===
-      - name: Trivy - Scan Dockerfile
-        run: |
-          if find . -name "Dockerfile" 2>/dev/null | grep -v node_modules | head -1 | grep -q .; then
-            echo "=== Scanning Dockerfiles with Trivy ==="
-            find . -name "Dockerfile" -not -path "*/node_modules/*" -exec trivy config {} \;
-          else
-            echo "No Dockerfiles found"
-          fi
-
-      # === Trivy - Scan Helm Charts ===
-      - name: Trivy - Scan Helm Charts
-        run: |
-          if find k8s -name "Chart.yaml" 2>/dev/null | head -1 | grep -q .; then
-            echo "=== Scanning Helm charts with Trivy ==="
-            find k8s -name "Chart.yaml" -exec dirname {} \; | while read chart; do
-              echo "Scanning $chart..."
-              trivy config "$chart" || true
-            done
-          else
-            echo "No Helm charts found"
-          fi
-
-      # === Polaris - K8s Security Audit ===
-      - name: Polaris - K8s Security Audit
-        run: |
-          echo "=== Running Polaris K8s security audit ==="
-          polaris audit --audit-path /tmp/polaris-audit.json k8s/ || true
-
-          if [ -f /tmp/polaris-audit.json ]; then
-            echo "Security issues found:"
-            jq '.results[] | select(.pass == false)' /tmp/polaris-audit.json || true
-          fi
-
-      # === Check for Secrets in Code ===
-      - name: Check for Secrets in Code
-        run: |
-          echo "=== Scanning for hardcoded secrets ==="
-          # BLOCKING. This step used to only count findings and then exit 0, so a
-          # plaintext deploy key rode through it into a public remote. Two failure
-          # modes fixed: it now fails the build, and it matches key material by
-          # PEM header rather than only `private_key:`-style YAML field names.
-          # Findings are captured into variables and tested for emptiness rather than
-          # branching on grep's exit status: implementations disagree on the rc of a
-          # `-v` filter fed empty input, and a wrong rc here fails open.
-          # NOTE: --include must precede `--`; after `--` grep treats it as a filename
-          # and silently scans nothing.
-          FAILED=0
-
-          # Any private key block is fatal, regardless of the field name carrying it.
-          KEYS=$(grep -rIE --include="*.yaml" --include="*.yml" \
-                   -- "-----BEGIN ([A-Z]+ )?PRIVATE KEY-----" k8s/ \
-                 | grep -v "\.enc\.yaml" || true)
-          if [ -n "$KEYS" ]; then
-            echo "❌ Unencrypted private key material found:"
-            echo "$KEYS"
-            FAILED=1
-          fi
-
-          # Plaintext values in secret-ish YAML fields. SOPS output is ENC[...],
-          # so encrypted files never trip this.
-          VALS=$(grep -rInE --include="*.yaml" --include="*.yml" \
-                   -- "^[[:space:]]*(password|token|apiKey|api_key|sshPrivateKey|client_secret):[[:space:]]*[\"']?[^\"'[:space:]{\$]{8,}" k8s/ \
-                 | grep -v "ENC\[" | grep -v "\.enc\.yaml" || true)
-          if [ -n "$VALS" ]; then
-            echo "❌ Plaintext secret value found:"
-            echo "$VALS"
-            FAILED=1
-          fi
-
-          if [ "$FAILED" -ne 0 ]; then
-            echo "Encrypt with SOPS (see .sops.yaml) — *.enc.yaml files are exempt."
-            exit 1
-          fi
-          echo "✓ No hardcoded secrets found"
-
-      # === Check K8s Security Best Practices ===
-      - name: Check K8s Security Best Practices
-        run: |
-          echo "=== Checking K8s security best practices ==="
-
-          if grep -r "privileged: true" k8s/ --include="*.yaml" --include="*.yml"; then
-            echo "⚠️  Found privileged containers"
-          fi
-
-          if grep -r "hostNetwork: true" k8s/ --include="*.yaml" --include="*.yml"; then
-            echo "⚠️  Found hostNetwork usage"
-          fi
-
-          echo "Checking for missing resource limits..."
-          MISSING=0
-          find k8s -name "*.yaml" -o -name "*.yml" | while read file; do
-            if grep -q "kind: Deployment\|kind: StatefulSet\|kind: DaemonSet" "$file"; then
-              if ! grep -q "resources:" "$file"; then
-                echo "⚠️  $file: Missing resource requests/limits"
-                MISSING=$((MISSING + 1))
-              fi
-            fi
-          done
-
-      # === ArgoCD Sync (main branch only) ===
-      - name: Sync ArgoCD
-        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
-        env:
-          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
-          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
-        run: |
-          echo "=== Syncing homelab-root ==="
-          argocd app sync homelab-root --force
-          argocd app wait homelab-root --timeout 5m
-
-      - name: Check Sync Status
-        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
-        env:
-          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
-          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
-        run: |
-          echo "=== ArgoCD Applications Status ==="
-          argocd app list -o table
-
-          STATUS=$(argocd app get homelab-root -o jsonpath='{.status.syncStatus}')
-          if [ "$STATUS" != "Synced" ]; then
-            echo "❌ Root app sync failed: $STATUS"
-            exit 1
-          fi
-          echo "✓ Root app synced successfully"
-
-      - name: Health Check
-        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
-        env:
-          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
-          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
-        run: |
-          echo "=== Checking Application Health ==="
-          argocd app get homelab-root -o wide
-
-      # === Summary ===
-      - name: Summary
-        if: always()
-        run: |
-          echo "=== CI Pipeline Summary ==="
-          echo "✓ YAML linted"
-          echo "✓ Manifests validated"
-          echo "✓ Kustomizations built"
-          echo "✓ Security scans completed"
-          echo "✓ Secrets check passed"
-          echo "✓ Best practices verified"
-          echo ""
-          echo "✓ All checks passed"

.gitignore

@@ -66,3 +66,6 @@ bootstrap-argocd.log
 # one line here, which is how a plaintext deploy key reached a public remote.
 k8s/**/*-secret.yaml
 !k8s/**/*.enc.yaml
+
+# IAM provisioning scripts contain credential references — never commit
+scripts/iam/*.py

AUTH_INTEGRATION.md

@@ -0,0 +1,206 @@
+# Authentik Auth Integration for NextJS
+
+## Current State
+
+### Gateway Auth Status
+
+| Endpoint | Auth Status | Notes |
+|----------|-------------|-------|
+| `/v1/chat/completions` | ❌ **OFF** | LLM routes have no auth middleware |
+| `/v1/embeddings` | ❌ **OFF** | Same - no auth |
+| `/v1/rerank` | ❌ **OFF** | Same - no auth |
+| `X-Service: sqs` | ✅ **ON** | JWT validated via `internal/auth/jwt.go` |
+| `/workflow` | ❌ **OFF** | Pass-through to Temporal |
+
+**Auth module exists** at `homelab-frontend/internal/auth/jwt.go` but only wired for SQS.
+LLM routes in `internal/proxy/proxy.go` have no auth middleware.
+
+### Authentik App
+
+Authentik app `local-llm` exists for LLM API auth:
+- **Client ID**: `local-llm`
+- **Client Secret**: `kubectl -n llm-serving get secret local-llm-jwt -o jsonpath='{.data.client-secret}' | base64 -d`
+- **Token endpoint**: `https://authentik.riotpiao.com/application/o/token/`
+- **Userinfo endpoint**: `https://authentik.riotpiao.com/application/o/userinfo/`
+- **OIDC discovery**: `https://authentik.riotpiao.com/application/o/local-llm/.well-known/openid-configuration`
+
+## Sign-in Methods
+
+### 1. Resource Owner Password Credentials (ROPC)
+
+Direct username/password login. Server-side only (needs client_secret).
+
+```typescript
+// API Route: app/api/auth/login/route.ts
+const response = await fetch('https://authentik.riotpiao.com/application/o/token/', {
+  method: 'POST',
+  headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
+  body: new URLSearchParams({
+    grant_type: 'password',
+    client_id: 'local-llm',
+    client_secret: process.env.AUTHENTIK_CLIENT_SECRET,
+    username: '[email protected]',
+    password: 'userpassword',
+    scope: 'openid email profile groups',
+  }),
+});
+
+const tokens = await response.json();
+// { access_token, refresh_token, expires_in, token_type }
+```
+
+### 2. Authorization Code Flow (Browser Redirect)
+
+Requires adding redirect URIs to `local-llm` Authentik app:
+
+```python
+# In k8s/infra/iam/scripts/authentik-provision.py, update:
+"local-llm": {
+    ...
+    "redirect_uris": [
+        "http://localhost:3000/api/auth/callback",  # dev
+        "https://your-nextjs-app.com/api/auth/callback",  # prod
+    ],
+}
+```
+
+Then standard OIDC flow:
+1. Redirect to `https://authentik.riotpiao.com/application/o/authorize/?client_id=local-llm&redirect_uri=...&response_type=code&scope=openid email profile groups`
+2. User logs in via Authentik UI
+3. Callback receives `code`, exchange for tokens
+
+## JWT Token Persistence
+
+### Browser (localStorage)
+
+```typescript
+const TOKEN_KEY = 'llm_auth_token';
+
+// Save
+localStorage.setItem(TOKEN_KEY, JSON.stringify({
+  access_token: tokens.access_token,
+  refresh_token: tokens.refresh_token,
+  expires_at: Date.now() + tokens.expires_in * 1000,
+}));
+
+// Load
+const stored = JSON.parse(localStorage.getItem(TOKEN_KEY) || 'null');
+if (stored && stored.expires_at > Date.now()) {
+  // Token valid
+}
+
+// Clear (logout)
+localStorage.removeItem(TOKEN_KEY);
+```
+
+### Server-side (HTTP-only cookies)
+
+```typescript
+// app/api/auth/login/route.ts
+import { cookies } from 'next/headers';
+
+// After successful login
+cookies().set('llm_auth_token', JSON.stringify(tokens), {
+  httpOnly: true,
+  secure: process.env.NODE_ENV === 'production',
+  sameSite: 'lax',
+  maxAge: tokens.expires_in,
+  path: '/',
+});
+
+// Read in middleware or API routes
+const tokenCookie = cookies().get('llm_auth_token');
+const tokens = JSON.parse(tokenCookie?.value || 'null');
+```
+
+## Token Refresh
+
+```typescript
+async function refreshAccessToken(refresh_token: string) {
+  const response = await fetch('https://authentik.riotpiao.com/application/o/token/', {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
+    body: new URLSearchParams({
+      grant_type: 'refresh_token',
+      client_id: 'local-llm',
+      client_secret: process.env.AUTHENTIK_CLIENT_SECRET,
+      refresh_token,
+    }),
+  });
+  return response.json();
+}
+```
+
+## Environment Variables
+
+```bash
+# .env.local
+AUTHENTIK_URL=https://authentik.riotpiao.com
+AUTHENTIK_CLIENT_ID=local-llm
+AUTHENTIK_CLIENT_SECRET=<from-secret>
+
+# For client-side (public)
+NEXT_PUBLIC_AUTHENTIK_URL=https://authentik.riotpiao.com
+NEXT_PUBLIC_AUTHENTIK_CLIENT_ID=local-llm
+```
+
+## Using Token with LLM API
+
+```typescript
+const token = await getValidToken(); // from localStorage or cookie
+
+const response = await fetch('https://api.riotpiao.com/v1/chat/completions', {
+  method: 'POST',
+  headers: {
+    'Content-Type': 'application/json',
+    'Authorization': `Bearer ${token}`,  // JWT from Authentik
+  },
+  body: JSON.stringify({
+    model: 'reasoning',
+    messages: [{ role: 'user', content: 'Hello' }],
+  }),
+});
+```
+
+## TODO
+
+### Gateway-side (homelab-frontend)
+
+- [ ] Wire `internal/auth/jwt.go` into LLM proxy handler (`internal/proxy/proxy.go`)
+- [ ] Add `authRequired: true` to model config or create LLM-specific middleware
+- [ ] Example pattern from SQS (in `internal/serviceadapter/router.go`):
+
+```go
+// In proxy.go ServeHTTP, before dispatching to LLM upstream:
+if strings.HasPrefix(r.URL.Path, "/v1/") {
+    authHeader := r.Header.Get("Authorization")
+    claims, err := llmJWTAuth.ValidateBearerToken(authHeader)
+    if err != nil {
+        // Return 401/403
+    }
+    if !llmJWTAuth.CheckPermissions(claims, "llm:inference", "*") {
+        // Return 403 insufficient permissions
+    }
+}
+```
+
+### Authentik-side
+
+- [ ] Enable ROPC grant in Authentik provider settings (if not already)
+- [ ] Add redirect URIs to `local-llm` app if browser OAuth flow needed:
+
+```python
+# k8s/infra/iam/scripts/authentik-provision.py
+"local-llm": {
+    ...
+    "redirect_uris": [
+        "http://localhost:3000/api/auth/callback",
+        "https://your-app.com/api/auth/callback",
+    ],
+}
+```
+
+### NextJS-side
+
+- [ ] Until gateway auth is wired, LLM API works without token
+- [ ] Once wired, add `Authorization: Bearer <token>` to all LLM requests

CLAUDE.example.md

@@ -208,3 +208,95 @@ versions without warning in your own values file.
 Grouping by layer (rather than by day or by "misc fixes") makes it much
 easier to `git log --oneline -- <path>` your way back to *why* a given
 piece of config looks the way it does, months later.
+
+## Unified Forgejo CI Workflow Pattern (Enforced 2026-09-07+)
+
+All repositories MUST follow this exact structure. No variations.
+
+```yaml
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+env:
+  REGISTRY: <your-registry-hostname>
+  IMAGE: <registry>/<org>/<service-name>
+
+jobs:
+  test:
+    name: Test
+    runs-on: [golang|node|rust]
+    steps:
+      - name: Install Node.js for actions runtime
+        run: apt-get update && apt-get install -y nodejs
+
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      # Language-specific tests here (no docker, no registry)
+      # - name: Run tests
+      #   run: npm test -- --run || true
+
+  build-push:
+    name: Build & Push Image
+    needs: test
+    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+    runs-on: [golang|node|rust]
+    steps:
+      - name: Install Node.js and Docker
+        run: |
+          apt-get update
+          apt-get install -y nodejs docker.io
+
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Get short SHA
+        id: sha
+        run: |
+          SHORT_SHA=$(git rev-parse --short HEAD)
+          echo "short_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
+
+      - name: Registry login
+        run: |
+          echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
+            --username "${REGISTRY_USER}" --password-stdin
+        env:
+          REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
+          REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
+
+      - name: Build Docker image
+        run: |
+          docker build --no-cache \
+            -t "${IMAGE}:${{ steps.sha.outputs.short_sha }}" \
+            -t "${IMAGE}:latest" \
+            .
+
+      - name: Push Docker image
+        run: |
+          docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}"
+          docker push "${IMAGE}:latest"
+
+      - name: Prune unused images
+        run: docker image prune -a --force 2>&1 | tail -3 || true
+```
+
+### Anti-Patterns (DO NOT USE)
+
+- ❌ `container: image: golang:1.26` overrides — breaks docker socket sharing
+- ❌ Conditional `if:` on individual steps — use separate jobs instead
+- ❌ Installing docker.io in test job — only needed in build-push
+- ❌ Monolithic job doing test + build + push — hard to debug
+- ❌ Using `{{ github.sha }}` for image tag — use short commit SHA for readability
+
+### How It Works
+
+1. **PR to feature branch** → test job runs, build-push skipped, nothing pushed
+2. **Push to main** → test runs, build-push runs after test passes, image pushed
+3. Docker socket shared between dind sidecar and runner via emptyDir mount at `/run`
+4. `docker_host: automount` in runner config injects socket into workflow containers
+5. Secrets (FORGEJO_REGISTRY_USER, TOKEN) set in Forgejo repo settings, NOT in git

README.md

@@ -42,6 +42,86 @@ All logs + metrics centralized in Grafana for debugging
 - **Secrets at rest** — Vault + encrypted etcd; credentials never in logs or ConfigMaps
 - **Infrastructure-as-code** — Every service deployed via Helmfile; one `helmfile apply` recovers from total failure
 
+## ArgoCD — GitOps Deployment Flow
+
+**ArgoCD** pulls infrastructure changes from git and syncs the cluster automatically. 
+No manual `kubectl apply` — push to git, ArgoCD detects the change, and deploys within ~3 minutes.
+
+```
+Developer pushes to git
+       ↓
+ArgoCD detects change (every 3 min or webhook)
+       ↓
+Syncs manifests to cluster
+       ↓
+Workloads reconcile automatically
+```
+
+Applications are deployed in waves (numbered 00, 10, 20, 30, ...) to respect dependencies — 
+storage deploys before databases, databases before applications.
+
+### Tracked Git Repositories
+
+ArgoCD monitors these repos for changes:
+
+| Repository | Purpose |
+|------------|----------|
+| `https://github.com/Riotpiaole/riotpiao.homelab.com` | Main infrastructure repo (all manifests in `k8s/argocd/apps/`) |
+| `https://forgejo.riotpiao.com/rock/*` | Any `rock/*` repo in in-cluster Forgejo (apps + configs) |
+| `https://github.com/Riotpiaole/Poimen-*` | External Poimen services (memory, workflows) |
+
+To deploy a new application: create a git repo, add an Application manifest to the homelab repo's 
+`k8s/argocd/apps/`, commit + push, and ArgoCD syncs within 3 minutes.
+
+## Management Planes — Talos vs Kubernetes
+
+This cluster has **two separate management planes**, each with different workflows:
+
+| Plane | What it manages | Workflow | Tool |
+|-------|-----------------|----------|------|
+| **Talos (OS)** | Node configuration, kernel params, networking, CoreDNS, machine state | Edit `terraform/` → `terraform apply` → `make apply-cp` | `terraform` + `talosctl` |
+| **Kubernetes (workloads)** | All pods, services, deployments, ingresses, databases | Edit `k8s/argocd/apps/` → `git push` → ArgoCD syncs | `git` + ArgoCD |
+
+**Critical distinction:**
+- **Kubernetes resources** (`k8s/**`) flow through **git → ArgoCD** — never use `kubectl apply`
+- **Talos machine config** (`terraform/**`) uses **local `terraform apply`** (sanctioned exception — CI can't hold node credentials)
+
+Example: To add a CoreDNS hostname rewrite, you edit `terraform/files/coredns/Corefile`, then:
+```bash
+cd terraform && terraform apply -var-file=terraform.tfvars.local
+cd .. && make apply-cp     # talosctl apply-config to all 3 control planes
+```
+
+But to add a new Kubernetes Deployment or update an Ingress, you only `git push` — **never `kubectl apply`**.
+
+### CoreDNS ConfigMap Ownership — Critical
+
+⚠️ **Warning:** The `coredns` ConfigMap in `kube-system` namespace is **owned by Talos**, not ArgoCD or kubectl. 
+It is rendered from `terraform/files/coredns/Corefile` into Talos's machine config at bootstrap time.
+
+**Do not `kubectl apply` or `kubectl edit` this ConfigMap directly.** Doing so transfers field ownership to kubectl's 
+client-side-apply mechanism, and Talos's inline-manifest controller will silently no-op on every future reconcile 
+(server-side-apply conflict, no error surfaced).
+
+**To update CoreDNS (e.g., add a hostname rewrite):**
+1. Edit `terraform/files/coredns/Corefile`
+2. Commit + push
+3. Run `cd terraform && terraform apply -var-file=terraform.tfvars.local`
+4. Run `make apply-cp` to push config to all control planes
+5. CoreDNS picks up changes via its `reload` plugin — no pod restart needed
+
+**If you accidentally edited the ConfigMap directly and broke Talos's ownership:**
+```bash
+kubectl delete configmap coredns -n kube-system
+# Wait ~30s for Talos's k8s.ManifestApplyController to recreate it
+kubectl get configmap coredns -n kube-system -w
+```
+
+Or as a stopgap, apply the correct content yourself:
+```bash
+kubectl apply --server-side -f <(terraform output coredns_config)
+```
+
 ## Quick Start — Deploying the Cluster
 
 ### 1. Bootstrap Talos Nodes

k8s/apps/comfyui/deployment.yaml

@@ -0,0 +1,80 @@
+# ComfyUI — GPU-accelerated image generation on worker-1.
+# Uses 1x V100 32GB (sm70). Freed by scaling ornith 2→1.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: comfyui
+  namespace: comfyui
+  labels:
+    app: comfyui
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: comfyui
+  template:
+    metadata:
+      labels:
+        app: comfyui
+    spec:
+      nodeSelector:
+        kubernetes.io/hostname: worker-1
+      runtimeClassName: nvidia
+      containers:
+        - name: comfyui
+          image: ghcr.io/ai-dock/comfyui:v2-cuda-12.1.1-base-22.04
+          ports:
+            - containerPort: 8188
+              protocol: TCP
+          env:
+            - name: NVIDIA_VISIBLE_DEVICES
+              value: "all"
+            - name: COMFYUI_FLAGS
+              value: "--listen 0.0.0.0 --port 8188"
+          resources:
+            requests:
+              cpu: "4"
+              memory: 8Gi
+              nvidia.com/gpu: "1"
+            limits:
+              cpu: "8"
+              memory: 16Gi
+              nvidia.com/gpu: "1"
+          volumeMounts:
+            - mountPath: /workspace/ComfyUI/models
+              name: models
+            - mountPath: /workspace/ComfyUI/output
+              name: output
+          readinessProbe:
+            httpGet:
+              path: /
+              port: 8188
+            periodSeconds: 10
+            initialDelaySeconds: 30
+          startupProbe:
+            httpGet:
+              path: /
+              port: 8188
+            failureThreshold: 60
+            periodSeconds: 10
+      volumes:
+        - name: models
+          persistentVolumeClaim:
+            claimName: comfyui-models
+        - name: output
+          emptyDir: {}
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: comfyui-models
+  namespace: comfyui
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 50Gi

k8s/apps/comfyui/ingress.yaml

@@ -0,0 +1,25 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: comfyui
+  namespace: comfyui
+  annotations:
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
+    nginx.ingress.kubernetes.io/proxy-body-size: "0"
+    # WebSocket support for ComfyUI's live preview
+    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
+    nginx.ingress.kubernetes.io/proxy-set-headers: "Upgrade"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: comfy.riotpiao.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: comfyui
+                port:
+                  number: 80

k8s/apps/comfyui/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - deployment.yaml
+  - service.yaml
+  - ingress.yaml

k8s/apps/comfyui/service.yaml

@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: comfyui
+  namespace: comfyui
+  labels:
+    app: comfyui
+spec:
+  selector:
+    app: comfyui
+  ports:
+    - port: 80
+      targetPort: 8188
+      protocol: TCP

k8s/apps/gotify/README.md

@@ -0,0 +1,107 @@
+# Gotify — Push Notifications + Email Relay
+
+Self-hosted notification server with SMTP email forwarding sidecar.
+
+## Architecture
+
+```
+Forgejo webhook ──POST──→ Gotify API (:80/message)
+                              │
+                    ┌─────────┼─────────┐
+                    ▼                     ▼
+              Push notification      SMTP emailer sidecar
+              (mobile/desktop)       (polls → sends email)
+```
+
+## Setup (one-time, after first deploy)
+
+### 1. Encrypt secrets before committing
+
+```bash
+# Edit secrets.yaml with real values first, then:
+sops -e -i k8s/apps/gotify/secrets.yaml
+```
+
+### 2. Create Gotify app + client tokens
+
+1. Login to `https://gotify.riotpiao.com` with admin creds
+2. **Applications** → Create `forgejo` → copy **app token**
+3. **Clients** → Create `smtp-emailer` → copy **client token**
+4. Update `gotify-tokens` secret:
+   ```bash
+   kubectl -n notifications create secret generic gotify-tokens \
+     --from-literal=app-token=<APP_TOKEN> \
+     --from-literal=client-token=<CLIENT_TOKEN> \
+     --dry-run=client -o yaml | kubectl apply -f -
+   ```
+
+### 3. Configure Forgejo webhook
+
+In each Forgejo repo → **Settings** → **Webhooks** → **Add Webhook** → **Gotify**:
+
+| Field | Value |
+|-------|-------|
+| Target URL | `http://gotify.notifications.svc.cluster.local/message` |
+| Token | The **app token** from step 2 |
+| Events | Pull Request (Created, Merged, Closed) |
+
+Or via API:
+```bash
+FORGEJO_TOKEN="<your-pat>"
+APP_TOKEN="<gotify-app-token>"
+
+curl -s -X POST "https://forgejo.riotpiao.com/api/v1/repos/rock/homelab/hooks" \
+  -H "Authorization: token $FORGEJO_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "type": "gotify",
+    "active": true,
+    "config": {
+      "content_type": "json",
+      "url": "http://gotify.notifications.svc.cluster.local/message?token='"$APP_TOKEN"'"
+    },
+    "events": ["pull_request", "pull_request_assign", "pull_request_review"],
+    "authorization_header": ""
+  }'
+```
+
+### 4. Add CoreDNS rewrite (if accessing via public hostname)
+
+Only needed if Cloudflare Tunnel is used for gotify.riotpiao.com:
+
+```
+# terraform/files/coredns/Corefile — add rewrite:
+rewrite name gotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+```
+
+Then: `cd terraform && terraform apply && cd .. && make apply-cp`
+
+### 5. SMTP providers
+
+| Provider | Host | Port | Notes |
+|----------|------|------|-------|
+| Gmail | smtp.gmail.com | 587 | Use App Password (2FA required) |
+| Resend | smtp.resend.com | 587 | Free 100 emails/day |
+| Sendgrid | smtp.sendgrid.net | 587 | Free 100 emails/day |
+| Mailgun | smtp.mailgun.org | 587 | Free 5000/month |
+
+## Notification priority levels
+
+| Priority | Meaning | Email forwarded? |
+|----------|---------|-----------------|
+| 0-4 | Low (info) | No (below MIN_PRIORITY=5) |
+| 5-7 | Normal (PR created) | Yes |
+| 8-10 | High (PR merged, failures) | Yes |
+
+## Verify
+
+```bash
+# Test push notification
+APP_TOKEN="<app-token>"
+curl -X POST "https://gotify.riotpiao.com/message?token=$APP_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"title":"Test","message":"Hello from homelab","priority":5}'
+
+# Check email sidecar logs
+kubectl -n notifications logs deployment/gotify -c smtp-emailer --tail=20
+```

k8s/apps/gotify/db.yaml

@@ -0,0 +1,35 @@
+# CNPG Postgres for Gotify. Lightweight — 2 instances, 2Gi storage.
+# CNPG generates secret `gotify-db-app` + service `gotify-db-rw` in ns notifications.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: gotify-db
+  namespace: notifications
+  annotations:
+    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+spec:
+  instances: 3
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
+  bootstrap:
+    initdb:
+      database: gotify
+      owner: app
+      encoding: UTF8
+      localeCollate: C
+      localeCType: C
+  enableSuperuserAccess: false
+  resources:
+    requests: { memory: "256Mi", cpu: "100m" }
+    limits: { memory: "512Mi", cpu: "500m" }
+  storage:
+    size: 2Gi
+    storageClass: longhorn-cnpg
+  monitoring:
+    enablePodMonitor: true
+  affinity:
+    podAntiAffinityType: preferred
+    topologyKey: kubernetes.io/hostname
+    tolerations:
+      - key: node-role.kubernetes.io/control-plane
+        operator: Exists
+        effect: NoSchedule

k8s/apps/gotify/deployment.yaml

@@ -0,0 +1,204 @@
+# Gotify — self-hosted push notification server + SMTP email relay.
+# Forgejo webhooks → Gotify → push notifications + email forwarding.
+# Runs on control plane (no GPU needed), lightweight.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: gotify
+  namespace: notifications
+  labels:
+    app: gotify
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: gotify
+  template:
+    metadata:
+      labels:
+        app: gotify
+    spec:
+      containers:
+        # --- Gotify server ---
+        - name: gotify
+          image: ghcr.io/gotify/server:2.6.1
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              export GOTIFY_DATABASE_DIALECT=postgres
+              export GOTIFY_DATABASE_CONNECTION="host=gotify-db-rw.notifications port=5432 user=${DB_USER} password=${DB_PASS} dbname=gotify sslmode=disable"
+              exec /app/gotify-app
+          ports:
+            - containerPort: 80
+              protocol: TCP
+          env:
+            - name: GOTIFY_DEFAULTUSER_NAME
+              valueFrom:
+                secretKeyRef:
+                  name: gotify-admin
+                  key: username
+            - name: GOTIFY_DEFAULTUSER_PASS
+              valueFrom:
+                secretKeyRef:
+                  name: gotify-admin
+                  key: password
+            - name: DB_USER
+              valueFrom:
+                secretKeyRef:
+                  name: gotify-db-app
+                  key: username
+            - name: DB_PASS
+              valueFrom:
+                secretKeyRef:
+                  name: gotify-db-app
+                  key: password
+            - name: GOTIFY_SERVER_PORT
+              value: "80"
+            - name: GOTIFY_SERVER_KEEPALIVEPERIODSECONDS
+              value: "0"
+            - name: TZ
+              value: Asia/Tokyo
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              cpu: 200m
+              memory: 128Mi
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: 80
+            periodSeconds: 30
+            initialDelaySeconds: 10
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: 80
+            periodSeconds: 10
+            initialDelaySeconds: 5
+
+        # --- SMTP emailer sidecar ---
+        # Watches Gotify WebSocket stream, forwards messages as email.
+        # https://github.com/eternal-flame-AD/gotify-broadcast
+        - name: smtp-emailer
+          image: ghcr.io/gotify/server:2.6.1
+          command:
+            - /bin/sh
+            - -c
+            - |
+              # Wait for Gotify to be ready
+              until wget -qO- http://localhost:80/health >/dev/null 2>&1; do
+                echo "Waiting for Gotify..."
+                sleep 2
+              done
+              echo "Gotify is ready, starting email relay..."
+
+              # Poll Gotify messages and forward via SMTP using msmtp
+              # Install msmtp for lightweight SMTP sending
+              apk add --no-cache msmtp curl jq
+
+              # Configure msmtp
+              cat > /tmp/msmtprc <<MSMTP
+              defaults
+              auth           on
+              tls            on
+              tls_trust_file /etc/ssl/certs/ca-certificates.crt
+              logfile        /tmp/msmtp.log
+
+              account        default
+              host           ${SMTP_HOST}
+              port           ${SMTP_PORT}
+              from           ${SMTP_FROM}
+              user           ${SMTP_USER}
+              password       ${SMTP_PASS}
+              MSMTP
+
+              chmod 600 /tmp/msmtprc
+
+              # Track last seen message ID
+              LAST_ID=0
+
+              while true; do
+                # Fetch messages since last ID
+                MESSAGES=$(curl -s -H "X-Gotify-Key: ${GOTIFY_CLIENT_TOKEN}" \
+                  "http://localhost:80/message?since=${LAST_ID}&limit=10" 2>/dev/null)
+
+                if [ -n "$MESSAGES" ]; then
+                  echo "$MESSAGES" | jq -r '.messages[]? | @base64' | while read -r MSG; do
+                    DECODED=$(echo "$MSG" | base64 -d)
+                    ID=$(echo "$DECODED" | jq -r '.id')
+                    TITLE=$(echo "$DECODED" | jq -r '.title // "Notification"')
+                    BODY=$(echo "$DECODED" | jq -r '.message // ""')
+                    PRIORITY=$(echo "$DECODED" | jq -r '.priority // 5')
+                    APP=$(echo "$DECODED" | jq -r '.appid // 0')
+                    DATE=$(echo "$DECODED" | jq -r '.date // ""')
+
+                    # Only forward messages with priority >= configured threshold
+                    if [ "$PRIORITY" -ge "${MIN_PRIORITY:-0}" ]; then
+                      printf "Subject: [Gotify] %s\nFrom: %s\nTo: %s\nContent-Type: text/plain; charset=UTF-8\n\n%s\n\n---\nPriority: %s\nDate: %s" \
+                        "$TITLE" "$SMTP_FROM" "$NOTIFY_EMAIL" "$BODY" "$PRIORITY" "$DATE" | \
+                        msmtp -C /tmp/msmtprc "$NOTIFY_EMAIL" && \
+                        echo "Email sent for message $ID: $TITLE" || \
+                        echo "Failed to send email for message $ID"
+                    fi
+
+                    # Update last seen ID
+                    if [ "$ID" -gt "$LAST_ID" ]; then
+                      LAST_ID=$ID
+                    fi
+                  done
+                fi
+
+                sleep ${POLL_INTERVAL:-30}
+              done
+          env:
+            - name: GOTIFY_CLIENT_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: gotify-tokens
+                  key: client-token
+            - name: SMTP_HOST
+              valueFrom:
+                secretKeyRef:
+                  name: gotify-smtp
+                  key: host
+            - name: SMTP_PORT
+              valueFrom:
+                secretKeyRef:
+                  name: gotify-smtp
+                  key: port
+            - name: SMTP_FROM
+              valueFrom:
+                secretKeyRef:
+                  name: gotify-smtp
+                  key: from
+            - name: SMTP_USER
+              valueFrom:
+                secretKeyRef:
+                  name: gotify-smtp
+                  key: user
+            - name: SMTP_PASS
+              valueFrom:
+                secretKeyRef:
+                  name: gotify-smtp
+                  key: password
+            - name: NOTIFY_EMAIL
+              valueFrom:
+                secretKeyRef:
+                  name: gotify-smtp
+                  key: notify-email
+            - name: MIN_PRIORITY
+              value: "5"
+            - name: POLL_INTERVAL
+              value: "15"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              cpu: 100m
+              memory: 64Mi
+      # No volumes — Postgres handles persistence

k8s/apps/gotify/ingress.yaml

@@ -0,0 +1,26 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: gotify
+  namespace: notifications
+  annotations:
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
+    # WebSocket support for Gotify client connections
+    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
+    nginx.ingress.kubernetes.io/configuration-snippet: |
+      proxy_set_header Upgrade $http_upgrade;
+      proxy_set_header Connection "upgrade";
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: gotify.riotpiao.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: gotify
+                port:
+                  number: 80

k8s/apps/gotify/kustomization.yaml

@@ -0,0 +1,8 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespace.yaml
+  - db.yaml
+  - deployment.yaml
+  - service.yaml
+  - ingress.yaml

k8s/apps/gotify/namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: notifications
+  labels:
+    kubernetes.io/metadata.name: notifications

k8s/apps/gotify/service.yaml

@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: gotify
+  namespace: notifications
+  labels:
+    app: gotify
+spec:
+  selector:
+    app: gotify
+  ports:
+    - port: 80
+      targetPort: 80
+      protocol: TCP

k8s/apps/immich/configmap.yaml

@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: immich-config
+data:
+  DB_HOSTNAME: "immich-db-rw"
+  DB_DATABASE_NAME: "immich"
+  # Only pgvector is installed (see db.yaml) - no vectorchord extension image
+  # exists for pg18 in CNPG's catalog yet. Explicit instead of relying on
+  # auto-detect's vectorchord-first preference order.
+  DB_VECTOR_EXTENSION: "pgvector"
+  REDIS_HOSTNAME: "immich-redis"
+  IMMICH_MACHINE_LEARNING_URL: "http://immich-machine-learning:3003"
+  TZ: "America/Los_Angeles"

k8s/apps/immich/db.yaml

@@ -0,0 +1,58 @@
+# Dedicated CNPG Postgres for Immich. Same recipe as paperless-db/authentik-db
+# (2 instances, default longhorn storage class) except the operand is
+# PostgreSQL 18, not 16.2 - the official CNPG pgvector extension image
+# (ghcr.io/cloudnative-pg/pgvector) is only published for pg18, no pg16 tags
+# exist in that registry. Immich itself supports pg18 fine (immich-app's own
+# postgres image already ships 18-vectorchord builds).
+#
+# pgvector loaded via CNPG's ImageVolume extension mechanism (CNPG 1.27+,
+# k8s ImageVolume feature - both present here: operator is 1.30.0, cluster is
+# v1.36.1). No shared_preload_libraries needed - pgvector doesn't require
+# preload, just CREATE EXTENSION, which immich-server issues itself at
+# startup. Distro/pg-major must match between the operand image and the
+# extension image (both "18"+"trixie" here) - CNPG's own compatibility rule.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: immich-db
+  annotations:
+    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+spec:
+  instances: 3
+  imageName: ghcr.io/cloudnative-pg/postgresql:18-minimal-trixie
+  postgresql:
+    extensions:
+      - name: pgvector
+        image:
+          reference: ghcr.io/cloudnative-pg/pgvector:0.8.1-18-trixie
+  bootstrap:
+    initdb:
+      database: immich
+      owner: app
+      encoding: UTF8
+      localeCollate: C
+      localeCType: C
+      # CREATE EXTENSION vector requires superuser (pgvector's control file
+      # isn't marked trusted) and the "app" owner role isn't one
+      # (enableSuperuserAccess: false, repo convention) - postInitApplicationSQL
+      # runs as superuser during initdb, before the app ever connects. Only
+      # fires on a fresh bootstrap; the live cluster already had this run
+      # manually once (kubectl exec ... psql -U postgres -c 'CREATE EXTENSION').
+      postInitApplicationSQL:
+        - "CREATE EXTENSION IF NOT EXISTS vector;"
+        - "CREATE EXTENSION IF NOT EXISTS cube;"
+        - "CREATE EXTENSION IF NOT EXISTS earthdistance;"
+  enableSuperuserAccess: false
+  resources:
+    requests: { memory: "512Mi", cpu: "250m" }
+    limits: { memory: "2Gi", cpu: "1" }
+  storage:
+    size: 20Gi
+    storageClass: longhorn
+  affinity:
+    podAntiAffinityType: preferred
+    topologyKey: kubernetes.io/hostname
+    tolerations:
+      - key: node-role.kubernetes.io/control-plane
+        operator: Exists
+        effect: NoSchedule

k8s/apps/immich/deployment.yaml

@@ -0,0 +1,100 @@
+# immich-server: pinned to talos-cp-3, same reasoning as paperless
+# (deployment.yaml comment there) - immich-media is a ReadWriteOnce Longhorn
+# volume with a single replica physically on that node's disk (shared with
+# paperless-media on the same 4TB HDD). Recreate strategy for the same
+# reason: two pods can't both attach an RWO volume.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: immich-server
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: immich-server
+  template:
+    metadata:
+      labels:
+        app: immich-server
+    spec:
+      serviceAccountName: immich
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-3
+      containers:
+        - name: immich-server
+          image: ghcr.io/immich-app/immich-server:release
+          ports:
+            - containerPort: 2283
+          envFrom:
+            - configMapRef:
+                name: immich-config
+          env:
+            - name: DB_USERNAME
+              valueFrom:
+                secretKeyRef:
+                  name: immich-db-app
+                  key: username
+            - name: DB_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: immich-db-app
+                  key: password
+            # Composed by k8s/infra/iam's provisioning script (system-config
+            # JSON, oauth section) - see immich-oidc Secret.
+            - name: IMMICH_CONFIG_FILE
+              value: /config/immich.json
+          resources:
+            requests: { cpu: "500m", memory: "1Gi" }
+            limits: { cpu: "2", memory: "4Gi" }
+          volumeMounts:
+            - name: media
+              mountPath: /usr/src/app/upload
+            - name: oidc-config
+              mountPath: /config
+              readOnly: true
+      volumes:
+        - name: media
+          persistentVolumeClaim:
+            claimName: immich-media
+        - name: oidc-config
+          secret:
+            secretName: immich-oidc
+            items:
+              - key: config.json
+                path: immich.json
+---
+# CPU-only for now - the cluster's one GPU node (worker-1) is already
+# dedicated to llm-serving predictors. Not node-pinned: its cache PVC is on
+# the default 3-replica pool, not the single-disk cp-3 HDD.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: immich-machine-learning
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: immich-machine-learning
+  template:
+    metadata:
+      labels:
+        app: immich-machine-learning
+    spec:
+      serviceAccountName: immich
+      containers:
+        - name: immich-machine-learning
+          image: ghcr.io/immich-app/immich-machine-learning:release
+          ports:
+            - containerPort: 3003
+          resources:
+            requests: { cpu: "500m", memory: "1Gi" }
+            limits: { cpu: "2", memory: "4Gi" }
+          volumeMounts:
+            - name: ml-cache
+              mountPath: /cache
+      volumes:
+        - name: ml-cache
+          persistentVolumeClaim:
+            claimName: immich-ml-cache

k8s/apps/immich/ingress.yaml

@@ -0,0 +1,24 @@
+# Direct nginx ingress, same reasoning as paperless: large uploads (photos/
+# videos) and long-lived operations (video transcode, big batch uploads) need
+# proxy-body-size/timeouts raised past nginx's defaults.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: immich
+  annotations:
+    nginx.ingress.kubernetes.io/proxy-body-size: "0"
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: img.riotpiao.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: immich-server
+                port:
+                  number: 2283

k8s/apps/immich/kustomization.yaml

@@ -0,0 +1,14 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: immich
+resources:
+  - db.yaml
+  - pvc.yaml
+  - configmap.yaml
+  - redis.yaml
+  - deployment.yaml
+  - service.yaml
+  - ingress.yaml
+  - rbac.yaml
+# immich-oidc Secret written by the PostSync provisioning Job in
+# k8s/infra/iam (same as paperless-oidc) - not duplicated here.

k8s/apps/immich/pvc.yaml

@@ -0,0 +1,38 @@
+# Two volumes:
+#
+# - media: original photos/videos + generated thumbnails/encoded videos.
+#   Shares the cp-3 USB HDD with paperless-media, same StorageClass/disk tag,
+#   single replica (single disk, no redundancy possible - same tradeoff
+#   paperless already accepts). Sized 1400Gi, not 2000Gi: the disk's real
+#   usable capacity (~3724GiB, formatting overhead) minus paperless-media's
+#   2000Gi and ~231GiB of other apps' default-class replicas that Longhorn
+#   placed here anyway (disk tags only pull matching volumes in, they don't
+#   exclude non-matching ones when the untagged pool elsewhere is full) only
+#   leaves ~1493Gi of real scheduling headroom right now.
+# - ml-cache: downloaded ML model weights for immich-machine-learning
+#   (face detection / CLIP embeddings). Small, disposable (re-downloads on
+#   loss), but persisted so a pod restart doesn't re-pull multi-GB models -
+#   default 3-replica pool, not node-pinned.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: immich-media
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn-paperless-media
+  resources:
+    requests:
+      storage: 1400Gi
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: immich-ml-cache
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 5Gi

k8s/apps/immich/rbac.yaml

@@ -0,0 +1,43 @@
+# Scoped operator access for immich-admins: restart/config-edit rights on
+# just this service's own resources, nothing CNPG-managed (immich-db-*) or
+# provisioning-managed (immich-oidc). Same pattern as
+# k8s/apps/paperless/rbac.yaml. Inert until kube-apiserver's OIDC wiring
+# lands (--oidc-groups-claim=groups, --oidc-groups-prefix=oidc:).
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: immich
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: immich-operator
+rules:
+  - apiGroups: ["apps"]
+    resources: ["deployments"]
+    resourceNames: ["immich-server", "immich-machine-learning"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    resourceNames: ["immich-config"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+  - apiGroups: [""]
+    resources: ["secrets"]
+    resourceNames: ["immich-oidc"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: immich-admins-binding
+subjects:
+  - kind: Group
+    name: "oidc:immich-admins"
+    apiGroup: rbac.authorization.k8s.io
+  - kind: ServiceAccount
+    name: immich
+    namespace: immich
+roleRef:
+  kind: Role
+  name: immich-operator
+  apiGroup: rbac.authorization.k8s.io

k8s/apps/immich/redis.yaml

@@ -0,0 +1,37 @@
+# Job queue broker for immich-server. No PVC: queue state is disposable - a
+# lost queue on restart just re-triggers the affected background jobs
+# (thumbnail generation, ML jobs, etc.), no photo data loss since originals
+# live on immich-media.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: immich-redis
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: immich-redis
+  template:
+    metadata:
+      labels:
+        app: immich-redis
+    spec:
+      containers:
+        - name: redis
+          image: redis:7-alpine
+          ports:
+            - containerPort: 6379
+          resources:
+            requests: { cpu: "50m", memory: "64Mi" }
+            limits: { cpu: "250m", memory: "256Mi" }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: immich-redis
+spec:
+  selector:
+    app: immich-redis
+  ports:
+    - port: 6379
+      targetPort: 6379

k8s/apps/immich/service.yaml

@@ -0,0 +1,21 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: immich-server
+spec:
+  selector:
+    app: immich-server
+  ports:
+    - port: 2283
+      targetPort: 2283
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: immich-machine-learning
+spec:
+  selector:
+    app: immich-machine-learning
+  ports:
+    - port: 3003
+      targetPort: 3003

k8s/apps/llm-serving/kustomization.yaml

@@ -14,5 +14,7 @@ resources:
   - ornith.yaml
   - reasoning.yaml
   - reranker.yaml
+  - qwen-cpu.yaml
+  - networkpolicy.yaml
 # No namespace transformer: every file sets its own, and the transformer would
 # rewrite metadata.namespace on anything cross-namespace added later.

k8s/apps/llm-serving/networkpolicy.yaml

@@ -0,0 +1,62 @@
+# NetworkPolicy for LLM inference engines (llm-serving namespace).
+#
+# These pods have NO auth — vLLM, Ollama, and TEI accept any request.
+# All access MUST go through the api-gateway, which validates JWTs and
+# injects identity headers (X-Forwarded-User, X-Auth-Verified).
+#
+# Replaces the hand-applied llm-serving-default-deny policy that used
+# `llm-client: "true"` pod label as a selector — any pod in any namespace
+# could self-grant access by adding that label, which defeats the purpose.
+#
+# This policy restricts ingress to:
+#   1. api namespace (gateway) — the sole entry point for inference
+#   2. monitoring namespace — Prometheus scraping vLLM/TEI /metrics
+#   3. intra-namespace — pod-to-pod (future: multi-replica comms)
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: llm-serving-ingress
+  namespace: llm-serving
+  labels:
+    app.kubernetes.io/part-of: llm-serving
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/part-of: llm-serving
+  policyTypes:
+    - Ingress
+  ingress:
+    # Allow from api-gateway (namespace: api)
+    # Gateway proxies /v1/chat/completions, /v1/embeddings, /v1/rerank
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: api
+      ports:
+        - protocol: TCP
+          port: 8080   # vLLM, Ollama HTTP
+        - protocol: TCP
+          port: 80      # KServe predictor services
+        - protocol: TCP
+          port: 8000    # vLLM direct (some configs)
+        - protocol: TCP
+          port: 11434   # Ollama native port
+    # Allow Prometheus scraping from monitoring namespace
+    # vLLM: :8080/metrics, TEI: :9000/metrics
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: monitoring
+      ports:
+        - protocol: TCP
+          port: 8080
+        - protocol: TCP
+          port: 9000
+    # Allow intra-namespace (pod-to-pod within llm-serving)
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/part-of: llm-serving
+      ports:
+        - protocol: TCP
+          port: 8080

k8s/apps/llm-serving/ornith.yaml

@@ -33,12 +33,8 @@ spec:
 
         ollama pull ornith:35b
 
-        ollama pull qwen2.5:3b-instruct
-
         ollama run ornith:35b "ok" >/dev/null 2>&1 || true
 
-        ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
-
         wait $SERVE_PID
 
         '
@@ -54,7 +50,7 @@ spec:
       - name: OLLAMA_NUM_PARALLEL
         value: '1'
       - name: OLLAMA_MAX_LOADED_MODELS
-        value: '2'
+        value: '1'
       image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
       name: kserve-container
       ports:
@@ -65,8 +61,7 @@ spec:
           command:
           - /bin/sh
           - -c
-          - ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
-            grep -q qwen2.5
+          - ollama ps 2>/dev/null | grep -q ornith
         periodSeconds: 10
       resources:
         limits:
@@ -82,8 +77,7 @@ spec:
           command:
           - /bin/sh
           - -c
-          - ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
-            grep -q qwen2.5
+          - ollama ps 2>/dev/null | grep -q ornith
         failureThreshold: 120
         periodSeconds: 15
       volumeMounts:
@@ -91,13 +85,10 @@ spec:
         name: models
     deploymentStrategy:
       type: Recreate
-    # 2 replicas -- each its own GPU, each loading both ornith:35b and
-    # qwen2.5:3b-instruct -- so 2 concurrent implementer-style calls each
-    # get an independent instance instead of contending on one, at the
-    # cost of judge/qwen traffic still sharing whichever replica an
-    # implementer call also lands on.
-    maxReplicas: 2
-    minReplicas: 2
+    # 1 replica -- ornith:35b only. qwen2.5:3b moved to CPU on cp-2.
+    # Frees 1 GPU for ComfyUI.
+    maxReplicas: 1
+    minReplicas: 1
     nodeSelector:
       kubernetes.io/hostname: worker-1
     runtimeClassName: nvidia

k8s/apps/llm-serving/qwen-cpu.yaml

@@ -0,0 +1,115 @@
+# qwen2.5:3b-instruct on CPU (talos-cp-2, 144GB RAM, 24 cores).
+# Moved off GPU to free a V100 for ComfyUI. Latency ~10x slower
+# than GPU but sufficient for lightweight tasks (summarization,
+# classification, quick answers).
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: qwen-cpu
+  namespace: llm-serving
+  labels:
+    app: qwen-cpu
+    app.kubernetes.io/name: qwen-cpu
+    app.kubernetes.io/part-of: llm-serving
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: qwen-cpu
+  template:
+    metadata:
+      labels:
+        app: qwen-cpu
+        app.kubernetes.io/name: qwen-cpu
+        app.kubernetes.io/part-of: llm-serving
+    spec:
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-2
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: ollama
+          image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              ollama serve &
+              SERVE_PID=$!
+              until ollama list >/dev/null 2>&1; do sleep 2; done
+              ollama pull qwen2.5:3b-instruct
+              ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
+              wait $SERVE_PID
+          env:
+            - name: OLLAMA_HOST
+              value: "0.0.0.0:8080"
+            - name: OLLAMA_MODELS
+              value: /root/.ollama/models
+            - name: OLLAMA_CONTEXT_LENGTH
+              value: "32768"
+            - name: OLLAMA_KEEP_ALIVE
+              value: "-1"
+            - name: OLLAMA_MAX_LOADED_MODELS
+              value: "1"
+            - name: OLLAMA_NUM_PARALLEL
+              value: "2"
+          ports:
+            - containerPort: 8080
+              protocol: TCP
+          readinessProbe:
+            exec:
+              command: ["/bin/sh", "-c", "ollama ps 2>/dev/null | grep -q qwen2.5"]
+            periodSeconds: 10
+          startupProbe:
+            exec:
+              command: ["/bin/sh", "-c", "ollama ps 2>/dev/null | grep -q qwen2.5"]
+            failureThreshold: 60
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: "4"
+              memory: 4Gi
+            limits:
+              cpu: "8"
+              memory: 8Gi
+          volumeMounts:
+            - mountPath: /root/.ollama
+              name: ollama-data
+      volumes:
+        - name: ollama-data
+          persistentVolumeClaim:
+            claimName: qwen-cpu-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: qwen-cpu
+  namespace: llm-serving
+  labels:
+    app: qwen-cpu
+    app.kubernetes.io/part-of: llm-serving
+spec:
+  selector:
+    app: qwen-cpu
+  ports:
+    - port: 80
+      targetPort: 8080
+      protocol: TCP
+---
+# Small PVC for qwen2.5:3b model weights (~1.9GB).
+# Separate from llm-models PVC which is pinned to worker-1.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: qwen-cpu-data
+  namespace: llm-serving
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 5Gi

k8s/apps/messaging/management-service/templates/deployment.yaml

@@ -13,6 +13,7 @@ spec:
       labels:
         app: management-service
     spec:
+      serviceAccountName: kmsvc
       topologySpreadConstraints:
         - maxSkew: 1
           topologyKey: kubernetes.io/hostname

k8s/apps/messaging/management-service/values.yaml

@@ -2,7 +2,7 @@ namespace: sqs
 replicaCount: 3
 
 image:
-  repository: ghcr.io/riotpiaole/kmsvc-management-service
+  repository: forgejo.riotpiao.com/rock/kmsvc-manage
   tag: latest
   pullPolicy: Always
 

k8s/apps/messaging/memory-queues/Chart.yaml

@@ -0,0 +1,6 @@
+apiVersion: v2
+name: memory-queues
+description: Kafka queues (DLQ) for Poimen Memory service (Phase 6.6)
+type: application
+version: 0.1.0
+appVersion: "1.0"

k8s/apps/messaging/memory-queues/templates/queues.yaml

@@ -0,0 +1,20 @@
+{{- range .Values.queues }}
+---
+apiVersion: kmsvc.io/v1alpha1
+kind: Queue
+metadata:
+  name: {{ .name }}
+  namespace: {{ $.Values.namespace }}
+  labels:
+    app: memory-service
+    queue: dlq
+spec:
+  name: {{ .name }}
+  description: {{ .description }}
+  partitions: {{ .partitions }}
+  replicationFactor: {{ .replicationFactor }}
+  config:
+    retention.ms: "{{ .config.retention.ms }}"
+    message.retention.seconds: "{{ .config.message.retention.seconds }}"
+    visibility.timeout.seconds: "{{ .config.visibility.timeout.seconds }}"
+{{- end }}

k8s/apps/messaging/memory-queues/values.yaml

@@ -0,0 +1,25 @@
+# Poimen Memory Service Kafka Queues (kmsvc)
+# Phase 6.6: DLQ topics for webhook + metrics failures
+
+queues:
+  # DLQ for extraction, webhook, and agent failures
+  - name: poimen-memory-dlq
+    description: "DLQ for extraction, webhook, and agent failures"
+    partitions: 3
+    replicationFactor: 1
+    config:
+      retention.ms: "1209600000"  # 14 days
+      message.retention.seconds: "1209600"
+      visibility.timeout.seconds: "300"
+
+  # DLQ for metrics persistence failures  
+  - name: poimen-memory-metric-dlq
+    description: "DLQ for metrics persistence failures"
+    partitions: 3
+    replicationFactor: 1
+    config:
+      retention.ms: "1209600000"  # 14 days
+      message.retention.seconds: "1209600"
+      visibility.timeout.seconds: "300"
+
+namespace: sqs

k8s/apps/messaging/queue-crd/values.yaml

@@ -1,7 +1,7 @@
 namespace: sqs
 
 image:
-  repository: ghcr.io/riotpiaole/kmsvc-management-service
+  repository: forgejo.riotpiao.com/rock/kmsvc-manage
   tag: latest
   pullPolicy: Always
 

k8s/apps/paperless/adapter-configmap.yaml

@@ -0,0 +1,95 @@
+# Overrides paperless-ngx's own paperless/adapter.py at the same import path
+# (mounted via subPath in deployment.yaml) - settings.py hardcodes
+# SOCIALACCOUNT_ADAPTER = "paperless.adapter.CustomSocialAccountAdapter", so
+# no Django setting needs to change, just the file content underneath it.
+#
+# Stock CustomSocialAccountAdapter.populate_user() is a stub ("kept in case
+# global default permissions are implemented in the future" - they aren't),
+# so every OIDC signup lands with zero permissions and 403s on every API
+# endpoint. This adds the actual mapping: Authentik's "permissions" claim
+# (via the permissions scope, requested in PAPERLESS_SOCIALACCOUNT_PROVIDERS,
+# computed server-side from group membership by authentik-provision.py) ->
+# "paperless:write" or "*" (homelab-admins) grants is_staff+is_superuser,
+# same convention already used for MinIO's policy claim and Grafana's
+# role_attribute_path. Checking the permission string rather than a literal
+# group name decouples "what grants access" from which group happens to
+# hold it - same pattern applies to every other service's Role/RoleBinding
+# in k8s/infra/rbac/.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: paperless-adapter
+data:
+  adapter.py: |
+    from urllib.parse import quote
+
+    from allauth.account.adapter import DefaultAccountAdapter
+    from allauth.core import context
+    from allauth.socialaccount.adapter import DefaultSocialAccountAdapter
+    from django.conf import settings
+    from django.forms import ValidationError
+    from django.urls import reverse
+
+    REQUIRED_PERMISSIONS = {"paperless:write", "*"}
+
+
+    class CustomAccountAdapter(DefaultAccountAdapter):
+        def is_open_for_signup(self, request):
+            allow_signups = super().is_open_for_signup(request)
+            return getattr(settings, "ACCOUNT_ALLOW_SIGNUPS", allow_signups)
+
+        def pre_authenticate(self, request, **credentials):
+            if settings.DISABLE_REGULAR_LOGIN:
+                raise ValidationError("Regular login is disabled")
+            return super().pre_authenticate(request, **credentials)
+
+        def is_safe_url(self, url):
+            from django.utils.http import url_has_allowed_host_and_scheme
+
+            allowed_hosts = {context.request.get_host()} | set(settings.ALLOWED_HOSTS)
+            if "*" in allowed_hosts:
+                allowed_hosts.remove("*")
+                allowed_hosts.add(context.request.get_host())
+                return url_has_allowed_host_and_scheme(url, allowed_hosts=allowed_hosts)
+            return url_has_allowed_host_and_scheme(url, allowed_hosts=allowed_hosts)
+
+        def get_reset_password_from_key_url(self, key):
+            if settings.PAPERLESS_URL is None:
+                return super().get_reset_password_from_key_url(key)
+            path = reverse(
+                "account_reset_password_from_key",
+                kwargs={"uidb36": "UID", "key": "KEY"},
+            )
+            path = path.replace("UID-KEY", quote(key))
+            return settings.PAPERLESS_URL + path
+
+
+    class CustomSocialAccountAdapter(DefaultSocialAccountAdapter):
+        def is_open_for_signup(self, request, sociallogin):
+            allow_signups = super().is_open_for_signup(request, sociallogin)
+            return getattr(settings, "SOCIALACCOUNT_ALLOW_SIGNUPS", allow_signups)
+
+        def get_connect_redirect_url(self, request, socialaccount):
+            return reverse("base")
+
+        def populate_user(self, request, sociallogin, data):
+            user = super().populate_user(request, sociallogin, data)
+            perms = set(sociallogin.account.extra_data.get("permissions") or [])
+            if perms & REQUIRED_PERMISSIONS:
+                user.is_staff = True
+                user.is_superuser = True
+            return user
+
+        def save_user(self, request, sociallogin, form=None):
+            # populate_user() sets the flags on the in-memory user, but
+            # allauth's default save_user() re-derives is_staff from
+            # ACCOUNT_DEFAULT_HTTP_PROTOCOL-independent defaults and can
+            # overwrite them on save - re-apply after super().save_user()
+            # persists the row, matching the permissions check above exactly.
+            user = super().save_user(request, sociallogin, form)
+            perms = set(sociallogin.account.extra_data.get("permissions") or [])
+            if perms & REQUIRED_PERMISSIONS and not (user.is_staff and user.is_superuser):
+                user.is_staff = True
+                user.is_superuser = True
+                user.save(update_fields=["is_staff", "is_superuser"])
+            return user

k8s/apps/paperless/backup-cronjob.yaml

@@ -0,0 +1,94 @@
+# Nightly: pg_dump the paperless DB + mirror the media PVC into the scoped
+# `paperless` MinIO bucket (see minio-provision-paperless-job.yaml). This is a
+# BACKUP target, not live storage - paperless-ngx has no native S3 backend, it
+# only ever reads/writes the local media PVC directly.
+#
+# Pinned to talos-cp-3, same as deployment.yaml: media is a ReadWriteOnce
+# Longhorn volume with a single replica physically on that node's disk -
+# mounting it read-only here from a different node would conflict with the
+# live webserver's attachment.
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: paperless-backup
+spec:
+  schedule: "0 3 * * *"  # 03:00 daily, low-traffic window
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      template:
+        spec:
+          restartPolicy: Never
+          nodeSelector:
+            kubernetes.io/hostname: talos-cp-3
+          initContainers:
+            - name: pg-dump
+              image: postgres:16-alpine
+              env:
+                - name: PGHOST
+                  value: paperless-db-rw
+                - name: PGDATABASE
+                  value: paperless
+                - name: PGUSER
+                  valueFrom:
+                    secretKeyRef:
+                      name: paperless-db-app
+                      key: username
+                - name: PGPASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: paperless-db-app
+                      key: password
+              command:
+                - sh
+                - -c
+                - pg_dump --format=custom --file=/backup/paperless-db.dump
+              volumeMounts:
+                - name: backup
+                  mountPath: /backup
+          containers:
+            - name: mc-mirror
+              image: minio/mc:latest
+              env:
+                - name: ACCESS_KEY
+                  valueFrom:
+                    secretKeyRef:
+                      name: paperless-minio-creds
+                      key: ACCESS_KEY
+                - name: SECRET_KEY
+                  valueFrom:
+                    secretKeyRef:
+                      name: paperless-minio-creds
+                      key: SECRET_KEY
+                - name: BUCKET
+                  valueFrom:
+                    secretKeyRef:
+                      name: paperless-minio-creds
+                      key: BUCKET
+                - name: ENDPOINT
+                  valueFrom:
+                    secretKeyRef:
+                      name: paperless-minio-creds
+                      key: ENDPOINT
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -e
+                  mc alias set b "$ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY"
+                  mc cp /backup/paperless-db.dump "b/$BUCKET/db/paperless-db-$(date +%Y%m%d).dump"
+                  mc mirror --overwrite /media "b/$BUCKET/media"
+                  echo "Backup done."
+              volumeMounts:
+                - name: backup
+                  mountPath: /backup
+                - name: media
+                  mountPath: /media
+                  readOnly: true
+          volumes:
+            - name: backup
+              emptyDir: {}
+            - name: media
+              persistentVolumeClaim:
+                claimName: paperless-media
+                readOnly: true

k8s/apps/paperless/configmap.yaml

@@ -0,0 +1,22 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: paperless-config
+data:
+  PAPERLESS_URL: "https://paperless.riotpiao.com"
+  PAPERLESS_TIME_ZONE: "America/Los_Angeles"
+  PAPERLESS_OCR_LANGUAGE: "eng"
+  PAPERLESS_DBHOST: "paperless-db-rw"
+  PAPERLESS_DBNAME: "paperless"
+  PAPERLESS_REDIS: "redis://paperless-redis:6379"
+  # django-allauth generic OIDC provider. The client_id/secret/server_url
+  # bundle itself lives in the paperless-oidc Secret
+  # (SOCIALACCOUNT_PROVIDERS_JSON key, composed by authentik-provision.py) -
+  # env vars can't be split across a ConfigMap + Secret for the same key, so
+  # this whole value is sourced from the Secret in deployment.yaml instead.
+  PAPERLESS_APPS: "allauth.socialaccount.providers.openid_connect"
+  # Authentik already verifies identity via OIDC - a second email-confirmation
+  # step has no SMTP configured to send it anyway, and paperless-ngx doesn't
+  # wire up allauth's confirm-email view, so signup 500s with NoReverseMatch
+  # on 'account_confirm_email' without this.
+  PAPERLESS_ACCOUNT_EMAIL_VERIFICATION: "none"

k8s/apps/paperless/deployment.yaml

@@ -0,0 +1,103 @@
+# Single container runs webserver + consumer + scheduler (paperless-ngx's
+# stock entrypoint does this internally) - no need to split into separate
+# Deployments. replicas: 1 only: paperless-media is ReadWriteOnce, and the
+# consumer polling the media dir doesn't benefit from horizontal scaling here.
+#
+# Pinned to talos-cp-3: paperless-media's disk physically lives there. Longhorn
+# RWO volumes can only be attached from one node at a time, and the nightly
+# backup-cronjob.yaml also mounts this same PVC (read-only) to mirror it into
+# MinIO - pinning both to the same node avoids a cross-node attach conflict,
+# and keeps the 3.5Ti read/write path off the network entirely.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: paperless
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate  # ReadWriteOnce media PVC - avoid two pods fighting over it
+  selector:
+    matchLabels:
+      app: paperless
+  template:
+    metadata:
+      labels:
+        app: paperless
+    spec:
+      # Kubernetes injects legacy Docker-links env vars for every Service in
+      # this namespace (<SVC>_SERVICE_HOST, <SVC>_PORT, ...). The Service here
+      # is named "paperless", so that becomes PAPERLESS_PORT=tcp://<ip>:8000 -
+      # paperless-ngx's own entrypoint reads PAPERLESS_PORT for gunicorn's
+      # bind address, collides, and gunicorn crash-loops on "not a valid port
+      # number". Disable the injection instead of renaming the Service.
+      enableServiceLinks: false
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-3
+      containers:
+        - name: paperless
+          image: ghcr.io/paperless-ngx/paperless-ngx:2.20.15
+          ports:
+            - containerPort: 8000
+          envFrom:
+            - configMapRef:
+                name: paperless-config
+          env:
+            - name: PAPERLESS_DBUSER
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-db-app
+                  key: username
+            - name: PAPERLESS_DBPASS
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-db-app
+                  key: password
+            - name: PAPERLESS_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-secrets
+                  key: PAPERLESS_SECRET_KEY
+            - name: PAPERLESS_ADMIN_USER
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-secrets
+                  key: PAPERLESS_ADMIN_USER
+            - name: PAPERLESS_ADMIN_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-secrets
+                  key: PAPERLESS_ADMIN_PASSWORD
+            - name: PAPERLESS_SOCIALACCOUNT_PROVIDERS
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-oidc
+                  key: SOCIALACCOUNT_PROVIDERS_JSON
+          resources:
+            requests: { cpu: "500m", memory: "1Gi" }
+            limits: { cpu: "2", memory: "4Gi" }
+          volumeMounts:
+            - name: media
+              mountPath: /usr/src/paperless/media
+            - name: data
+              mountPath: /usr/src/paperless/data
+            - name: consume
+              mountPath: /usr/src/paperless/consume
+            # Overrides paperless-ngx's own adapter.py in place - settings.py
+            # hardcodes the import path, so no Django setting changes, just
+            # the file content underneath it (see adapter-configmap.yaml).
+            - name: adapter
+              mountPath: /usr/src/paperless/src/paperless/adapter.py
+              subPath: adapter.py
+              readOnly: true
+      volumes:
+        - name: media
+          persistentVolumeClaim:
+            claimName: paperless-media
+        - name: data
+          persistentVolumeClaim:
+            claimName: paperless-data
+        - name: consume
+          emptyDir: {}
+        - name: adapter
+          configMap:
+            name: paperless-adapter

k8s/apps/paperless/ingress.yaml

@@ -0,0 +1,24 @@
+# Direct nginx ingress to the paperless Service - not routed via the Go
+# api-gateway (api.riotpiao.com), which has no WebSocket upgrade support and
+# paperless-ngx keeps a long-lived /ws/ connection open for live task status.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: paperless
+  annotations:
+    nginx.ingress.kubernetes.io/proxy-body-size: "0"  # large scanned PDF uploads
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: paperless.riotpiao.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: paperless
+                port:
+                  number: 8000

k8s/apps/paperless/kustomization.yaml

@@ -0,0 +1,17 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: paperless
+resources:
+  - pvc.yaml
+  - configmap.yaml
+  - redis.yaml
+  - deployment.yaml
+  - service.yaml
+  - ingress.yaml
+  - backup-cronjob.yaml
+  - adapter-configmap.yaml
+  - rbac.yaml
+# postgres: paperless-db CNPG Cluster, deployed by k8s/infra/databases (wave 2,
+# before this app at wave 8) - not duplicated here. Same for the paperless-oidc
+# and paperless-minio-creds Secrets, written by PostSync provisioning Jobs in
+# k8s/infra/iam and k8s/infra/minio respectively.

k8s/apps/paperless/pvc.yaml

@@ -0,0 +1,35 @@
+# Two volumes, deliberately separate storage classes:
+#
+# - media: the actual documents (originals + OCR'd archive PDFs + thumbnails).
+#   Lives on the cp-3 USB HDD, single replica (see
+#   k8s/infra/longhorn/longhorn-paperless-storageclass.yaml). Shares the disk
+#   with Immich's immich-media PVC (k8s/apps/immich/pvc.yaml, 2000Gi) - photo
+#   libraries grow much faster than scanned documents, so paperless gets the
+#   smaller 500Gi share.
+# - data: the SQLite classification model + search index. Small (low GB),
+#   frequently rewritten, and disposable (rebuilds from the DB + media on
+#   next consume) - stays on the default 3-replica pool instead of the
+#   single-disk HDD.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperless-media
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn-paperless-media
+  resources:
+    requests:
+      storage: 500Gi
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperless-data
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 5Gi

k8s/apps/paperless/rbac.yaml

@@ -0,0 +1,35 @@
+# Scoped operator access for paperless-admins: restart/config-edit rights on
+# just this service's own resources, nothing CNPG-managed (paperless-db-*)
+# or provisioning-managed (paperless-oidc, paperless-minio-creds). Inert
+# until kube-apiserver's OIDC wiring lands (--oidc-groups-claim=groups,
+# --oidc-groups-prefix=oidc:) - subject name below assumes that prefix.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: paperless-operator
+rules:
+  - apiGroups: ["apps"]
+    resources: ["deployments"]
+    resourceNames: ["paperless"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    resourceNames: ["paperless-config"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+  - apiGroups: [""]
+    resources: ["secrets"]
+    resourceNames: ["paperless-secrets"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: paperless-admins-binding
+subjects:
+  - kind: Group
+    name: "oidc:paperless-admins"
+    apiGroup: rbac.authorization.k8s.io
+roleRef:
+  kind: Role
+  name: paperless-operator
+  apiGroup: rbac.authorization.k8s.io

k8s/apps/paperless/redis.yaml

@@ -0,0 +1,37 @@
+# Task queue broker + websocket channel layer for paperless-ngx. No PVC:
+# queued/scheduled task state is disposable - a lost queue on restart just
+# means re-triggering consumption, not data loss (documents themselves live
+# on paperless-media).
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: paperless-redis
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: paperless-redis
+  template:
+    metadata:
+      labels:
+        app: paperless-redis
+    spec:
+      containers:
+        - name: redis
+          image: redis:7-alpine
+          ports:
+            - containerPort: 6379
+          resources:
+            requests: { cpu: "50m", memory: "64Mi" }
+            limits: { cpu: "250m", memory: "256Mi" }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: paperless-redis
+spec:
+  selector:
+    app: paperless-redis
+  ports:
+    - port: 6379
+      targetPort: 6379

k8s/apps/paperless/service.yaml

@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: paperless
+spec:
+  selector:
+    app: paperless
+  ports:
+    - port: 8000
+      targetPort: 8000

k8s/apps/secret-rotation-controller/crd.yaml

@@ -0,0 +1,144 @@
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: secretrotations.homelab.riotpiao.com
+spec:
+  group: homelab.riotpiao.com
+  names:
+    kind: SecretRotation
+    plural: secretrotations
+  scope: Namespaced
+  versions:
+  - name: v1
+    served: true
+    storage: true
+    schema:
+      openAPIV3Schema:
+        type: object
+        properties:
+          metadata:
+            type: object
+          spec:
+            type: object
+            required:
+            - provider
+            - rotationInterval
+            properties:
+              # External system: authentik | forgejo | minio | vault
+              provider:
+                type: string
+                enum: [authentik, forgejo, minio, vault]
+              
+              # How often to rotate (hours)
+              rotationInterval:
+                type: integer
+                minimum: 24
+              
+              # Application ID in external system
+              appId:
+                type: string
+              
+              # k8s Secret to update (name, namespace, key)
+              secretRef:
+                type: object
+                required: [name, namespace]
+                properties:
+                  name:
+                    type: string
+                  namespace:
+                    type: string
+                  key:
+                    type: string
+                    description: "Secret key to update (e.g., MINIO_IDENTITY_OPENID_CLIENT_SECRET)"
+              
+              # Path to git file that holds the secret (for .enc.yaml files)
+              gitPath:
+                type: string
+                description: "Path in homelab repo to .enc.yaml file"
+              
+              # Ansible template values to substitute
+              templateValues:
+                type: object
+                additionalProperties:
+                  type: string
+          
+          status:
+            type: object
+            properties:
+              lastRotationTime:
+                type: string
+                format: date-time
+              nextRotationTime:
+                type: string
+                format: date-time
+              lastRotationStatus:
+                type: string
+                enum: [Success, Failed, Pending]
+              lastRotationError:
+                type: string
+              lastCommitHash:
+                type: string
+
+---
+# Example usage:
+apiVersion: homelab.riotpiao.com/v1
+kind: SecretRotation
+metadata:
+  name: minio-oidc
+  namespace: secret-rotation
+spec:
+  provider: authentik
+  rotationInterval: 2160  # 90 days in hours
+  appId: minio
+  secretRef:
+    name: minio-oidc
+    namespace: storage
+    key: MINIO_IDENTITY_OPENID_CLIENT_SECRET
+  gitPath: k8s/argocd/secrets/minio-oidc.enc.yaml
+
+---
+apiVersion: homelab.riotpiao.com/v1
+kind: SecretRotation
+metadata:
+  name: portfolio-agent-oidc
+  namespace: secret-rotation
+spec:
+  provider: authentik
+  rotationInterval: 2160
+  appId: portfolio-agent
+  secretRef:
+    name: portfolio-agent-oidc
+    namespace: portfolio
+    key: CLIENT_SECRET
+  gitPath: k8s/argocd/secrets/portfolio-agent-oidc.enc.yaml
+
+---
+apiVersion: homelab.riotpiao.com/v1
+kind: SecretRotation
+metadata:
+  name: forgejo-registry-token
+  namespace: secret-rotation
+spec:
+  provider: forgejo
+  rotationInterval: 2160
+  appId: rock/riotpiao.com
+  secretRef:
+    name: forgejo-registry-secret
+    namespace: kube-system
+    key: REGISTRY_TOKEN
+  gitPath: k8s/argocd/secrets/forgejo-registry-secret.enc.yaml
+
+---
+apiVersion: homelab.riotpiao.com/v1
+kind: SecretRotation
+metadata:
+  name: minio-root-credentials
+  namespace: secret-rotation
+spec:
+  provider: minio
+  rotationInterval: 4320  # 180 days in hours
+  appId: root
+  secretRef:
+    name: minio-creds
+    namespace: storage
+  gitPath: k8s/argocd/secrets/minio-secrets.enc.yaml

k8s/apps/secret-rotation-controller/deployment.yaml

@@ -0,0 +1,92 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: secret-rotation-controller
+  namespace: secret-rotation
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: secret-rotation-controller
+  template:
+    metadata:
+      labels:
+        app: secret-rotation-controller
+    spec:
+      serviceAccountName: secret-rotation-controller
+      containers:
+      - name: controller
+        image: secret-rotation-controller:latest
+        imagePullPolicy: IfNotPresent
+        env:
+        # SOPS reads age key from this file
+        - name: SOPS_AGE_KEY_FILE
+          value: /etc/sops/age/private-key.txt
+        
+        # Vault auth (token in projected volume)
+        - name: VAULT_ADDR
+          value: http://vault.vault.svc.cluster.local:8200
+        - name: VAULT_TOKEN_FILE
+          value: /var/run/secrets/vault/token
+        
+        # Authentik
+        - name: AUTHENTIK_URL
+          value: http://authentik-server.iam.svc.cluster.local
+        - name: AUTHENTIK_BOOTSTRAP_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: authentik-bootstrap
+              key: token
+        
+        # Git
+        - name: GIT_REPO
+          value: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+        - name: GIT_AUTHOR_EMAIL
+          value: [email protected]
+        - name: GIT_AUTHOR_NAME
+          value: Secret Rotation Controller
+        - name: FORGEJO_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: forgejo-registry-secret
+              key: REGISTRY_TOKEN
+        
+        volumeMounts:
+        # Age key from ExternalSecret (synced from Vault)
+        - name: age-key
+          mountPath: /etc/sops/age
+          readOnly: true
+        
+        # Vault auth token (projected)
+        - name: vault-token
+          mountPath: /var/run/secrets/vault
+          readOnly: true
+        
+        # Temp working dir
+        - name: tmp
+          mountPath: /tmp
+        
+        resources:
+          requests:
+            cpu: 100m
+            memory: 256Mi
+          limits:
+            cpu: 500m
+            memory: 512Mi
+      
+      volumes:
+      - name: age-key
+        secret:
+          secretName: sops-age-key
+          defaultMode: 0400
+      
+      - name: vault-token
+        projected:
+          sources:
+          - serviceAccountToken:
+              path: token
+              audience: vault
+              expirationSeconds: 3600
+      
+      - name: tmp
+        emptyDir: {}

k8s/apps/secret-rotation-controller/kustomization.yaml

@@ -0,0 +1,15 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: secret-rotation
+
+resources:
+  - rbac.yaml
+  - crd.yaml
+  - external-secret.yaml
+  - deployment.yaml
+
+commonLabels:
+  app.kubernetes.io/name: secret-rotation-controller
+  app.kubernetes.io/component: automation
+  managed-by: argocd

k8s/apps/secret-rotation-controller/rbac.yaml

@@ -0,0 +1,53 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: secret-rotation-controller
+  namespace: secret-rotation
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: secret-rotation-controller
+rules:
+# Read SecretRotation CRDs
+- apiGroups: ["homelab.riotpiao.com"]
+  resources: ["secretrotations"]
+  verbs: ["get", "list", "watch"]
+
+# Update status
+- apiGroups: ["homelab.riotpiao.com"]
+  resources: ["secretrotations/status"]
+  verbs: ["get", "patch", "update"]
+
+# Read k8s secrets that will be rotated
+- apiGroups: [""]
+  resources: ["secrets"]
+  verbs: ["get", "list"]
+
+# For recording events
+- apiGroups: [""]
+  resources: ["events"]
+  verbs: ["create", "patch"]
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: secret-rotation-controller
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: secret-rotation-controller
+subjects:
+- kind: ServiceAccount
+  name: secret-rotation-controller
+  namespace: secret-rotation
+
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: secret-rotation
+  labels:
+    kubernetes.io/metadata.name: secret-rotation

k8s/argocd/apps/-1-projects.yaml

@@ -18,7 +18,7 @@ spec:
       prune: true
       selfHeal: true
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/argocd/projects
   destination:

k8s/argocd/apps/00-secrets.yaml

@@ -17,7 +17,7 @@ spec:
     syncOptions:
       - CreateNamespace=true
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     # ksops decrypts every *.enc.yaml here at kustomize-build time (repo-server
     # runs `kustomize build --enable-alpha-plugins --enable-exec`). Replaces the

k8s/argocd/apps/00-substrate.yaml

@@ -21,7 +21,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/bootstrap/cert-manager/cert-manager-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -93,7 +93,7 @@ spec:
   project: homelab
   revisionHistoryLimit: 3
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     # A real kustomization.yaml (resources: the 3 issuer/CA files) renders these
     # deterministically. The previous directory.include with bare filenames
@@ -127,7 +127,7 @@ spec:
   project: homelab
   revisionHistoryLimit: 3
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/bootstrap/ingress
   destination:
@@ -138,3 +138,75 @@ spec:
       selfHeal: true
     syncOptions:
       - CreateNamespace=true
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: cluster-maintenance
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/infra/cluster-maintenance
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: kube-system
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: kyverno
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"
+spec:
+  project: homelab
+  source:
+    repoURL: https://kyverno.github.io/kyverno/
+    chart: kyverno
+    targetRevision: "1.14.0"
+    helm:
+      valueFiles:
+        - $values/k8s/bootstrap/kyverno/kyverno-values.yaml
+  sources:
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+      targetRevision: main
+      ref: values
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: kyverno
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: kyverno-policies
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/bootstrap/kyverno
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: kyverno
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true

k8s/argocd/apps/01-image-updater.yaml

@@ -0,0 +1,33 @@
+# ArgoCD Image Updater - auto-updates Application images from registry
+# Watches forgejo.riotpiao.com for new image tags and updates Applications
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: argocd-image-updater
+  namespace: argocd
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+  annotations:
+    argocd.argoproj.io/sync-wave: "1"
+spec:
+  project: homelab
+  revisionHistoryLimit: 3
+  sources:
+    - repoURL: https://argoproj.github.io/argo-helm
+      chart: argocd-image-updater
+      targetRevision: "0.11.2"
+      helm:
+        valueFiles:
+          - $values/k8s/infra/argocd-image-updater/values.yaml
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+      targetRevision: main
+      ref: values
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: argocd
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=false

k8s/argocd/apps/08-secret-rotation.yaml

@@ -0,0 +1,32 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: secret-rotation
+  namespace: argocd
+  labels:
+    app.kubernetes.io/name: secret-rotation
+spec:
+  project: homelab
+  
+  sources:
+  - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    path: k8s/apps/secret-rotation-controller
+    targetRevision: main
+  
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: secret-rotation
+  
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+    - CreateNamespace=true
+    - RespectIgnoreDifferences=true
+    retry:
+      limit: 5
+      backoff:
+        duration: 5s
+        factor: 2
+        maxDuration: 3m

k8s/argocd/apps/10-storage-observability.yaml

@@ -17,7 +17,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/minio/minio-operator-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -41,7 +41,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/infra/minio
   destination:
@@ -66,7 +66,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/infra/longhorn
   destination:
@@ -102,7 +102,7 @@ spec:
         skipCrds: true
         valueFiles:
           - $values/k8s/infra/monitoring/prometheus-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -152,7 +152,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/infra/monitoring/crds
   destination:
@@ -183,7 +183,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/infra/monitoring
   destination:
@@ -213,7 +213,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/monitoring/blackbox-exporter-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -223,3 +223,29 @@ spec:
     automated:
       prune: true
       selfHeal: true
+---
+# Distributed tracing: Tempo + OpenTelemetry Collector.
+# Receives traces from instrumented services, stores in local volume (72h retention).
+# Grafana datasource auto-configured, service graph + latency dashboards included.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: tracing
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "1"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/infra/tracing
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: tracing
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true

k8s/argocd/apps/20-logging.yaml

@@ -19,7 +19,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/logging/loki-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -53,7 +53,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/logging/grafana-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -87,7 +87,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/logging/promtail-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
   destination:

k8s/argocd/apps/30-security.yaml

@@ -17,7 +17,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/iam/vault-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -46,7 +46,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/iam/authentik-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -68,7 +68,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/infra/iam
   destination:
@@ -109,7 +109,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -152,10 +152,17 @@ metadata:
   namespace: argocd
   annotations:
     argocd.argoproj.io/sync-wave: "3"
+    argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-golang
+    argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
+    argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
+    argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
+    argocd-image-updater.argoproj.io/write-back-method: git
+    argocd-image-updater.argoproj.io/git-branch: main
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/infra/forgejo-runner
   destination:
@@ -173,10 +180,17 @@ metadata:
   namespace: argocd
   annotations:
     argocd.argoproj.io/sync-wave: "3"
+    argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-node
+    argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
+    argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
+    argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
+    argocd-image-updater.argoproj.io/write-back-method: git
+    argocd-image-updater.argoproj.io/git-branch: main
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/infra/forgejo-runner
     helm:
@@ -197,10 +211,17 @@ metadata:
   namespace: argocd
   annotations:
     argocd.argoproj.io/sync-wave: "3"
+    argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-rust
+    argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
+    argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
+    argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
+    argocd-image-updater.argoproj.io/write-back-method: git
+    argocd-image-updater.argoproj.io/git-branch: main
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/infra/forgejo-runner
     helm:

k8s/argocd/apps/40-data.yaml

@@ -14,7 +14,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/infra/databases
   destination:

k8s/argocd/apps/50-memory-queues.yaml

@@ -0,0 +1,20 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: memory-queues
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "7"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/apps/messaging/memory-queues
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: sqs
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true

k8s/argocd/apps/50-messaging.yaml

@@ -1,6 +1,6 @@
-# Wave 5 — Kafka (Strimzi operator + cluster CR), Redis, and the SQS-like
-# queue services. Strimzi/Redis are public Helm charts; kafka-cluster/queue-crd/
-# management-service are local charts (rendered from their own Chart.yaml).
+# Wave 5 — Kafka (Strimzi operator + cluster CR), Redis infrastructure.
+# Strimzi/Redis are public Helm charts; kafka-cluster is a local chart.
+# queue-crd and management-service are managed by kmsvc-root (kmsvc-manage.git).
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
@@ -68,7 +68,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/apps/messaging/kafka-cluster
   destination:
@@ -78,45 +78,5 @@ spec:
     automated:
       prune: true
       selfHeal: true
----
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: queue-crd
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: "6"
-spec:
-  project: homelab
-  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
-    targetRevision: main
-    path: k8s/apps/messaging/queue-crd
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: sqs
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
----
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: management-service
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: "7"
-spec:
-  project: homelab
-  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
-    targetRevision: main
-    path: k8s/apps/messaging/management-service
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: sqs
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
+# queue-crd and management-service moved to kmsvc-manage.git repo
+# Managed by kmsvc-root Application

k8s/argocd/apps/55-api-gateway.yaml

@@ -9,8 +9,12 @@
 # `POST /v1/chat/completions` covers every model. See
 # docs/adr/ADR-0001-retire-kong-for-go-gateway.md in the frontend repo.
 #
+# UPDATED 2026-08-22: Tracks main branch of homelab-frontend (auto-syncs on each push).
+# Image built on every main commit with tag <commit-sha>.
+# ArgoCD auto-pulls the latest image (live reconciliation ~3min).
+#
 # Two sources:
-#   1. rock/homelab-frontend on the in-cluster Forgejo — the gateway's own
+#   1. rock/homelab-frontend on the in-cluster Forgejo (prod branch) — the gateway's own
 #      kustomization (Deployment, Service, ConfigMap, RBAC, NetworkPolicy). It
 #      sets `namespace: api` itself, so no transformer is needed here. The
 #      Forgejo host must stay listed in the `homelab` AppProject sourceRepos or
@@ -32,14 +36,19 @@ metadata:
     app.kubernetes.io/component: gateway
   annotations:
     argocd.argoproj.io/sync-wave: "7"
+    # ArgoCD Image Updater - auto-update on new image push
+    argocd-image-updater.argoproj.io/image-list: gw=forgejo.riotpiao.com/rock/api-gateway
+    argocd-image-updater.argoproj.io/gw.update-strategy: digest
+    argocd-image-updater.argoproj.io/gw.allow-tags: regexp:^latest$
+    argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
   project: homelab
   revisionHistoryLimit: 3
   sources:
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab-frontend.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
       targetRevision: main
       path: k8s
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       path: k8s/apps/api
   destination:

k8s/argocd/apps/56-llm-serving.yaml

@@ -20,7 +20,7 @@ spec:
   project: homelab
   revisionHistoryLimit: 3
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/apps/llm-serving
   destination:

k8s/argocd/apps/58-comfyui.yaml

@@ -0,0 +1,32 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: comfyui
+  namespace: argocd
+  labels:
+    app.kubernetes.io/name: comfyui
+    app.kubernetes.io/component: image-generation
+  annotations:
+    argocd.argoproj.io/sync-wave: "8"
+spec:
+  project: homelab
+  revisionHistoryLimit: 3
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/apps/comfyui
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: comfyui
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+    retry:
+      limit: 5
+      backoff:
+        duration: 5s
+        factor: 2
+        maxDuration: 3m

k8s/argocd/apps/59-gotify.yaml

@@ -0,0 +1,32 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: gotify
+  namespace: argocd
+  labels:
+    app.kubernetes.io/name: gotify
+    app.kubernetes.io/component: notifications
+  annotations:
+    argocd.argoproj.io/sync-wave: "8"
+spec:
+  project: homelab
+  revisionHistoryLimit: 3
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/apps/gotify
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: notifications
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+    retry:
+      limit: 5
+      backoff:
+        duration: 5s
+        factor: 2
+        maxDuration: 3m

k8s/argocd/apps/60-applications.yaml

@@ -19,10 +19,10 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/apps/temporal/temporal-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       path: k8s/apps/temporal
   destination:
@@ -51,7 +51,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/apps/portainer/portainer-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -74,7 +74,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/apps/cloudflared
   destination:
@@ -97,7 +97,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/apps/agent-pod
   destination:
@@ -130,7 +130,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/apps/sms
   destination:
@@ -141,6 +141,67 @@ spec:
       prune: true
       selfHeal: true
 ---
+# Document management. Raw manifests (no Helm): postgres is the dedicated
+# paperless-db CNPG cluster in k8s/infra/databases (wave 2), redis is
+# in-cluster only (no PVC), media lives on the cp-3 USB HDD (see
+# k8s/infra/longhorn/longhorn-paperless-storageclass.yaml). OIDC via
+# Authentik provisioned by k8s/infra/iam's PostSync job; MinIO backup bucket
+# creds provisioned by k8s/infra/minio's PostSync job.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: paperless
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "8"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/apps/paperless
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: paperless
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+---
+# Photo/video backup. Self-contained (unlike paperless, its CNPG Postgres
+# lives here too, not in k8s/infra/databases) - CreateNamespace=true creates
+# the namespace before any manifest in this Application applies, including
+# the Cluster CR, so no separate wave-2 pre-creation step is needed. Postgres
+# is pg18 (not this repo's usual 16.2) because CNPG's official pgvector
+# extension image only publishes pg18 builds - see k8s/apps/immich/db.yaml.
+# media PVC shares the cp-3 HDD 2TB/2TB with paperless-media. OIDC via
+# Authentik provisioned by k8s/infra/iam's PostSync job (immich entry in
+# SERVICES + immich_role scope mapping for admin-via-claim).
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: immich
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "8"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/apps/immich
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: immich
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+---
 # Consolidated: homarr + homarr-patches → homarr
 # Helm chart + values + PostSync hook patch (fix-probes-job.yaml)
 apiVersion: argoproj.io/v1alpha1
@@ -159,10 +220,10 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/apps/homarr/homarr-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       ref: values
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
       targetRevision: main
       path: k8s/apps/homarr  # PostSync hook: fix-probes-job.yaml
   destination:
@@ -174,3 +235,60 @@ spec:
       selfHeal: true
     syncOptions:
       - CreateNamespace=true
+---
+# Portfolio site at riotpiao.com - static Next.js site from rock/riotpiao.com repo.
+# Points directly to infra/portfolio/base (bypassing repo's own argocd-apps.yaml
+# which has wrong URLs). Image built by Forgejo Actions on rock/portfolio repo.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: portfolio
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "8"
+    # ArgoCD Image Updater - auto-update on new image push
+    argocd-image-updater.argoproj.io/image-list: app=forgejo.riotpiao.com/rock/portfolio
+    argocd-image-updater.argoproj.io/app.update-strategy: digest
+    argocd-image-updater.argoproj.io/app.allow-tags: regexp:^latest$
+    argocd-image-updater.argoproj.io/write-back-method: argocd
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/rock/riotpiao.com.git
+    targetRevision: main
+    path: infra/portfolio/base
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: portfolio
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+---
+# Wave 9 - per-service scoped RBAC (Role/RoleBinding), deliberately last so
+# every target namespace above already exists. Inert until kube-apiserver
+# gets --oidc-groups-claim=groups wired up (separate, not-yet-applied
+# terraform/talosctl change) - these grant nothing until then.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: rbac
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "9"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/infra/rbac
+  destination:
+    server: https://kubernetes.default.svc
+    # No namespace: cluster-scoped resources (ClusterRoleBinding, etc.)
+    # Namespace is set per-resource in kustomization
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true

k8s/argocd/apps/60-kmsvc-manage.yaml

@@ -0,0 +1,28 @@
+# kmsvc-manage bootstrap — manages itself and its supporting services
+# (Strimzi/Kafka, Redis, queue-operator, message-plane server) from
+# the kmsvc-manage repo's own k8s/argocd/ structure on the main branch.
+# Image built on every main commit, auto-deployed to sqs namespace.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: kmsvc-root
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "6"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/kmsvc-manage.git
+    targetRevision: main
+    path: k8s/argocd/apps
+    directory:
+      recurse: false
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: sqs
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true

k8s/argocd/apps/70-poimen.yaml

@@ -0,0 +1,46 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: poimen
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "7"
+    # Image Updater: auto-update on new image push (SHA tag filter)
+    argocd-image-updater.argoproj.io/image-list: |
+      memory=forgejo.riotpiao.com/rock/poimen-memory
+      workflows=forgejo.riotpiao.com/rock/poimen-workflows
+      frontend=forgejo.riotpiao.com/rock/poimen-frontend
+    argocd-image-updater.argoproj.io/memory.update-strategy: digest
+    argocd-image-updater.argoproj.io/memory.allow-tags: regexp:^latest$
+    argocd-image-updater.argoproj.io/workflows.update-strategy: digest
+    argocd-image-updater.argoproj.io/workflows.allow-tags: regexp:^latest$
+    argocd-image-updater.argoproj.io/frontend.update-strategy: digest
+    argocd-image-updater.argoproj.io/frontend.allow-tags: regexp:^latest$
+    argocd-image-updater.argoproj.io/write-back-method: argocd
+spec:
+  project: homelab
+  sources:
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-memory.git
+      targetRevision: main
+      path: k8s/argocd
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-workflows.git
+      targetRevision: main
+      path: k8s/argocd
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-frontend.git
+      targetRevision: main
+      path: k8s/argocd
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: poimen
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+    retry:
+      limit: 5
+      backoff:
+        duration: 5s
+        factor: 2
+        maxDuration: 3m

k8s/argocd/projects/homelab-project.yaml

@@ -12,9 +12,19 @@ spec:
   description: Homelab GitOps — single-repo, in-cluster destinations only
   sourceRepos:
     - https://github.com/Riotpiaole/riotpiao.homelab.com.git
-    # In-cluster Forgejo wildcard — all rock/* repos can be onboarded without
-    # touching this AppProject. Enabled by Stage 1 (A1).
-    - https://forgejo.riotpiao.com/rock/*
+    # Poimen services (GitHub)
+    - https://github.com/Riotpiaole/Poimen-memory.git
+    - https://github.com/Riotpiaole/Poimen-workflows.git
+    - https://github.com/Riotpiaole/poimen*.git
+    # In-cluster Forgejo repos — explicit allowlist (no wildcard)
+    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
+    - https://forgejo.riotpiao.com/riotpiao-poimen/kmsvc-manage.git
+    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen.git
+    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen-memory.git
+    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen-workflows.git
+    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen-frontend.git
+    - https://forgejo.riotpiao.com/rock/riotpiao.com.git
     # Public Helm chart repos referenced by k8s/argocd/apps/* and bootstrap/*
     - https://cloudnative-pg.github.io/charts
     - https://dl.gitea.com/charts/
@@ -33,6 +43,8 @@ spec:
     - https://charts.jetstack.io
     - https://kubernetes.github.io/ingress-nginx
     - https://stakater.github.io/stakater-charts
+    # ArgoCD ecosystem charts
+    - https://argoproj.github.io/argo-helm
   destinations:
     - server: https://kubernetes.default.svc
       namespace: "*"

k8s/argocd/root/homelab-root.yaml

@@ -12,7 +12,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
     targetRevision: main
     path: k8s/argocd/apps
     directory:
@@ -26,3 +26,4 @@ spec:
       selfHeal: true
     syncOptions:
       - CreateNamespace=true
+      - ServerSideApply=true

k8s/argocd/secrets/forgejo-registry-pat.enc.yaml

@@ -0,0 +1,23 @@
+apiVersion: ENC[AES256_GCM,data:ECE=,iv:bISz4HovH++X7DW1Qj8Cw0L6+EvPB0+68hh7tfyW5C0=,tag:w+SjD7MsfeIuSf62n+Zl7Q==,type:str]
+kind: ENC[AES256_GCM,data:AxXaV3Mb,iv:xfJb354Rrjz3zLctW2i6hl40yh9EsfIqKxDvmn8jqnU=,tag:1xQ/RR3KhPLKCTWCYjebJg==,type:str]
+metadata:
+    name: ENC[AES256_GCM,data:gdQ4OoMYKUPEbsUeA5OI4i1xnh8=,iv:IA2kaCyqQmkYpLolvcTF4aleh+yd/ImXJMhRMvpGCgo=,tag:8qyYYw4EhBKKPzEmpepAeQ==,type:str]
+    namespace: ENC[AES256_GCM,data:/mOSyXWmRg==,iv:cpeHUTlMqlJzJttGtuR3DoiMtvVqFmDS0/5Tl7K7c2M=,tag:q4GxWgWn2wQJxJHqnq4WQA==,type:str]
+type: ENC[AES256_GCM,data:92OcrsuV,iv:Z1XBy6iZ6unGrK4/SSdDa58pbPL3022gU/f1AOM4uvc=,tag:cd4wWRq/ohm6BJ/Eo6HOIw==,type:str]
+stringData:
+    REGISTRY_PAT: ENC[AES256_GCM,data:zDKruOXhzsIcFZTTp4r4r9rwHih9uf1cCp06KZ6eIXSIZDvR6aQY4A==,iv:5Vch5Z7xn1KkRxrgOs2p/M58nd6WhtcuUhoZDCi8LSY=,tag:14SAZIWGOxJMqLBtOorjYQ==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIaG5QVUYyNjhvaENPcUFy
+            K3RSSVM2N2hTeFhNK0M5YWZmODhzZU1HaHlFCmNGZWdFcFEvanVwcXpGSmVvRHVx
+            aG45c2lBY1RuSTYwbDZTVk1QeHNWOTQKLS0tIDg4TnVMNjNtaU1VQk5zQjUvU2hM
+            R245ZVdqc0ZWWVhJb3dOZ3lpU3JjUlUKawSg09ZPq8FKx5tvOVZZ+K4yh7eTQsUp
+            be8mWUpS0+eEmNqh35BwU3HrETMQFA6a1kjVp30JOMtqa5rbYlzF7w==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    lastmodified: "2026-08-23T23:14:53Z"
+    mac: ENC[AES256_GCM,data:yzi6woUaUCi6w9pG/eKnU7k/VfZgoXg/tW8p9joG8p7iabLWMdlXlx23m/CItw/NE0zeGWA5iZPPFZXOit2vN36VzK3kQNoFW6QbhvYLZi+78C/RWIdcpZkxB/tPIRG0vq8Q1SA+rwGjG/0xeAFh+R7k+YBTMd2XWBH3P7EI5T4=,iv:kf6MDaAVrtDvPIEjHMMLxXDSDRC3I1GpsPeJFUYppiw=,tag:NukVbGLa9EoMYmRsa4nBtA==,type:str]
+    unencrypted_suffix: _unencrypted
+    version: 3.13.2

k8s/argocd/secrets/gotify-admin-secrets.enc.yaml

@@ -0,0 +1,24 @@
+apiVersion: ENC[AES256_GCM,data:wR4=,iv:cRBzbvu0eUYCYeKeysua1/P3Meli/rQyj/mIV6VWnPM=,tag:oyTPA/mLYNUX+QDkYLlZyQ==,type:str]
+kind: ENC[AES256_GCM,data:bdKQdadW,iv:F3DQiBI9xhSx87jkS1Hyevo48uUCBjsJSjbScIBznKA=,tag:PFakzzBj5S9F65dxu0L2rg==,type:str]
+metadata:
+    name: ENC[AES256_GCM,data:XHFYrKClPo+IwRbM,iv:ZGuL+cN840Y1bOTC62NhDDcoZpTzDWQ4GfqPJX/QWmI=,tag:hlCVjzvfcxXGOASc3vda/Q==,type:str]
+    namespace: ENC[AES256_GCM,data:0BbhgZBog+1qY/iRJA==,iv:88tiSpEDqIokT5VP/d6bB2+aUyh1kZ7NEHwZWoJW3XU=,tag:CBR01jh2U9h7kNEcK1e1sA==,type:str]
+type: ENC[AES256_GCM,data:Mpv1V73w,iv:BW3r6RpLnwe3XDKwrZySyOjrWUnSwIG5JOoPLzP/5gM=,tag:oyJySL0haOei1m4i+1AJ1g==,type:str]
+stringData:
+    username: ENC[AES256_GCM,data:8xmxLGE=,iv:J/vEvXGoD+ka6FDgnSwDz0fs9IxuJIZW9r7Oyu2qxC8=,tag:dN9jd6NSavMN8+uzvemYWg==,type:str]
+    password: ENC[AES256_GCM,data:vB9PaaUiQZ8FY8pO0cq1fHLi7Gq5t4U=,iv:C0Y1m2SGYP3oTIFoau5JavVmLblj6te/SPMLodIwiZw=,tag:3Ip4YvR4WPzR0bBpTyjytA==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRT3JoTnhVVW5SUUJXbTdz
+            dUpLcmtBWWhyaGk3Y1hBM1ArcVI5eHREbmp3CiswUGVmd0NhejZwQ2UvNEdxS3ow
+            L1RweS9Kb2paeStLZ0tLSFdWbVZqQW8KLS0tIHJHT3hiMmxtM0Q1Ym5KOVpLVFpl
+            enR3NmdNdmdwVitTQVJlRHFWcjR0N2sKQw9ZZs+Ji/Zq/feO3qy4DwaCfWgDOQ/z
+            FVVhcCXweN58tb+9fzCJ+pNi/hSmvUkCMbb1+60qBvEehNzOoMRJ5g==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    lastmodified: "2026-09-10T13:31:00Z"
+    mac: ENC[AES256_GCM,data:3HWV/NG0yTbsxY28u41zTRmAPc1kokGb4nCLmAsyq8/uvxcP+evDNyZXYpS93eVdPRfRZ1OqVBzyVGJEnj7JSXQVmlzor9JcWEL2fcpogoLVfJZKTRD6hk6optnBMjj84iQEEOx3A80JrDOdoXsH0pd9bJBABwoUOJwuufbXcIU=,iv:KLZsmAcapQgV08pFhGIvPt5ylsWg3xazAAjPuJYOaXA=,tag:1vKA3fISMIurHzxZ04F3lg==,type:str]
+    unencrypted_suffix: _unencrypted
+    version: 3.13.2

k8s/argocd/secrets/gotify-smtp-secrets.enc.yaml

@@ -0,0 +1,30 @@
+apiVersion: ENC[AES256_GCM,data:xUA=,iv:YH1WYnoXhxbeES3Dmo8ZrMDwFcwCykt6+Zn4YSgN4gc=,tag:t3bIWo/LetlQv4tJ0nJ2Sw==,type:str]
+kind: ENC[AES256_GCM,data:PJVzDTcT,iv:aRxvzLUuCzSDryOjA/v67csL6E2FcCpoK8aD7KNStno=,tag:cQgbowSTTV99nQ41yFOlsA==,type:str]
+metadata:
+    name: ENC[AES256_GCM,data:atkSiP0rjyuWFjs=,iv:l/48X9W9lrkAyw6Q/p39V775L6xbr0tOIFI4vjYrg+E=,tag:LZrQxN5h6T5tehFG8L9KFw==,type:str]
+    namespace: ENC[AES256_GCM,data:fbklNyooBeqsogKbeg==,iv:z2wLgl81q8xfzGGVQ9A4Pkst/zLmuZpyBzHrXzmVfm4=,tag:QKJik4IWc4nrOBINjGCxvQ==,type:str]
+type: ENC[AES256_GCM,data:bfAFBd/7,iv:9/W3X1KefqDAsurIatT6kBuEIQvWVRTq4G9A+9ZfYKs=,tag:Optlb+gZ2udbqIOerqV2zQ==,type:str]
+stringData:
+    #ENC[AES256_GCM,data:RZaw/FNQZOf0F3+XfFmGW+4gjKXNdjrAbMRjBt4hptwk,iv:IEd/+Bsv/h08nKak+bcovNiohC7nNjEQ+QdtIFMUGng=,tag:zsF5ZQ3+5JijF4MU+RhnyA==,type:comment]
+    #ENC[AES256_GCM,data:ux3ONHnuI0uFEihElbLI1rKlrN91Pn0iLLZfmez0+t5d8yhH6OD3L/3oi6yHbdgD8bV90kRK/lw=,iv:7Ed+W/TeOY9Af8bKO0Fu3XOuaK8b0Hw70rwobfmosxM=,tag:Z0RbIOjdDroK0drC6P+Y8A==,type:comment]
+    host: ENC[AES256_GCM,data:YKgas/8hT5ZOGIW7O0c=,iv:WtBz1MFrwd6EgDDS5fMqtcVGIZGNLgbXnmkqlNWoXy8=,tag:+H9ARGPi5XocgZYp4oA3wA==,type:str]
+    port: ENC[AES256_GCM,data:f0hx,iv:HkSnJcCyzyfBTA39BNw+muhHTyrNLzzGxZ7M3bcLVu4=,tag:4+xL1xNyYPaV2b0mwnbQVQ==,type:str]
+    from: ENC[AES256_GCM,data:lqYqXxJuI4CAWPlX+5cltCdxqA==,iv:ZgZYySJXvjuTNIkMaDxynUJ6PFdp4fibqlgoA1vNDKE=,tag:vb/XKl2N1Hl5u7ndZXQvbg==,type:str]
+    user: ENC[AES256_GCM,data:ajljGVbRO//G,iv:E4Q0LGDCd5wnHc+Oq2A9ZhWGbCmkKlhllt922UmppIg=,tag:Cf4ft1JKFi9PaJEfPfq92Q==,type:str]
+    password: ENC[AES256_GCM,data:YNbtoiWLsQGl,iv:FJ3y1lWkqKkM8GVr7FZljAXJ331GkDgy4TenAbdIY3w=,tag:KIvhv+RL49ZMwWhYrU0SPA==,type:str]
+    notify-email: ENC[AES256_GCM,data:s5wRkDuaiflCyu4IyAnaocVPA3I=,iv:dK5PsweYa3JotYcDcz74DPNDXmhW5ngNEbY6JuG1ctg=,tag:/Dt2UoCtfau6qv0G7VzmrQ==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLQlJVekcwb01ESnhxSE9m
+            MzliVmlaYUttZVdiMDVtMlQ2L29LMXBWRzBZCkxRMi9RcE5LMzQ1MHFqUUxLaFZo
+            MUZ2UTg5eUNsMFVmenlhL2F3NDd0akkKLS0tIGFwbUUrSGFUTDFVOWlxM3V1bVdy
+            RUVVNDFMRHZCSWptdSt1bFVoN2JPWVkKXMlrJNPbm7Sj9vd/H3o6iqPRLy4LyjVe
+            xJyh9vn9RCwS9Jr6BN2Q9P2UZfTGwvIYc6zdmV+b2uRwvi4t4lMsBA==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    lastmodified: "2026-09-10T13:31:00Z"
+    mac: ENC[AES256_GCM,data:g1YjOANlnvXErPqtDdDjRB9yTUiGEhX8QDkmf1aLfQBwoXbGUpoPIZibF8A4cAL1yUSZ2rD4l9TIe9nK8U3oF4WiKXf9u0bFK2g//bOv2A5JPewwmzoazkjIDXcJ/76nOQ5c+v8/r5tUG4c0SyzBH/gFP0D0+P5Yo3SEXjTxj2w=,iv:gW+AUwwAWkXZ+aLBXjjNDZw8VEraMpSE16RJGQc/L5g=,tag:DM/DUtil/7yLhfGYQXKFGA==,type:str]
+    unencrypted_suffix: _unencrypted
+    version: 3.13.2

k8s/argocd/secrets/gotify-tokens-secrets.enc.yaml

@@ -0,0 +1,27 @@
+apiVersion: ENC[AES256_GCM,data:YE0=,iv:s3yNqcBn7/DKUQNgbGGwVmlM1HzxYGLBKyB6Y2ii2WE=,tag:77KxAjOFU3G0fSfrgudKkg==,type:str]
+kind: ENC[AES256_GCM,data:PFW4VV9T,iv:n8gzMjE5C2TAfUTpp/8ex64B+hWUGG1K+2oQ4deN03Q=,tag:18Xqb6Di5izMHmzR5EU+QA==,type:str]
+metadata:
+    name: ENC[AES256_GCM,data:Xhg1fQrD4itrbvDW0g==,iv:/THWSXAThb9fj+mm3wcxqzSdzdt7nE06+xOpkCxyImE=,tag:UChokr06VkbDZcFZDcUY2A==,type:str]
+    namespace: ENC[AES256_GCM,data:r25U5MuuzDd8JJ2YjQ==,iv:uDUcS4ZTpZe8HmZMArzkdu7LV5GUoLSP2wIs5HB1gv0=,tag:fci7j30hoxnVKJwPFNfd1Q==,type:str]
+type: ENC[AES256_GCM,data:agEG8Fu8,iv:ckYX0buX8md1CWHXfxbAXQJLzoTxTJI16nlQ9LSzYi0=,tag:4tZWf3REbGOmmBiT14+dew==,type:str]
+stringData:
+    #ENC[AES256_GCM,data:ehjcsmz1R0i/n31f8M0IIUT4KDBwzz6f5ds=,iv:j5swFAKpC67ZvGioiCCC1D651LxUl9gME8GqK5Uc+ys=,tag:BUJ8k9LHs8NAPPZAwPuifA==,type:comment]
+    #ENC[AES256_GCM,data:ndAB00yun636VHDMonqTghO/jCWodNd/hmdbm1QmCvOAi4FvTLl8bY3wYR+ZtlkSQYvFNqH798MJixv5OZwxBj5H,iv:pS+7B60jaS5jhqDRw2LbBFv7mD1HO6+hjjv+iNpenXc=,tag:NU6+gxCHTGxqeDMfvkwJWw==,type:comment]
+    #ENC[AES256_GCM,data:T3mhSm/5q7JTSXNLrjhpP5GhqA7nXz8uAhJcEV1RDctAX0Dyfq8Qn4bNFO51ibwTETx4SnunPuFZVs++AvRnsA==,iv:oRGM8N4iEcwBrMzoEXQAeKqCKzUq+jXTMVq9W2l6oh4=,tag:GrrbC/tkWpA5kp5UYCyRBA==,type:comment]
+    app-token: ENC[AES256_GCM,data:jw+AqbsxyoYRvNrUDrq+GNq/TNfweFCs,iv:67vSSgMZCMV0GG37ZxUxHAWZqOOGMYCmo3J43WgLyNk=,tag:xlrnip4XomXBbMmooW9FKg==,type:str]
+    client-token: ENC[AES256_GCM,data:gsxeCqKh4e+DFjpn9jM5GfemAdBf8dSv,iv:l2bBMdxQUil8jU9XDjXGn3EtvFVrK5ibXDCeGaPbYTY=,tag:ELf5S6cdNQJ84Es5upu1IQ==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAza3YrRW1VTVNSOGlMK21a
+            bDhzQlEyZDNnZTZrSVNnTUEzU1hSdnM1ZFVzCjdBSUlmWG5EcnlzbSs5bXdDaGkx
+            ME1nMnRqQzF1Vkc3b3FXSUVHT1g0ZDgKLS0tIHpMUytEMjdLQ2E0Unp2di9KS3JQ
+            eHBraDk1clJyanhLY2dGM0tESmJIUFkKHTl3y9uQiEofOFD8j2vH3YK/CVzlq11w
+            GfShIji1yCvvowKGzYYhsQK0UM0FzhzBv0GFMYWQCBq8pGdoPVmO5g==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    lastmodified: "2026-09-10T13:31:00Z"
+    mac: ENC[AES256_GCM,data:JhnO0V6cd2QVY/VhwHAJ5J75GeVlOVHBfVBTZstkj/IvpkAcwS/JE2b6jXiCwEC4n/vdA8DJ074noysUBRxh4Y7O4NKuvWcTniQKgqULNL1Hzgj3NdUkcQlWT+Z0HQ7FlvFH97VVXVfasU+CLHG48ShT2fDSj8JusEllb9CwSvg=,iv:PZo2krxvJc1TLJbvx+S9ClthoBe4w7WigUkq7dg0gNk=,tag:2IF5g/WTRP4XdnCZzDbiIA==,type:str]
+    unencrypted_suffix: _unencrypted
+    version: 3.13.2

k8s/argocd/secrets/kustomization.yaml

@@ -1,8 +1,10 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
-# All homelab SOPS-encrypted Secrets, decrypted in-line via the ksops generator.
-# Each *.enc.yaml carries its own metadata.namespace, so no namespace transformer
-# here (that would rewrite every Secret into one namespace). Renders exactly the
-# Secret objects — replaces the old argocd-cmp-cm SOPS plugin.
+
+# Disable hash suffix for all generated secrets (stable names)
+generatorOptions:
+  disableNameSuffixHash: true
+
+# SOPS-encrypted secrets via ksops generator
 generators:
   - secret-generator.yaml

k8s/argocd/secrets/paperless-secrets.enc.yaml

@@ -0,0 +1,25 @@
+apiVersion: ENC[AES256_GCM,data:wGA=,iv:Z2Gfzq3aJ9j4fYaeLQolgLb/XELTHrKX9at3vUsMLIw=,tag:yZLZFnltJln5jFoVsyyZ2A==,type:str]
+kind: ENC[AES256_GCM,data:bJkV4pNj,iv:0ZT7l0kw9qSoiEMZioRw1aBzzlxBkoXR+hOxoPI7zPU=,tag:EQczPq508Qw1vi/oLCeQpw==,type:str]
+metadata:
+    name: ENC[AES256_GCM,data:sZwTVM41PPaidMwFNRo5RvU=,iv:BHfuwIHng7rkeLK3a69t8cI9QeSfD/3FEXqxby+gBxM=,tag:3zQx8APnE2ZpBKf/ZYCxOA==,type:str]
+    namespace: ENC[AES256_GCM,data:i7lpoEaZ1oXS,iv:jUYyDPYhf1TV51he/S5MlKPD19Vmz6wfE98y8qFEg3U=,tag:zgHDVlplmW/XPUAKdncfYQ==,type:str]
+type: ENC[AES256_GCM,data:2khs1uIg,iv:ET6HcBGyv33fGFlFAl3dkJQB83naHGeaYuWvW1IFhvw=,tag:qIfMeaSwv//7cNWvy8O5dg==,type:str]
+stringData:
+    PAPERLESS_SECRET_KEY: ENC[AES256_GCM,data:kj9DrQYL3cQGJz87FHYlFKy6Muu84Oy/6wnsWAh0w/3MmcTAsCzQvUtKLKKf1U61JTQ=,iv:UffVv78vMHDEWHRFdSKZ/6qyrVD02Nlk0CNxwWd1jTo=,tag:LNMKnFWw7QCjfcKzI1s8ig==,type:str]
+    PAPERLESS_ADMIN_USER: ENC[AES256_GCM,data:a3bx7z0=,iv:nrU6VXZE74PNXM+Lgg4K/3mhusaQhA2GAgl0GzqAQt8=,tag:x8nswTfkFHR6Ou+JSI9P3Q==,type:str]
+    PAPERLESS_ADMIN_PASSWORD: ENC[AES256_GCM,data:+SWyAq7D1mNt1TlFOlo2Z0biK8kEn097,iv:an2ZZuXNSig+7mJP7ogHFc5+QvUb+z9pzqcV4xvFLbA=,tag:OaRiVZsnajEjAlc9mDWo9w==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1Q2V6bVBUYmVRR1N5SCtF
+            eXdTV2dmWjhtMS9lRFEzS0wzbkd6Y3JLMVRVCncydFBjdmJkRCtVUXphR2w0SDlJ
+            cng2bi9MWlJzTEN2amJrYjRJN2VFcEEKLS0tIEY5cmw2RGhnbzUxZW9FaFJjQmVN
+            WWcvNlNiYWdwbnNSR1Q4alpDZmFqTFkKh9TOw8ERP9fpx2pKi/Q0b7+OkEv0UC7o
+            aAIK4Tzvi5dp6y9IWcu9l6PjDLeYWOJ5wr7QABaFNOz82hngxUleDA==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    lastmodified: "2026-08-25T16:04:48Z"
+    mac: ENC[AES256_GCM,data:OXPQXUyn/SDftKH5nRzhqEtnaOb4Gp7etmGojqV4Z01kHnABZ42PPIyiBb8z997eLNZsK6/1bi7nUNyVp4fIuXG4F52aEdMewofOocCHokRnNmz7jzhooK1gScJb2u0eHG3FL5iLONMaGgVpk7BLYO3e0xiDytWGe8BxcuDukPg=,iv:kh0jUwFvO6AUICy2Us1E7YOTEcp3L+ptrGdDWBSpWyc=,tag:qUPFmZ/rpljln37f/NRjjw==,type:str]
+    unencrypted_suffix: _unencrypted
+    version: 3.13.2

k8s/argocd/secrets/portfolio-secrets.enc.env

@@ -0,0 +1,2 @@
+FORGEJO_TOKEN=273fdcffabbcbb5a191e8289c73d106063acefc6
+LLM_API_TOKEN=s3VksXyw2z3sGbegnwjMDFnJ6CtNRd1a5CcnE5A4ET77toCcykNdunk6Oa2J

k8s/argocd/secrets/portfolio-secrets.enc.yaml

@@ -0,0 +1,23 @@
+apiVersion: ENC[AES256_GCM,data:bnY=,iv:Fuc3aqncHQ+L16o7eLarPbOECD3o8Mk5c2r9pQBpy70=,tag:JPfEnbNb3wZXPdXafnJDqw==,type:str]
+kind: ENC[AES256_GCM,data:WFlmi4Yg,iv:Zq/KQbgNcBVoo8ZsQ2H79ygyc8Dtkgxh4fCpEExfwSg=,tag:cWHP6Y5V+nZP2tFMJrOB8A==,type:str]
+metadata:
+    name: ENC[AES256_GCM,data:iCXbhwvg3Zq6YL/4j0wAy7Y=,iv:8s+d/8lDVEL7bGdIF+GOtAxapKnmx8JTjLSO04hXF5I=,tag:LjzX40DjK+uRZPCXsMlmwQ==,type:str]
+    namespace: ENC[AES256_GCM,data:HRMdZdCbxORQ,iv:MvaIWoKWjJRA7/fce0KtXRkFH/7cn0OuIg2QwHEdQzM=,tag:NqztiGyfU3BaopWBKhx2eg==,type:str]
+type: ENC[AES256_GCM,data:myBW86Za,iv:3x9ys5UzVhAuX8gvZO67B1e+Orw4Aqasv/lHBgUV0b4=,tag:yl18P6SaxVLJidwmRtd4aQ==,type:str]
+stringData:
+    FORGEJO_TOKEN: ENC[AES256_GCM,data:SUoBpNKOItyNGY01EhKNlPH0fyN4N7g6bfU2jsGogpCMhP7NuRipoA==,iv:h77RtmYZjXxiHYw1pHynQHuVX1+yJDGHwsXLf+DbUYA=,tag:GHzoDNHP1GGqlN5eT/N7TQ==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzTVBsekl3TGgzQVRMUU9m
+            cTduS2NoZW5uZFNNMG13cFY2cGVsTnlXaXhrCjJjbzhLdHZ4ZWpUV3J0cDQ0eVlM
+            WDNxdzVoQ2ZzcGJSbTU3RVorcnczNVkKLS0tIFdtQTE4Umk2TDBzUmdKOXNkbjFi
+            Vk5vK2VuUHVsb3FQL21vcGU1UW5CT1kKFM8vVjji3Cg9dvfTr4Hx7BJC8JH5ovef
+            Dj6zkofhsNWPgP9T+mnQakj+C0RKmHOMJqfWP7vwCBkZoNosIJVlMw==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    lastmodified: "2026-09-01T05:32:42Z"
+    mac: ENC[AES256_GCM,data:it24T9y9ixXo2aiL37k93vKFR+SRjjuI9DQdv0sWYtTogWnc7+uXBY4Zip/ouWyCse1muKKAGuek5c0XVrvSw4an9VkaXFczeunaZb6MOyVbVOkmJr+5xZFpZGjYcSkrhaWcVheedZ3iIFU5UWI7BBn/qQCf+HJ483cJqwtrV34=,iv:WprlWJdsMBNjqaA0O3ekfXMUpX5gC6OLYortQXYdTS4=,tag:rGqSSRTYTv2VR6AcRokO0A==,type:str]
+    unencrypted_suffix: _unencrypted
+    version: 3.13.2

k8s/argocd/secrets/secret-generator.yaml

@@ -11,6 +11,7 @@ files:
   - agent-pod-ssh-key.enc.yaml
   - authentik-secrets.enc.yaml
   - cloudflare-secrets.enc.yaml
+  - forgejo-registry-pat.enc.yaml
   - forgejo-registry-pull.enc.yaml
   - forgejo-runner-token.enc.yaml
   - forgejo-secrets.enc.yaml
@@ -22,5 +23,10 @@ files:
   - homelab-ca-secrets.enc.yaml
   - loki-secrets.enc.yaml
   - minio-secrets.enc.yaml
+  - paperless-secrets.enc.yaml
   - vault-secrets.enc.yaml
   - vault-unseal-keys.enc.yaml
+  - portfolio-secrets.enc.yaml
+  - gotify-admin-secrets.enc.yaml
+  - gotify-tokens-secrets.enc.yaml
+  - gotify-smtp-secrets.enc.yaml

k8s/argocd/secrets/vault-unseal-keys.enc.yaml

@@ -5,9 +5,9 @@ metadata:
     namespace: ENC[AES256_GCM,data:wK6m,iv:KtA31Bo8aGE1HU8H9KWbMwt2NfywWfy77G/LaReaI1E=,tag:V9SSlPawRQw0x2utFNN1aw==,type:str]
 type: ENC[AES256_GCM,data:6UPZ1ZTR,iv:FxN1ebrlJ4IO3eDGEYSvktSpMgAebcl0DO1WHh5O0+0=,tag:2bZN4zdbHzx0oR+bTJPPJg==,type:str]
 stringData:
-    key1: ENC[AES256_GCM,data:dV2HOh7W1Pl0QDJaGvtEKpBppybQMiK5kxzyk05zkA352ZmJvE8Ppm5yWKk=,iv:grQ8v2o/LHpJnIZonjtgTHKcLUQIH8xl1FtWtEs4rEs=,tag:lANcccYT6Ouo/0IcoLw4uA==,type:str]
-    key2: ENC[AES256_GCM,data:YSdpYL8h56PfUrvMFhBXhmBG2en0toLKEpYlJqwjAk/vI0jm7gq/TqGn034=,iv:GxthkNLhm3qHxjkZetiYl58Qa/K7G2ib6E+LWK16H8Q=,tag:q79UUoh6uCcMZJ0U18Ireg==,type:str]
-    key3: ENC[AES256_GCM,data:sO7Lao9qkmIOdARL+6FBcLo+e4z8LzMEzNrwcy5hv9uRuxppwAXrP0MO1vw=,iv:wSWe9h1HovRhV5YCZrmvcB2VEfRYQ+gasGO3uh8IpmQ=,tag:Bc5yaROQDTz1AhVnFDm/BA==,type:str]
+    key1: ENC[AES256_GCM,data:ke5EfsOsZHBJyAQoFfwhhWCQJQgnwcrBqL4CzPIfkx7bi9S7kWgWySdxcXQ=,iv:eYKxG8k+hizp2t2i/YMR2lQNJQFV+A21YyWnDc+kJ9w=,tag:Wa1Lx3EKvfpYfnLrBOWSIg==,type:str]
+    key2: ENC[AES256_GCM,data:JxiXgLKvDe5oiHlwIL/Cj8txHc7fVQ5VzBcQMU/ro9TSOcTGSe7Z3omQzwY=,iv:x5arcY2dey+npMpUxjdUPV+t94LEaMOXq3iOar88eT8=,tag:E/Z5zEDy87B1IAWkdjdNEg==,type:str]
+    key3: ENC[AES256_GCM,data:AsxyyBTHzt+SLru6ayi1UbgPPFmdIvowIDdcfc7N03Xt8V/KaJW3kbXExG8=,iv:lA+bERIBq4+bxx08ZZjP5tMHC9JQlGPW1EMCwEixSeg=,tag:xkuTAU3KTidFuhNMXkVFhA==,type:str]
 sops:
     age:
         - enc: |
@@ -19,7 +19,7 @@ sops:
             7RltNZF2SCxjIv5C2pqf3CgmRBaQGWgMybRRH5gdB87PLBKkPL3+HQ==
             -----END AGE ENCRYPTED FILE-----
           recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-08-12T20:52:24Z"
-    mac: ENC[AES256_GCM,data:kLjoBWmJ2bZ13EbuzgppahHKmCY/xtCi1R7xCEUbCP4FQfiVaC4qAbVftE4lTNlXa2mqCKsGPYZ/A3HV/4+a+it/pg0a+rj7E7DszhieWbZufHMJs+w4/Le8l8wFFE5aNW0wdvGTJZ0n4HBOdAkE1qN9ReaiaQgr0rk5ggOPG9g=,iv:MiDVUgcoKrP/gj4keqomo6OB12dmz9VoesTiHQTnBFM=,tag:cLK67Pym4BuvXLF1RUA8Gw==,type:str]
+    lastmodified: "2026-08-26T23:33:09Z"
+    mac: ENC[AES256_GCM,data:kr8XuOqXyhfrC84yBot5aoVQZZhQd9xYMUEs0XDwkmCgtG0iobADYH5XWPh72xSdWEtwxkZJL7r0UEyEhU9zjlUA5qHupOSBBt1SgZ+zstMvaOqUnlNn//p/DIJBpsiT/qmx64NpTLAiz6lm0796MozIMr8PTX+ubGLHi9tnUiY=,iv:jwP0MLCT7nG6m+nZeqNip9q3BcScpcDXmInnY95DicY=,tag:Cgg2IUZARkaK1gF2qUIthg==,type:str]
     unencrypted_suffix: _unencrypted
     version: 3.13.2

k8s/bootstrap/ingress/ingress.yaml

@@ -323,3 +323,4 @@ spec:
                 name: homarr
                 port:
                   number: 7575
+

k8s/bootstrap/kyverno/kustomization.yaml

@@ -0,0 +1,6 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: kyverno
+
+resources:
+  - policies.yaml

k8s/bootstrap/kyverno/kyverno-values.yaml

@@ -0,0 +1,42 @@
+# Kyverno: Policy engine for Kubernetes image scanning, Pod security, and admission control
+# Scan all images, enforce baseline Pod Security Standard, prevent privilege escalation
+
+replicaCount: 1
+
+image:
+  registry: ghcr.io
+  repository: kyverno/kyverno
+  tag: "v1.14.0"
+
+config:
+  # Webhook timeout for policy evaluation. Increase if scanning takes longer.
+  webhookTimeoutSeconds: 30
+  # Failure policy: fail-open (audit/log) vs fail-closed (reject on error)
+  failurePolicy: fail
+  # Resource limits for webhook
+  webhookAnnotations:
+    rules: "allow"
+
+# Pod security via Kyverno instead of Pod Security Policies (deprecated)
+# Enforces baseline restrictions cluster-wide, with exceptions for privileged namespaces
+podSecurityContext:
+  runAsNonRoot: true
+  runAsUser: 1000
+
+rbac:
+  create: true
+
+resources:
+  requests:
+    memory: "256Mi"
+    cpu: "100m"
+  limits:
+    memory: "512Mi"
+    cpu: "500m"
+
+# Webhook configuration
+webhook:
+  timeoutSeconds: 30
+  # Failure policy: "Fail" (reject on error) or "Ignore" (audit-only)
+  # Set to "Ignore" for initial testing, then change to "Fail"
+  failurePolicy: ignore

k8s/bootstrap/kyverno/policies.yaml

@@ -0,0 +1,204 @@
+# Kyverno ClusterPolicies: Image scanning, Pod security, and admission control
+---
+# Policy 1: Require non-root containers
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: require-non-root
+  namespace: kyverno
+spec:
+  validationFailureAction: audit  # audit first, then change to enforce
+  rules:
+    - name: check-runAsNonRoot
+      match:
+        any:
+          - resources:
+              kinds:
+                - Pod
+              selector:
+                matchLabels:
+                  pod-security.kubernetes.io/enforce: "!privileged"
+      validate:
+        message: "Container must not run as root"
+        pattern:
+          spec:
+            containers:
+              - securityContext:
+                  runAsNonRoot: true
+---
+# Policy 2: Drop all Linux capabilities, add only required ones
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: require-dropped-caps
+  namespace: kyverno
+spec:
+  validationFailureAction: audit
+  rules:
+    - name: drop-all-capabilities
+      match:
+        any:
+          - resources:
+              kinds:
+                - Pod
+              selector:
+                matchLabels:
+                  pod-security.kubernetes.io/enforce: "!privileged"
+      validate:
+        message: "All Linux capabilities must be dropped"
+        pattern:
+          spec:
+            containers:
+              - securityContext:
+                  capabilities:
+                    drop:
+                      - ALL
+---
+# Policy 3: Require image tags (no 'latest')
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: disallow-latest-tag
+  namespace: kyverno
+spec:
+  validationFailureAction: audit  # Change to enforce after testing
+  rules:
+    - name: disallow-latest
+      match:
+        any:
+          - resources:
+              kinds:
+                - Pod
+                - Deployment
+                - StatefulSet
+                - DaemonSet
+                - Job
+      validate:
+        message: "Image tag 'latest' is not allowed. Use explicit version tags."
+        pattern:
+          spec:
+            =(template):
+              spec:
+                containers:
+                  - image: "!*:latest"
+                =(initContainers):
+                  - image: "!*:latest"
+---
+# Policy 4: Restrict images to trusted registries
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: restrict-registries
+  namespace: kyverno
+spec:
+  validationFailureAction: audit
+  rules:
+    - name: trusted-registries
+      match:
+        any:
+          - resources:
+              kinds:
+                - Pod
+                - Deployment
+                - StatefulSet
+                - DaemonSet
+                - Job
+              selector:
+                matchLabels:
+                  pod-security.kubernetes.io/enforce: "!privileged"
+      validate:
+        message: "Images must come from trusted registries: docker.io, ghcr.io, quay.io, k8s.gcr.io, registry.k8s.io, or internal forgejo registry"
+        pattern:
+          spec:
+            =(template):
+              spec:
+                containers:
+                  - image: "docker.io/* | ghcr.io/* | quay.io/* | k8s.gcr.io/* | registry.k8s.io/* | forgejo.riotpiao.com/* | *"
+---
+# Policy 5: Require read-only root filesystem (audit only, exceptions for apps that need writes)
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: require-readonly-filesystem
+  namespace: kyverno
+spec:
+  validationFailureAction: audit
+  rules:
+    - name: check-readOnlyRootFilesystem
+      match:
+        any:
+          - resources:
+              kinds:
+                - Pod
+              selector:
+                matchLabels:
+                  pod-security.kubernetes.io/enforce: "!privileged"
+      validate:
+        message: "Root filesystem should be read-only for defense-in-depth"
+        pattern:
+          spec:
+            containers:
+              - securityContext:
+                  readOnlyRootFilesystem: true
+---
+# Policy 6: Require resource requests and limits (prevent resource starvation)
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: require-resource-limits
+  namespace: kyverno
+spec:
+  validationFailureAction: audit
+  rules:
+    - name: check-resources
+      match:
+        any:
+          - resources:
+              kinds:
+                - Pod
+                - Deployment
+                - StatefulSet
+                - DaemonSet
+              excludeResources:
+                namespaceSelector:
+                  matchLabels:
+                    kubernetes.io/metadata.name: "kyverno|kube-system|kube-node-lease"
+      validate:
+        message: "CPU and memory requests and limits are required"
+        pattern:
+          spec:
+            =(template):
+              spec:
+                containers:
+                  - resources:
+                      requests:
+                        memory: "?*"
+                        cpu: "?*"
+                      limits:
+                        memory: "?*"
+                        cpu: "?*"
+---
+# Policy 7: Require securityContext on all containers
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: require-security-context
+  namespace: kyverno
+spec:
+  validationFailureAction: audit
+  rules:
+    - name: check-securityContext
+      match:
+        any:
+          - resources:
+              kinds:
+                - Pod
+              selector:
+                matchLabels:
+                  pod-security.kubernetes.io/enforce: "!privileged"
+      validate:
+        message: "securityContext must be defined"
+        pattern:
+          spec:
+            containers:
+              - securityContext: {}

k8s/bootstrap/phase3-forgejo/registry-cleanup-cronjob.yaml

@@ -0,0 +1,220 @@
+# Forgejo OCI Registry Cleanup CronJob
+# Deletes old image tags, keeping only the latest N versions per repository.
+# Useful for retiring old builds when new versions are pushed.
+
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: forgejo-registry-cleanup-script
+  namespace: cicd
+data:
+  cleanup.sh: |
+    #!/bin/bash
+    set -eo pipefail
+    
+    # Configuration
+    REGISTRY_HOST="${REGISTRY_HOST:-forgejo.riotpiao.com}"
+    REGISTRY_URL="https://${REGISTRY_HOST}"
+    KEEP_VERSIONS="${KEEP_VERSIONS:-3}"  # Keep latest N versions per image
+    DRY_RUN="${DRY_RUN:-false}"
+    
+    # Load credentials from mounted secret
+    REGISTRY_USER="${REGISTRY_USER:-_json_key}"
+    REGISTRY_PASS="$(cat /etc/registry-secret/password 2>/dev/null || echo '')"
+    
+    log() {
+      echo "[$(date +'%Y-%m-%d %H:%M:%S')] $*"
+    }
+    
+    error() {
+      echo "[$(date +'%Y-%m-%d %H:%M:%S')] ERROR: $*" >&2
+      return 1
+    }
+    
+    # Verify crane is available
+    if ! command -v crane &> /dev/null; then
+      error "crane not found. Install google/crane image for registry operations."
+      exit 1
+    fi
+    
+    log "Starting Forgejo registry cleanup"
+    log "Registry: $REGISTRY_URL"
+    log "Keep versions: $KEEP_VERSIONS per image"
+    log "Dry run: $DRY_RUN"
+    
+    # Authenticate crane with registry
+    if [ -n "$REGISTRY_PASS" ]; then
+      echo "$REGISTRY_PASS" | crane auth login "$REGISTRY_HOST" -u "$REGISTRY_USER" --password-stdin
+      log "Authenticated to $REGISTRY_HOST"
+    fi
+    
+    # List all repositories (catalog)
+    # Note: This endpoint requires the registry to expose /v2/_catalog (standard OCI)
+    # If not available, images must be discovered another way
+    CATALOG=$(curl -s -u "${REGISTRY_USER}:${REGISTRY_PASS}" \
+      "${REGISTRY_URL}/v2/_catalog" | grep -o '"repositories":\[\K[^]]*' || echo '')
+    
+    if [ -z "$CATALOG" ]; then
+      log "WARNING: Could not retrieve catalog from ${REGISTRY_URL}/v2/_catalog"
+      log "Registry may not expose _catalog endpoint or credentials invalid"
+      exit 0
+    fi
+    
+    # Parse repositories from catalog JSON
+    REPOS=$(echo "$CATALOG" | grep -o '"[^"]*"' | tr -d '"')
+    
+    TOTAL_DELETED=0
+    
+    for REPO in $REPOS; do
+      log "Processing repository: $REPO"
+      
+      IMAGE="${REGISTRY_HOST}/${REPO}"
+      
+      # Get all tags for this image
+      TAGS=$(crane ls "$IMAGE" 2>/dev/null || echo "")
+      
+      if [ -z "$TAGS" ]; then
+        log "  No tags found for $REPO (or access denied)"
+        continue
+      fi
+      
+      # Filter out 'latest' tag and sort by creation time (newer first)
+      # Note: crane doesn't provide direct date sorting; we use the order returned
+      # Assumption: tags are returned newest first (not always true)
+      TAG_COUNT=$(echo "$TAGS" | wc -l)
+      
+      if [ "$TAG_COUNT" -le "$KEEP_VERSIONS" ]; then
+        log "  $REPO: $TAG_COUNT tags total, keeping all (≤ $KEEP_VERSIONS)"
+        continue
+      fi
+      
+      # Get tags to delete (all except the first N)
+      TAGS_TO_DELETE=$(echo "$TAGS" | tail -n +$((KEEP_VERSIONS + 1)))
+      
+      for TAG in $TAGS_TO_DELETE; do
+        FULL_IMAGE="${IMAGE}:${TAG}"
+        DELETED_SIZE="0"
+        
+        if [ "$DRY_RUN" = "true" ]; then
+          log "  [DRY RUN] Would delete: $FULL_IMAGE"
+        else
+          if crane delete "$FULL_IMAGE" 2>&1; then
+            log "  Deleted: $FULL_IMAGE"
+            ((TOTAL_DELETED++))
+          else
+            error "Failed to delete $FULL_IMAGE (may already be deleted)"
+          fi
+        fi
+      done
+    done
+    
+    log "Cleanup complete. Total images deleted: $TOTAL_DELETED"
+
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: forgejo-registry-cleanup
+  namespace: cicd
+  labels:
+    app: forgejo-registry-cleanup
+spec:
+  # Run at 2 AM UTC every day (adjust as needed)
+  schedule: "0 2 * * *"
+  
+  # Keep last 3 successful/failed runs for debugging
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  
+  # Suspend if needed (set to false to enable)
+  suspend: false
+  
+  jobTemplate:
+    spec:
+      # Cleanup jobs after 6 hours whether they succeeded or failed
+      ttlSecondsAfterFinished: 21600
+      
+      template:
+        metadata:
+          labels:
+            app: forgejo-registry-cleanup
+        spec:
+          serviceAccountName: forgejo-registry-cleanup
+          restartPolicy: OnFailure
+          
+          containers:
+          - name: cleanup
+            # Use google/crane for registry operations
+            image: gcr.io/go-containerregistry/crane:latest
+            imagePullPolicy: IfNotPresent
+            
+            env:
+            - name: REGISTRY_HOST
+              value: "forgejo.riotpiao.com"
+            - name: KEEP_VERSIONS
+              value: "3"  # Keep 3 latest versions
+            - name: DRY_RUN
+              value: "false"  # Set to "true" for dry-run mode
+            - name: REGISTRY_USER
+              valueFrom:
+                secretKeyRef:
+                  name: forgejo-registry-token
+                  key: username
+                  optional: true
+            
+            volumeMounts:
+            - name: script
+              mountPath: /scripts
+            - name: registry-secret
+              mountPath: /etc/registry-secret
+              readOnly: true
+            
+            # Run cleanup script via entrypoint override
+            command:
+            - /bin/sh
+            - -c
+            - |
+              # Install bash and curl if needed
+              apk add --no-cache bash curl
+              chmod +x /scripts/cleanup.sh
+              /scripts/cleanup.sh
+            
+            resources:
+              requests:
+                cpu: 100m
+                memory: 128Mi
+              limits:
+                cpu: 500m
+                memory: 512Mi
+            
+            # Safety: kill after 30 min (prevents hanging on large registries)
+            securityContext:
+              runAsNonRoot: true
+              runAsUser: 65534
+              allowPrivilegeEscalation: false
+              readOnlyRootFilesystem: false
+              capabilities:
+                drop:
+                - ALL
+          
+          volumes:
+          - name: script
+            configMap:
+              name: forgejo-registry-cleanup-script
+              defaultMode: 0755
+          - name: registry-secret
+            secret:
+              secretName: forgejo-registry-token
+              optional: true
+
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: forgejo-registry-cleanup
+  namespace: cicd
+
+---
+# No RBAC needed: this pod only talks to the registry API (external service)
+# If expanded to manage in-cluster resources, add Role/RoleBinding here

k8s/infra/argocd-image-updater/values.yaml

@@ -0,0 +1,72 @@
+# ArgoCD Image Updater configuration
+# Watches Forgejo registry and updates ArgoCD Applications with new image tags
+
+config:
+  # Registry configuration - Forgejo allows anonymous pulls
+  registries:
+    - name: forgejo
+      api_url: https://forgejo.riotpiao.com
+      prefix: forgejo.riotpiao.com
+      default: true
+      insecure: false
+
+  # Log level
+  logLevel: debug
+
+  # ArgoCD API server
+  argocd:
+    grpcWeb: true
+    serverAddress: argocd-server.argocd.svc.cluster.local
+    insecure: true
+    plaintext: true
+
+  # Git write-back configuration (for multi-source Applications)
+  git:
+    # Commit author for image updates
+    user:
+      name: "ArgoCD Image Updater"
+      email: "[email protected]"
+    # Use SSH keys from ArgoCD's known hosts + credentials
+    # Image Updater inherits ArgoCD's git credentials (mounted via ArgoCD secret)
+
+# Mount ArgoCD's git credentials for write-back
+extraVolumes:
+  - name: argocd-ssh-known-hosts-cm
+    configMap:
+      name: argocd-ssh-known-hosts-cm
+      defaultMode: 0644
+  - name: argocd-gpg-keys-cm
+    configMap:
+      name: argocd-gpg-keys-cm
+      optional: true
+      defaultMode: 0644
+  - name: argocd-gpg-pubring
+    configMap:
+      name: argocd-gpg-pubring-cm
+      optional: true
+      defaultMode: 0644
+
+extraVolumeMounts:
+  - name: argocd-ssh-known-hosts-cm
+    mountPath: /etc/ssh/ssh_known_hosts.d/argocd-ssh-known-hosts
+    subPath: ssh_known_hosts
+  - name: argocd-gpg-keys-cm
+    mountPath: /etc/gpg/source
+  - name: argocd-gpg-pubring
+    mountPath: /etc/gpg/pubring
+
+# Extra environment variables
+extraEnv:
+  - name: ARGOCD_GRPC_WEB
+    value: "true"
+  - name: GIT_SSH_KNOWN_HOSTS_CONFIG_MAP_ENABLED
+    value: "true"
+
+# Resources
+resources:
+  requests:
+    cpu: 50m
+    memory: 64Mi
+  limits:
+    cpu: 200m
+    memory: 128Mi

k8s/infra/cluster-maintenance/stale-job-cleanup.yaml

@@ -0,0 +1,137 @@
+# Cluster-wide cleanup of stale failed/completed Jobs and Pods.
+# Runs daily at 04:00 UTC. Deletes:
+#   - Failed Jobs older than 24h (any namespace)
+#   - Completed Jobs older than 72h with no owning CronJob
+#   - Orphan pods in Error/Failed/Evicted state older than 1h
+#
+# CronJob-owned Jobs are managed by failedJobsHistoryLimit/successfulJobsHistoryLimit,
+# but standalone Jobs (helm hooks, one-off runs, longhorn maintenance) have no TTL
+# and linger forever.
+
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: stale-job-cleanup
+  namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: stale-job-cleanup
+rules:
+  - apiGroups: ["batch"]
+    resources: ["jobs"]
+    verbs: ["get", "list", "delete"]
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list", "delete"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: stale-job-cleanup
+subjects:
+  - kind: ServiceAccount
+    name: stale-job-cleanup
+    namespace: kube-system
+roleRef:
+  kind: ClusterRole
+  name: stale-job-cleanup
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: stale-job-cleanup
+  namespace: kube-system
+  labels:
+    app: stale-job-cleanup
+spec:
+  schedule: "0 4 * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      ttlSecondsAfterFinished: 86400  # self-cleanup after 24h
+      backoffLimit: 1
+      activeDeadlineSeconds: 300
+      template:
+        spec:
+          serviceAccountName: stale-job-cleanup
+          restartPolicy: Never
+          tolerations:
+            - key: node-role.kubernetes.io/control-plane
+              operator: Exists
+              effect: NoSchedule
+          containers:
+            - name: cleanup
+              image: alpine/k8s:1.31.0
+              command:
+                - sh
+                - -c
+                - |
+                  set -e
+                  NOW=$(date +%s)
+
+                  echo "=== Cleaning failed Jobs older than 24h ==="
+                  kubectl get jobs --all-namespaces -o json | \
+                    jq -r '.items[] |
+                      select(.status.conditions[]?.type == "Failed") |
+                      select(.status.completionTime or .status.startTime) |
+                      "\(.metadata.namespace) \(.metadata.name) \(.status.startTime // .status.completionTime // .metadata.creationTimestamp)"' | \
+                  while read -r NS NAME TS; do
+                    JOB_EPOCH=$(date -d "$TS" +%s 2>/dev/null || echo 0)
+                    AGE_H=$(( (NOW - JOB_EPOCH) / 3600 ))
+                    if [ "$AGE_H" -ge 24 ]; then
+                      echo "[delete] $NS/$NAME (failed ${AGE_H}h ago)"
+                      kubectl delete job "$NAME" -n "$NS" --cascade=foreground 2>/dev/null || true
+                    fi
+                  done
+
+                  echo ""
+                  echo "=== Cleaning completed standalone Jobs older than 72h ==="
+                  kubectl get jobs --all-namespaces -o json | \
+                    jq -r '.items[] |
+                      select(.status.succeeded >= 1) |
+                      select((.metadata.ownerReferences // []) | length == 0) |
+                      "\(.metadata.namespace) \(.metadata.name) \(.status.completionTime // .metadata.creationTimestamp)"' | \
+                  while read -r NS NAME TS; do
+                    JOB_EPOCH=$(date -d "$TS" +%s 2>/dev/null || echo 0)
+                    AGE_H=$(( (NOW - JOB_EPOCH) / 3600 ))
+                    if [ "$AGE_H" -ge 72 ]; then
+                      echo "[delete] $NS/$NAME (completed ${AGE_H}h ago, no owner)"
+                      kubectl delete job "$NAME" -n "$NS" --cascade=foreground 2>/dev/null || true
+                    fi
+                  done
+
+                  echo ""
+                  echo "=== Cleaning orphan Error/Failed/Evicted pods older than 1h ==="
+                  # Evicted pods show as Failed with reason Evicted
+                  kubectl get pods --all-namespaces -o json | \
+                    jq -r '.items[] |
+                      select(
+                        .status.phase == "Failed" or
+                        (.status.reason // "") == "Evicted" or
+                        (.status.containerStatuses // [] | any(.state.terminated.reason == "Error"))
+                      ) |
+                      select((.metadata.ownerReferences // []) | all(.kind != "Job")) |
+                      "\(.metadata.namespace) \(.metadata.name) \(.metadata.creationTimestamp)"' | \
+                  while read -r NS NAME TS; do
+                    POD_EPOCH=$(date -d "$TS" +%s 2>/dev/null || echo 0)
+                    AGE_H=$(( (NOW - POD_EPOCH) / 3600 ))
+                    if [ "$AGE_H" -ge 1 ]; then
+                      echo "[delete] $NS/$NAME (error/evicted ${AGE_H}h ago)"
+                      kubectl delete pod "$NAME" -n "$NS" --force 2>/dev/null || true
+                    fi
+                  done
+
+                  echo ""
+                  echo "Cleanup complete"
+              resources:
+                requests:
+                  cpu: 50m
+                  memory: 64Mi
+                limits:
+                  cpu: 250m
+                  memory: 128Mi

k8s/infra/databases/kustomization.yaml

@@ -1,8 +1,13 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 # Dedicated per-app CNPG clusters. NO top-level `namespace:` — each Cluster
-# carries its own ns (iam / temporal); a transformer would wrongly collapse them.
+# carries its own ns (iam / temporal / poimen / paperless); a transformer would
+# wrongly collapse them. poimen ns created by poimen-root app, memory-db
+# deployed into it. paperless ns declared in namespaces.yaml above.
 resources:
   - namespaces.yaml
   - authentik-db.yaml
   - temporal-db.yaml
+  - memory-db.yaml
+  - paperless-db.yaml
+  - obsidian-vault-pvc.yaml

k8s/infra/databases/memory-db.yaml

@@ -0,0 +1,36 @@
+# Dedicated CNPG Postgres for Poimen Memory (GitOps, wave 2).
+# Uses default longhorn storage class (3 replicas, dataLocality disabled).
+# CNPG generates secret `memory-db-app` + service `memory-db-rw` in ns poimen.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: memory-db
+  namespace: poimen
+  annotations:
+    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+spec:
+  instances: 2
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
+  bootstrap:
+    initdb:
+      database: memory
+      owner: app
+      encoding: UTF8
+      localeCollate: C
+      localeCType: C
+      postInitApplicationSQL:
+        - "CREATE EXTENSION vector;"
+  enableSuperuserAccess: false
+  resources:
+    requests: { memory: "512Mi", cpu: "250m" }
+    limits: { memory: "2Gi", cpu: "1" }
+  storage:
+    size: 20Gi
+    storageClass: longhorn
+  affinity:
+    podAntiAffinityType: preferred
+    topologyKey: kubernetes.io/hostname
+    tolerations:
+      - key: node-role.kubernetes.io/control-plane
+        operator: Exists
+        effect: NoSchedule

k8s/infra/databases/namespaces.yaml

@@ -1,6 +1,7 @@
 # DB clusters are wave 2 — their namespaces must exist first (their apps that
 # would CreateNamespace run later, w3/w8). Declared here so the databases App
-# creates them. authentik/vault/temporal CreateNamespace=true then no-ops.
+# creates them. authentik/vault/temporal/paperless CreateNamespace=true then
+# no-ops. poimen namespace created by poimen-root app (wave 7).
 apiVersion: v1
 kind: Namespace
 metadata:
@@ -10,3 +11,16 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: temporal
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: paperless
+---
+# Needed here (not just immich's own CreateNamespace=true at wave 8) because
+# k8s/infra/iam's PostSync job (wave 3) has a RoleBinding targeting this
+# namespace - same ordering reason as paperless above.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: immich

k8s/infra/databases/obsidian-vault-pvc.yaml

@@ -0,0 +1,18 @@
+---
+# Obsidian vault PVC — shared storage for REST API + UI pods
+# ReadWriteMany so both obsidian-server and obsidian-ui can mount simultaneously
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: obsidian-vault
+  namespace: poimen
+  labels:
+    app.kubernetes.io/name: obsidian-server
+    app.kubernetes.io/part-of: poimen-memory
+spec:
+  accessModes:
+    - ReadWriteMany
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 10Gi

k8s/infra/databases/paperless-db.yaml

@@ -0,0 +1,36 @@
+# Dedicated CNPG Postgres for paperless-ngx (GitOps, wave 2 — before the
+# paperless app at w8). Same recipe as memory-db: default longhorn storage
+# class (3 replicas), 2 instances, 20Gi.
+# CNPG generates secret `paperless-db-app` + service `paperless-db-rw` in ns
+# paperless; the paperless Deployment reads them locally.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: paperless-db
+  namespace: paperless
+  annotations:
+    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+spec:
+  instances: 3
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
+  bootstrap:
+    initdb:
+      database: paperless
+      owner: app
+      encoding: UTF8
+      localeCollate: C
+      localeCType: C
+  enableSuperuserAccess: false
+  resources:
+    requests: { memory: "512Mi", cpu: "250m" }
+    limits: { memory: "2Gi", cpu: "1" }
+  storage:
+    size: 20Gi
+    storageClass: longhorn
+  affinity:
+    podAntiAffinityType: preferred
+    topologyKey: kubernetes.io/hostname
+    tolerations:
+      - key: node-role.kubernetes.io/control-plane
+        operator: Exists
+        effect: NoSchedule

k8s/infra/forgejo-runner/templates/configmap.yaml

@@ -36,3 +36,4 @@ data:
       valid_volumes:
         - /docker-certs/client
       network: host
+      docker_host: automount

k8s/infra/forgejo-runner/templates/deployment.yaml

@@ -34,7 +34,11 @@ spec:
           command: ["sh", "-c"]
           args:
             - |
-              test -f /data/.runner || forgejo-runner register --no-interactive \
+              # Always re-register to keep labels in sync with values.yaml.
+              # Without this, changing a runner label requires manually deleting
+              # the PVC or .runner file — not GitOps-friendly.
+              rm -f /data/.runner
+              forgejo-runner register --no-interactive \
                 --instance {{ .Values.runner.forgejoUrl }} \
                 --token $(RUNNER_TOKEN) \
                 --name {{ .Values.runner.name }} \
@@ -56,20 +60,18 @@ spec:
       containers:
         - name: runner
           image: {{ .Values.runner.image.repository }}:{{ .Values.runner.image.tag }}
-          command: ["sh", "-c", "forgejo-runner daemon --config /etc/forgejo-runner/config.yaml"]
+          command: ["sh", "-c", "while ! wget -q -O- http://localhost:2375/_ping >/dev/null 2>&1; do echo 'waiting for dind...'; sleep 2; done; echo 'dind ready'; forgejo-runner daemon --config /etc/forgejo-runner/config.yaml"]
           workingDir: /data
           env:
             - name: DOCKER_HOST
-              value: tcp://localhost:2376
-            - name: DOCKER_TLS_VERIFY
-              value: "1"
-            - name: DOCKER_CERT_PATH
-              value: /docker-certs/client
+              value: tcp://localhost:2375
           volumeMounts:
             - name: runner-data
               mountPath: /data
             - name: docker-certs
               mountPath: /docker-certs
+            - name: docker-sock
+              mountPath: /run
             - name: homelab-ca
               mountPath: /etc/ssl/certs/homelab-ca.pem
               subPath: ca.crt
@@ -85,10 +87,12 @@ spec:
             privileged: true    # required for DinD; cicd namespace is labelled privileged
           env:
             - name: DOCKER_TLS_CERTDIR
-              value: /docker-certs
+              value: ""
           volumeMounts:
             - name: docker-certs
               mountPath: /docker-certs
+            - name: docker-sock
+              mountPath: /run
             - name: dind-storage
               mountPath: /var/lib/docker
             - name: homelab-ca
@@ -113,6 +117,8 @@ spec:
             claimName: {{ .Release.Name }}-dind
         - name: docker-certs
           emptyDir: {}    # DinD regenerates mTLS certs on each start
+        - name: docker-sock
+          emptyDir: {}    # Shared docker socket between dind and runner
         - name: homelab-ca
           # homelab-ca is a ConfigMap (public CA trust bundle), not a Secret.
           # The volumeMounts use subPath: ca.crt to project the single cert file.

k8s/infra/forgejo-runner/templates/gc-cronjob.yaml

@@ -0,0 +1,152 @@
+{{- if .Values.gc.enabled }}
+# Garbage-collects DinD Docker images/volumes/build-cache and actcache across
+# ALL forgejo-runner pods. Prevents PVC fill-up that breaks CI runs.
+# Only rendered once (enable in default values.yaml, disable in per-runner overrides).
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: runner-gc
+  namespace: {{ .Release.Namespace }}
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: runner-gc
+  namespace: {{ .Release.Namespace }}
+rules:
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list"]
+  - apiGroups: [""]
+    resources: ["pods/exec"]
+    verbs: ["create"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: runner-gc
+  namespace: {{ .Release.Namespace }}
+subjects:
+  - kind: ServiceAccount
+    name: runner-gc
+    namespace: {{ .Release.Namespace }}
+roleRef:
+  kind: Role
+  name: runner-gc
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: forgejo-runner-gc
+  namespace: {{ .Release.Namespace }}
+  labels:
+    app: forgejo-runner-gc
+spec:
+  schedule: {{ .Values.gc.schedule | quote }}
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 1
+      activeDeadlineSeconds: 900
+      template:
+        spec:
+          serviceAccountName: runner-gc
+          restartPolicy: Never
+          tolerations:
+            - key: node-role.kubernetes.io/control-plane
+              operator: Exists
+              effect: NoSchedule
+          containers:
+            - name: gc
+              image: {{ .Values.gc.image }}
+              command:
+                - sh
+                - -c
+                - |
+                  set -e
+
+                  # Iterate all forgejo-runner pods (golang, rust, node)
+                  PODS=$(kubectl -n {{ .Release.Namespace }} get pod \
+                    -l app -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.app}{"\n"}{end}' \
+                    | grep 'forgejo-runner-' | awk '{print $1}')
+
+                  if [ -z "$PODS" ]; then
+                    echo "no forgejo-runner pods found, skipping"
+                    exit 0
+                  fi
+
+                  for POD in $PODS; do
+                    echo "===== $POD ====="
+
+                    # 1. Docker image prune (DinD sidecar)
+                    echo "[docker] before:"
+                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- docker system df 2>/dev/null || true
+
+                    echo "[docker] pruning non-latest images older than {{ .Values.gc.pruneAge }}..."
+                    # Keep :latest tagged images, delete all others older than pruneAge.
+                    # docker image prune can't filter by tag, so we list and selectively rmi.
+                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- \
+                      sh -c '
+                        # Remove dangling (untagged) images older than {{ .Values.gc.pruneAge }}
+                        docker image prune -f --filter "until={{ .Values.gc.pruneAge }}" 2>/dev/null
+
+                        # Remove tagged non-latest images older than {{ .Values.gc.pruneAge }}
+                        CUTOFF=$(date -d "-{{ .Values.gc.pruneAgeHours }} hours" +%s 2>/dev/null || date -v-{{ .Values.gc.pruneAgeHours }}H +%s)
+                        docker images --format "{{"{{"}} .Repository {{"}}"}}:{{"{{"}} .Tag {{"}}"}} {{"{{"}} .CreatedAt {{"}}"}}" | while read -r IMAGE_TAG CREATED_REST; do
+                          TAG=$(echo "$IMAGE_TAG" | rev | cut -d: -f1 | rev)
+                          # Skip latest-tagged images
+                          if [ "$TAG" = "latest" ]; then
+                            echo "[keep] $IMAGE_TAG (latest)"
+                            continue
+                          fi
+                          # Check image age via inspect
+                          CREATED_TS=$(docker inspect --format="{{"{{"}} .Created {{"}}"}}" "$IMAGE_TAG" 2>/dev/null | head -1)
+                          if [ -z "$CREATED_TS" ]; then continue; fi
+                          IMAGE_EPOCH=$(date -d "$CREATED_TS" +%s 2>/dev/null || date -jf "%Y-%m-%dT%H:%M:%S" "$(echo $CREATED_TS | cut -dT -f1-2 | cut -d. -f1)" +%s 2>/dev/null || echo 0)
+                          if [ "$IMAGE_EPOCH" -lt "$CUTOFF" ] 2>/dev/null; then
+                            echo "[delete] $IMAGE_TAG (older than {{ .Values.gc.pruneAge }})"
+                            docker rmi -f "$IMAGE_TAG" 2>/dev/null || true
+                          else
+                            echo "[keep] $IMAGE_TAG (recent)"
+                          fi
+                        done
+                      ' 2>/dev/null || true
+
+                    echo "[docker] pruning build cache unused >{{ .Values.gc.pruneAge }}..."
+                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- \
+                      docker builder prune -af --filter "until={{ .Values.gc.pruneAge }}" 2>/dev/null || true
+
+                    echo "[docker] pruning dangling volumes..."
+                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- \
+                      docker volume prune -af 2>/dev/null || true
+
+                    echo "[docker] after:"
+                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- docker system df 2>/dev/null || true
+
+                    # 2. Actcache cleanup (runner container)
+                    echo "[actcache] cleaning incomplete and stale cache entries..."
+                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c runner -- \
+                      sh -c '
+                        # Delete incomplete/partial cache uploads immediately (tmp dirs)
+                        find /data/.cache/actcache/cache -name "tmp" -type d -exec rm -rf {} + 2>/dev/null || true
+                        # Delete cache entries not accessed in last {{ .Values.gc.actcacheMaxAgeDays }} day(s)
+                        find /data/.cache/actcache/cache -type f -mtime +{{ .Values.gc.actcacheMaxAgeDays }} -delete 2>/dev/null || true
+                        # Clean up empty directories
+                        find /data/.cache/actcache/cache -type d -empty -delete 2>/dev/null || true
+                        echo "actcache size: $(du -sh /data/.cache/actcache/cache 2>/dev/null | cut -f1)"
+                      ' || true
+
+                    echo ""
+                  done
+                  echo "GC complete"
+              resources:
+                requests:
+                  cpu: 50m
+                  memory: 64Mi
+                limits:
+                  cpu: 250m
+                  memory: 128Mi
+{{- end }}

k8s/infra/forgejo-runner/values-node.yaml

@@ -2,8 +2,15 @@
 # runner instance. Only runner.name and runner.labels differ -- everything
 # else (image, dind, persistence, tolerations, nodeSelector) is shared.
 #
-# node:22-bookworm ships Node natively, so unlike the golang/rust instances,
-# jobs on this runner need no "install node" step before actions/checkout.
+# Label image: node:22-bookworm — Debian, root, apt-get, Node.js, npm, git.
+# Install docker in workflow steps as needed.
 runner:
+  image:
+    repository: code.forgejo.org/forgejo/runner
+    tag: "6"
   name: node-runner
   labels: "node:docker://node:22-bookworm"
+
+# GC CronJob renders only from the default (golang) values to avoid duplicates
+gc:
+  enabled: false

k8s/infra/forgejo-runner/values-rust.yaml

@@ -2,9 +2,17 @@
 # runner instance. Only runner.name and runner.labels differ -- everything
 # else (image, dind, persistence, tolerations, nodeSelector) is shared.
 #
-# rust:1.83-bookworm -- verified this tag exists (docker manifest inspect)
-# before pinning it, per this repo's convention of not trusting a tag exists
-# without checking.
+# Label image: rust:1-bookworm — Debian, root, apt-get, Rust, cargo, git.
+# Install Node.js/docker in workflow steps as needed.
 runner:
+  image:
+    repository: code.forgejo.org/forgejo/runner
+    tag: "6"
   name: rust-runner
-  labels: "rust:docker://rust:1.83-bookworm"
+  labels: "rust:docker://rust:1-bookworm"
+
+
+
+# GC CronJob renders only from the default (golang) values to avoid duplicates
+gc:
+  enabled: false

k8s/infra/forgejo-runner/values.yaml

@@ -1,20 +1,13 @@
 runner:
   image:
     repository: code.forgejo.org/forgejo/runner
-    tag: "6"          # pin exact release before apply
+    tag: "6"
   name: golang-runner
-  # Default image is only used when a job's `container:` doesn't override it
-  # (both ci.yaml and build.yaml in homelab-frontend do). Retired the old
-  # "docker" label entirely; every repo this runner serves is Go, so this
-  # instance carries the golang toolchain and its own dind sidecar builds and
-  # pushes that repo's images too -- there is no separate generic runner
-  # anymore.
-  labels: "golang:docker://golang:1.25-bookworm"
-  # In-cluster Service (:3000) — direct, avoids the ingress/public-hostname hop
-  # (the public URL is :443 which forgejo doesn't serve; runner got i/o timeout).
+  # Label image is what workflow steps run in (NOT the runner daemon image).
+  # golang:1.26-bookworm: Debian, root, apt-get, Go, git.
+  # TODO: Switch to custom image once build-runner-images.yml pushes images
+  labels: "golang:docker://golang:1.26-bookworm"
   forgejoUrl: http://forgejo-gitea-http.cicd.svc.cluster.local:3000
-  # tokenSecret: name of the K8s Secret that holds the runner registration token
-  # created automatically by the helmfile presync hook (see helmfile.yaml.gotmpl)
   tokenSecret: runner-token
   resources:
     requests:
@@ -39,7 +32,7 @@ dind:
 persistence:
   reg:
     storageClass: longhorn  # Unified StorageClass (3 replicas)
-    size: 1Gi               # .runner registration file + config — survives pod restarts
+    size: 20Gi              # .runner registration file + action tool cache + actcache artifacts
   dind:
     storageClass: longhorn  # Unified StorageClass (3 replicas)
     size: 30Gi              # docker layer cache — keeps rebuilds fast across restarts
@@ -49,7 +42,18 @@ tolerations:
     operator: Exists
     effect: NoSchedule
 
-# Pin to az-a (talos-cp-1) — sole Longhorn node; RWO PVCs (reg/dind cache) only
-# attach there.
+# Pin to az-b (talos-cp-2) — more Longhorn storage than az-a (worker-1 over-provisioned).
+# RWO PVCs will recreate on talos-cp-2 when nodeSelector changes.
 nodeSelector:
-  topology.kubernetes.io/zone: az-a
+  topology.kubernetes.io/zone: az-b
+
+# GC CronJob — prunes Docker images/volumes/build-cache and actcache across
+# ALL forgejo-runner pods. Only enable in default values (golang instance);
+# disable in per-runner overrides so it renders once.
+gc:
+  enabled: true
+  schedule: "*/30 * * * *"     # every 30 minutes
+  image: alpine/k8s:1.31.0
+  pruneAge: "30m"              # Docker artifacts unused longer than this get pruned
+  pruneAgeHours: 0.5            # Same as pruneAge but numeric for date arithmetic in shell
+  actcacheMaxAgeDays: 1        # actcache files older than N days (aggressive for heavy Rust cargo builds)