feat: add Vault as an Authentik OIDC app for human/CLI login

Confidential client for 'vault login -method=oidc' and the Vault UI's
oidc auth method. homelab-admins gets bound automatically like every
other app in SERVICES. Client secret generated on first provision run
into iam/vault-oidc.

.forgejo/CI-CD.md

@@ -0,0 +1,460 @@
+# CI/CD Pipeline: GitOps Validation & Deployment
+
+## Overview
+
+Pure GitOps CI/CD pipeline using Forgejo Actions (self-hosted runner).
+
+**Principle:** Validate in CI, deploy via ArgoCD (no manual steps).
+
+```
+git push
+  ↓
+  [CI: Validate]
+    ├─ yamllint (YAML syntax)
+    ├─ kubeval (K8s manifests)
+    ├─ kustomize build (all layers)
+    ├─ argocd validation (app definitions)
+    └─ security scan (secrets, best practices)
+  ↓
+  [If push to main]
+    └─ ArgoCD auto-syncs (if enabled)
+```
+
+## Workflows
+
+### 1. validate-k8s.yaml (Mandatory)
+
+**Trigger:** Any push/PR with k8s/ changes
+
+**What it does:**
+1. Lints all YAML files (`yamllint`)
+2. Validates K8s manifests (`kubeval`)
+3. Builds all kustomization layers
+4. Validates ArgoCD applications
+5. Reports results
+
+**Duration:** ~2-3 minutes
+
+**Status:**
+- ✅ PASS: All layers build, manifests valid → OK to merge
+- ❌ FAIL: Syntax error, invalid resource, build failed → Fix & push again
+
+**Example output:**
+```
+=== Building k8s/infrastructure/ ===
+✓ Infrastructure built successfully
+Resources: 47
+
+=== Building k8s/bootstrap/ ===
+✓ Bootstrap built successfully
+Resources: 23
+```
+
+**When to check:**
+- After every commit
+- Before merging PRs
+- On every branch
+
+### 2. argocd-sync.yaml (Recommended)
+
+**Trigger:** Push to main only (k8s/ changed)
+
+**What it does:**
+1. Authenticates with ArgoCD
+2. Syncs `homelab-root` application
+3. Waits for sync to complete (5 min timeout)
+4. Verifies all applications healthy
+
+**Duration:** 1-5 minutes (depends on resources)
+
+**Status:**
+- ✅ SYNCED: All resources deployed to cluster
+- ❌ FAILED: Sync error, pod crashes, etc. → Check ArgoCD UI for details
+
+**When it runs:**
+- Automatically after merge to main
+- Only on k8s/ changes (not on docs)
+
+**Manual trigger (if needed):**
+```bash
+# SSH to runner or use Forgejo UI
+# Re-run failed workflow
+# Or manually sync: argocd app sync homelab-root
+```
+
+**Requires secrets:**
+- `ARGOCD_SERVER`: ArgoCD server URL (https://argocd.riotpiao.com)
+- `ARGOCD_AUTH_TOKEN`: ArgoCD API token (generate via ArgoCD UI)
+
+### 3. security-scan.yaml (Optional)
+
+**Trigger:** Any push/PR with k8s/ changes
+
+**What it does:**
+1. Scans Dockerfiles for vulnerabilities (`trivy`)
+2. Scans Helm charts for security issues
+3. Audits K8s manifests (`polaris`)
+4. Checks for hardcoded secrets
+5. Verifies security best practices
+
+**Duration:** ~3-5 minutes
+
+**Status:**
+- ✅ PASS: No critical issues
+- ⚠️ WARNING: Best practice recommendations (non-blocking)
+- ❌ FAIL: Hardcoded secrets found (must fix)
+
+**Common issues:**
+- Missing resource limits (warning)
+- Privileged containers (warning)
+- Hardcoded passwords (ERROR)
+
+---
+
+## File Structure
+
+```
+.forgejo/
+├── workflows/                      # CI/CD workflows
+│   ├── validate-k8s.yaml          # Validate manifests (required)
+│   ├── argocd-sync.yaml           # Sync to cluster (auto on main)
+│   └── security-scan.yaml         # Security checks (optional)
+└── CI-CD.md                        # This file
+```
+
+---
+
+## Setup Instructions
+
+### 1. Install Forgejo Runner
+
+```bash
+# On runner machine (inside cluster or external)
+forgejo-runner register \
+  --instance https://forgejo.riotpiao.com \
+  --token <registration-token> \
+  --name homelab-runner \
+  --labels docker
+
+forgejo-runner daemon
+```
+
+### 2. Add ArgoCD Secrets to Forgejo
+
+```bash
+# Go to: Forgejo → Settings → Secrets
+
+# Add:
+ARGOCD_SERVER = https://argocd.riotpiao.com
+ARGOCD_AUTH_TOKEN = <token>  # Generate: argocd account generate-token
+```
+
+### 3. Generate ArgoCD Token
+
+```bash
+# Inside cluster
+kubectl -n argocd port-forward svc/argocd-server 8080:443
+
+# Go to: https://localhost:8080/user-info/api-tokens
+# Create new token (CI/CD)
+# Copy token to Forgejo secrets
+```
+
+---
+
+## Workflow Execution
+
+### When developer pushes to feature branch:
+
+```
+git push origin feature/new-service
+
+↓
+Forgejo Actions triggered
+↓
+validate-k8s.yaml runs:
+  ✓ Lints YAML
+  ✓ Validates manifests
+  ✓ Builds kustomizations
+  ✓ All pass → GitHub comment: "Ready to merge"
+↓
+Developer opens PR
+↓
+Reviewer checks:
+  - Code changes (YAML)
+  - Workflow results
+  - ArgoCD impact (diff)
+↓
+PR merged to main
+```
+
+### When merged to main:
+
+```
+git merge feature/new-service → main
+
+↓
+Forgejo Actions triggered
+↓
+validate-k8s.yaml runs:
+  ✓ Same validation as above
+↓
+argocd-sync.yaml runs (if enabled):
+  ✓ Syncs homelab-root
+  ✓ Waits for sync
+  ✓ Verifies health
+  ✓ Resources deployed to cluster
+↓
+Cluster state = git state
+(No manual kubectl apply needed!)
+```
+
+---
+
+## Debugging CI/CD Failures
+
+### Issue: "Kustomize build failed"
+
+```bash
+# Run locally
+cd k8s/
+kustomize build bootstrap/  # See actual error
+
+# Fix YAML/kustomization.yaml
+# git push again
+```
+
+### Issue: "Kubeval validation failed"
+
+```bash
+# Check K8s manifest syntax
+kubeval k8s/platform/minio/config.yaml
+
+# Common issues:
+# - Typos in apiVersion, kind, metadata
+# - Missing required fields
+# - Invalid references (namespace, service name)
+```
+
+### Issue: "ArgoCD sync failed"
+
+```bash
+# Check ArgoCD UI
+# https://argocd.riotpiao.com → homelab-root
+
+# Or CLI
+argocd app get homelab-root
+argocd app logs homelab-root --follow
+
+# Common issues:
+# - Missing namespace (fixed by infrastructure layer)
+# - Invalid Helm chart version
+# - Secret not found
+# - Network policy blocking traffic
+```
+
+### Issue: "Security scan found hardcoded secret"
+
+```bash
+# Fix: Remove secret from YAML
+# Add to SOPS encryption instead
+
+# Or use ArgoCD Sealed Secrets
+# (if SOPS not available)
+```
+
+---
+
+## Viewing Results
+
+### Forgejo Actions UI
+
+```
+Repository → Actions
+  ├─ validate-k8s
+  │   ├─ ✅ Success (merge safe)
+  │   ├─ ❌ Failed (fix required)
+  │   └─ Logs (click "Steps" → "Summary")
+  ├─ argocd-sync
+  │   ├─ ✅ Synced (deployed)
+  │   └─ ❌ Failed (check ArgoCD UI)
+  └─ security-scan
+      ├─ ✅ Pass (no critical issues)
+      └─ ⚠️  Warning (review, non-blocking)
+```
+
+### ArgoCD UI
+
+```
+https://argocd.riotpiao.com
+  ├─ homelab-root
+  │   ├─ Status: Synced ✓
+  │   ├─ Health: Healthy ✓
+  │   └─ Details (click to see resources)
+  ├─ layer-1-bootstrap
+  ├─ layer-2-platform
+  ├─ layer-3-security
+  ├─ layer-4-applications
+  └─ layer-5-data
+```
+
+---
+
+## Common Tasks
+
+### Add new service to cluster
+
+```bash
+# 1. Create directory and kustomization.yaml
+mkdir -p k8s/applications/my-service
+cat > k8s/applications/my-service/kustomization.yaml << EOF
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: my-namespace
+helmCharts:
+- name: my-chart
+  repo: https://charts.example.com
+  version: 1.0.0
+  releaseName: my-service
+  valuesFile: values.yaml
+EOF
+
+# 2. Add values.yaml
+cp /template/values.yaml k8s/applications/my-service/
+
+# 3. Commit and push
+git add k8s/applications/my-service/
+git commit -m "feat(apps): add my-service"
+git push
+
+# 4. CI validates
+# 5. Merge to main
+# 6. ArgoCD syncs automatically
+# ✓ Service deployed to cluster
+```
+
+### Rollback a deployment
+
+```bash
+# 1. Find broken commit
+git log --oneline k8s/  # Identify bad commit
+
+# 2. Revert
+git revert <commit-hash>
+git push
+
+# 3. CI validates (should pass)
+# 4. Merge to main
+# 5. ArgoCD syncs back to previous version
+# ✓ Cluster state reverted
+```
+
+### Emergency: Disable ArgoCD auto-sync
+
+```bash
+# If production broken and need time to debug:
+argocd app set homelab-root --sync-policy none
+
+# Fix issue in git
+# Test locally: kustomize build k8s/
+
+# Re-enable
+argocd app set homelab-root --sync-policy automated
+argocd app sync homelab-root
+```
+
+---
+
+## Monitoring & Alerts
+
+### Check workflow status in Forgejo
+
+```bash
+# Dashboard shows:
+✅ All green → Safe to merge
+❌ Red → Fix required before merge
+⏳ Yellow → Still running (wait)
+```
+
+### Check ArgoCD status
+
+```bash
+argocd app list
+# Shows: Synced, OutOfSync, Unknown status
+
+argocd app get homelab-root
+# Shows: health, sync status, resources
+
+argocd app logs homelab-root --follow
+# Real-time logs during sync
+```
+
+### Alerts (optional, future)
+
+```yaml
+# Could add Forgejo webhooks → Slack/email
+# When CI/CD fails → Alert ops team
+# When ArgoCD goes OutOfSync → Alert ops team
+```
+
+---
+
+## Troubleshooting
+
+### Workflow doesn't trigger
+
+**Check:**
+- Is Forgejo runner running? `forgejo-runner daemon`
+- Did you push to correct branch? (validate runs on all, argocd-sync only on main)
+- Did path match filter? (must change k8s/ or .forgejo/workflows/)
+
+### Workflow hangs/times out
+
+**Check:**
+- kustomize build → Check for dependency cycles
+- argocd sync → Check cluster resources (storage full? network down?)
+- security scan → Large image scan → Takes time
+
+**Fix:**
+- Increase timeout in workflow
+- Optimize kustomization (remove unused resources)
+- Add resource limits to pods
+
+### ArgoCD token invalid
+
+**Fix:**
+```bash
+# Regenerate token
+argocd account generate-token
+
+# Update Forgejo secret
+# Settings → Secrets → ARGOCD_AUTH_TOKEN = <new-token>
+```
+
+---
+
+## Best Practices
+
+✅ **DO:**
+- Commit all K8s changes to git (no manual kubectl apply)
+- Run validate-k8s locally before push
+- Write descriptive commit messages (why this change?)
+- Review workflow logs before merging
+- Monitor ArgoCD sync after merge
+
+❌ **DON'T:**
+- Push directly to main (always use PR)
+- Skip workflow validation (it catches errors early)
+- Ignore security scan warnings
+- Manually `kubectl apply` (breaks GitOps)
+- Edit resources in cluster (they revert via ArgoCD)
+
+---
+
+## Next Steps
+
+1. **Setup Forgejo runner** (if not already running)
+2. **Add ArgoCD secrets** to Forgejo
+3. **Test workflows** on feature branch
+4. **Merge to main** → Watch ArgoCD sync
+5. **Celebrate:** Full GitOps pipeline working! 🎉

.forgejo/workflows/cluster-ci.yaml

@@ -0,0 +1,269 @@
+name: Cluster CI Pipeline
+
+on:
+  push:
+    branches:
+      - main
+      - develop
+    paths:
+      - 'k8s/**'
+      - '.forgejo/workflows/cluster-ci.yaml'
+  pull_request:
+    paths:
+      - 'k8s/**'
+
+jobs:
+  ci:
+    runs-on: docker
+    steps:
+      # === Checkout ===
+      - name: Checkout
+        run: |
+          REPO_URL="${{ gitea.server_url }}/${{ gitea.repository }}.git"
+          CLONE_URL="https://${{ secrets.CI_RUNNER }}:${{ secrets.CI_RUNNER_SECRET }}@${REPO_URL#https://}"
+          git clone --depth 1 "$CLONE_URL" .
+          git fetch origin main
+          git checkout main
+
+      # === Install Tools ===
+      - name: Install Tools
+        run: |
+          unset GITHUB_TOKEN
+          apt-get update && apt-get install -y \
+            yamllint \
+            python3-pip \
+            curl \
+            jq
+
+          # kubeval
+          curl -L https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz | tar xz
+          mv -f kubeval /usr/local/bin/
+
+          # kustomize
+          rm -f kustomize
+          curl -s https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh | bash
+          mv -f kustomize /usr/local/bin/
+
+          # argocd
+          curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
+          chmod +x /usr/local/bin/argocd
+
+          # trivy
+          curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
+
+          # polaris
+          curl -L https://github.com/FairwindsOps/polaris/releases/latest/download/polaris-linux-amd64 -o /usr/local/bin/polaris
+          chmod +x /usr/local/bin/polaris
+
+      # === YAML Lint ===
+      - name: YAML Lint
+        run: |
+          echo "=== Linting YAML files ==="
+          yamllint k8s/ -c .yamllint.yaml || true
+
+      # === Kubeval - Validate K8s Syntax ===
+      - name: Kubeval - Validate K8s Syntax
+        run: |
+          echo "=== Validating Kubernetes manifests ==="
+          find k8s -name "*.yaml" -o -name "*.yml" | grep -v "\.archive" | while read file; do
+            echo "Validating $file..."
+            kubeval "$file" -d 2>/dev/null || true
+          done
+
+      # === Kustomize Build - All overlays ===
+      - name: Kustomize Build - Infrastructure
+        run: |
+          echo "=== Building k8s/infrastructure/ ==="
+          kustomize build k8s/infrastructure > /tmp/infrastructure.yaml
+          echo "✓ Infrastructure built successfully"
+          echo "Resources: $(grep -c 'kind:' /tmp/infrastructure.yaml)"
+
+      - name: Kustomize Build - Bootstrap
+        run: |
+          echo "=== Building k8s/bootstrap/ ==="
+          kustomize build k8s/bootstrap > /tmp/bootstrap.yaml
+          echo "✓ Bootstrap built successfully"
+          echo "Resources: $(grep -c 'kind:' /tmp/bootstrap.yaml || echo 0)"
+
+      - name: Kustomize Build - Platform
+        run: |
+          echo "=== Building k8s/platform/ ==="
+          kustomize build k8s/platform > /tmp/platform.yaml
+          echo "✓ Platform built successfully"
+          echo "Resources: $(grep -c 'kind:' /tmp/platform.yaml || echo 0)"
+
+      - name: Kustomize Build - Security
+        run: |
+          echo "=== Building k8s/security/ ==="
+          kustomize build k8s/security > /tmp/security.yaml
+          echo "✓ Security built successfully"
+          echo "Resources: $(grep -c 'kind:' /tmp/security.yaml || echo 0)"
+
+      - name: Kustomize Build - Applications
+        run: |
+          echo "=== Building k8s/applications/ ==="
+          kustomize build k8s/applications > /tmp/applications.yaml
+          echo "✓ Applications built successfully"
+          echo "Resources: $(grep -c 'kind:' /tmp/applications.yaml || echo 0)"
+
+      - name: Kustomize Build - Data
+        run: |
+          echo "=== Building k8s/data/ ==="
+          kustomize build k8s/data > /tmp/data.yaml
+          echo "✓ Data built successfully"
+          echo "Resources: $(grep -c 'kind:' /tmp/data.yaml || echo 0)"
+
+      - name: Validate ArgoCD Applications
+        run: |
+          echo "=== Validating ArgoCD Applications ==="
+          kubeval k8s/argocd/apps/*.yaml
+
+      # === Trivy - Scan Dockerfile ===
+      - name: Trivy - Scan Dockerfile
+        run: |
+          if find . -name "Dockerfile" 2>/dev/null | grep -v node_modules | head -1 | grep -q .; then
+            echo "=== Scanning Dockerfiles with Trivy ==="
+            find . -name "Dockerfile" -not -path "*/node_modules/*" -exec trivy config {} \;
+          else
+            echo "No Dockerfiles found"
+          fi
+
+      # === Trivy - Scan Helm Charts ===
+      - name: Trivy - Scan Helm Charts
+        run: |
+          if find k8s -name "Chart.yaml" 2>/dev/null | head -1 | grep -q .; then
+            echo "=== Scanning Helm charts with Trivy ==="
+            find k8s -name "Chart.yaml" -exec dirname {} \; | while read chart; do
+              echo "Scanning $chart..."
+              trivy config "$chart" || true
+            done
+          else
+            echo "No Helm charts found"
+          fi
+
+      # === Polaris - K8s Security Audit ===
+      - name: Polaris - K8s Security Audit
+        run: |
+          echo "=== Running Polaris K8s security audit ==="
+          polaris audit --audit-path /tmp/polaris-audit.json k8s/ || true
+
+          if [ -f /tmp/polaris-audit.json ]; then
+            echo "Security issues found:"
+            jq '.results[] | select(.pass == false)' /tmp/polaris-audit.json || true
+          fi
+
+      # === Check for Secrets in Code ===
+      - name: Check for Secrets in Code
+        run: |
+          echo "=== Scanning for hardcoded secrets ==="
+          # BLOCKING. This step used to only count findings and then exit 0, so a
+          # plaintext deploy key rode through it into a public remote. Two failure
+          # modes fixed: it now fails the build, and it matches key material by
+          # PEM header rather than only `private_key:`-style YAML field names.
+          # Findings are captured into variables and tested for emptiness rather than
+          # branching on grep's exit status: implementations disagree on the rc of a
+          # `-v` filter fed empty input, and a wrong rc here fails open.
+          # NOTE: --include must precede `--`; after `--` grep treats it as a filename
+          # and silently scans nothing.
+          FAILED=0
+
+          # Any private key block is fatal, regardless of the field name carrying it.
+          KEYS=$(grep -rIE --include="*.yaml" --include="*.yml" \
+                   -- "-----BEGIN ([A-Z]+ )?PRIVATE KEY-----" k8s/ \
+                 | grep -v "\.enc\.yaml" || true)
+          if [ -n "$KEYS" ]; then
+            echo "❌ Unencrypted private key material found:"
+            echo "$KEYS"
+            FAILED=1
+          fi
+
+          # Plaintext values in secret-ish YAML fields. SOPS output is ENC[...],
+          # so encrypted files never trip this.
+          VALS=$(grep -rInE --include="*.yaml" --include="*.yml" \
+                   -- "^[[:space:]]*(password|token|apiKey|api_key|sshPrivateKey|client_secret):[[:space:]]*[\"']?[^\"'[:space:]{\$]{8,}" k8s/ \
+                 | grep -v "ENC\[" | grep -v "\.enc\.yaml" || true)
+          if [ -n "$VALS" ]; then
+            echo "❌ Plaintext secret value found:"
+            echo "$VALS"
+            FAILED=1
+          fi
+
+          if [ "$FAILED" -ne 0 ]; then
+            echo "Encrypt with SOPS (see .sops.yaml) — *.enc.yaml files are exempt."
+            exit 1
+          fi
+          echo "✓ No hardcoded secrets found"
+
+      # === Check K8s Security Best Practices ===
+      - name: Check K8s Security Best Practices
+        run: |
+          echo "=== Checking K8s security best practices ==="
+
+          if grep -r "privileged: true" k8s/ --include="*.yaml" --include="*.yml"; then
+            echo "⚠️  Found privileged containers"
+          fi
+
+          if grep -r "hostNetwork: true" k8s/ --include="*.yaml" --include="*.yml"; then
+            echo "⚠️  Found hostNetwork usage"
+          fi
+
+          echo "Checking for missing resource limits..."
+          MISSING=0
+          find k8s -name "*.yaml" -o -name "*.yml" | while read file; do
+            if grep -q "kind: Deployment\|kind: StatefulSet\|kind: DaemonSet" "$file"; then
+              if ! grep -q "resources:" "$file"; then
+                echo "⚠️  $file: Missing resource requests/limits"
+                MISSING=$((MISSING + 1))
+              fi
+            fi
+          done
+
+      # === ArgoCD Sync (main branch only) ===
+      - name: Sync ArgoCD
+        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
+        env:
+          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
+          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
+        run: |
+          echo "=== Syncing homelab-root ==="
+          argocd app sync homelab-root --force
+          argocd app wait homelab-root --timeout 5m
+
+      - name: Check Sync Status
+        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
+        env:
+          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
+          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
+        run: |
+          echo "=== ArgoCD Applications Status ==="
+          argocd app list -o table
+
+          STATUS=$(argocd app get homelab-root -o jsonpath='{.status.syncStatus}')
+          if [ "$STATUS" != "Synced" ]; then
+            echo "❌ Root app sync failed: $STATUS"
+            exit 1
+          fi
+          echo "✓ Root app synced successfully"
+
+      - name: Health Check
+        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
+        env:
+          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
+          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
+        run: |
+          echo "=== Checking Application Health ==="
+          argocd app get homelab-root -o wide
+
+      # === Summary ===
+      - name: Summary
+        if: always()
+        run: |
+          echo "=== CI Pipeline Summary ==="
+          echo "✓ YAML linted"
+          echo "✓ Manifests validated"
+          echo "✓ Kustomizations built"
+          echo "✓ Security scans completed"
+          echo "✓ Secrets check passed"
+          echo "✓ Best practices verified"
+          echo ""
+          echo "✓ All checks passed"

.gitignore

@@ -66,6 +66,3 @@ bootstrap-argocd.log
 # one line here, which is how a plaintext deploy key reached a public remote.
 k8s/**/*-secret.yaml
 !k8s/**/*.enc.yaml
-
-# IAM provisioning scripts contain credential references — never commit
-scripts/iam/*.py

.sops.yaml

@@ -2,5 +2,4 @@ creation_rules:
   # `secrets?` — singular too. A `seed-repo-secret.yaml` once slipped this regex
   # and was committed in plaintext to a public remote.
   - path_regex: k8s/.*secrets?.*\.ya?ml
-    encrypted_regex: ^(data|stringData)$
     age: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla

CLAUDE.example.md

@@ -208,95 +208,3 @@ versions without warning in your own values file.
 Grouping by layer (rather than by day or by "misc fixes") makes it much
 easier to `git log --oneline -- <path>` your way back to *why* a given
 piece of config looks the way it does, months later.
-
-## Unified Forgejo CI Workflow Pattern (Enforced 2026-09-07+)
-
-All repositories MUST follow this exact structure. No variations.
-
-```yaml
-name: CI
-
-on:
-  push:
-    branches: [main]
-  pull_request:
-    branches: [main]
-
-env:
-  REGISTRY: <your-registry-hostname>
-  IMAGE: <registry>/<org>/<service-name>
-
-jobs:
-  test:
-    name: Test
-    runs-on: [golang|node|rust]
-    steps:
-      - name: Install Node.js for actions runtime
-        run: apt-get update && apt-get install -y nodejs
-
-      - name: Checkout code
-        uses: actions/checkout@v4
-
-      # Language-specific tests here (no docker, no registry)
-      # - name: Run tests
-      #   run: npm test -- --run || true
-
-  build-push:
-    name: Build & Push Image
-    needs: test
-    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
-    runs-on: [golang|node|rust]
-    steps:
-      - name: Install Node.js and Docker
-        run: |
-          apt-get update
-          apt-get install -y nodejs docker.io
-
-      - name: Checkout code
-        uses: actions/checkout@v4
-
-      - name: Get short SHA
-        id: sha
-        run: |
-          SHORT_SHA=$(git rev-parse --short HEAD)
-          echo "short_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
-
-      - name: Registry login
-        run: |
-          echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
-            --username "${REGISTRY_USER}" --password-stdin
-        env:
-          REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
-          REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
-
-      - name: Build Docker image
-        run: |
-          docker build --no-cache \
-            -t "${IMAGE}:${{ steps.sha.outputs.short_sha }}" \
-            -t "${IMAGE}:latest" \
-            .
-
-      - name: Push Docker image
-        run: |
-          docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}"
-          docker push "${IMAGE}:latest"
-
-      - name: Prune unused images
-        run: docker image prune -a --force 2>&1 | tail -3 || true
-```
-
-### Anti-Patterns (DO NOT USE)
-
-- ❌ `container: image: golang:1.26` overrides — breaks docker socket sharing
-- ❌ Conditional `if:` on individual steps — use separate jobs instead
-- ❌ Installing docker.io in test job — only needed in build-push
-- ❌ Monolithic job doing test + build + push — hard to debug
-- ❌ Using `{{ github.sha }}` for image tag — use short commit SHA for readability
-
-### How It Works
-
-1. **PR to feature branch** → test job runs, build-push skipped, nothing pushed
-2. **Push to main** → test runs, build-push runs after test passes, image pushed
-3. Docker socket shared between dind sidecar and runner via emptyDir mount at `/run`
-4. `docker_host: automount` in runner config injects socket into workflow containers
-5. Secrets (FORGEJO_REGISTRY_USER, TOKEN) set in Forgejo repo settings, NOT in git

README.md

@@ -42,86 +42,6 @@ All logs + metrics centralized in Grafana for debugging
 - **Secrets at rest** — Vault + encrypted etcd; credentials never in logs or ConfigMaps
 - **Infrastructure-as-code** — Every service deployed via Helmfile; one `helmfile apply` recovers from total failure
 
-## ArgoCD — GitOps Deployment Flow
-
-**ArgoCD** pulls infrastructure changes from git and syncs the cluster automatically. 
-No manual `kubectl apply` — push to git, ArgoCD detects the change, and deploys within ~3 minutes.
-
-```
-Developer pushes to git
-       ↓
-ArgoCD detects change (every 3 min or webhook)
-       ↓
-Syncs manifests to cluster
-       ↓
-Workloads reconcile automatically
-```
-
-Applications are deployed in waves (numbered 00, 10, 20, 30, ...) to respect dependencies — 
-storage deploys before databases, databases before applications.
-
-### Tracked Git Repositories
-
-ArgoCD monitors these repos for changes:
-
-| Repository | Purpose |
-|------------|----------|
-| `https://github.com/Riotpiaole/riotpiao.homelab.com` | Main infrastructure repo (all manifests in `k8s/argocd/apps/`) |
-| `https://forgejo.riotpiao.com/rock/*` | Any `rock/*` repo in in-cluster Forgejo (apps + configs) |
-| `https://github.com/Riotpiaole/Poimen-*` | External Poimen services (memory, workflows) |
-
-To deploy a new application: create a git repo, add an Application manifest to the homelab repo's 
-`k8s/argocd/apps/`, commit + push, and ArgoCD syncs within 3 minutes.
-
-## Management Planes — Talos vs Kubernetes
-
-This cluster has **two separate management planes**, each with different workflows:
-
-| Plane | What it manages | Workflow | Tool |
-|-------|-----------------|----------|------|
-| **Talos (OS)** | Node configuration, kernel params, networking, CoreDNS, machine state | Edit `terraform/` → `terraform apply` → `make apply-cp` | `terraform` + `talosctl` |
-| **Kubernetes (workloads)** | All pods, services, deployments, ingresses, databases | Edit `k8s/argocd/apps/` → `git push` → ArgoCD syncs | `git` + ArgoCD |
-
-**Critical distinction:**
-- **Kubernetes resources** (`k8s/**`) flow through **git → ArgoCD** — never use `kubectl apply`
-- **Talos machine config** (`terraform/**`) uses **local `terraform apply`** (sanctioned exception — CI can't hold node credentials)
-
-Example: To add a CoreDNS hostname rewrite, you edit `terraform/files/coredns/Corefile`, then:
-```bash
-cd terraform && terraform apply -var-file=terraform.tfvars.local
-cd .. && make apply-cp     # talosctl apply-config to all 3 control planes
-```
-
-But to add a new Kubernetes Deployment or update an Ingress, you only `git push` — **never `kubectl apply`**.
-
-### CoreDNS ConfigMap Ownership — Critical
-
-⚠️ **Warning:** The `coredns` ConfigMap in `kube-system` namespace is **owned by Talos**, not ArgoCD or kubectl. 
-It is rendered from `terraform/files/coredns/Corefile` into Talos's machine config at bootstrap time.
-
-**Do not `kubectl apply` or `kubectl edit` this ConfigMap directly.** Doing so transfers field ownership to kubectl's 
-client-side-apply mechanism, and Talos's inline-manifest controller will silently no-op on every future reconcile 
-(server-side-apply conflict, no error surfaced).
-
-**To update CoreDNS (e.g., add a hostname rewrite):**
-1. Edit `terraform/files/coredns/Corefile`
-2. Commit + push
-3. Run `cd terraform && terraform apply -var-file=terraform.tfvars.local`
-4. Run `make apply-cp` to push config to all control planes
-5. CoreDNS picks up changes via its `reload` plugin — no pod restart needed
-
-**If you accidentally edited the ConfigMap directly and broke Talos's ownership:**
-```bash
-kubectl delete configmap coredns -n kube-system
-# Wait ~30s for Talos's k8s.ManifestApplyController to recreate it
-kubectl get configmap coredns -n kube-system -w
-```
-
-Or as a stopgap, apply the correct content yourself:
-```bash
-kubectl apply --server-side -f <(terraform output coredns_config)
-```
-
 ## Quick Start — Deploying the Cluster
 
 ### 1. Bootstrap Talos Nodes

k8s/apps/comfyui/deployment.yaml

@@ -1,82 +0,0 @@
-# ComfyUI — GPU-accelerated image generation on worker-1.
-# Uses 1x V100 32GB (sm70). Freed by scaling ornith 2→1.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: comfyui
-  namespace: comfyui
-  labels:
-    app: comfyui
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app: comfyui
-  template:
-    metadata:
-      labels:
-        app: comfyui
-    spec:
-      nodeSelector:
-        kubernetes.io/hostname: worker-1
-      runtimeClassName: nvidia
-      # k8s Service named 'comfyui' injects COMFYUI_PORT=tcp://... into pod env,
-      # which clobbers ai-dock's own COMFYUI_PORT variable (expects a port number).
-      # Disable service link injection to avoid the collision.
-      enableServiceLinks: false
-      containers:
-        - name: comfyui
-          image: ghcr.io/ai-dock/comfyui:v2-cuda-12.1.1-base-22.04
-          ports:
-            - containerPort: 8188
-              protocol: TCP
-          env:
-            - name: NVIDIA_VISIBLE_DEVICES
-              value: "all"
-          resources:
-            requests:
-              cpu: "4"
-              memory: 8Gi
-              nvidia.com/gpu: "1"
-            limits:
-              cpu: "8"
-              memory: 16Gi
-              nvidia.com/gpu: "1"
-          volumeMounts:
-            - mountPath: /workspace/ComfyUI/models
-              name: models
-            - mountPath: /workspace/ComfyUI/output
-              name: output
-          readinessProbe:
-            httpGet:
-              path: /
-              port: 8188
-            periodSeconds: 10
-            initialDelaySeconds: 30
-          startupProbe:
-            httpGet:
-              path: /
-              port: 8188
-            failureThreshold: 120
-            periodSeconds: 10
-      volumes:
-        - name: models
-          persistentVolumeClaim:
-            claimName: comfyui-models
-        - name: output
-          emptyDir: {}
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: comfyui-models
-  namespace: comfyui
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: longhorn
-  resources:
-    requests:
-      storage: 50Gi

k8s/apps/comfyui/ingress.yaml

@@ -1,33 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: comfyui
-  namespace: comfyui
-  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-prod
-    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
-    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
-    nginx.ingress.kubernetes.io/proxy-body-size: "0"
-    # WebSocket support for ComfyUI's live preview
-    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
-    nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr"
-    nginx.ingress.kubernetes.io/configuration-snippet: |
-      proxy_set_header Upgrade $http_upgrade;
-      proxy_set_header Connection "upgrade";
-spec:
-  ingressClassName: nginx
-  tls:
-    - secretName: comfyui-tls
-      hosts:
-        - comfyui.riotpiao.com
-  rules:
-    - host: comfyui.riotpiao.com
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: comfyui
-                port:
-                  number: 80

k8s/apps/comfyui/kustomization.yaml

@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-
-resources:
-  - deployment.yaml
-  - service.yaml
-  - ingress.yaml

k8s/apps/comfyui/service.yaml

@@ -1,14 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: comfyui
-  namespace: comfyui
-  labels:
-    app: comfyui
-spec:
-  selector:
-    app: comfyui
-  ports:
-    - port: 80
-      targetPort: 8188
-      protocol: TCP

k8s/apps/gotify/README.md

@@ -1,107 +0,0 @@
-# Gotify — Push Notifications + Email Relay
-
-Self-hosted notification server with SMTP email forwarding sidecar.
-
-## Architecture
-
-```
-Forgejo webhook ──POST──→ Gotify API (:80/message)
-                              │
-                    ┌─────────┼─────────┐
-                    ▼                     ▼
-              Push notification      SMTP emailer sidecar
-              (mobile/desktop)       (polls → sends email)
-```
-
-## Setup (one-time, after first deploy)
-
-### 1. Encrypt secrets before committing
-
-```bash
-# Edit secrets.yaml with real values first, then:
-sops -e -i k8s/apps/gotify/secrets.yaml
-```
-
-### 2. Create Gotify app + client tokens
-
-1. Login to `https://gotify.riotpiao.com` with admin creds
-2. **Applications** → Create `forgejo` → copy **app token**
-3. **Clients** → Create `smtp-emailer` → copy **client token**
-4. Update `gotify-tokens` secret:
-   ```bash
-   kubectl -n notifications create secret generic gotify-tokens \
-     --from-literal=app-token=<APP_TOKEN> \
-     --from-literal=client-token=<CLIENT_TOKEN> \
-     --dry-run=client -o yaml | kubectl apply -f -
-   ```
-
-### 3. Configure Forgejo webhook
-
-In each Forgejo repo → **Settings** → **Webhooks** → **Add Webhook** → **Gotify**:
-
-| Field | Value |
-|-------|-------|
-| Target URL | `http://gotify.notifications.svc.cluster.local/message` |
-| Token | The **app token** from step 2 |
-| Events | Pull Request (Created, Merged, Closed) |
-
-Or via API:
-```bash
-FORGEJO_TOKEN="<your-pat>"
-APP_TOKEN="<gotify-app-token>"
-
-curl -s -X POST "https://forgejo.riotpiao.com/api/v1/repos/rock/homelab/hooks" \
-  -H "Authorization: token $FORGEJO_TOKEN" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "type": "gotify",
-    "active": true,
-    "config": {
-      "content_type": "json",
-      "url": "http://gotify.notifications.svc.cluster.local/message?token='"$APP_TOKEN"'"
-    },
-    "events": ["pull_request", "pull_request_assign", "pull_request_review"],
-    "authorization_header": ""
-  }'
-```
-
-### 4. Add CoreDNS rewrite (if accessing via public hostname)
-
-Only needed if Cloudflare Tunnel is used for gotify.riotpiao.com:
-
-```
-# terraform/files/coredns/Corefile — add rewrite:
-rewrite name gotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
-```
-
-Then: `cd terraform && terraform apply && cd .. && make apply-cp`
-
-### 5. SMTP providers
-
-| Provider | Host | Port | Notes |
-|----------|------|------|-------|
-| Gmail | smtp.gmail.com | 587 | Use App Password (2FA required) |
-| Resend | smtp.resend.com | 587 | Free 100 emails/day |
-| Sendgrid | smtp.sendgrid.net | 587 | Free 100 emails/day |
-| Mailgun | smtp.mailgun.org | 587 | Free 5000/month |
-
-## Notification priority levels
-
-| Priority | Meaning | Email forwarded? |
-|----------|---------|-----------------|
-| 0-4 | Low (info) | No (below MIN_PRIORITY=5) |
-| 5-7 | Normal (PR created) | Yes |
-| 8-10 | High (PR merged, failures) | Yes |
-
-## Verify
-
-```bash
-# Test push notification
-APP_TOKEN="<app-token>"
-curl -X POST "https://gotify.riotpiao.com/message?token=$APP_TOKEN" \
-  -H "Content-Type: application/json" \
-  -d '{"title":"Test","message":"Hello from homelab","priority":5}'
-
-# Check email sidecar logs
-kubectl -n notifications logs deployment/gotify -c smtp-emailer --tail=20
-```

k8s/apps/gotify/db.yaml

@@ -1,35 +0,0 @@
-# CNPG Postgres for Gotify. Lightweight — 2 instances, 2Gi storage.
-# CNPG generates secret `gotify-db-app` + service `gotify-db-rw` in ns notifications.
-apiVersion: postgresql.cnpg.io/v1
-kind: Cluster
-metadata:
-  name: gotify-db
-  namespace: notifications
-  annotations:
-    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
-spec:
-  instances: 3
-  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
-  bootstrap:
-    initdb:
-      database: gotify
-      owner: app
-      encoding: UTF8
-      localeCollate: C
-      localeCType: C
-  enableSuperuserAccess: false
-  resources:
-    requests: { memory: "256Mi", cpu: "100m" }
-    limits: { memory: "512Mi", cpu: "500m" }
-  storage:
-    size: 2Gi
-    storageClass: longhorn-cnpg
-  monitoring:
-    enablePodMonitor: true
-  affinity:
-    podAntiAffinityType: preferred
-    topologyKey: kubernetes.io/hostname
-    tolerations:
-      - key: node-role.kubernetes.io/control-plane
-        operator: Exists
-        effect: NoSchedule

k8s/apps/gotify/deployment.yaml

@@ -1,204 +0,0 @@
-# Gotify — self-hosted push notification server + SMTP email relay.
-# Forgejo webhooks → Gotify → push notifications + email forwarding.
-# Runs on control plane (no GPU needed), lightweight.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: gotify
-  namespace: notifications
-  labels:
-    app: gotify
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app: gotify
-  template:
-    metadata:
-      labels:
-        app: gotify
-    spec:
-      containers:
-        # --- Gotify server ---
-        - name: gotify
-          image: ghcr.io/gotify/server:2.6.1
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              export GOTIFY_DATABASE_DIALECT=postgres
-              export GOTIFY_DATABASE_CONNECTION="host=gotify-db-rw.notifications port=5432 user=${DB_USER} password=${DB_PASS} dbname=gotify sslmode=disable"
-              exec /app/gotify-app
-          ports:
-            - containerPort: 80
-              protocol: TCP
-          env:
-            - name: GOTIFY_DEFAULTUSER_NAME
-              valueFrom:
-                secretKeyRef:
-                  name: gotify-admin
-                  key: username
-            - name: GOTIFY_DEFAULTUSER_PASS
-              valueFrom:
-                secretKeyRef:
-                  name: gotify-admin
-                  key: password
-            - name: DB_USER
-              valueFrom:
-                secretKeyRef:
-                  name: gotify-db-app
-                  key: username
-            - name: DB_PASS
-              valueFrom:
-                secretKeyRef:
-                  name: gotify-db-app
-                  key: password
-            - name: GOTIFY_SERVER_PORT
-              value: "80"
-            - name: GOTIFY_SERVER_KEEPALIVEPERIODSECONDS
-              value: "0"
-            - name: TZ
-              value: Asia/Tokyo
-          resources:
-            requests:
-              cpu: 50m
-              memory: 64Mi
-            limits:
-              cpu: 200m
-              memory: 128Mi
-          livenessProbe:
-            httpGet:
-              path: /health
-              port: 80
-            periodSeconds: 30
-            initialDelaySeconds: 10
-          readinessProbe:
-            httpGet:
-              path: /health
-              port: 80
-            periodSeconds: 10
-            initialDelaySeconds: 5
-
-        # --- SMTP emailer sidecar ---
-        # Watches Gotify WebSocket stream, forwards messages as email.
-        # https://github.com/eternal-flame-AD/gotify-broadcast
-        - name: smtp-emailer
-          image: ghcr.io/gotify/server:2.6.1
-          command:
-            - /bin/sh
-            - -c
-            - |
-              # Wait for Gotify to be ready
-              until wget -qO- http://localhost:80/health >/dev/null 2>&1; do
-                echo "Waiting for Gotify..."
-                sleep 2
-              done
-              echo "Gotify is ready, starting email relay..."
-
-              # Poll Gotify messages and forward via SMTP using msmtp
-              # Install msmtp for lightweight SMTP sending
-              apk add --no-cache msmtp curl jq
-
-              # Configure msmtp
-              cat > /tmp/msmtprc <<MSMTP
-              defaults
-              auth           on
-              tls            on
-              tls_trust_file /etc/ssl/certs/ca-certificates.crt
-              logfile        /tmp/msmtp.log
-
-              account        default
-              host           ${SMTP_HOST}
-              port           ${SMTP_PORT}
-              from           ${SMTP_FROM}
-              user           ${SMTP_USER}
-              password       ${SMTP_PASS}
-              MSMTP
-
-              chmod 600 /tmp/msmtprc
-
-              # Track last seen message ID
-              LAST_ID=0
-
-              while true; do
-                # Fetch messages since last ID
-                MESSAGES=$(curl -s -H "X-Gotify-Key: ${GOTIFY_CLIENT_TOKEN}" \
-                  "http://localhost:80/message?since=${LAST_ID}&limit=10" 2>/dev/null)
-
-                if [ -n "$MESSAGES" ]; then
-                  echo "$MESSAGES" | jq -r '.messages[]? | @base64' | while read -r MSG; do
-                    DECODED=$(echo "$MSG" | base64 -d)
-                    ID=$(echo "$DECODED" | jq -r '.id')
-                    TITLE=$(echo "$DECODED" | jq -r '.title // "Notification"')
-                    BODY=$(echo "$DECODED" | jq -r '.message // ""')
-                    PRIORITY=$(echo "$DECODED" | jq -r '.priority // 5')
-                    APP=$(echo "$DECODED" | jq -r '.appid // 0')
-                    DATE=$(echo "$DECODED" | jq -r '.date // ""')
-
-                    # Only forward messages with priority >= configured threshold
-                    if [ "$PRIORITY" -ge "${MIN_PRIORITY:-0}" ]; then
-                      printf "Subject: [Gotify] %s\nFrom: %s\nTo: %s\nContent-Type: text/plain; charset=UTF-8\n\n%s\n\n---\nPriority: %s\nDate: %s" \
-                        "$TITLE" "$SMTP_FROM" "$NOTIFY_EMAIL" "$BODY" "$PRIORITY" "$DATE" | \
-                        msmtp -C /tmp/msmtprc "$NOTIFY_EMAIL" && \
-                        echo "Email sent for message $ID: $TITLE" || \
-                        echo "Failed to send email for message $ID"
-                    fi
-
-                    # Update last seen ID
-                    if [ "$ID" -gt "$LAST_ID" ]; then
-                      LAST_ID=$ID
-                    fi
-                  done
-                fi
-
-                sleep ${POLL_INTERVAL:-30}
-              done
-          env:
-            - name: GOTIFY_CLIENT_TOKEN
-              valueFrom:
-                secretKeyRef:
-                  name: gotify-tokens
-                  key: client-token
-            - name: SMTP_HOST
-              valueFrom:
-                secretKeyRef:
-                  name: gotify-smtp
-                  key: host
-            - name: SMTP_PORT
-              valueFrom:
-                secretKeyRef:
-                  name: gotify-smtp
-                  key: port
-            - name: SMTP_FROM
-              valueFrom:
-                secretKeyRef:
-                  name: gotify-smtp
-                  key: from
-            - name: SMTP_USER
-              valueFrom:
-                secretKeyRef:
-                  name: gotify-smtp
-                  key: user
-            - name: SMTP_PASS
-              valueFrom:
-                secretKeyRef:
-                  name: gotify-smtp
-                  key: password
-            - name: NOTIFY_EMAIL
-              valueFrom:
-                secretKeyRef:
-                  name: gotify-smtp
-                  key: notify-email
-            - name: MIN_PRIORITY
-              value: "5"
-            - name: POLL_INTERVAL
-              value: "15"
-          resources:
-            requests:
-              cpu: 10m
-              memory: 32Mi
-            limits:
-              cpu: 100m
-              memory: 64Mi
-      # No volumes — Postgres handles persistence

k8s/apps/gotify/ingress.yaml

@@ -1,26 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: gotify
-  namespace: notifications
-  annotations:
-    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
-    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
-    # WebSocket support for Gotify client connections
-    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
-    nginx.ingress.kubernetes.io/configuration-snippet: |
-      proxy_set_header Upgrade $http_upgrade;
-      proxy_set_header Connection "upgrade";
-spec:
-  ingressClassName: nginx
-  rules:
-    - host: gotify.riotpiao.com
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: gotify
-                port:
-                  number: 80

k8s/apps/gotify/kustomization.yaml

@@ -1,8 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - namespace.yaml
-  - db.yaml
-  - deployment.yaml
-  - service.yaml
-  - ingress.yaml

k8s/apps/gotify/namespace.yaml

@@ -1,6 +0,0 @@
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: notifications
-  labels:
-    kubernetes.io/metadata.name: notifications

k8s/apps/gotify/service.yaml

@@ -1,14 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: gotify
-  namespace: notifications
-  labels:
-    app: gotify
-spec:
-  selector:
-    app: gotify
-  ports:
-    - port: 80
-      targetPort: 80
-      protocol: TCP

k8s/apps/immich/db.yaml

@@ -18,7 +18,7 @@ metadata:
   annotations:
     argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
 spec:
-  instances: 3
+  instances: 2
   imageName: ghcr.io/cloudnative-pg/postgresql:18-minimal-trixie
   postgresql:
     extensions:

k8s/apps/immich/deployment.yaml

@@ -19,7 +19,6 @@ spec:
       labels:
         app: immich-server
     spec:
-      serviceAccountName: immich
       nodeSelector:
         kubernetes.io/hostname: talos-cp-3
       containers:
@@ -82,7 +81,6 @@ spec:
       labels:
         app: immich-machine-learning
     spec:
-      serviceAccountName: immich
       containers:
         - name: immich-machine-learning
           image: ghcr.io/immich-app/immich-machine-learning:release

k8s/apps/immich/rbac.yaml

@@ -3,11 +3,6 @@
 # provisioning-managed (immich-oidc). Same pattern as
 # k8s/apps/paperless/rbac.yaml. Inert until kube-apiserver's OIDC wiring
 # lands (--oidc-groups-claim=groups, --oidc-groups-prefix=oidc:).
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: immich
----
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
@@ -34,9 +29,6 @@ subjects:
   - kind: Group
     name: "oidc:immich-admins"
     apiGroup: rbac.authorization.k8s.io
-  - kind: ServiceAccount
-    name: immich
-    namespace: immich
 roleRef:
   kind: Role
   name: immich-operator

k8s/apps/llm-serving/embeddings.yaml

@@ -45,14 +45,6 @@ spec:
       volumeMounts:
       - mountPath: /mnt/models
         name: models
-    podMetadata:
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "8080"
-        prometheus.io/path: "/metrics"
-      labels:
-        app.kubernetes.io/name: llm-embeddings
-        app.kubernetes.io/part-of: llm-serving
     maxReplicas: 1
     minReplicas: 1
     nodeSelector:

k8s/apps/llm-serving/kustomization.yaml

@@ -14,7 +14,5 @@ resources:
   - ornith.yaml
   - reasoning.yaml
   - reranker.yaml
-  - qwen-cpu.yaml
-  - networkpolicy.yaml
 # No namespace transformer: every file sets its own, and the transformer would
 # rewrite metadata.namespace on anything cross-namespace added later.

k8s/apps/llm-serving/networkpolicy.yaml

@@ -1,76 +0,0 @@
-# NetworkPolicy for LLM inference engines (llm-serving namespace).
-#
-# These pods have NO auth — vLLM, Ollama, and TEI accept any request.
-# All access MUST go through the api-gateway, which validates JWTs and
-# injects identity headers (X-Forwarded-User, X-Auth-Verified).
-#
-# Replaces the hand-applied llm-serving-default-deny policy that used
-# `llm-client: "true"` pod label as a selector — any pod in any namespace
-# could self-grant access by adding that label, which defeats the purpose.
-#
-# This policy restricts ingress to:
-#   1. api namespace (gateway) — the sole entry point for inference
-#   2. monitoring namespace — Prometheus scraping vLLM/TEI /metrics
-#   3. intra-namespace — pod-to-pod (future: multi-replica comms)
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: llm-serving-ingress
-  namespace: llm-serving
-  labels:
-    app.kubernetes.io/part-of: llm-serving
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/part-of: llm-serving
-  policyTypes:
-    - Ingress
-  ingress:
-    # Allow from api-gateway (namespace: api)
-    # Gateway proxies /v1/chat/completions, /v1/embeddings, /v1/rerank
-    - from:
-        - namespaceSelector:
-            matchLabels:
-              kubernetes.io/metadata.name: api
-      ports:
-        - protocol: TCP
-          port: 8080   # vLLM, Ollama HTTP
-        - protocol: TCP
-          port: 80      # KServe predictor services
-        - protocol: TCP
-          port: 8000    # vLLM direct (some configs)
-        - protocol: TCP
-          port: 11434   # Ollama native port
-    # Allow Prometheus scraping from monitoring namespace
-    # vLLM: :8080/metrics, TEI: :9000/metrics
-    - from:
-        - namespaceSelector:
-            matchLabels:
-              kubernetes.io/metadata.name: monitoring
-      ports:
-        - protocol: TCP
-          port: 8080
-        - protocol: TCP
-          port: 9000
-    # Allow from paperless namespace (paperless-ai auto-tagging)
-    # Bypasses gateway until service-account JWT token exchange is implemented.
-    # paperless-ai-agent has llm:inference role in Authentik.
-    - from:
-        - namespaceSelector:
-            matchLabels:
-              kubernetes.io/metadata.name: paperless
-      ports:
-        - protocol: TCP
-          port: 8080
-        - protocol: TCP
-          port: 80
-        - protocol: TCP
-          port: 8000
-    # Allow intra-namespace (pod-to-pod within llm-serving)
-    - from:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/part-of: llm-serving
-      ports:
-        - protocol: TCP
-          port: 8080

k8s/apps/llm-serving/ornith.yaml

@@ -33,8 +33,12 @@ spec:
 
         ollama pull ornith:35b
 
+        ollama pull qwen2.5:3b-instruct
+
         ollama run ornith:35b "ok" >/dev/null 2>&1 || true
 
+        ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
+
         wait $SERVE_PID
 
         '
@@ -50,7 +54,7 @@ spec:
       - name: OLLAMA_NUM_PARALLEL
         value: '1'
       - name: OLLAMA_MAX_LOADED_MODELS
-        value: '1'
+        value: '2'
       image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
       name: kserve-container
       ports:
@@ -61,7 +65,8 @@ spec:
           command:
           - /bin/sh
           - -c
-          - ollama ps 2>/dev/null | grep -q ornith
+          - ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
+            grep -q qwen2.5
         periodSeconds: 10
       resources:
         limits:
@@ -77,26 +82,22 @@ spec:
           command:
           - /bin/sh
           - -c
-          - ollama ps 2>/dev/null | grep -q ornith
+          - ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
+            grep -q qwen2.5
         failureThreshold: 120
         periodSeconds: 15
       volumeMounts:
       - mountPath: /mnt/models
         name: models
-    podMetadata:
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "8080"
-        prometheus.io/path: "/metrics"
-      labels:
-        app.kubernetes.io/name: llm-ornith
-        app.kubernetes.io/part-of: llm-serving
     deploymentStrategy:
       type: Recreate
-    # 1 replica -- ornith:35b only. qwen2.5:3b moved to CPU on cp-2.
+    # 2 replicas -- each its own GPU, each loading both ornith:35b and
-    # Frees 1 GPU for ComfyUI.
+    # qwen2.5:3b-instruct -- so 2 concurrent implementer-style calls each
-    maxReplicas: 1
+    # get an independent instance instead of contending on one, at the
-    minReplicas: 1
+    # cost of judge/qwen traffic still sharing whichever replica an
+    # implementer call also lands on.
+    maxReplicas: 2
+    minReplicas: 2
     nodeSelector:
       kubernetes.io/hostname: worker-1
     runtimeClassName: nvidia

k8s/apps/llm-serving/qwen-cpu.yaml

@@ -1,115 +0,0 @@
-# qwen2.5:3b-instruct on CPU (talos-cp-2, 144GB RAM, 24 cores).
-# Moved off GPU to free a V100 for ComfyUI. Latency ~10x slower
-# than GPU but sufficient for lightweight tasks (summarization,
-# classification, quick answers).
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: qwen-cpu
-  namespace: llm-serving
-  labels:
-    app: qwen-cpu
-    app.kubernetes.io/name: qwen-cpu
-    app.kubernetes.io/part-of: llm-serving
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app: qwen-cpu
-  template:
-    metadata:
-      labels:
-        app: qwen-cpu
-        app.kubernetes.io/name: qwen-cpu
-        app.kubernetes.io/part-of: llm-serving
-    spec:
-      nodeSelector:
-        kubernetes.io/hostname: talos-cp-2
-      tolerations:
-        - key: node-role.kubernetes.io/control-plane
-          operator: Exists
-          effect: NoSchedule
-      containers:
-        - name: ollama
-          image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              ollama serve &
-              SERVE_PID=$!
-              until ollama list >/dev/null 2>&1; do sleep 2; done
-              ollama pull qwen2.5:3b-instruct
-              ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
-              wait $SERVE_PID
-          env:
-            - name: OLLAMA_HOST
-              value: "0.0.0.0:8080"
-            - name: OLLAMA_MODELS
-              value: /root/.ollama/models
-            - name: OLLAMA_CONTEXT_LENGTH
-              value: "32768"
-            - name: OLLAMA_KEEP_ALIVE
-              value: "-1"
-            - name: OLLAMA_MAX_LOADED_MODELS
-              value: "1"
-            - name: OLLAMA_NUM_PARALLEL
-              value: "2"
-          ports:
-            - containerPort: 8080
-              protocol: TCP
-          readinessProbe:
-            exec:
-              command: ["/bin/sh", "-c", "ollama ps 2>/dev/null | grep -q qwen2.5"]
-            periodSeconds: 10
-          startupProbe:
-            exec:
-              command: ["/bin/sh", "-c", "ollama ps 2>/dev/null | grep -q qwen2.5"]
-            failureThreshold: 60
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: "4"
-              memory: 4Gi
-            limits:
-              cpu: "8"
-              memory: 8Gi
-          volumeMounts:
-            - mountPath: /root/.ollama
-              name: ollama-data
-      volumes:
-        - name: ollama-data
-          persistentVolumeClaim:
-            claimName: qwen-cpu-data
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: qwen-cpu
-  namespace: llm-serving
-  labels:
-    app: qwen-cpu
-    app.kubernetes.io/part-of: llm-serving
-spec:
-  selector:
-    app: qwen-cpu
-  ports:
-    - port: 80
-      targetPort: 8080
-      protocol: TCP
----
-# Small PVC for qwen2.5:3b model weights (~1.9GB).
-# Separate from llm-models PVC which is pinned to worker-1.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: qwen-cpu-data
-  namespace: llm-serving
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: longhorn
-  resources:
-    requests:
-      storage: 5Gi

k8s/apps/llm-serving/reasoning.yaml

@@ -104,14 +104,6 @@ spec:
         name: models
       - mountPath: /dev/shm
         name: shm
-    podMetadata:
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "8080"
-        prometheus.io/path: "/metrics"
-      labels:
-        app.kubernetes.io/name: llm-reasoning
-        app.kubernetes.io/part-of: llm-serving
     deploymentStrategy:
       type: Recreate
     maxReplicas: 1

k8s/apps/llm-serving/reranker.yaml

@@ -45,14 +45,6 @@ spec:
       volumeMounts:
       - mountPath: /mnt/models
         name: models
-    podMetadata:
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "8080"
-        prometheus.io/path: "/metrics"
-      labels:
-        app.kubernetes.io/name: llm-reranker
-        app.kubernetes.io/part-of: llm-serving
     maxReplicas: 1
     minReplicas: 1
     nodeSelector:

k8s/apps/messaging/management-service/templates/deployment.yaml

@@ -13,7 +13,6 @@ spec:
       labels:
         app: management-service
     spec:
-      serviceAccountName: kmsvc
       topologySpreadConstraints:
         - maxSkew: 1
           topologyKey: kubernetes.io/hostname

k8s/apps/messaging/management-service/values.yaml

@@ -2,7 +2,7 @@ namespace: sqs
 replicaCount: 3
 
 image:
-  repository: forgejo.riotpiao.com/rock/kmsvc-manage
+  repository: ghcr.io/riotpiaole/kmsvc-management-service
   tag: latest
   pullPolicy: Always
 

k8s/apps/messaging/memory-queues/Chart.yaml

@@ -1,6 +0,0 @@
-apiVersion: v2
-name: memory-queues
-description: Kafka queues (DLQ) for Poimen Memory service (Phase 6.6)
-type: application
-version: 0.1.0
-appVersion: "1.0"

k8s/apps/messaging/memory-queues/templates/queues.yaml

@@ -1,20 +0,0 @@
-{{- range .Values.queues }}
----
-apiVersion: kmsvc.io/v1alpha1
-kind: Queue
-metadata:
-  name: {{ .name }}
-  namespace: {{ $.Values.namespace }}
-  labels:
-    app: memory-service
-    queue: dlq
-spec:
-  name: {{ .name }}
-  description: {{ .description }}
-  partitions: {{ .partitions }}
-  replicationFactor: {{ .replicationFactor }}
-  config:
-    retention.ms: "{{ .config.retention.ms }}"
-    message.retention.seconds: "{{ .config.message.retention.seconds }}"
-    visibility.timeout.seconds: "{{ .config.visibility.timeout.seconds }}"
-{{- end }}

k8s/apps/messaging/memory-queues/values.yaml

@@ -1,25 +0,0 @@
-# Poimen Memory Service Kafka Queues (kmsvc)
-# Phase 6.6: DLQ topics for webhook + metrics failures
-
-queues:
-  # DLQ for extraction, webhook, and agent failures
-  - name: poimen-memory-dlq
-    description: "DLQ for extraction, webhook, and agent failures"
-    partitions: 3
-    replicationFactor: 1
-    config:
-      retention.ms: "1209600000"  # 14 days
-      message.retention.seconds: "1209600"
-      visibility.timeout.seconds: "300"
-
-  # DLQ for metrics persistence failures  
-  - name: poimen-memory-metric-dlq
-    description: "DLQ for metrics persistence failures"
-    partitions: 3
-    replicationFactor: 1
-    config:
-      retention.ms: "1209600000"  # 14 days
-      message.retention.seconds: "1209600"
-      visibility.timeout.seconds: "300"
-
-namespace: sqs

k8s/apps/messaging/queue-crd/values.yaml

@@ -1,7 +1,7 @@
 namespace: sqs
 
 image:
-  repository: forgejo.riotpiao.com/rock/kmsvc-manage
+  repository: ghcr.io/riotpiaole/kmsvc-management-service
   tag: latest
   pullPolicy: Always
 

k8s/apps/paperless/backup-cronjob.yaml

@@ -48,7 +48,7 @@ spec:
                   mountPath: /backup
           containers:
             - name: mc-mirror
-              image: quay.io/minio/mc:latest
+              image: minio/mc:latest
               env:
                 - name: ACCESS_KEY
                   valueFrom:

k8s/apps/paperless/kustomization.yaml

@@ -11,7 +11,6 @@ resources:
   - backup-cronjob.yaml
   - adapter-configmap.yaml
   - rbac.yaml
-  - paperless-ai.yaml
 # postgres: paperless-db CNPG Cluster, deployed by k8s/infra/databases (wave 2,
 # before this app at wave 8) - not duplicated here. Same for the paperless-oidc
 # and paperless-minio-creds Secrets, written by PostSync provisioning Jobs in

k8s/apps/paperless/paperless-ai.yaml

@@ -1,114 +0,0 @@
-# Secret paperless-ai-config managed via SOPS (argocd/secrets/paperless-ai-secrets.enc.yaml)
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: paperless-ai
-  namespace: paperless
-  labels:
-    app.kubernetes.io/name: paperless-ai
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: paperless-ai
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: paperless-ai
-    spec:
-      tolerations:
-        - key: node-role.kubernetes.io/control-plane
-          operator: Exists
-          effect: NoSchedule
-      volumes:
-        - name: paperless-ai-data
-          persistentVolumeClaim:
-            claimName: paperless-ai-data
-      initContainers:
-        - name: fetch-llm-token
-          image: curlimages/curl:8.12.0
-          command:
-            - sh
-            - -c
-            - |
-              set -e
-              echo "[init] Fetching JWT token from Authentik for paperless-ai-agent..."
-              
-              # Get JWT token via client_credentials grant
-              # Client secret sourced from environment (injected from paperless-ai-config Secret)
-              TOKEN_RESPONSE=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
-                -d "grant_type=client_credentials" \
-                -d "client_id=paperless-ai-agent" \
-                -d "client_secret=${LLM_AUTH_CLIENT_SECRET}" \
-                -d "scope=openid llm:inference" 2>/dev/null)
-              
-              # Extract token
-              TOKEN=$(echo "$TOKEN_RESPONSE" | grep -o '"access_token":"[^"]*' | cut -d'"' -f4)
-              
-              if [ -z "$TOKEN" ]; then
-                echo "[error] Failed to get token. Response: $TOKEN_RESPONSE"
-                exit 1
-              fi
-              
-              # Store token in file for main container to read
-              mkdir -p /data
-              echo "$TOKEN" > /data/llm_token.txt
-              echo "[init] Token fetched and stored successfully"
-          env:
-            - name: LLM_AUTH_CLIENT_SECRET
-              valueFrom:
-                secretKeyRef:
-                  name: paperless-ai-config
-                  key: LLM_AUTH_CLIENT_SECRET
-          volumeMounts:
-            - name: paperless-ai-data
-              mountPath: /data
-      containers:
-        - name: paperless-ai
-          image: clusterzx/paperless-ai:latest
-          env:
-            # Paperless-ngx connection
-            - name: PAPERLESS_API_URL
-              value: "http://paperless.paperless.svc.cluster.local:8000"
-            - name: PAPERLESS_API_TOKEN
-              valueFrom:
-                secretKeyRef:
-                  name: paperless-ai-config
-                  key: PAPERLESS_API_TOKEN
-            - name: PAPERLESS_USERNAME
-              value: "admin"
-            # LLM API — via public gateway with JWT auth
-            - name: AI_PROVIDER
-              value: "custom"
-            - name: CUSTOM_BASE_URL
-              value: "https://api.riotpiao.com/v1"
-            # Token will be read from file at runtime by the application
-            # The init container fetches it and stores in /app/data/llm_token.txt
-            - name: CUSTOM_API_KEY_FILE
-              value: "/app/data/llm_token.txt"
-            - name: CUSTOM_MODEL
-              value: "qwen2.5:3b-instruct"
-            # Behavior - scan for new documents and tag them
-            - name: SCAN_INTERVAL
-              value: "60"
-            - name: PROCESS_PREDEFINED_DOCUMENTS
-              value: "no"
-            - name: ADD_AI_TAG
-              value: "yes"
-            - name: AI_TAG_NAME
-              value: "ai-processed"
-            - name: USE_PROMPT_TAGS
-              value: "yes"
-            - name: ADD_AI_DESCRIPTIONS
-              value: "yes"
-          volumeMounts:
-            - name: paperless-ai-data
-              mountPath: /app/data
-          resources:
-            requests:
-              cpu: 100m
-              memory: 512Mi
-            limits:
-              cpu: "1"
-              memory: 2Gi

k8s/apps/secret-rotation-controller/crd.yaml

@@ -1,144 +0,0 @@
-apiVersion: apiextensions.k8s.io/v1
-kind: CustomResourceDefinition
-metadata:
-  name: secretrotations.homelab.riotpiao.com
-spec:
-  group: homelab.riotpiao.com
-  names:
-    kind: SecretRotation
-    plural: secretrotations
-  scope: Namespaced
-  versions:
-  - name: v1
-    served: true
-    storage: true
-    schema:
-      openAPIV3Schema:
-        type: object
-        properties:
-          metadata:
-            type: object
-          spec:
-            type: object
-            required:
-            - provider
-            - rotationInterval
-            properties:
-              # External system: authentik | forgejo | minio | vault
-              provider:
-                type: string
-                enum: [authentik, forgejo, minio, vault]
-              
-              # How often to rotate (hours)
-              rotationInterval:
-                type: integer
-                minimum: 24
-              
-              # Application ID in external system
-              appId:
-                type: string
-              
-              # k8s Secret to update (name, namespace, key)
-              secretRef:
-                type: object
-                required: [name, namespace]
-                properties:
-                  name:
-                    type: string
-                  namespace:
-                    type: string
-                  key:
-                    type: string
-                    description: "Secret key to update (e.g., MINIO_IDENTITY_OPENID_CLIENT_SECRET)"
-              
-              # Path to git file that holds the secret (for .enc.yaml files)
-              gitPath:
-                type: string
-                description: "Path in homelab repo to .enc.yaml file"
-              
-              # Ansible template values to substitute
-              templateValues:
-                type: object
-                additionalProperties:
-                  type: string
-          
-          status:
-            type: object
-            properties:
-              lastRotationTime:
-                type: string
-                format: date-time
-              nextRotationTime:
-                type: string
-                format: date-time
-              lastRotationStatus:
-                type: string
-                enum: [Success, Failed, Pending]
-              lastRotationError:
-                type: string
-              lastCommitHash:
-                type: string
-
----
-# Example usage:
-apiVersion: homelab.riotpiao.com/v1
-kind: SecretRotation
-metadata:
-  name: minio-oidc
-  namespace: secret-rotation
-spec:
-  provider: authentik
-  rotationInterval: 2160  # 90 days in hours
-  appId: minio
-  secretRef:
-    name: minio-oidc
-    namespace: storage
-    key: MINIO_IDENTITY_OPENID_CLIENT_SECRET
-  gitPath: k8s/argocd/secrets/minio-oidc.enc.yaml
-
----
-apiVersion: homelab.riotpiao.com/v1
-kind: SecretRotation
-metadata:
-  name: portfolio-agent-oidc
-  namespace: secret-rotation
-spec:
-  provider: authentik
-  rotationInterval: 2160
-  appId: portfolio-agent
-  secretRef:
-    name: portfolio-agent-oidc
-    namespace: portfolio
-    key: CLIENT_SECRET
-  gitPath: k8s/argocd/secrets/portfolio-agent-oidc.enc.yaml
-
----
-apiVersion: homelab.riotpiao.com/v1
-kind: SecretRotation
-metadata:
-  name: forgejo-registry-token
-  namespace: secret-rotation
-spec:
-  provider: forgejo
-  rotationInterval: 2160
-  appId: rock/riotpiao.com
-  secretRef:
-    name: forgejo-registry-secret
-    namespace: kube-system
-    key: REGISTRY_TOKEN
-  gitPath: k8s/argocd/secrets/forgejo-registry-secret.enc.yaml
-
----
-apiVersion: homelab.riotpiao.com/v1
-kind: SecretRotation
-metadata:
-  name: minio-root-credentials
-  namespace: secret-rotation
-spec:
-  provider: minio
-  rotationInterval: 4320  # 180 days in hours
-  appId: root
-  secretRef:
-    name: minio-creds
-    namespace: storage
-  gitPath: k8s/argocd/secrets/minio-secrets.enc.yaml

k8s/apps/secret-rotation-controller/deployment.yaml

@@ -1,92 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: secret-rotation-controller
-  namespace: secret-rotation
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: secret-rotation-controller
-  template:
-    metadata:
-      labels:
-        app: secret-rotation-controller
-    spec:
-      serviceAccountName: secret-rotation-controller
-      containers:
-      - name: controller
-        image: secret-rotation-controller:latest
-        imagePullPolicy: IfNotPresent
-        env:
-        # SOPS reads age key from this file
-        - name: SOPS_AGE_KEY_FILE
-          value: /etc/sops/age/private-key.txt
-        
-        # Vault auth (token in projected volume)
-        - name: VAULT_ADDR
-          value: http://vault.vault.svc.cluster.local:8200
-        - name: VAULT_TOKEN_FILE
-          value: /var/run/secrets/vault/token
-        
-        # Authentik
-        - name: AUTHENTIK_URL
-          value: http://authentik-server.iam.svc.cluster.local
-        - name: AUTHENTIK_BOOTSTRAP_TOKEN
-          valueFrom:
-            secretKeyRef:
-              name: authentik-bootstrap
-              key: token
-        
-        # Git
-        - name: GIT_REPO
-          value: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-        - name: GIT_AUTHOR_EMAIL
-          value: [email protected]
-        - name: GIT_AUTHOR_NAME
-          value: Secret Rotation Controller
-        - name: FORGEJO_TOKEN
-          valueFrom:
-            secretKeyRef:
-              name: forgejo-registry-secret
-              key: REGISTRY_TOKEN
-        
-        volumeMounts:
-        # Age key from ExternalSecret (synced from Vault)
-        - name: age-key
-          mountPath: /etc/sops/age
-          readOnly: true
-        
-        # Vault auth token (projected)
-        - name: vault-token
-          mountPath: /var/run/secrets/vault
-          readOnly: true
-        
-        # Temp working dir
-        - name: tmp
-          mountPath: /tmp
-        
-        resources:
-          requests:
-            cpu: 100m
-            memory: 256Mi
-          limits:
-            cpu: 500m
-            memory: 512Mi
-      
-      volumes:
-      - name: age-key
-        secret:
-          secretName: sops-age-key
-          defaultMode: 0400
-      
-      - name: vault-token
-        projected:
-          sources:
-          - serviceAccountToken:
-              path: token
-              audience: vault
-              expirationSeconds: 3600
-      
-      - name: tmp
-        emptyDir: {}

k8s/apps/secret-rotation-controller/kustomization.yaml

@@ -1,15 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-
-namespace: secret-rotation
-
-resources:
-  - rbac.yaml
-  - crd.yaml
-  - external-secret.yaml
-  - deployment.yaml
-
-commonLabels:
-  app.kubernetes.io/name: secret-rotation-controller
-  app.kubernetes.io/component: automation
-  managed-by: argocd

k8s/apps/secret-rotation-controller/rbac.yaml

@@ -1,53 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: secret-rotation-controller
-  namespace: secret-rotation
-
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: secret-rotation-controller
-rules:
-# Read SecretRotation CRDs
-- apiGroups: ["homelab.riotpiao.com"]
-  resources: ["secretrotations"]
-  verbs: ["get", "list", "watch"]
-
-# Update status
-- apiGroups: ["homelab.riotpiao.com"]
-  resources: ["secretrotations/status"]
-  verbs: ["get", "patch", "update"]
-
-# Read k8s secrets that will be rotated
-- apiGroups: [""]
-  resources: ["secrets"]
-  verbs: ["get", "list"]
-
-# For recording events
-- apiGroups: [""]
-  resources: ["events"]
-  verbs: ["create", "patch"]
-
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: secret-rotation-controller
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: secret-rotation-controller
-subjects:
-- kind: ServiceAccount
-  name: secret-rotation-controller
-  namespace: secret-rotation
-
----
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: secret-rotation
-  labels:
-    kubernetes.io/metadata.name: secret-rotation

k8s/argocd/apps/-1-projects.yaml

@@ -18,7 +18,7 @@ spec:
       prune: true
       selfHeal: true
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/argocd/projects
   destination:

k8s/argocd/apps/00-secrets.yaml

@@ -17,7 +17,7 @@ spec:
     syncOptions:
       - CreateNamespace=true
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     # ksops decrypts every *.enc.yaml here at kustomize-build time (repo-server
     # runs `kustomize build --enable-alpha-plugins --enable-exec`). Replaces the

k8s/argocd/apps/00-substrate.yaml

@@ -21,7 +21,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/bootstrap/cert-manager/cert-manager-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -93,7 +93,7 @@ spec:
   project: homelab
   revisionHistoryLimit: 3
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     # A real kustomization.yaml (resources: the 3 issuer/CA files) renders these
     # deterministically. The previous directory.include with bare filenames
@@ -127,7 +127,7 @@ spec:
   project: homelab
   revisionHistoryLimit: 3
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/bootstrap/ingress
   destination:
@@ -138,75 +138,3 @@ spec:
       selfHeal: true
     syncOptions:
       - CreateNamespace=true
----
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: cluster-maintenance
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: "0"
-spec:
-  project: homelab
-  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    targetRevision: main
-    path: k8s/infra/cluster-maintenance
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: kube-system
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
----
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: kyverno
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: "0"
-spec:
-  project: homelab
-  source:
-    repoURL: https://kyverno.github.io/kyverno/
-    chart: kyverno
-    targetRevision: "1.14.0"
-    helm:
-      valueFiles:
-        - $values/k8s/bootstrap/kyverno/kyverno-values.yaml
-  sources:
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-      targetRevision: main
-      ref: values
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: kyverno
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true
----
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: kyverno-policies
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: "0"
-spec:
-  project: homelab
-  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    targetRevision: main
-    path: k8s/bootstrap/kyverno
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: kyverno
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true

k8s/argocd/apps/01-image-updater.yaml

@@ -1,33 +0,0 @@
-# ArgoCD Image Updater - auto-updates Application images from registry
-# Watches forgejo.riotpiao.com for new image tags and updates Applications
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: argocd-image-updater
-  namespace: argocd
-  finalizers:
-    - resources-finalizer.argocd.argoproj.io
-  annotations:
-    argocd.argoproj.io/sync-wave: "1"
-spec:
-  project: homelab
-  revisionHistoryLimit: 3
-  sources:
-    - repoURL: https://argoproj.github.io/argo-helm
-      chart: argocd-image-updater
-      targetRevision: "0.11.2"
-      helm:
-        valueFiles:
-          - $values/k8s/infra/argocd-image-updater/values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-      targetRevision: main
-      ref: values
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: argocd
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=false

k8s/argocd/apps/06-ci-cd.yaml

@@ -1,55 +0,0 @@
-# Tekton Pipelines v0.68.0
-#
-# Install method: vendored release.yaml in k8s/infra/tekton/
-# downloaded from https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.68.0/release.yaml
-#
-# To upgrade:
-#   1. Download new release.yaml from https://github.com/tektoncd/pipeline/releases
-#   2. Replace k8s/infra/tekton/release.yaml
-#   3. Commit and push — ArgoCD syncs automatically
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: tekton-pipelines
-  namespace: argocd
-  labels:
-    app.kubernetes.io/name: tekton-pipelines
-    app.kubernetes.io/part-of: homelab-infra
-    wave: "06"
-spec:
-  project: homelab
-
-  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    targetRevision: main
-    path: k8s/infra/tekton
-
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: tekton-pipelines
-
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-    - CreateNamespace=true
-    - ServerSideApply=true
-    retry:
-      limit: 5
-      backoff:
-        duration: 5s
-        factor: 2
-        maxDuration: 3m
-
-  ignoreDifferences:
-  - group: admissionregistration.k8s.io
-    kind: ValidatingWebhookConfiguration
-    jsonPointers:
-    - /webhooks/0/clientConfig/caBundle
-    - /webhooks
-  - group: admissionregistration.k8s.io
-    kind: MutatingWebhookConfiguration
-    jsonPointers:
-    - /webhooks/0/clientConfig/caBundle
-    - /webhooks

k8s/argocd/apps/08-secret-rotation.yaml

@@ -1,32 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: secret-rotation
-  namespace: argocd
-  labels:
-    app.kubernetes.io/name: secret-rotation
-spec:
-  project: homelab
-  
-  sources:
-  - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    path: k8s/apps/secret-rotation-controller
-    targetRevision: main
-  
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: secret-rotation
-  
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-    - CreateNamespace=true
-    - RespectIgnoreDifferences=true
-    retry:
-      limit: 5
-      backoff:
-        duration: 5s
-        factor: 2
-        maxDuration: 3m

k8s/argocd/apps/10-storage-observability.yaml

@@ -17,7 +17,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/minio/minio-operator-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -41,7 +41,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/infra/minio
   destination:
@@ -66,7 +66,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/infra/longhorn
   destination:
@@ -102,7 +102,7 @@ spec:
         skipCrds: true
         valueFiles:
           - $values/k8s/infra/monitoring/prometheus-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -152,7 +152,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/infra/monitoring/crds
   destination:
@@ -183,7 +183,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/infra/monitoring
   destination:
@@ -213,7 +213,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/monitoring/blackbox-exporter-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -223,29 +223,3 @@ spec:
     automated:
       prune: true
       selfHeal: true
----
-# Distributed tracing: Tempo + OpenTelemetry Collector.
-# Receives traces from instrumented services, stores in local volume (72h retention).
-# Grafana datasource auto-configured, service graph + latency dashboards included.
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: tracing
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: "1"
-spec:
-  project: homelab
-  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    targetRevision: main
-    path: k8s/infra/tracing
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: tracing
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true

k8s/argocd/apps/20-logging.yaml

@@ -19,7 +19,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/logging/loki-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -53,7 +53,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/logging/grafana-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -87,7 +87,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/logging/promtail-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
   destination:

k8s/argocd/apps/30-security.yaml

@@ -17,7 +17,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/iam/vault-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -46,7 +46,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/infra/iam/authentik-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -68,7 +68,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/infra/iam
   destination:
@@ -109,7 +109,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -152,17 +152,10 @@ metadata:
   namespace: argocd
   annotations:
     argocd.argoproj.io/sync-wave: "3"
-    argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-golang
-    argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
-    argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
-    argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
-    argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
-    argocd-image-updater.argoproj.io/write-back-method: git
-    argocd-image-updater.argoproj.io/git-branch: main
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/infra/forgejo-runner
   destination:
@@ -180,17 +173,10 @@ metadata:
   namespace: argocd
   annotations:
     argocd.argoproj.io/sync-wave: "3"
-    argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-node
-    argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
-    argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
-    argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
-    argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
-    argocd-image-updater.argoproj.io/write-back-method: git
-    argocd-image-updater.argoproj.io/git-branch: main
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/infra/forgejo-runner
     helm:
@@ -211,17 +197,10 @@ metadata:
   namespace: argocd
   annotations:
     argocd.argoproj.io/sync-wave: "3"
-    argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-rust
-    argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
-    argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
-    argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
-    argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
-    argocd-image-updater.argoproj.io/write-back-method: git
-    argocd-image-updater.argoproj.io/git-branch: main
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/infra/forgejo-runner
     helm:

k8s/argocd/apps/40-data.yaml

@@ -14,7 +14,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/infra/databases
   destination:

k8s/argocd/apps/50-memory-queues.yaml

@@ -1,20 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: memory-queues
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: "7"
-spec:
-  project: homelab
-  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    targetRevision: main
-    path: k8s/apps/messaging/memory-queues
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: sqs
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true

k8s/argocd/apps/50-messaging.yaml

@@ -1,6 +1,6 @@
-# Wave 5 — Kafka (Strimzi operator + cluster CR), Redis infrastructure.
+# Wave 5 — Kafka (Strimzi operator + cluster CR), Redis, and the SQS-like
-# Strimzi/Redis are public Helm charts; kafka-cluster is a local chart.
+# queue services. Strimzi/Redis are public Helm charts; kafka-cluster/queue-crd/
-# queue-crd and management-service are managed by kmsvc-root (kmsvc-manage.git).
+# management-service are local charts (rendered from their own Chart.yaml).
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
@@ -68,7 +68,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/apps/messaging/kafka-cluster
   destination:
@@ -78,5 +78,45 @@ spec:
     automated:
       prune: true
       selfHeal: true
-# queue-crd and management-service moved to kmsvc-manage.git repo
+---
-# Managed by kmsvc-root Application
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: queue-crd
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "6"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    targetRevision: main
+    path: k8s/apps/messaging/queue-crd
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: sqs
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: management-service
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "7"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    targetRevision: main
+    path: k8s/apps/messaging/management-service
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: sqs
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true

k8s/argocd/apps/55-api-gateway.yaml

@@ -36,19 +36,14 @@ metadata:
     app.kubernetes.io/component: gateway
   annotations:
     argocd.argoproj.io/sync-wave: "7"
-    # ArgoCD Image Updater - auto-update on new image push
-    argocd-image-updater.argoproj.io/image-list: gw=forgejo.riotpiao.com/rock/api-gateway
-    argocd-image-updater.argoproj.io/gw.update-strategy: digest
-    argocd-image-updater.argoproj.io/gw.allow-tags: regexp:^latest$
-    argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
   project: homelab
   revisionHistoryLimit: 3
   sources:
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab-frontend.git
       targetRevision: main
       path: k8s
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       path: k8s/apps/api
   destination:

k8s/argocd/apps/56-llm-serving.yaml

@@ -20,7 +20,7 @@ spec:
   project: homelab
   revisionHistoryLimit: 3
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/apps/llm-serving
   destination:

k8s/argocd/apps/58-comfyui.yaml

@@ -1,32 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: comfyui
-  namespace: argocd
-  labels:
-    app.kubernetes.io/name: comfyui
-    app.kubernetes.io/component: image-generation
-  annotations:
-    argocd.argoproj.io/sync-wave: "8"
-spec:
-  project: homelab
-  revisionHistoryLimit: 3
-  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    targetRevision: main
-    path: k8s/apps/comfyui
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: comfyui
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true
-    retry:
-      limit: 5
-      backoff:
-        duration: 5s
-        factor: 2
-        maxDuration: 3m

k8s/argocd/apps/59-gotify.yaml

@@ -1,32 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: gotify
-  namespace: argocd
-  labels:
-    app.kubernetes.io/name: gotify
-    app.kubernetes.io/component: notifications
-  annotations:
-    argocd.argoproj.io/sync-wave: "8"
-spec:
-  project: homelab
-  revisionHistoryLimit: 3
-  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    targetRevision: main
-    path: k8s/apps/gotify
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: notifications
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true
-    retry:
-      limit: 5
-      backoff:
-        duration: 5s
-        factor: 2
-        maxDuration: 3m

k8s/argocd/apps/60-applications.yaml

@@ -19,10 +19,10 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/apps/temporal/temporal-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       path: k8s/apps/temporal
   destination:
@@ -51,7 +51,7 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/apps/portainer/portainer-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
   destination:
@@ -74,7 +74,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/apps/cloudflared
   destination:
@@ -97,7 +97,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/apps/agent-pod
   destination:
@@ -130,7 +130,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/apps/sms
   destination:
@@ -157,7 +157,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/apps/paperless
   destination:
@@ -189,7 +189,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/apps/immich
   destination:
@@ -220,10 +220,10 @@ spec:
       helm:
         valueFiles:
           - $values/k8s/apps/homarr/homarr-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       ref: values
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
       targetRevision: main
       path: k8s/apps/homarr  # PostSync hook: fix-probes-job.yaml
   destination:
@@ -236,37 +236,6 @@ spec:
     syncOptions:
       - CreateNamespace=true
 ---
-# Portfolio site at riotpiao.com - static Next.js site from rock/riotpiao.com repo.
-# Points directly to infra/portfolio/base (bypassing repo's own argocd-apps.yaml
-# which has wrong URLs). Image built by Forgejo Actions on rock/portfolio repo.
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: portfolio
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: "8"
-    # ArgoCD Image Updater - auto-update on new image push
-    argocd-image-updater.argoproj.io/image-list: app=forgejo.riotpiao.com/rock/portfolio
-    argocd-image-updater.argoproj.io/app.update-strategy: digest
-    argocd-image-updater.argoproj.io/app.allow-tags: regexp:^latest$
-    argocd-image-updater.argoproj.io/write-back-method: argocd
-spec:
-  project: homelab
-  source:
-    repoURL: https://forgejo.riotpiao.com/rock/riotpiao.com.git
-    targetRevision: main
-    path: infra/portfolio/base
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: portfolio
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true
----
 # Wave 9 - per-service scoped RBAC (Role/RoleBinding), deliberately last so
 # every target namespace above already exists. Inert until kube-apiserver
 # gets --oidc-groups-claim=groups wired up (separate, not-yet-applied
@@ -281,13 +250,12 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/infra/rbac
   destination:
     server: https://kubernetes.default.svc
-    # No namespace: cluster-scoped resources (ClusterRoleBinding, etc.)
+    namespace: default
-    # Namespace is set per-resource in kustomization
   syncPolicy:
     automated:
       prune: true

k8s/argocd/apps/60-kmsvc-manage.yaml

@@ -12,7 +12,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/kmsvc-manage.git
+    repoURL: https://forgejo.riotpiao.com/rock/kmsvc-manage.git
     targetRevision: main
     path: k8s/argocd/apps
     directory:

k8s/argocd/apps/70-poimen.yaml

@@ -1,28 +1,27 @@
+# Poimen project collection — manages poimen-memory, poimen-workflows, and poiman
+# Each repo tracks its own main branch (no prod branch). Poiman is the primary
+# orchestrator with k8s/argocd/ containing the AppProject and deployment structure.
+#
+# CI: All three repos trigger on main branch pushes (no image builds yet).
+# Future: Add build workflows for poiman once container runtime needs are clear.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
-  name: poimen
+  name: poimen-root
   namespace: argocd
+  labels:
+    app.kubernetes.io/name: poimen
+    app.kubernetes.io/component: orchestrator
   annotations:
     argocd.argoproj.io/sync-wave: "7"
-    # Image Updater: auto-update on new image push (SHA tag filter)
-    argocd-image-updater.argoproj.io/image-list: |
-      memory=forgejo.riotpiao.com/rock/poimen-memory
-      workflows=forgejo.riotpiao.com/rock/poimen-workflows
-      frontend=forgejo.riotpiao.com/rock/poimen-frontend
-    argocd-image-updater.argoproj.io/memory.update-strategy: digest
-    argocd-image-updater.argoproj.io/memory.allow-tags: regexp:^latest$
-    argocd-image-updater.argoproj.io/workflows.update-strategy: digest
-    argocd-image-updater.argoproj.io/workflows.allow-tags: regexp:^latest$
-    argocd-image-updater.argoproj.io/frontend.update-strategy: digest
-    argocd-image-updater.argoproj.io/frontend.allow-tags: regexp:^latest$
-    argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-workflows.git
+    repoURL: https://forgejo.riotpiao.com/rock/poimen.git
     targetRevision: main
-    path: k8s
+    path: k8s/argocd
+    directory:
+      recurse: false
   destination:
     server: https://kubernetes.default.svc
     namespace: poimen

k8s/argocd/projects/homelab-project.yaml

@@ -16,15 +16,9 @@ spec:
     - https://github.com/Riotpiaole/Poimen-memory.git
     - https://github.com/Riotpiaole/Poimen-workflows.git
     - https://github.com/Riotpiaole/poimen*.git
-    # In-cluster Forgejo repos — explicit allowlist (no wildcard)
+    # In-cluster Forgejo wildcard — all rock/* repos can be onboarded without
-    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    # touching this AppProject. Enabled by Stage 1 (A1). Includes poimen-* repos.
-    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
+    - https://forgejo.riotpiao.com/rock/*
-    - https://forgejo.riotpiao.com/riotpiao-poimen/kmsvc-manage.git
-    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen.git
-    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen-memory.git
-    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen-workflows.git
-    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen-frontend.git
-    - https://forgejo.riotpiao.com/rock/riotpiao.com.git
     # Public Helm chart repos referenced by k8s/argocd/apps/* and bootstrap/*
     - https://cloudnative-pg.github.io/charts
     - https://dl.gitea.com/charts/
@@ -43,8 +37,6 @@ spec:
     - https://charts.jetstack.io
     - https://kubernetes.github.io/ingress-nginx
     - https://stakater.github.io/stakater-charts
-    # ArgoCD ecosystem charts
-    - https://argoproj.github.io/argo-helm
   destinations:
     - server: https://kubernetes.default.svc
       namespace: "*"

k8s/argocd/root/homelab-root.yaml

@@ -12,7 +12,7 @@ metadata:
 spec:
   project: homelab
   source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
     targetRevision: main
     path: k8s/argocd/apps
     directory:
@@ -26,4 +26,3 @@ spec:
       selfHeal: true
     syncOptions:
       - CreateNamespace=true
-      - ServerSideApply=true

k8s/argocd/secrets/forgejo-smtp-secrets.enc.yaml

@@ -1,24 +0,0 @@
-apiVersion: ENC[AES256_GCM,data:uS8=,iv:EEoo9U+C244eAJMSTOQVkf5AE6BeHrc5DWPjtWnPRdk=,tag:H+YmBSGvcON3wh0p22LXDQ==,type:str]
-kind: ENC[AES256_GCM,data:j1/PFkTS,iv:ja4q8X+nzE/ZczwcY+Qe2DnnsxG64W1fkRPQvOaoqvA=,tag:WiilMGagudEKUlc2JdbGyg==,type:str]
-metadata:
-    name: ENC[AES256_GCM,data:J9iAIHboMyHu6xn/,iv:p3z3uzlkM7VDzOT3WDCelCdZ2t+QqSPmFtqYfvXPQMs=,tag:rKRtpUexVkuZKOJ34a0Xjw==,type:str]
-    namespace: ENC[AES256_GCM,data:su0FvA==,iv:6SPvwxZ/4aoL0Z07zdPC9r6L7HOHuKv3RodXpVcii04=,tag:njwkj+yvSgN8sliJP8T/Kw==,type:str]
-type: ENC[AES256_GCM,data:Qd6WJN1c,iv:M3fc78LvemcEbWzesuYeQ/GZyIOl7Eg/0EmUe3p0qAk=,tag:Tnzwewn0vwdowxy/EyuPdA==,type:str]
-stringData:
-    user: ENC[AES256_GCM,data:/Z7gPrsTUZOLzfoZ8cZGD10wrZE=,iv:0ydBSeq+yHB2b1J4W7FwIv55Eh+N3qfQ6Q7PwoUAvYo=,tag:CO19F8nElaYZyFiFR6N/Tg==,type:str]
-    password: ENC[AES256_GCM,data:AOl4StpeQIHSLX+oEFnkfg==,iv:mYsZ+5sum6YqyUPndtPCniTraxldKc803ULlKs8Gsaw=,tag:hK3w51ifZ/omAL7XDf8seg==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaMlhKVEM2bFdFOFAwV1lp
-            WUd6SEN4ZTF1TEpGYXhOYmJLK0tpcWZyc2w4ClVzZmI2Qk1KUnV1OXpyTEV2WWFa
-            VlJ2eHRSaEhqUnA2dUJVbWJUcEgxcFkKLS0tIG9IaFVnenNpSFRzdStiWjJvakVL
-            aDk2bTZGR3Zya3ROUS9vd1hEQVRFaG8KbeXA6IebHEaB79N6u795336aHesHOgzO
-            uZvvBUzSBy3t3jfFk8bJP4aH79I33Ha2eK5rsvdsiv/orwCMXUINKg==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-09-12T21:00:46Z"
-    mac: ENC[AES256_GCM,data:DjzPtR+Ueihh166Bvd3jCLtZFQdrvrxOoF87cv6lxKGWPEptT5vbw/EfuIFJBb6lvEJFDWKRC/r5ASdGwComYsPn0DZs4BPCzeKBtLfP9K2OGCXnAC7eGqt4mzMrrW0CQd1QSGcuXw8CY7uJGkMPGPKe3/0mQWiis2OSpHxTM18=,iv:/QqSEFU9RuX9z5Z6aSaD7KlP/cgGTOYvTH+VJOnDTYM=,tag:yFbFXn+iWqQZx7wW4dKppg==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.2

k8s/argocd/secrets/gotify-admin-secrets.enc.yaml

@@ -1,24 +0,0 @@
-apiVersion: ENC[AES256_GCM,data:wR4=,iv:cRBzbvu0eUYCYeKeysua1/P3Meli/rQyj/mIV6VWnPM=,tag:oyTPA/mLYNUX+QDkYLlZyQ==,type:str]
-kind: ENC[AES256_GCM,data:bdKQdadW,iv:F3DQiBI9xhSx87jkS1Hyevo48uUCBjsJSjbScIBznKA=,tag:PFakzzBj5S9F65dxu0L2rg==,type:str]
-metadata:
-    name: ENC[AES256_GCM,data:XHFYrKClPo+IwRbM,iv:ZGuL+cN840Y1bOTC62NhDDcoZpTzDWQ4GfqPJX/QWmI=,tag:hlCVjzvfcxXGOASc3vda/Q==,type:str]
-    namespace: ENC[AES256_GCM,data:0BbhgZBog+1qY/iRJA==,iv:88tiSpEDqIokT5VP/d6bB2+aUyh1kZ7NEHwZWoJW3XU=,tag:CBR01jh2U9h7kNEcK1e1sA==,type:str]
-type: ENC[AES256_GCM,data:Mpv1V73w,iv:BW3r6RpLnwe3XDKwrZySyOjrWUnSwIG5JOoPLzP/5gM=,tag:oyJySL0haOei1m4i+1AJ1g==,type:str]
-stringData:
-    username: ENC[AES256_GCM,data:8xmxLGE=,iv:J/vEvXGoD+ka6FDgnSwDz0fs9IxuJIZW9r7Oyu2qxC8=,tag:dN9jd6NSavMN8+uzvemYWg==,type:str]
-    password: ENC[AES256_GCM,data:vB9PaaUiQZ8FY8pO0cq1fHLi7Gq5t4U=,iv:C0Y1m2SGYP3oTIFoau5JavVmLblj6te/SPMLodIwiZw=,tag:3Ip4YvR4WPzR0bBpTyjytA==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRT3JoTnhVVW5SUUJXbTdz
-            dUpLcmtBWWhyaGk3Y1hBM1ArcVI5eHREbmp3CiswUGVmd0NhejZwQ2UvNEdxS3ow
-            L1RweS9Kb2paeStLZ0tLSFdWbVZqQW8KLS0tIHJHT3hiMmxtM0Q1Ym5KOVpLVFpl
-            enR3NmdNdmdwVitTQVJlRHFWcjR0N2sKQw9ZZs+Ji/Zq/feO3qy4DwaCfWgDOQ/z
-            FVVhcCXweN58tb+9fzCJ+pNi/hSmvUkCMbb1+60qBvEehNzOoMRJ5g==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-09-10T13:31:00Z"
-    mac: ENC[AES256_GCM,data:3HWV/NG0yTbsxY28u41zTRmAPc1kokGb4nCLmAsyq8/uvxcP+evDNyZXYpS93eVdPRfRZ1OqVBzyVGJEnj7JSXQVmlzor9JcWEL2fcpogoLVfJZKTRD6hk6optnBMjj84iQEEOx3A80JrDOdoXsH0pd9bJBABwoUOJwuufbXcIU=,iv:KLZsmAcapQgV08pFhGIvPt5ylsWg3xazAAjPuJYOaXA=,tag:1vKA3fISMIurHzxZ04F3lg==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.2

k8s/argocd/secrets/gotify-smtp-secrets.enc.yaml

@@ -1,28 +0,0 @@
-apiVersion: ENC[AES256_GCM,data:FOg=,iv:15mfPeWXV5LQEYahaYvNf1z2bHbxk4j5i8DXT6mdG/0=,tag:1f9/jnnu+wGeeiXGNFgKzA==,type:str]
-kind: ENC[AES256_GCM,data:dOFfNQiM,iv:HVLosbRpcCgu4iiYKV8MDLiQ0uhj8DXBfVpRBW2NFNo=,tag:mCbRIzD1OHRhBLc+7xcVug==,type:str]
-metadata:
-    name: ENC[AES256_GCM,data:kHoKhGyiIQvCfng=,iv:dAhxjeLlXK3yueMKfrHxmh9YmNA7AYKFBquLikCNzcE=,tag:3Xoaw7YNBCU/GINlaQOpBw==,type:str]
-    namespace: ENC[AES256_GCM,data:fTJMZhbqSQWD3/cnWg==,iv:D7zWa91+Fgerfyrywf8VA5YNzGrThhLz7CvYXucfiq0=,tag:RcIegCl2UR1JsbfIQm928w==,type:str]
-type: ENC[AES256_GCM,data:Qh51bTsM,iv:2Htv0Xze7Hd1m6zkP6rtFXAlg8qm0AKylSjwJxRjpBk=,tag:NCu5iVUTNVfYgErZSvwHIg==,type:str]
-stringData:
-    host: ENC[AES256_GCM,data:aZXSvQ8B7h78q1kHmh0=,iv:uUyL9X3ehIHqztGAtXtAQWgduvbSsSwZBZlTFMdOlBo=,tag:OM/vdO19VKDSa4W5WQAKNQ==,type:str]
-    port: ENC[AES256_GCM,data:5ZBB,iv:1/XAfq+PJKdBsufx+EPvvB/cm9nbEwYigc8eACXjR8g=,tag:WlNyz7gTPh0KMe5oKVd0BA==,type:str]
-    from: ENC[AES256_GCM,data:2/akr8cXgmgQGnqJaFcLJMDcWw==,iv:0rT/6aqyXxn1jIJ5umYCDZR4S8Pbh4vUgaXNrxd3JF8=,tag:k0pQrPOrqWTyE1Xu9oSzVA==,type:str]
-    user: ENC[AES256_GCM,data:d9BLaFYOYm++HZMzlf1gVauKkUQ=,iv:Wd48T5UPVn6z3bS0t02X9GbrnWal3QoIQv2EgM9z9Wg=,tag:cmg2KnLY4Atwco+j2wVdeg==,type:str]
-    password: ENC[AES256_GCM,data:pmqEj9623A6bThKrkCCjuA==,iv:M/QwJ0s//UvuOjOljl67EDhvZt/9Wp8JicvCcows51A=,tag:bfRyIdj1cgIZxU3WPbKteg==,type:str]
-    notify-email: ENC[AES256_GCM,data:+Xo22g8U2wDKsnPnFq/+GKxgYOI=,iv:PE+C1etlp6046ragiv1iMDQbhfo5IULd/5akD0+Sc6A=,tag:cCZmXByA85fuZL9yozVLjw==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpVVdxQTZFTHcwMXFFZEFk
-            ckNsYk4rdWtkeXFMVnlyVmdaRDBuRWsvZkQ4CitSSTBXOTZhWVpUZFFlR0JITVV4
-            Uy9lSlNHNjFNaFhHNTN0WnRaRlNNVGcKLS0tIHRlMnZpQVlScGRYYm5nT3Z5TWd6
-            ak1UZ1RobkpQZHBXR3MyenBDcU53Mk0KvH9O6bgwrjay0+1/A6TGX8GhITiDjWoO
-            RNX4fDtqNwhzmCfXbVjK30vlBjOFe+Bb7Z2n+hWMmHHDgFdS0xIAzA==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-09-12T20:57:18Z"
-    mac: ENC[AES256_GCM,data:V/wgjnpUBvaV39BCTlecCwQy2/1h+0vixEXleJc/I9y+AOvuwVZNH7M86hdjoAdsKiIoNYySj4Flz+MJ9C8jQoAxBTDPbolP9UwDFWQ5KMJTKPPDoJGLNjy7bUq51WoyePUM6WWAYIiWHIB3OvAxHaSzECzL+ZoAhQy92cPKa1o=,iv:vLtIoD/C5yeXPEr+2Lim6XnWzAj42vr9qQgsxKpuhA8=,tag:PBq8qPHrx4RRgAKCWrxG3Q==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.2

k8s/argocd/secrets/gotify-tokens-secrets.enc.yaml

@@ -1,27 +0,0 @@
-apiVersion: ENC[AES256_GCM,data:YE0=,iv:s3yNqcBn7/DKUQNgbGGwVmlM1HzxYGLBKyB6Y2ii2WE=,tag:77KxAjOFU3G0fSfrgudKkg==,type:str]
-kind: ENC[AES256_GCM,data:PFW4VV9T,iv:n8gzMjE5C2TAfUTpp/8ex64B+hWUGG1K+2oQ4deN03Q=,tag:18Xqb6Di5izMHmzR5EU+QA==,type:str]
-metadata:
-    name: ENC[AES256_GCM,data:Xhg1fQrD4itrbvDW0g==,iv:/THWSXAThb9fj+mm3wcxqzSdzdt7nE06+xOpkCxyImE=,tag:UChokr06VkbDZcFZDcUY2A==,type:str]
-    namespace: ENC[AES256_GCM,data:r25U5MuuzDd8JJ2YjQ==,iv:uDUcS4ZTpZe8HmZMArzkdu7LV5GUoLSP2wIs5HB1gv0=,tag:fci7j30hoxnVKJwPFNfd1Q==,type:str]
-type: ENC[AES256_GCM,data:agEG8Fu8,iv:ckYX0buX8md1CWHXfxbAXQJLzoTxTJI16nlQ9LSzYi0=,tag:4tZWf3REbGOmmBiT14+dew==,type:str]
-stringData:
-    #ENC[AES256_GCM,data:ehjcsmz1R0i/n31f8M0IIUT4KDBwzz6f5ds=,iv:j5swFAKpC67ZvGioiCCC1D651LxUl9gME8GqK5Uc+ys=,tag:BUJ8k9LHs8NAPPZAwPuifA==,type:comment]
-    #ENC[AES256_GCM,data:ndAB00yun636VHDMonqTghO/jCWodNd/hmdbm1QmCvOAi4FvTLl8bY3wYR+ZtlkSQYvFNqH798MJixv5OZwxBj5H,iv:pS+7B60jaS5jhqDRw2LbBFv7mD1HO6+hjjv+iNpenXc=,tag:NU6+gxCHTGxqeDMfvkwJWw==,type:comment]
-    #ENC[AES256_GCM,data:T3mhSm/5q7JTSXNLrjhpP5GhqA7nXz8uAhJcEV1RDctAX0Dyfq8Qn4bNFO51ibwTETx4SnunPuFZVs++AvRnsA==,iv:oRGM8N4iEcwBrMzoEXQAeKqCKzUq+jXTMVq9W2l6oh4=,tag:GrrbC/tkWpA5kp5UYCyRBA==,type:comment]
-    app-token: ENC[AES256_GCM,data:jw+AqbsxyoYRvNrUDrq+GNq/TNfweFCs,iv:67vSSgMZCMV0GG37ZxUxHAWZqOOGMYCmo3J43WgLyNk=,tag:xlrnip4XomXBbMmooW9FKg==,type:str]
-    client-token: ENC[AES256_GCM,data:gsxeCqKh4e+DFjpn9jM5GfemAdBf8dSv,iv:l2bBMdxQUil8jU9XDjXGn3EtvFVrK5ibXDCeGaPbYTY=,tag:ELf5S6cdNQJ84Es5upu1IQ==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAza3YrRW1VTVNSOGlMK21a
-            bDhzQlEyZDNnZTZrSVNnTUEzU1hSdnM1ZFVzCjdBSUlmWG5EcnlzbSs5bXdDaGkx
-            ME1nMnRqQzF1Vkc3b3FXSUVHT1g0ZDgKLS0tIHpMUytEMjdLQ2E0Unp2di9KS3JQ
-            eHBraDk1clJyanhLY2dGM0tESmJIUFkKHTl3y9uQiEofOFD8j2vH3YK/CVzlq11w
-            GfShIji1yCvvowKGzYYhsQK0UM0FzhzBv0GFMYWQCBq8pGdoPVmO5g==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-09-10T13:31:00Z"
-    mac: ENC[AES256_GCM,data:JhnO0V6cd2QVY/VhwHAJ5J75GeVlOVHBfVBTZstkj/IvpkAcwS/JE2b6jXiCwEC4n/vdA8DJ074noysUBRxh4Y7O4NKuvWcTniQKgqULNL1Hzgj3NdUkcQlWT+Z0HQ7FlvFH97VVXVfasU+CLHG48ShT2fDSj8JusEllb9CwSvg=,iv:PZo2krxvJc1TLJbvx+S9ClthoBe4w7WigUkq7dg0gNk=,tag:2IF5g/WTRP4XdnCZzDbiIA==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.2

k8s/argocd/secrets/kustomization.yaml

@@ -1,10 +1,8 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
-
+# All homelab SOPS-encrypted Secrets, decrypted in-line via the ksops generator.
-# Disable hash suffix for all generated secrets (stable names)
+# Each *.enc.yaml carries its own metadata.namespace, so no namespace transformer
-generatorOptions:
+# here (that would rewrite every Secret into one namespace). Renders exactly the
-  disableNameSuffixHash: true
+# Secret objects — replaces the old argocd-cmp-cm SOPS plugin.
-
-# SOPS-encrypted secrets via ksops generator
 generators:
   - secret-generator.yaml

k8s/argocd/secrets/paperless-ai-secrets.enc.yaml

@@ -1,23 +0,0 @@
-apiVersion: ENC[AES256_GCM,data:l7I=,iv:NZY7r3JVW3zVwxeiScvWKpAQUDa9+nckHd0qWVrGU88=,tag:qpowp5OILdYKtTux/nlWpg==,type:str]
-kind: ENC[AES256_GCM,data:6oeEbuIk,iv:5flI9TtcYQ961wOYPBPhvQpitHFYAf8yMdNBXqytbJs=,tag:WNhlbKmSeZEqZ9ZgiB/BYg==,type:str]
-metadata:
-    name: ENC[AES256_GCM,data:fvJMsgy+wPZqMCdxm9hoV3n/+Q==,iv:I3rVtHIJBOME+bxhPws58Zjhj5i+KT5UtB9o7Vus5EU=,tag:6h4FnUu7PGxlQPIQxPYCRg==,type:str]
-    namespace: ENC[AES256_GCM,data:CDriLoLovROW,iv:rHXcN1xm5+t2D/Tq/2sx9lQFF8anD5jH24ZntPuBA8U=,tag:XstJAz6S8IM1c/pp9Cg0Ww==,type:str]
-type: ENC[AES256_GCM,data:JdTwBbag,iv:9Ys15Ketl0ghNK0u0N8IOpUt7+KoloutiG5M9zHPXxw=,tag:teau/udf41Ty34A5wLAm6Q==,type:str]
-stringData:
-    PAPERLESS_API_TOKEN: ENC[AES256_GCM,data:TuQeDx8po3h4loTRABlItVYQJ7gFjnmIn3zQQGtUcoNFMKpkqLK/GQ==,iv:TDg0stpca5pDtatqu8DFU7R0Bm/S/BI9ZoiG4K8mCT4=,tag:BfjX25V5gL7AeIsscClRAg==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKVDN3aFVqVmFXY1VQVXZP
-            OUsrNHdNdlRvRFgvTDQrNE5uL2xaazVENTBjCkNPbGR5Vk16RXNDOW15OGNTRmFR
-            U0JOeTllaWU1dzNVY3lBbEVyVG5tOEkKLS0tIEZvajlvcUtJdFNNUkkzV3FKOFRj
-            UUE2TDBzT0xVc2E1NlUvQXAyZytMZEUKBv+ChaoQCstA742L3Bq5mBJlW/UC4Pyw
-            ZvFAyYbs1NaEqhjtHq+4T62jTWcH/St/vKgUuFQ9LCUQhYd8DUzAow==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-09-12T21:04:26Z"
-    mac: ENC[AES256_GCM,data:0ks96xQzOa8ygNDDYfKV8EJ8dWLBaC0PCnLG73NinMByF/KoWkCdwMDPC34W9xxVoTD2NiF1i/3pgCG4QFezTE7tPHPPKdYcQfwda9fkooiXW4Nh2M/sgbq/xgsy9N3qpHD/O5iILgpehb9y0DuErOyxcn2AIYizynU7m8e63pc=,iv:fvPWBsfE6YHskKzdlxJidp14dUgRR0XdMkEu6IxMMSo=,tag:5FiuIrFOg/GXvlQVy7drJQ==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.2

k8s/argocd/secrets/portfolio-secrets.enc.env

@@ -1,2 +0,0 @@
-FORGEJO_TOKEN=273fdcffabbcbb5a191e8289c73d106063acefc6
-LLM_API_TOKEN=s3VksXyw2z3sGbegnwjMDFnJ6CtNRd1a5CcnE5A4ET77toCcykNdunk6Oa2J

k8s/argocd/secrets/portfolio-secrets.enc.yaml

@@ -1,23 +0,0 @@
-apiVersion: ENC[AES256_GCM,data:bnY=,iv:Fuc3aqncHQ+L16o7eLarPbOECD3o8Mk5c2r9pQBpy70=,tag:JPfEnbNb3wZXPdXafnJDqw==,type:str]
-kind: ENC[AES256_GCM,data:WFlmi4Yg,iv:Zq/KQbgNcBVoo8ZsQ2H79ygyc8Dtkgxh4fCpEExfwSg=,tag:cWHP6Y5V+nZP2tFMJrOB8A==,type:str]
-metadata:
-    name: ENC[AES256_GCM,data:iCXbhwvg3Zq6YL/4j0wAy7Y=,iv:8s+d/8lDVEL7bGdIF+GOtAxapKnmx8JTjLSO04hXF5I=,tag:LjzX40DjK+uRZPCXsMlmwQ==,type:str]
-    namespace: ENC[AES256_GCM,data:HRMdZdCbxORQ,iv:MvaIWoKWjJRA7/fce0KtXRkFH/7cn0OuIg2QwHEdQzM=,tag:NqztiGyfU3BaopWBKhx2eg==,type:str]
-type: ENC[AES256_GCM,data:myBW86Za,iv:3x9ys5UzVhAuX8gvZO67B1e+Orw4Aqasv/lHBgUV0b4=,tag:yl18P6SaxVLJidwmRtd4aQ==,type:str]
-stringData:
-    FORGEJO_TOKEN: ENC[AES256_GCM,data:SUoBpNKOItyNGY01EhKNlPH0fyN4N7g6bfU2jsGogpCMhP7NuRipoA==,iv:h77RtmYZjXxiHYw1pHynQHuVX1+yJDGHwsXLf+DbUYA=,tag:GHzoDNHP1GGqlN5eT/N7TQ==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzTVBsekl3TGgzQVRMUU9m
-            cTduS2NoZW5uZFNNMG13cFY2cGVsTnlXaXhrCjJjbzhLdHZ4ZWpUV3J0cDQ0eVlM
-            WDNxdzVoQ2ZzcGJSbTU3RVorcnczNVkKLS0tIFdtQTE4Umk2TDBzUmdKOXNkbjFi
-            Vk5vK2VuUHVsb3FQL21vcGU1UW5CT1kKFM8vVjji3Cg9dvfTr4Hx7BJC8JH5ovef
-            Dj6zkofhsNWPgP9T+mnQakj+C0RKmHOMJqfWP7vwCBkZoNosIJVlMw==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-09-01T05:32:42Z"
-    mac: ENC[AES256_GCM,data:it24T9y9ixXo2aiL37k93vKFR+SRjjuI9DQdv0sWYtTogWnc7+uXBY4Zip/ouWyCse1muKKAGuek5c0XVrvSw4an9VkaXFczeunaZb6MOyVbVOkmJr+5xZFpZGjYcSkrhaWcVheedZ3iIFU5UWI7BBn/qQCf+HJ483cJqwtrV34=,iv:WprlWJdsMBNjqaA0O3ekfXMUpX5gC6OLYortQXYdTS4=,tag:rGqSSRTYTv2VR6AcRokO0A==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.2

k8s/argocd/secrets/secret-generator.yaml

@@ -26,9 +26,3 @@ files:
   - paperless-secrets.enc.yaml
   - vault-secrets.enc.yaml
   - vault-unseal-keys.enc.yaml
-  - portfolio-secrets.enc.yaml
-  - gotify-admin-secrets.enc.yaml
-  - gotify-tokens-secrets.enc.yaml
-  - gotify-smtp-secrets.enc.yaml
-  - forgejo-smtp-secrets.enc.yaml
-  - paperless-ai-secrets.enc.yaml

k8s/bootstrap/ingress/ingress.yaml

@@ -323,4 +323,3 @@ spec:
                 name: homarr
                 port:
                   number: 7575
-

k8s/bootstrap/kyverno/kustomization.yaml

@@ -1,6 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-namespace: kyverno
-
-resources:
-  - policies.yaml

k8s/bootstrap/kyverno/kyverno-values.yaml

@@ -1,42 +0,0 @@
-# Kyverno: Policy engine for Kubernetes image scanning, Pod security, and admission control
-# Scan all images, enforce baseline Pod Security Standard, prevent privilege escalation
-
-replicaCount: 1
-
-image:
-  registry: ghcr.io
-  repository: kyverno/kyverno
-  tag: "v1.14.0"
-
-config:
-  # Webhook timeout for policy evaluation. Increase if scanning takes longer.
-  webhookTimeoutSeconds: 30
-  # Failure policy: fail-open (audit/log) vs fail-closed (reject on error)
-  failurePolicy: fail
-  # Resource limits for webhook
-  webhookAnnotations:
-    rules: "allow"
-
-# Pod security via Kyverno instead of Pod Security Policies (deprecated)
-# Enforces baseline restrictions cluster-wide, with exceptions for privileged namespaces
-podSecurityContext:
-  runAsNonRoot: true
-  runAsUser: 1000
-
-rbac:
-  create: true
-
-resources:
-  requests:
-    memory: "256Mi"
-    cpu: "100m"
-  limits:
-    memory: "512Mi"
-    cpu: "500m"
-
-# Webhook configuration
-webhook:
-  timeoutSeconds: 30
-  # Failure policy: "Fail" (reject on error) or "Ignore" (audit-only)
-  # Set to "Ignore" for initial testing, then change to "Fail"
-  failurePolicy: ignore

k8s/bootstrap/kyverno/policies.yaml

@@ -1,204 +0,0 @@
-# Kyverno ClusterPolicies: Image scanning, Pod security, and admission control
----
-# Policy 1: Require non-root containers
-apiVersion: kyverno.io/v1
-kind: ClusterPolicy
-metadata:
-  name: require-non-root
-  namespace: kyverno
-spec:
-  validationFailureAction: audit  # audit first, then change to enforce
-  rules:
-    - name: check-runAsNonRoot
-      match:
-        any:
-          - resources:
-              kinds:
-                - Pod
-              selector:
-                matchLabels:
-                  pod-security.kubernetes.io/enforce: "!privileged"
-      validate:
-        message: "Container must not run as root"
-        pattern:
-          spec:
-            containers:
-              - securityContext:
-                  runAsNonRoot: true
----
-# Policy 2: Drop all Linux capabilities, add only required ones
-apiVersion: kyverno.io/v1
-kind: ClusterPolicy
-metadata:
-  name: require-dropped-caps
-  namespace: kyverno
-spec:
-  validationFailureAction: audit
-  rules:
-    - name: drop-all-capabilities
-      match:
-        any:
-          - resources:
-              kinds:
-                - Pod
-              selector:
-                matchLabels:
-                  pod-security.kubernetes.io/enforce: "!privileged"
-      validate:
-        message: "All Linux capabilities must be dropped"
-        pattern:
-          spec:
-            containers:
-              - securityContext:
-                  capabilities:
-                    drop:
-                      - ALL
----
-# Policy 3: Require image tags (no 'latest')
-apiVersion: kyverno.io/v1
-kind: ClusterPolicy
-metadata:
-  name: disallow-latest-tag
-  namespace: kyverno
-spec:
-  validationFailureAction: audit  # Change to enforce after testing
-  rules:
-    - name: disallow-latest
-      match:
-        any:
-          - resources:
-              kinds:
-                - Pod
-                - Deployment
-                - StatefulSet
-                - DaemonSet
-                - Job
-      validate:
-        message: "Image tag 'latest' is not allowed. Use explicit version tags."
-        pattern:
-          spec:
-            =(template):
-              spec:
-                containers:
-                  - image: "!*:latest"
-                =(initContainers):
-                  - image: "!*:latest"
----
-# Policy 4: Restrict images to trusted registries
-apiVersion: kyverno.io/v1
-kind: ClusterPolicy
-metadata:
-  name: restrict-registries
-  namespace: kyverno
-spec:
-  validationFailureAction: audit
-  rules:
-    - name: trusted-registries
-      match:
-        any:
-          - resources:
-              kinds:
-                - Pod
-                - Deployment
-                - StatefulSet
-                - DaemonSet
-                - Job
-              selector:
-                matchLabels:
-                  pod-security.kubernetes.io/enforce: "!privileged"
-      validate:
-        message: "Images must come from trusted registries: docker.io, ghcr.io, quay.io, k8s.gcr.io, registry.k8s.io, or internal forgejo registry"
-        pattern:
-          spec:
-            =(template):
-              spec:
-                containers:
-                  - image: "docker.io/* | ghcr.io/* | quay.io/* | k8s.gcr.io/* | registry.k8s.io/* | forgejo.riotpiao.com/* | *"
----
-# Policy 5: Require read-only root filesystem (audit only, exceptions for apps that need writes)
-apiVersion: kyverno.io/v1
-kind: ClusterPolicy
-metadata:
-  name: require-readonly-filesystem
-  namespace: kyverno
-spec:
-  validationFailureAction: audit
-  rules:
-    - name: check-readOnlyRootFilesystem
-      match:
-        any:
-          - resources:
-              kinds:
-                - Pod
-              selector:
-                matchLabels:
-                  pod-security.kubernetes.io/enforce: "!privileged"
-      validate:
-        message: "Root filesystem should be read-only for defense-in-depth"
-        pattern:
-          spec:
-            containers:
-              - securityContext:
-                  readOnlyRootFilesystem: true
----
-# Policy 6: Require resource requests and limits (prevent resource starvation)
-apiVersion: kyverno.io/v1
-kind: ClusterPolicy
-metadata:
-  name: require-resource-limits
-  namespace: kyverno
-spec:
-  validationFailureAction: audit
-  rules:
-    - name: check-resources
-      match:
-        any:
-          - resources:
-              kinds:
-                - Pod
-                - Deployment
-                - StatefulSet
-                - DaemonSet
-              excludeResources:
-                namespaceSelector:
-                  matchLabels:
-                    kubernetes.io/metadata.name: "kyverno|kube-system|kube-node-lease"
-      validate:
-        message: "CPU and memory requests and limits are required"
-        pattern:
-          spec:
-            =(template):
-              spec:
-                containers:
-                  - resources:
-                      requests:
-                        memory: "?*"
-                        cpu: "?*"
-                      limits:
-                        memory: "?*"
-                        cpu: "?*"
----
-# Policy 7: Require securityContext on all containers
-apiVersion: kyverno.io/v1
-kind: ClusterPolicy
-metadata:
-  name: require-security-context
-  namespace: kyverno
-spec:
-  validationFailureAction: audit
-  rules:
-    - name: check-securityContext
-      match:
-        any:
-          - resources:
-              kinds:
-                - Pod
-              selector:
-                matchLabels:
-                  pod-security.kubernetes.io/enforce: "!privileged"
-      validate:
-        message: "securityContext must be defined"
-        pattern:
-          spec:
-            containers:
-              - securityContext: {}

k8s/bootstrap/phase1-storage/longhorn-values.yaml

@@ -12,7 +12,6 @@ defaultSettings:
   replicaSoftAntiAffinity: false  # REQUIRED for true HA
   replicaAutoBalance: best-effort
   storageMinimalAvailablePercentage: 10
-  storageOverProvisioningPercentage: 200  # Actual usage is ~10% of scheduled; 200% unblocks all 3-replica scheduling on cp-1
   
   # Performance tuning
   defaultDataPath: /var/lib/longhorn

k8s/bootstrap/phase3-forgejo/forgejo-values.yaml

@@ -80,14 +80,6 @@ gitea:
     actions:
       ENABLED: true
 
-    mailer:
-      ENABLED: true
-      PROTOCOL: smtp+starttls
-      SMTP_ADDR: smtp.gmail.com
-      SMTP_PORT: 587
-      FROM: "Forgejo <[email protected]>"
-      # USER and PASSWD injected via env vars below (GITEA__MAILER__USER, GITEA__MAILER__PASSWD)
-
 # Persistence (shared storage for repos)
 persistence:
   enabled: true
@@ -156,13 +148,3 @@ deployment:
         secretKeyRef:
           name: forgejo-db-app
           key: password
-    - name: GITEA__MAILER__USER
-      valueFrom:
-        secretKeyRef:
-          name: forgejo-smtp
-          key: user
-    - name: GITEA__MAILER__PASSWD
-      valueFrom:
-        secretKeyRef:
-          name: forgejo-smtp
-          key: password

k8s/infra/argocd-image-updater/values.yaml

@@ -1,70 +0,0 @@
-# ArgoCD Image Updater configuration
-# Watches Forgejo registry and updates ArgoCD Applications with new image tags
-
-config:
-  # Registry configuration - Forgejo allows anonymous pulls
-  registries:
-    - name: forgejo
-      api_url: https://forgejo.riotpiao.com
-      prefix: forgejo.riotpiao.com
-      default: true
-      insecure: false
-
-  # Log level
-  logLevel: debug
-
-  # ArgoCD API server
-  argocd:
-    grpcWeb: true
-    serverAddress: argocd-server.argocd.svc.cluster.local
-    insecure: true
-    plaintext: true
-
-  # Git write-back configuration (for multi-source Applications)
-  git:
-    # Commit author for image updates
-    user:
-      name: "ArgoCD Image Updater"
-      email: "[email protected]"
-    # Use SSH keys from ArgoCD's known hosts + credentials
-    # Image Updater inherits ArgoCD's git credentials (mounted via ArgoCD secret)
-
-# Mount ArgoCD's git credentials for write-back
-extraVolumes:
-  - name: argocd-ssh-known-hosts-cm
-    configMap:
-      name: argocd-ssh-known-hosts-cm
-      defaultMode: 0644
-  - name: argocd-gpg-keys-cm
-    configMap:
-      name: argocd-gpg-keys-cm
-      optional: true
-      defaultMode: 0644
-  - name: argocd-gpg-pubring
-    configMap:
-      name: argocd-gpg-pubring-cm
-      optional: true
-      defaultMode: 0644
-
-extraVolumeMounts:
-  - name: argocd-ssh-known-hosts-cm
-    mountPath: /etc/ssh/ssh_known_hosts.d/argocd-ssh-known-hosts
-    subPath: ssh_known_hosts
-  - name: argocd-gpg-keys-cm
-    mountPath: /etc/gpg/source
-  - name: argocd-gpg-pubring
-    mountPath: /etc/gpg/pubring
-
-# Extra environment variables
-extraEnv:
-  - name: GIT_SSH_KNOWN_HOSTS_CONFIG_MAP_ENABLED
-    value: "true"
-
-# Resources
-resources:
-  requests:
-    cpu: 50m
-    memory: 64Mi
-  limits:
-    cpu: 200m
-    memory: 128Mi

k8s/infra/cluster-maintenance/stale-job-cleanup.yaml

@@ -1,139 +0,0 @@
-# Cluster-wide cleanup of stale failed/completed Jobs and Pods.
-# Runs every 2 minutes. Deletes:
-#   - Failed Jobs older than 2m (any namespace)
-#   - Completed standalone Jobs older than 2m with no owning CronJob
-#   - Orphan pods in Error/Failed/Evicted/Completed state older than 2m
-#
-# CronJob-owned Jobs are managed by failedJobsHistoryLimit/successfulJobsHistoryLimit,
-# but standalone Jobs (helm hooks, one-off runs, CI TaskRuns) have no TTL
-# and linger forever. Aggressive schedule keeps the cluster clean.
-
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: stale-job-cleanup
-  namespace: kube-system
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: stale-job-cleanup
-rules:
-  - apiGroups: ["batch"]
-    resources: ["jobs"]
-    verbs: ["get", "list", "delete"]
-  - apiGroups: [""]
-    resources: ["pods"]
-    verbs: ["get", "list", "delete"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: stale-job-cleanup
-subjects:
-  - kind: ServiceAccount
-    name: stale-job-cleanup
-    namespace: kube-system
-roleRef:
-  kind: ClusterRole
-  name: stale-job-cleanup
-  apiGroup: rbac.authorization.k8s.io
----
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: stale-job-cleanup
-  namespace: kube-system
-  labels:
-    app: stale-job-cleanup
-spec:
-  schedule: "*/2 * * * *"
-  concurrencyPolicy: Forbid
-  successfulJobsHistoryLimit: 3
-  failedJobsHistoryLimit: 3
-  jobTemplate:
-    spec:
-      ttlSecondsAfterFinished: 86400  # self-cleanup after 24h
-      backoffLimit: 1
-      activeDeadlineSeconds: 300
-      template:
-        spec:
-          serviceAccountName: stale-job-cleanup
-          restartPolicy: Never
-          tolerations:
-            - key: node-role.kubernetes.io/control-plane
-              operator: Exists
-              effect: NoSchedule
-          containers:
-            - name: cleanup
-              image: alpine/k8s:1.31.0
-              command:
-                - sh
-                - -c
-                - |
-                  set -e
-                  NOW=$(date +%s)
-
-                  TTL=120  # 2 minutes in seconds
-
-                  echo "=== Cleaning failed Jobs older than 2m ==="
-                  kubectl get jobs --all-namespaces -o json | \
-                    jq -r '.items[] |
-                      select(.status.conditions[]?.type == "Failed") |
-                      select(.status.completionTime or .status.startTime) |
-                      "\(.metadata.namespace) \(.metadata.name) \(.status.startTime // .status.completionTime // .metadata.creationTimestamp)"' | \
-                  while read -r NS NAME TS; do
-                    JOB_EPOCH=$(date -d "$TS" +%s 2>/dev/null || echo 0)
-                    AGE=$(( NOW - JOB_EPOCH ))
-                    if [ "$AGE" -ge "$TTL" ]; then
-                      echo "[delete] $NS/$NAME (failed ${AGE}s ago)"
-                      kubectl delete job "$NAME" -n "$NS" --cascade=foreground 2>/dev/null || true
-                    fi
-                  done
-
-                  echo ""
-                  echo "=== Cleaning completed standalone Jobs older than 2m ==="
-                  kubectl get jobs --all-namespaces -o json | \
-                    jq -r '.items[] |
-                      select(.status.succeeded >= 1) |
-                      select((.metadata.ownerReferences // []) | length == 0) |
-                      "\(.metadata.namespace) \(.metadata.name) \(.status.completionTime // .metadata.creationTimestamp)"' | \
-                  while read -r NS NAME TS; do
-                    JOB_EPOCH=$(date -d "$TS" +%s 2>/dev/null || echo 0)
-                    AGE=$(( NOW - JOB_EPOCH ))
-                    if [ "$AGE" -ge "$TTL" ]; then
-                      echo "[delete] $NS/$NAME (completed ${AGE}s ago, no owner)"
-                      kubectl delete job "$NAME" -n "$NS" --cascade=foreground 2>/dev/null || true
-                    fi
-                  done
-
-                  echo ""
-                  echo "=== Cleaning orphan Error/Failed/Evicted/Completed pods older than 2m ==="
-                  kubectl get pods --all-namespaces -o json | \
-                    jq -r '.items[] |
-                      select(
-                        .status.phase == "Failed" or
-                        .status.phase == "Succeeded" or
-                        (.status.reason // "") == "Evicted" or
-                        (.status.containerStatuses // [] | any(.state.terminated.reason == "Error"))
-                      ) |
-                      select((.metadata.ownerReferences // []) | all(.kind != "Job")) |
-                      "\(.metadata.namespace) \(.metadata.name) \(.metadata.creationTimestamp)"' | \
-                  while read -r NS NAME TS; do
-                    POD_EPOCH=$(date -d "$TS" +%s 2>/dev/null || echo 0)
-                    AGE=$(( NOW - POD_EPOCH ))
-                    if [ "$AGE" -ge "$TTL" ]; then
-                      echo "[delete] $NS/$NAME (stale ${AGE}s ago)"
-                      kubectl delete pod "$NAME" -n "$NS" --force 2>/dev/null || true
-                    fi
-                  done
-
-                  echo ""
-                  echo "Cleanup complete"
-              resources:
-                requests:
-                  cpu: 50m
-                  memory: 64Mi
-                limits:
-                  cpu: 250m
-                  memory: 128Mi

k8s/infra/databases/memory-db.yaml

@@ -9,7 +9,7 @@ metadata:
   annotations:
     argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
 spec:
-  instances: 3
+  instances: 2
   imageName: ghcr.io/cloudnative-pg/postgresql:16.2
   bootstrap:
     initdb:

k8s/infra/databases/paperless-db.yaml

@@ -11,7 +11,7 @@ metadata:
   annotations:
     argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
 spec:
-  instances: 3
+  instances: 2
   imageName: ghcr.io/cloudnative-pg/postgresql:16.2
   bootstrap:
     initdb:

k8s/infra/forgejo-runner/templates/cilium-apiserver-egress.yaml

@@ -1,25 +0,0 @@
-# CiliumNetworkPolicy for kube-apiserver access.
-#
-# Standard K8s NetworkPolicy ipBlock rules don't work for the API server
-# under Cilium — the except clause on 192.168.1.0/24 blocks the post-DNAT
-# destination even when a separate rule re-allows a /32 or subnet.
-#
-# Cilium's native `kube-apiserver` entity resolves this correctly: it
-# tracks the API server endpoints regardless of ClusterIP vs node-IP
-# routing, so the policy stays valid across node changes and NAT paths.
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: {{ .Release.Name }}-apiserver
-  namespace: {{ .Release.Namespace }}
-spec:
-  endpointSelector:
-    matchLabels:
-      app: {{ .Release.Name }}
-  egress:
-  - toEntities:
-    - kube-apiserver
-    toPorts:
-    - ports:
-      - port: "6443"
-        protocol: TCP

k8s/infra/forgejo-runner/templates/configmap.yaml

@@ -36,4 +36,3 @@ data:
       valid_volumes:
         - /docker-certs/client
       network: host
-      docker_host: automount

k8s/infra/forgejo-runner/templates/deployment.yaml

@@ -34,11 +34,7 @@ spec:
           command: ["sh", "-c"]
           args:
             - |
-              # Always re-register to keep labels in sync with values.yaml.
+              test -f /data/.runner || forgejo-runner register --no-interactive \
-              # Without this, changing a runner label requires manually deleting
-              # the PVC or .runner file — not GitOps-friendly.
-              rm -f /data/.runner
-              forgejo-runner register --no-interactive \
                 --instance {{ .Values.runner.forgejoUrl }} \
                 --token $(RUNNER_TOKEN) \
                 --name {{ .Values.runner.name }} \
@@ -60,18 +56,20 @@ spec:
       containers:
         - name: runner
           image: {{ .Values.runner.image.repository }}:{{ .Values.runner.image.tag }}
-          command: ["sh", "-c", "while ! wget -q -O- http://localhost:2375/_ping >/dev/null 2>&1; do echo 'waiting for dind...'; sleep 2; done; echo 'dind ready'; forgejo-runner daemon --config /etc/forgejo-runner/config.yaml"]
+          command: ["sh", "-c", "forgejo-runner daemon --config /etc/forgejo-runner/config.yaml"]
           workingDir: /data
           env:
             - name: DOCKER_HOST
-              value: tcp://localhost:2375
+              value: tcp://localhost:2376
+            - name: DOCKER_TLS_VERIFY
+              value: "1"
+            - name: DOCKER_CERT_PATH
+              value: /docker-certs/client
           volumeMounts:
             - name: runner-data
               mountPath: /data
             - name: docker-certs
               mountPath: /docker-certs
-            - name: docker-sock
-              mountPath: /run
             - name: homelab-ca
               mountPath: /etc/ssl/certs/homelab-ca.pem
               subPath: ca.crt
@@ -87,12 +85,10 @@ spec:
             privileged: true    # required for DinD; cicd namespace is labelled privileged
           env:
             - name: DOCKER_TLS_CERTDIR
-              value: ""
+              value: /docker-certs
           volumeMounts:
             - name: docker-certs
               mountPath: /docker-certs
-            - name: docker-sock
-              mountPath: /run
             - name: dind-storage
               mountPath: /var/lib/docker
             - name: homelab-ca
@@ -117,8 +113,6 @@ spec:
             claimName: {{ .Release.Name }}-dind
         - name: docker-certs
           emptyDir: {}    # DinD regenerates mTLS certs on each start
-        - name: docker-sock
-          emptyDir: {}    # Shared docker socket between dind and runner
         - name: homelab-ca
           # homelab-ca is a ConfigMap (public CA trust bundle), not a Secret.
           # The volumeMounts use subPath: ca.crt to project the single cert file.

k8s/infra/forgejo-runner/templates/gc-cronjob.yaml

@@ -1,152 +0,0 @@
-{{- if .Values.gc.enabled }}
-# Garbage-collects DinD Docker images/volumes/build-cache and actcache across
-# ALL forgejo-runner pods. Prevents PVC fill-up that breaks CI runs.
-# Only rendered once (enable in default values.yaml, disable in per-runner overrides).
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: runner-gc
-  namespace: {{ .Release.Namespace }}
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: runner-gc
-  namespace: {{ .Release.Namespace }}
-rules:
-  - apiGroups: [""]
-    resources: ["pods"]
-    verbs: ["get", "list"]
-  - apiGroups: [""]
-    resources: ["pods/exec"]
-    verbs: ["create"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: runner-gc
-  namespace: {{ .Release.Namespace }}
-subjects:
-  - kind: ServiceAccount
-    name: runner-gc
-    namespace: {{ .Release.Namespace }}
-roleRef:
-  kind: Role
-  name: runner-gc
-  apiGroup: rbac.authorization.k8s.io
----
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: forgejo-runner-gc
-  namespace: {{ .Release.Namespace }}
-  labels:
-    app: forgejo-runner-gc
-spec:
-  schedule: {{ .Values.gc.schedule | quote }}
-  concurrencyPolicy: Forbid
-  successfulJobsHistoryLimit: 3
-  failedJobsHistoryLimit: 3
-  jobTemplate:
-    spec:
-      backoffLimit: 1
-      activeDeadlineSeconds: 900
-      template:
-        spec:
-          serviceAccountName: runner-gc
-          restartPolicy: Never
-          tolerations:
-            - key: node-role.kubernetes.io/control-plane
-              operator: Exists
-              effect: NoSchedule
-          containers:
-            - name: gc
-              image: {{ .Values.gc.image }}
-              command:
-                - sh
-                - -c
-                - |
-                  set -e
-
-                  # Iterate all forgejo-runner pods (golang, rust, node)
-                  PODS=$(kubectl -n {{ .Release.Namespace }} get pod \
-                    -l app -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.app}{"\n"}{end}' \
-                    | grep 'forgejo-runner-' | awk '{print $1}')
-
-                  if [ -z "$PODS" ]; then
-                    echo "no forgejo-runner pods found, skipping"
-                    exit 0
-                  fi
-
-                  for POD in $PODS; do
-                    echo "===== $POD ====="
-
-                    # 1. Docker image prune (DinD sidecar)
-                    echo "[docker] before:"
-                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- docker system df 2>/dev/null || true
-
-                    echo "[docker] pruning non-latest images older than {{ .Values.gc.pruneAge }}..."
-                    # Keep :latest tagged images, delete all others older than pruneAge.
-                    # docker image prune can't filter by tag, so we list and selectively rmi.
-                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- \
-                      sh -c '
-                        # Remove dangling (untagged) images older than {{ .Values.gc.pruneAge }}
-                        docker image prune -f --filter "until={{ .Values.gc.pruneAge }}" 2>/dev/null
-
-                        # Remove tagged non-latest images older than {{ .Values.gc.pruneAge }}
-                        CUTOFF=$(date -d "-{{ .Values.gc.pruneAgeHours }} hours" +%s 2>/dev/null || date -v-{{ .Values.gc.pruneAgeHours }}H +%s)
-                        docker images --format "{{"{{"}} .Repository {{"}}"}}:{{"{{"}} .Tag {{"}}"}} {{"{{"}} .CreatedAt {{"}}"}}" | while read -r IMAGE_TAG CREATED_REST; do
-                          TAG=$(echo "$IMAGE_TAG" | rev | cut -d: -f1 | rev)
-                          # Skip latest-tagged images
-                          if [ "$TAG" = "latest" ]; then
-                            echo "[keep] $IMAGE_TAG (latest)"
-                            continue
-                          fi
-                          # Check image age via inspect
-                          CREATED_TS=$(docker inspect --format="{{"{{"}} .Created {{"}}"}}" "$IMAGE_TAG" 2>/dev/null | head -1)
-                          if [ -z "$CREATED_TS" ]; then continue; fi
-                          IMAGE_EPOCH=$(date -d "$CREATED_TS" +%s 2>/dev/null || date -jf "%Y-%m-%dT%H:%M:%S" "$(echo $CREATED_TS | cut -dT -f1-2 | cut -d. -f1)" +%s 2>/dev/null || echo 0)
-                          if [ "$IMAGE_EPOCH" -lt "$CUTOFF" ] 2>/dev/null; then
-                            echo "[delete] $IMAGE_TAG (older than {{ .Values.gc.pruneAge }})"
-                            docker rmi -f "$IMAGE_TAG" 2>/dev/null || true
-                          else
-                            echo "[keep] $IMAGE_TAG (recent)"
-                          fi
-                        done
-                      ' 2>/dev/null || true
-
-                    echo "[docker] pruning build cache unused >{{ .Values.gc.pruneAge }}..."
-                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- \
-                      docker builder prune -af --filter "until={{ .Values.gc.pruneAge }}" 2>/dev/null || true
-
-                    echo "[docker] pruning dangling volumes..."
-                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- \
-                      docker volume prune -af 2>/dev/null || true
-
-                    echo "[docker] after:"
-                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- docker system df 2>/dev/null || true
-
-                    # 2. Actcache cleanup (runner container)
-                    echo "[actcache] cleaning incomplete and stale cache entries..."
-                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c runner -- \
-                      sh -c '
-                        # Delete incomplete/partial cache uploads immediately (tmp dirs)
-                        find /data/.cache/actcache/cache -name "tmp" -type d -exec rm -rf {} + 2>/dev/null || true
-                        # Delete cache entries not accessed in last {{ .Values.gc.actcacheMaxAgeDays }} day(s)
-                        find /data/.cache/actcache/cache -type f -mtime +{{ .Values.gc.actcacheMaxAgeDays }} -delete 2>/dev/null || true
-                        # Clean up empty directories
-                        find /data/.cache/actcache/cache -type d -empty -delete 2>/dev/null || true
-                        echo "actcache size: $(du -sh /data/.cache/actcache/cache 2>/dev/null | cut -f1)"
-                      ' || true
-
-                    echo ""
-                  done
-                  echo "GC complete"
-              resources:
-                requests:
-                  cpu: 50m
-                  memory: 64Mi
-                limits:
-                  cpu: 250m
-                  memory: 128Mi
-{{- end }}

k8s/infra/forgejo-runner/values-node.yaml

@@ -2,15 +2,8 @@
 # runner instance. Only runner.name and runner.labels differ -- everything
 # else (image, dind, persistence, tolerations, nodeSelector) is shared.
 #
-# Label image: node:22-bookworm — Debian, root, apt-get, Node.js, npm, git.
+# node:22-bookworm ships Node natively, so unlike the golang/rust instances,
-# Install docker in workflow steps as needed.
+# jobs on this runner need no "install node" step before actions/checkout.
 runner:
-  image:
-    repository: code.forgejo.org/forgejo/runner
-    tag: "6"
   name: node-runner
   labels: "node:docker://node:22-bookworm"
-
-# GC CronJob renders only from the default (golang) values to avoid duplicates
-gc:
-  enabled: false

k8s/infra/forgejo-runner/values-rust.yaml

@@ -2,17 +2,9 @@
 # runner instance. Only runner.name and runner.labels differ -- everything
 # else (image, dind, persistence, tolerations, nodeSelector) is shared.
 #
-# Label image: rust:1-bookworm — Debian, root, apt-get, Rust, cargo, git.
+# rust:1.83-bookworm -- verified this tag exists (docker manifest inspect)
-# Install Node.js/docker in workflow steps as needed.
+# before pinning it, per this repo's convention of not trusting a tag exists
+# without checking.
 runner:
-  image:
-    repository: code.forgejo.org/forgejo/runner
-    tag: "6"
   name: rust-runner
-  labels: "rust:docker://rust:1-bookworm"
+  labels: "rust:docker://rust:1.83-bookworm"
-
-
-
-# GC CronJob renders only from the default (golang) values to avoid duplicates
-gc:
-  enabled: false

k8s/infra/forgejo-runner/values.yaml

@@ -1,13 +1,20 @@
 runner:
   image:
     repository: code.forgejo.org/forgejo/runner
-    tag: "6"
+    tag: "6"          # pin exact release before apply
   name: golang-runner
-  # Label image is what workflow steps run in (NOT the runner daemon image).
+  # Default image is only used when a job's `container:` doesn't override it
-  # golang:1.26-bookworm: Debian, root, apt-get, Go, git.
+  # (both ci.yaml and build.yaml in homelab-frontend do). Retired the old
-  # TODO: Switch to custom image once build-runner-images.yml pushes images
+  # "docker" label entirely; every repo this runner serves is Go, so this
-  labels: "golang:docker://golang:1.26-bookworm"
+  # instance carries the golang toolchain and its own dind sidecar builds and
+  # pushes that repo's images too -- there is no separate generic runner
+  # anymore.
+  labels: "golang:docker://golang:1.25-bookworm"
+  # In-cluster Service (:3000) — direct, avoids the ingress/public-hostname hop
+  # (the public URL is :443 which forgejo doesn't serve; runner got i/o timeout).
   forgejoUrl: http://forgejo-gitea-http.cicd.svc.cluster.local:3000
+  # tokenSecret: name of the K8s Secret that holds the runner registration token
+  # created automatically by the helmfile presync hook (see helmfile.yaml.gotmpl)
   tokenSecret: runner-token
   resources:
     requests:
@@ -32,7 +39,7 @@ dind:
 persistence:
   reg:
     storageClass: longhorn  # Unified StorageClass (3 replicas)
-    size: 20Gi              # .runner registration file + action tool cache + actcache artifacts
+    size: 1Gi               # .runner registration file + config — survives pod restarts
   dind:
     storageClass: longhorn  # Unified StorageClass (3 replicas)
     size: 30Gi              # docker layer cache — keeps rebuilds fast across restarts
@@ -42,18 +49,7 @@ tolerations:
     operator: Exists
     effect: NoSchedule
 
-# Pin to az-b (talos-cp-2) — more Longhorn storage than az-a (worker-1 over-provisioned).
+# Pin to az-a (talos-cp-1) — sole Longhorn node; RWO PVCs (reg/dind cache) only
-# RWO PVCs will recreate on talos-cp-2 when nodeSelector changes.
+# attach there.
 nodeSelector:
-  topology.kubernetes.io/zone: az-b
+  topology.kubernetes.io/zone: az-a
-
-# GC CronJob — prunes Docker images/volumes/build-cache and actcache across
-# ALL forgejo-runner pods. Only enable in default values (golang instance);
-# disable in per-runner overrides so it renders once.
-gc:
-  enabled: true
-  schedule: "*/30 * * * *"     # every 30 minutes
-  image: alpine/k8s:1.31.0
-  pruneAge: "30m"              # Docker artifacts unused longer than this get pruned
-  pruneAgeHours: 0.5            # Same as pruneAge but numeric for date arithmetic in shell
-  actcacheMaxAgeDays: 1        # actcache files older than N days (aggressive for heavy Rust cargo builds)

k8s/infra/iam/authentik-provision-job.yaml

@@ -0,0 +1,195 @@
+# Authentik OAuth provisioning — PostSync hook, reruns on every ArgoCD sync
+# (hook-delete-policy: BeforeHookCreation deletes the previous run's Job before
+# creating a new one, so this stays reconciled the same way the rest of the
+# cluster does — no separate manual bootstrap step like setup_talos_iam.sh /
+# provision_oidc.py, which never got migrated off the old helmfile workflow).
+#
+# What it does (see scripts/authentik-provision.py docstring): creates the
+# "groups" scope mapping, homelab-admins / grafana-admins groups, the "rock"
+# admin user, OAuth2 providers + Applications for grafana/minio/forgejo/argocd,
+# and binds homelab-admins to all of them. The script is generated into the
+# authentik-provision-script ConfigMap by kustomize configMapGenerator (see
+# kustomization.yaml), not embedded here.
+#
+# RBAC: this Job only touches Secrets (get existing client secrets, create new
+# ones for forgejo/argocd/rock) across the namespaces those services live in.
+# It never touches any other resource type.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: authentik-provisioner
+  namespace: iam
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: authentik-provisioner
+rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "list", "create", "update", "patch"]
+---
+# One RoleBinding per namespace the script touches (least-privilege: Secrets
+# only, and only in these 5 namespaces — not a cluster-wide ClusterRoleBinding).
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: authentik-provisioner
+  namespace: iam
+subjects:
+  - kind: ServiceAccount
+    name: authentik-provisioner
+    namespace: iam
+roleRef:
+  kind: ClusterRole
+  name: authentik-provisioner
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: authentik-provisioner
+  namespace: cicd
+subjects:
+  - kind: ServiceAccount
+    name: authentik-provisioner
+    namespace: iam
+roleRef:
+  kind: ClusterRole
+  name: authentik-provisioner
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: authentik-provisioner
+  namespace: argocd
+subjects:
+  - kind: ServiceAccount
+    name: authentik-provisioner
+    namespace: iam
+roleRef:
+  kind: ClusterRole
+  name: authentik-provisioner
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: authentik-provisioner
+  namespace: logging
+subjects:
+  - kind: ServiceAccount
+    name: authentik-provisioner
+    namespace: iam
+roleRef:
+  kind: ClusterRole
+  name: authentik-provisioner
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: authentik-provisioner
+  namespace: storage
+subjects:
+  - kind: ServiceAccount
+    name: authentik-provisioner
+    namespace: iam
+roleRef:
+  kind: ClusterRole
+  name: authentik-provisioner
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: authentik-provisioner
+  namespace: paperless
+subjects:
+  - kind: ServiceAccount
+    name: authentik-provisioner
+    namespace: iam
+roleRef:
+  kind: ClusterRole
+  name: authentik-provisioner
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: authentik-provisioner
+  namespace: immich
+subjects:
+  - kind: ServiceAccount
+    name: authentik-provisioner
+    namespace: iam
+roleRef:
+  kind: ClusterRole
+  name: authentik-provisioner
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: authentik-provision
+  namespace: iam
+  annotations:
+    argocd.argoproj.io/hook: PostSync
+    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
+spec:
+  ttlSecondsAfterFinished: 600
+  backoffLimit: 3
+  template:
+    spec:
+      serviceAccountName: authentik-provisioner
+      restartPolicy: Never
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: provision
+          image: python:3.12-alpine
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+          env:
+            - name: AUTHENTIK_BOOTSTRAP_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: authentik-secrets
+                  key: AUTHENTIK_BOOTSTRAP_TOKEN
+          volumeMounts:
+            - name: script
+              mountPath: /script
+          command:
+            - /bin/sh
+            - -c
+            - |
+              set -e
+              echo "waiting for authentik-server..."
+              until wget -q -O /dev/null http://authentik-server.iam.svc.cluster.local/-/health/ready/ 2>/dev/null; do
+                sleep 5
+              done
+              echo "installing kubectl (via python urllib - no apk/curl: this"
+              echo "container runs as non-root UID 1000 and can't write to"
+              echo "apk's directories or /usr/local/bin, both root-owned in"
+              echo "the python:3.12-alpine image; /tmp is world-writable)..."
+              python3 -c "
+              import urllib.request, os, stat
+              kver = urllib.request.urlopen('https://dl.k8s.io/release/stable.txt').read().decode().strip()
+              url = f'https://dl.k8s.io/release/{kver}/bin/linux/amd64/kubectl'
+              urllib.request.urlretrieve(url, '/tmp/kubectl')
+              st = os.stat('/tmp/kubectl')
+              os.chmod('/tmp/kubectl', st.st_mode | stat.S_IEXEC)
+              "
+              export PATH="/tmp:$PATH"
+              echo "running provisioning script..."
+              python3 /script/authentik-provision.py
+      volumes:
+        - name: script
+          configMap:
+            name: authentik-provision-script

k8s/infra/iam/authentik-values.yaml

@@ -153,11 +153,9 @@ server:
   # checks aren't treated as failures. (Only these fields are overridden; the
   # chart deep-merges the rest of each probe, incl. the httpGet path.)
   livenessProbe:
-    initialDelaySeconds: 300  # skip probe until 5min passed (migrations finish)
     timeoutSeconds: 15
     failureThreshold: 6
   readinessProbe:
-    initialDelaySeconds: 300  # skip probe until migrations complete
     timeoutSeconds: 15
     failureThreshold: 6
   startupProbe:
@@ -217,13 +215,6 @@ worker:
   podAnnotations:
     configmap.reloader.stakater.com/reload: "homelab-ca"
     homelab.io/restart-at: "2026-06-21T13-40"
-  livenessProbe:
-    initialDelaySeconds: 300  # skip probe until 5min passed (migrations finish)
-  readinessProbe:
-    initialDelaySeconds: 300  # skip probe until migrations complete
-  startupProbe:
-    initialDelaySeconds: 30  # let server finish DB work first
-    failureThreshold: 120
   metrics:
     enabled: true
     serviceMonitor:

k8s/infra/iam/kustomization.yaml

@@ -1,12 +1,35 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
-
+# NOTE: no top-level `namespace:` transformer here (removed) - it used to
+# force-rewrite metadata.namespace to "iam" on every resource in this
+# kustomization, which was harmless while every manifest here only ever
+# targeted the iam namespace itself. authentik-provision-job.yaml's
+# RoleBindings deliberately target cicd/argocd/logging/storage (least-
+# privilege access for the authentik-provisioner ServiceAccount to touch
+# Secrets in those namespaces) - the namespace transformer would have
+# silently rewritten all of them back to iam, breaking the RBAC. Every
+# manifest in this directory already sets its own explicit
+# metadata.namespace, so dropping the transformer changes nothing for the
+# existing resources/.
 resources:
+  - authentik-provision-job.yaml
   - rbac-dashboard-rolebinding.yaml
 
-# IAM provisioning is manual-only (security-sensitive).
+# Provisioning/verification python lives in scripts/*.py (real files, linted +
-# Script: scripts/iam/authentik-provision.py
+# diff-friendly) and is generated into ConfigMaps here rather than embedded in
-# Run:
+# the job YAML. disableNameSuffixHash keeps the names stable so the Jobs'
-#   export AUTHENTIK_BOOTSTRAP_TOKEN=$(kubectl -n iam get secret authentik-secrets \
+# configMap volume refs and PostSync hook-delete semantics keep working; each
-#     -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_TOKEN}' | base64 -d)
+# hook Job is recreated per sync so it always mounts the latest script.
-#   python3 scripts/iam/authentik-provision.py
+configMapGenerator:
+  - name: authentik-provision-script
+    namespace: iam
+    files:
+      - authentik-provision.py=scripts/authentik-provision.py
+
+generatorOptions:
+  disableNameSuffixHash: true
+# authentik-migrations-job.yaml removed — redundant + broken. The authentik
+# `server` entrypoint runs migrations itself; this standalone job lacked the
+# authentik-secrets envFrom (Secret key missing) and always failed.
+# SOPS secrets (*.enc.yaml) handled by ArgoCD SOPS plugin at sync time
+# authentik/vault deployed via ArgoCD Helm source

k8s/infra/iam/scripts/authentik-provision.py

@@ -0,0 +1,659 @@
+#!/usr/bin/env python3
+"""
+Authentik OAuth provisioning - idempotent, safe to re-run (ArgoCD PostSync hook).
+
+Creates/updates, in order:
+  1. A custom "groups" OAuth2 scope mapping (Authentik ships openid/email/profile
+     by default but NOT groups - required for ArgoCD RBAC group mapping and
+     Grafana's role_attribute_path, both of which read a `groups` claim).
+  2. Groups: homelab-admins (is_superuser=true), grafana-admins.
+  3. User "rock": created if missing, always (re-)synced into both groups above.
+     Password is generated once and only written to the k8s Secret
+     rock-credentials (iam ns) the first time the user is created - re-runs
+     never rotate an existing password.
+  4. OAuth2/OIDC providers + Applications for: grafana, minio, forgejo, argocd.
+     Client secrets are read from existing k8s Secrets (grafana-oidc, minio-oidc)
+     if present, or generated once and written out (forgejo-oidc, oidc-secret)
+     the first time.
+  5. PolicyBinding of homelab-admins -> every Application above, so "rock" (and
+     anyone else in that group) has guaranteed access regardless of each app's
+     default visibility.
+
+Talks to Authentik over the in-cluster Service (authentik-server.iam.svc:80),
+authenticating with the bootstrap token. Everything is done with GET-then-
+create-or-patch so this can be re-run on every ArgoCD sync without duplicating
+or clobbering objects (PostSync hook, not a one-shot Job with hook-delete).
+
+kubectl is used only to read/write the small set of Secrets this script
+touches - it shells out rather than using the Python k8s client to keep the
+container image to stdlib Python + the kubectl binary, no pip installs.
+"""
+import json
+import os
+import secrets
+import string
+import subprocess
+import sys
+import urllib.error
+import urllib.request
+
+AUTHENTIK_URL = "http://authentik-server.iam.svc.cluster.local"
+TOKEN = os.environ["AUTHENTIK_BOOTSTRAP_TOKEN"]
+
+
+def api(method, path, data=None):
+    url = f"{AUTHENTIK_URL}{path}"
+    body = json.dumps(data).encode() if data is not None else None
+    req = urllib.request.Request(
+        url,
+        data=body,
+        method=method,
+        headers={
+            "Authorization": f"Bearer {TOKEN}",
+            "Content-Type": "application/json",
+        },
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=30) as resp:
+            raw = resp.read()
+            return resp.status, (json.loads(raw) if raw else {})
+    except urllib.error.HTTPError as e:
+        raw = e.read()
+        try:
+            parsed = json.loads(raw) if raw else {}
+        except json.JSONDecodeError:
+            parsed = {"raw": raw.decode(errors="replace")}
+        return e.code, parsed
+
+
+def die(msg):
+    print(f"FATAL: {msg}", file=sys.stderr)
+    sys.exit(1)
+
+
+def gen_secret(n=40):
+    alphabet = string.ascii_letters + string.digits
+    return "".join(secrets.choice(alphabet) for _ in range(n))
+
+
+def kubectl_get_secret_key(namespace, name, key):
+    """Returns decoded value, or None if the secret/key doesn't exist."""
+    p = subprocess.run(
+        ["kubectl", "-n", namespace, "get", "secret", name, "-o", f"jsonpath={{.data.{key}}}"],
+        capture_output=True, text=True,
+    )
+    if p.returncode != 0 or not p.stdout.strip():
+        return None
+    import base64
+    return base64.b64decode(p.stdout).decode()
+
+
+def kubectl_create_secret(namespace, name, literals: dict, labels: dict = None):
+    """Idempotent: create-or-update via dry-run|apply, same pattern used
+    elsewhere in this repo (setup_vault.sh, apply-vault-secrets.sh)."""
+    args = ["kubectl", "-n", namespace, "create", "secret", "generic", name]
+    for k, v in literals.items():
+        args += [f"--from-literal={k}={v}"]
+    args += ["--dry-run=client", "-o", "yaml"]
+    render = subprocess.run(args, capture_output=True, text=True)
+    if render.returncode != 0:
+        die(f"rendering secret {namespace}/{name}: {render.stderr}")
+    apply = subprocess.run(["kubectl", "apply", "-f", "-"], input=render.stdout,
+                            capture_output=True, text=True)
+    if apply.returncode != 0:
+        die(f"applying secret {namespace}/{name}: {apply.stderr}")
+    print(f"  secret {namespace}/{name}: {apply.stdout.strip()}")
+    if labels:
+        # argocd's `$secret:key` substitution only reads Secrets carrying
+        # app.kubernetes.io/part-of: argocd — without it OIDC login fails with
+        # oauth2 "invalid_client" (empty client_secret sent to the IdP).
+        label_args = ["kubectl", "-n", namespace, "label", "secret", name,
+                      "--overwrite"] + [f"{k}={v}" for k, v in labels.items()]
+        subprocess.run(label_args, capture_output=True, text=True)
+
+
+def get_or_create(list_path, create_path, query, payload, patch_existing=None):
+    status, res = api("GET", f"{list_path}?{query}")
+    if status != 200:
+        die(f"GET {list_path}?{query} -> {status} {res}")
+    results = res.get("results", [])
+    if results:
+        obj = results[0]
+        if patch_existing:
+            status, obj2 = api("PATCH", f"{create_path}{obj['pk']}/", patch_existing)
+            if status not in (200, 201):
+                die(f"PATCH {create_path}{obj['pk']}/ -> {status} {obj2}")
+            return obj2
+        return obj
+    status, obj = api("POST", create_path, payload)
+    if status not in (200, 201):
+        die(f"POST {create_path} -> {status} {obj}")
+    return obj
+
+
+# -----------------------------------------------------------------------------
+print("[1/5] Ensuring custom 'groups' scope mapping exists...")
+groups_mapping = get_or_create(
+    "/api/v3/propertymappings/provider/scope/",
+    "/api/v3/propertymappings/provider/scope/",
+    "scope_name=groups",
+    {
+        "name": "homelab: groups claim",
+        "scope_name": "groups",
+        # request.user.ak_groups is deprecated in authentik 2026.x (logs a
+        # deprecation warning on every token issue) -> use request.user.groups.
+        "expression": (
+            "return {\"groups\": [group.name for group in request.user.groups.all()]}"
+        ),
+    },
+    # Force the expression onto the already-created mapping on re-run.
+    patch_existing={
+        "expression": (
+            "return {\"groups\": [group.name for group in request.user.groups.all()]}"
+        ),
+    },
+)
+GROUPS_MAPPING_PK = groups_mapping["pk"]
+
+# Generic "permissions" claim, computed from group membership - lets each app
+# (and eventually k8s RBAC via --oidc-groups-claim) check a permission string
+# like "paperless:write" instead of hardcoding a group name. homelab-admins
+# gets "*" (everything); every other admin group gets its own read+write pair.
+# k8s-devops-admin is declared but has no k8s Role/RoleBinding target yet -
+# foundation for a future short-lived federated-operator credential.
+_PERMISSIONS_EXPR = """
+GROUP_PERMISSIONS = {
+    "homelab-admins": ["*"],
+    "grafana-admins": ["grafana:read", "grafana:write"],
+    "minio-admins": ["minio:read", "minio:write"],
+    "forgejo-admins": ["forgejo:read", "forgejo:write"],
+    "homarr-admins": ["homarr:read", "homarr:write"],
+    "portainer-admins": ["portainer:read", "portainer:write"],
+    "kmsvc-admins": ["kmsvc:read", "kmsvc:write"],
+    "temporal-admins": ["temporal:read", "temporal:write"],
+    "llm-admins": ["llm:read", "llm:write"],
+    "paperless-admins": ["paperless:read", "paperless:write"],
+    "immich-admins": ["immich:read", "immich:write"],
+    "poimen-memory-admins": ["poimen-memory:read", "poimen-memory:write"],
+    "k8s-devops-admin": ["k8s:devops"],
+}
+perms = set()
+for group in request.user.groups.all():
+    perms.update(GROUP_PERMISSIONS.get(group.name, []))
+return {"permissions": sorted(perms)}
+""".strip()
+permissions_mapping = get_or_create(
+    "/api/v3/propertymappings/provider/scope/",
+    "/api/v3/propertymappings/provider/scope/",
+    "scope_name=permissions",
+    {
+        "name": "homelab: permissions claim",
+        "scope_name": "permissions",
+        "expression": _PERMISSIONS_EXPR,
+    },
+    patch_existing={"expression": _PERMISSIONS_EXPR},
+)
+PERMISSIONS_MAPPING_PK = permissions_mapping["pk"]
+
+# Immich reads a "immich_role" claim on every login (not just user-creation -
+# fixed upstream in immich-app/immich#29991) and syncs isAdmin from it, so
+# this is the actual mechanism that makes "rock" an Immich admin - not
+# Immich's first-user-is-admin fallback, which races badly with OAuth login.
+_IMMICH_ROLE_EXPR = (
+    "return {\"immich_role\": \"admin\" "
+    "if request.user.ak_groups.filter(name__in=[\"homelab-admins\", \"immich-admins\"]).exists() "
+    "else \"user\"}"
+)
+immich_role_mapping = get_or_create(
+    "/api/v3/propertymappings/provider/scope/",
+    "/api/v3/propertymappings/provider/scope/",
+    "scope_name=immich_role",
+    {
+        "name": "homelab: immich role claim",
+        "scope_name": "immich_role",
+        "expression": _IMMICH_ROLE_EXPR,
+    },
+    patch_existing={"expression": _IMMICH_ROLE_EXPR},
+)
+IMMICH_ROLE_MAPPING_PK = immich_role_mapping["pk"]
+
+# MinIO maps OIDC users to a MinIO policy via a "policy" claim
+# (MINIO_IDENTITY_OPENID_CLAIM_NAME=policy). Emit consoleAdmin (full admin) for
+# homelab-admins members, readonly for everyone else. Without this claim MinIO
+# assigns no policy and OIDC users get no access.
+_POLICY_EXPR = (
+    "return {\"policy\": \"consoleAdmin\" "
+    "if request.user.ak_groups.filter(name=\"homelab-admins\").exists() "
+    "else \"readonly\"}"
+)
+policy_mapping = get_or_create(
+    "/api/v3/propertymappings/provider/scope/",
+    "/api/v3/propertymappings/provider/scope/",
+    "scope_name=minio",
+    {
+        "name": "homelab: minio policy claim",
+        "scope_name": "minio",
+        "expression": _POLICY_EXPR,
+    },
+    patch_existing={"expression": _POLICY_EXPR},
+)
+POLICY_MAPPING_PK = policy_mapping["pk"]
+
+# Fetch the standard openid/email/profile mapping pks (shipped by default).
+status, res = api("GET", "/api/v3/propertymappings/provider/scope/")
+by_scope = {m["scope_name"]: m["pk"] for m in res["results"]}
+SCOPE_PKS = [by_scope["openid"], by_scope["email"], by_scope["profile"], GROUPS_MAPPING_PK, PERMISSIONS_MAPPING_PK]
+
+status, res = api("GET", "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent")
+AUTHORIZATION_FLOW_PK = res["results"][0]["pk"]
+status, res = api("GET", "/api/v3/flows/instances/?slug=default-provider-invalidation-flow")
+INVALIDATION_FLOW_PK = res["results"][0]["pk"]
+status, res = api("GET", "/api/v3/crypto/certificatekeypairs/?has_key=true")
+SIGNING_KEY_PK = res["results"][0]["pk"]
+
+# -----------------------------------------------------------------------------
+print("[2/5] Ensuring homelab-admins + per-service admin groups exist...")
+homelab_admins = get_or_create(
+    "/api/v3/core/groups/", "/api/v3/core/groups/",
+    "name=homelab-admins",
+    {"name": "homelab-admins", "is_superuser": True},
+)
+# App-scoped, not Authentik superusers (unlike homelab-admins) - each maps to
+# read+write in its own service via the "permissions" claim above (k8s Role/
+# RoleBinding in k8s/infra/rbac/, or an app's own adapter e.g. paperless's).
+# k8s-devops-admin is declared with no target yet - foundation for a future
+# short-lived federated-operator credential.
+SERVICE_ADMIN_GROUP_NAMES = [
+    "grafana-admins", "minio-admins", "forgejo-admins", "homarr-admins",
+    "portainer-admins", "kmsvc-admins", "temporal-admins", "llm-admins",
+    "paperless-admins", "immich-admins", "poimen-memory-admins",
+    "k8s-devops-admin",
+]
+service_admin_groups = {}
+for group_name in SERVICE_ADMIN_GROUP_NAMES:
+    service_admin_groups[group_name] = get_or_create(
+        "/api/v3/core/groups/", "/api/v3/core/groups/",
+        f"name={group_name}",
+        {"name": group_name, "is_superuser": False},
+    )
+grafana_admins = service_admin_groups["grafana-admins"]
+paperless_admins = service_admin_groups["paperless-admins"]
+
+# -----------------------------------------------------------------------------
+print("[3/5] Ensuring user 'rock' exists with admin group membership...")
+status, res = api("GET", "/api/v3/core/users/?username=rock")
+rock_password = None
+if res.get("results"):
+    rock = res["results"][0]
+    status, rock = api("PATCH", f"/api/v3/core/users/{rock['pk']}/", {
+        "groups": [homelab_admins["pk"]] + [g["pk"] for g in service_admin_groups.values()],
+        "is_active": True,
+        # email is REQUIRED: Grafana's OIDC login reads the email claim from
+        # userinfo; an empty email makes Grafana fall back to a GitHub-style
+        # <userinfo>/emails call, which Authentik 404s -> login fails entirely.
+        "email": "[email protected]",
+    })
+    if status not in (200, 201):
+        die(f"PATCH user rock -> {status} {rock}")
+    print("  rock already exists, group membership synced (password unchanged)")
+else:
+    rock_password = gen_secret(24)
+    status, rock = api("POST", "/api/v3/core/users/", {
+        "username": "rock",
+        "name": "Rock",
+        "is_active": True,
+        # Required for Grafana OIDC (see PATCH branch above).
+        "email": "[email protected]",
+        "groups": [homelab_admins["pk"]] + [g["pk"] for g in service_admin_groups.values()],
+        "path": "users",
+        "type": "internal",
+    })
+    if status not in (200, 201):
+        die(f"POST user rock -> {status} {rock}")
+    status, pw_res = api("POST", f"/api/v3/core/users/{rock['pk']}/set_password/",
+                          {"password": rock_password})
+    if status not in (200, 204):
+        die(f"set_password for rock -> {status} {pw_res}")
+    kubectl_create_secret("iam", "rock-credentials", {
+        "username": "rock",
+        "password": rock_password,
+    })
+    print("  rock created, credentials stored in iam/rock-credentials")
+
+# -----------------------------------------------------------------------------
+print("[4/5] Ensuring OAuth2 providers + applications for grafana/minio/forgejo/argocd...")
+
+SERVICES = {
+    "grafana": {
+        "client_secret_source": ("logging", "grafana-oidc", "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET"),
+        "redirect_uris": ["https://grafana.riotpiao.com/login/generic_oauth"],
+        "launch_url": "https://grafana.riotpiao.com",
+        "display_name": "Grafana",
+    },
+    "minio": {
+        "client_secret_source": ("storage", "minio-oidc", "MINIO_IDENTITY_OPENID_CLIENT_SECRET"),
+        "redirect_uris": ["https://minio.riotpiao.com/oauth_callback"],
+        "launch_url": "https://minio.riotpiao.com",
+        "display_name": "MinIO",
+    },
+    "forgejo": {
+        # No secret exists yet for forgejo - generate + store on first run.
+        "client_secret_source": ("cicd", "forgejo-oidc", "CLIENT_SECRET"),
+        "generate_if_missing": True,
+        "redirect_uris": [
+            "https://forgejo.riotpiao.com/user/oauth2/authentik/callback",
+            "https://forgejo.riotpiao.com/user/oauth2/openidconnect/callback",
+        ],
+        "launch_url": "https://forgejo.riotpiao.com",
+        "display_name": "Forgejo",
+    },
+    "argocd": {
+        # oidc-secret uses hyphenated keys (client-id/client-secret) per
+        # argocd-values.yaml's `$oidc-secret:client-id` / `:client-secret` refs.
+        "client_secret_source": ("argocd", "oidc-secret", "client-secret"),
+        "generate_if_missing": True,
+        "extra_secret_literals": {"client-id": "argocd"},
+        # argocd only reads $secret refs from Secrets labelled part-of: argocd.
+        "secret_labels": {"app.kubernetes.io/part-of": "argocd"},
+        "redirect_uris": ["https://argocd.riotpiao.com/auth/callback"],
+        "launch_url": "https://argocd.riotpiao.com",
+        "display_name": "Argo CD",
+    },
+    "homarr": {
+        "client_secret_source": ("dashboard", "homarr-oidc", "client-secret"),
+        "generate_if_missing": True,
+        "extra_secret_literals": {"client-id": "homarr"},
+        "redirect_uris": ["https://homarr.riotpiao.com/api/auth/callback/oidc"],
+        "launch_url": "https://homarr.riotpiao.com",
+        "display_name": "Homarr",
+    },
+    "paperless": {
+        # No secret exists yet for paperless - generate + store on first run.
+        # django-allauth's generic openid_connect provider callback path is
+        # /accounts/oidc/<provider_id>/login/callback/ - provider_id "authentik"
+        # is set in PAPERLESS_SOCIALACCOUNT_PROVIDERS (see configmap.yaml).
+        "client_secret_source": ("paperless", "paperless-oidc", "CLIENT_SECRET"),
+        "generate_if_missing": True,
+        "redirect_uris": ["https://paperless.riotpiao.com/accounts/oidc/authentik/login/callback/"],
+        "launch_url": "https://paperless.riotpiao.com",
+        "display_name": "Paperless-ngx",
+    },
+    "immich": {
+        # No secret exists yet for immich - generate + store on first run.
+        "client_secret_source": ("immich", "immich-oidc", "CLIENT_SECRET"),
+        "generate_if_missing": True,
+        # /auth/login + /user-settings are Immich's own web callback routes;
+        # /api/oauth/mobile-redirect forwards to the app.immich:///oauth-callback
+        # custom scheme Authentik can't register directly (see docs.immich.app/
+        # administration/oauth - "custom scheme" workaround).
+        "redirect_uris": [
+            "https://img.riotpiao.com/auth/login",
+            "https://img.riotpiao.com/user-settings",
+            "https://img.riotpiao.com/api/oauth/mobile-redirect",
+        ],
+        "launch_url": "https://img.riotpiao.com",
+        "display_name": "Immich",
+    },
+    "vault": {
+        # Human/CLI login only (`vault login -method=oidc`) - not wired to any
+        # workload. No secret exists yet - generate + store on first run.
+        # localhost:8250/oidc/callback is the vault CLI's documented fixed
+        # callback port for `vault login -method=oidc`; the other is the
+        # browser/UI flow's callback path (mount path "oidc").
+        "client_secret_source": ("iam", "vault-oidc", "CLIENT_SECRET"),
+        "generate_if_missing": True,
+        "extra_secret_literals": {"client-id": "vault"},
+        "redirect_uris": [
+            "https://vault.riotpiao.com/ui/vault/auth/oidc/oidc/callback",
+            "http://localhost:8250/oidc/callback",
+        ],
+        "launch_url": "https://vault.riotpiao.com",
+        "display_name": "Vault",
+    },
+}
+
+app_pks_for_binding = []
+
+for name, cfg in SERVICES.items():
+    # MinIO also needs the "policy" claim (via the minio scope mapping) so its
+    # MINIO_IDENTITY_OPENID_CLAIM_NAME=policy maps homelab-admins -> consoleAdmin.
+    # Immich needs "immich_role" so its OAuth roleClaim can grant admin.
+    provider_mappings = SCOPE_PKS + ([POLICY_MAPPING_PK] if name == "minio" else []) \
+        + ([IMMICH_ROLE_MAPPING_PK] if name == "immich" else [])
+    ns, secret_name, key = cfg["client_secret_source"]
+    client_secret = kubectl_get_secret_key(ns, secret_name, key)
+    if client_secret is None:
+        if not cfg.get("generate_if_missing"):
+            print(f"  WARNING: {ns}/{secret_name} key {key} not found and "
+                  f"generate_if_missing not set for '{name}' - skipping provider/app")
+            continue
+        client_secret = gen_secret(40)
+        literals = {key: client_secret}
+        literals.update(cfg.get("extra_secret_literals", {}))
+        kubectl_create_secret(ns, secret_name, literals,
+                              labels=cfg.get("secret_labels"))
+        print(f"  {name}: generated new client secret -> {ns}/{secret_name}")
+    else:
+        print(f"  {name}: using existing client secret from {ns}/{secret_name}")
+
+    if name == "paperless":
+        # paperless-ngx's django-allauth OIDC config takes client_id/secret
+        # bundled inside one JSON blob (PAPERLESS_SOCIALACCOUNT_PROVIDERS), not
+        # discrete env vars - compose it here and store it alongside
+        # CLIENT_SECRET so the Deployment can source it directly via
+        # secretKeyRef, no shell wrapper needed. Runs every time (not just on
+        # generate), so it stays in sync if the client_secret is ever rotated
+        # by hand.
+        providers_json = json.dumps({
+            "openid_connect": {
+                "APPS": [{
+                    "provider_id": "authentik",
+                    "name": "Authentik",
+                    "client_id": "paperless",
+                    "secret": client_secret,
+                    "settings": {
+                        "server_url": "https://authentik.riotpiao.com/application/o/paperless/.well-known/openid-configuration",
+                        # "groups"/"permissions" aren't default OIDC scopes -
+                        # must be requested explicitly for Authentik's scope
+                        # mappings above to actually be returned. paperless's
+                        # adapter.py ConfigMap reads the "permissions" claim
+                        # to grant is_staff+is_superuser.
+                        "scope": ["openid", "profile", "email", "groups", "permissions"],
+                    },
+                }],
+            },
+        })
+        kubectl_create_secret("paperless", "paperless-oidc", {
+            "CLIENT_SECRET": client_secret,
+            "SOCIALACCOUNT_PROVIDERS_JSON": providers_json,
+        })
+
+    if name == "immich":
+        # Immich reads its whole system-config from IMMICH_CONFIG_FILE (a
+        # mounted JSON file, see k8s/apps/immich/deployment.yaml), not
+        # discrete env vars. "immich_role" must be in `scope` for Authentik
+        # to actually include that claim in the token (non-default scopes
+        # are opt-in per-client, same reason paperless requests "permissions"
+        # explicitly). roleClaim is re-evaluated on every login (immich-app/
+        # immich#29991) so this is the actual admin-grant mechanism for rock,
+        # not Immich's racy first-user-is-admin fallback.
+        immich_config_json = json.dumps({
+            "oauth": {
+                "enabled": True,
+                "issuerUrl": "https://authentik.riotpiao.com/application/o/immich/",
+                "clientId": "immich",
+                "clientSecret": client_secret,
+                "scope": "openid email profile immich_role",
+                "roleClaim": "immich_role",
+                "autoRegister": True,
+                "autoLaunch": False,
+                "buttonText": "Login with Authentik",
+                "mobileRedirectUri": "app.immich:///oauth-callback",
+            },
+        })
+        kubectl_create_secret("immich", "immich-oidc", {
+            "CLIENT_SECRET": client_secret,
+            "config.json": immich_config_json,
+        })
+
+    provider = get_or_create(
+        "/api/v3/providers/oauth2/", "/api/v3/providers/oauth2/",
+        f"name={name}",
+        {
+            "name": name,
+            "client_id": name,
+            "client_secret": client_secret,
+            "client_type": "confidential",
+            "authorization_flow": AUTHORIZATION_FLOW_PK,
+            "invalidation_flow": INVALIDATION_FLOW_PK,
+            "signing_key": SIGNING_KEY_PK,
+            "property_mappings": provider_mappings,
+            "sub_mode": "hashed_user_id",
+            "include_claims_in_id_token": True,
+            # authentik 2026.x requires grant_types to be set explicitly; the
+            # API defaults it to [] when omitted, which makes /authorize reject
+            # every login with "Invalid grant_type for provider" ->
+            # invalid_request. authorization_code = the web SSO flow all these
+            # apps use; refresh_token = long-lived sessions (offline_access).
+            "grant_types": ["authorization_code", "refresh_token"],
+            "redirect_uris": [
+                {"matching_mode": "strict", "url": u} for u in cfg["redirect_uris"]
+            ],
+        },
+        # Keep the redirect_uris/mappings/grant_types in sync on re-run, but
+        # never touch client_secret again once created (that's the source of
+        # truth in the k8s Secret, and re-sending it here is harmless anyway).
+        patch_existing={
+            "property_mappings": provider_mappings,
+            "grant_types": ["authorization_code", "refresh_token"],
+            "redirect_uris": [
+                {"matching_mode": "strict", "url": u} for u in cfg["redirect_uris"]
+            ],
+        },
+    )
+
+    # superuser_full_list=true is REQUIRED on the LIST: the applications list
+    # applies access-policy filtering to the results array (these apps are bound
+    # to homelab-admins, and the bootstrap-token user akadmin is not a member),
+    # so without it the GET returns an empty results list even though the app
+    # exists -> fall through to POST -> 400 "already exists".
+    #
+    # We deliberately do NOT patch_existing here: the application DETAIL endpoint
+    # (PATCH /applications/{pk}/) enforces the same access policy and does NOT
+    # honor superuser_full_list, so PATCH-by-pk returns 404 for akadmin once the
+    # homelab-admins binding exists. That 404 aborted the loop before later
+    # providers got their grant_types. slug/provider/launch_url are set at
+    # creation and are stable (provider is get_or_create'd by name, stable pk),
+    # so find-or-create is sufficient.
+    application = get_or_create(
+        "/api/v3/core/applications/", "/api/v3/core/applications/",
+        f"slug={name}&superuser_full_list=true",
+        {
+            "name": cfg["display_name"],
+            "slug": name,
+            "provider": provider["pk"],
+            "meta_launch_url": cfg["launch_url"],
+        },
+    )
+    app_pks_for_binding.append((name, application["pk"]))
+    print(f"  {name}: provider pk={provider['pk']} application pk={application['pk']}")
+
+# -----------------------------------------------------------------------------
+# Separate from the SERVICES loop above: this is a PUBLIC client (PKCE, no
+# client_secret) for `kubectl` OIDC login, not a confidential-client app
+# login. Foundation for k8s/infra/rbac/ - kube-apiserver's --oidc-* flags
+# (controlplane.tftpl) validate tokens issued against this provider.
+# Redirect URI matches kubelogin's (int128/kubelogin) documented default;
+# adjust here if a different kubectl OIDC plugin/port is actually used.
+print("Ensuring public OAuth2 client 'kubernetes' for kubectl OIDC login...")
+k8s_provider = get_or_create(
+    "/api/v3/providers/oauth2/", "/api/v3/providers/oauth2/",
+    "name=kubernetes",
+    {
+        "name": "kubernetes",
+        "client_id": "kubernetes",
+        "client_type": "public",
+        "authorization_flow": AUTHORIZATION_FLOW_PK,
+        "invalidation_flow": INVALIDATION_FLOW_PK,
+        "signing_key": SIGNING_KEY_PK,
+        "property_mappings": SCOPE_PKS,
+        "sub_mode": "hashed_user_id",
+        "include_claims_in_id_token": True,
+        "grant_types": ["authorization_code", "refresh_token"],
+        "redirect_uris": [
+            {"matching_mode": "strict", "url": "http://localhost:8000"},
+        ],
+    },
+    patch_existing={
+        "property_mappings": SCOPE_PKS,
+        "grant_types": ["authorization_code", "refresh_token"],
+        "redirect_uris": [
+            {"matching_mode": "strict", "url": "http://localhost:8000"},
+        ],
+    },
+)
+k8s_application = get_or_create(
+    "/api/v3/core/applications/", "/api/v3/core/applications/",
+    "slug=kubernetes&superuser_full_list=true",
+    {
+        "name": "Kubernetes",
+        "slug": "kubernetes",
+        "provider": k8s_provider["pk"],
+        "meta_launch_url": "https://authentik.riotpiao.com",
+    },
+)
+app_pks_for_binding.append(("kubernetes", k8s_application["pk"]))
+print(f"  kubernetes: provider pk={k8s_provider['pk']} application pk={k8s_application['pk']}")
+
+# -----------------------------------------------------------------------------
+print("[5/5] Binding homelab-admins to every application (guaranteed access for rock)...")
+for name, app_pk in app_pks_for_binding:
+    get_or_create(
+        "/api/v3/policies/bindings/", "/api/v3/policies/bindings/",
+        f"target={app_pk}&group={homelab_admins['pk']}",
+        {
+            "target": app_pk,
+            "group": homelab_admins["pk"],
+            "order": 0,
+            "enabled": True,
+        },
+    )
+    print(f"  {name}: homelab-admins bound")
+
+# Per-service admin groups are app-scoped (unlike homelab-admins' blanket
+# binding above) - only grants visibility/access to that one application.
+# portainer/kmsvc/temporal/llm-serving have no Authentik Application (no OIDC
+# login integration), so their groups exist for the "permissions" claim /
+# future k8s RBAC only - nothing to bind here.
+SERVICE_GROUP_TO_APP_SLUG = {
+    "grafana-admins": "grafana",
+    "minio-admins": "minio",
+    "forgejo-admins": "forgejo",
+    "homarr-admins": "homarr",
+    "paperless-admins": "paperless",
+    "immich-admins": "immich",
+}
+for group_name, app_slug in SERVICE_GROUP_TO_APP_SLUG.items():
+    app_pk = next((pk for n, pk in app_pks_for_binding if n == app_slug), None)
+    if not app_pk:
+        continue
+    group_pk = service_admin_groups[group_name]["pk"]
+    get_or_create(
+        "/api/v3/policies/bindings/", "/api/v3/policies/bindings/",
+        f"target={app_pk}&group={group_pk}",
+        {
+            "target": app_pk,
+            "group": group_pk,
+            "order": 0,
+            "enabled": True,
+        },
+    )
+    print(f"  {app_slug}: {group_name} bound")
+
+print("\nDone. Summary:")
+print("  groups:  homelab-admins (superuser) + " + ", ".join(SERVICE_ADMIN_GROUP_NAMES))
+print("  user:    rock -> homelab-admins + all service admin groups")
+print(f"  apps:    {', '.join(n for n, _ in app_pks_for_binding)}")
+if rock_password:
+    print("  NOTE: rock's password was generated this run - see")
+    print("  kubectl -n iam get secret rock-credentials -o jsonpath='{.data.password}' | base64 -d")

k8s/infra/logging/grafana-values.yaml

@@ -44,10 +44,6 @@ grafana.ini:
   server:
     root_url: https://grafana.riotpiao.com
 
-  # Allow embedding dashboards in iframes (NextJS integration)
-  security:
-    allow_embedding: true
-
   # No anonymous read access — every user must log in via Authentik SSO.
   auth.anonymous:
     enabled: false
@@ -68,14 +64,15 @@ grafana.ini:
   # doesn't return localhost redirects in its token responses.
   #
   # role_attribute_path: JMESPath expression evaluated against the userinfo
-  # response. akadmin gets GrafanaAdmin (server admin, can impersonate);
+  # response. Members of the 'grafana-admins' Authentik group get Admin role;
-  # homelab-admins members get Admin (org admin); everyone else Viewer.
+  # everyone else gets Viewer. The group name must match exactly what Authentik
+  # sends in the 'groups' claim.
   auth.generic_oauth:
     enabled: true
     name: Authentik
     allow_sign_up: true
     client_id: grafana
-    scopes: openid email profile groups
+    scopes: openid email profile
     auth_url: https://authentik.riotpiao.com/application/o/authorize/
     token_url: https://authentik.riotpiao.com/application/o/token/
     api_url: https://authentik.riotpiao.com/application/o/userinfo/
@@ -86,8 +83,7 @@ grafana.ini:
     email_attribute_path: email
     login_attribute_path: preferred_username
     name_attribute_path: name
-    role_attribute_path: "preferred_username == 'akadmin' && 'GrafanaAdmin' || contains(groups[*], 'homelab-admins') && 'Admin' || 'Viewer'"
+    role_attribute_path: "contains(groups[*], 'grafana-admins') && 'Admin' || 'Viewer'"
-    allow_assign_grafana_admin: true
     use_pkce: false
     use_refresh_token: false
     skip_org_role_sync: false

k8s/infra/longhorn/longhorn-cnpg-storageclass.yaml

@@ -35,5 +35,8 @@ parameters:
   mkfsParams: "-O ^64bit,^metadata_csum"
 mountOptions:
   - "noatime"
+  # Critical: mount with postgres UID/GID (26:26) to avoid permission issues
+  - "uid=26"
+  - "gid=26"
 reclaimPolicy: Delete
 volumeBindingMode: Immediate

k8s/infra/minio/minio-provision-paperless-job.yaml

@@ -59,7 +59,7 @@ spec:
               mountPath: /shared
       containers:
         - name: provision
-          image: quay.io/minio/mc:latest
+          image: minio/mc:latest
           volumeMounts:
             - name: shared
               mountPath: /shared
@@ -81,9 +81,6 @@ spec:
               mc alias set m http://minio-cluster-hl.storage.svc.cluster.local:9000 \
                 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"
 
-              echo "Ensuring paperless bucket exists..."
-              mc mb --ignore-existing m/paperless
-
               echo "Checking for existing paperless-minio-creds secret..."
               if kubectl -n paperless get secret paperless-minio-creds >/dev/null 2>&1; then
                 ACCESS_KEY=$(kubectl -n paperless get secret paperless-minio-creds -o jsonpath='{.data.ACCESS_KEY}' | base64 -d)

k8s/infra/minio/minio-tenant.yaml

@@ -96,13 +96,8 @@ spec:
       console: https://minio.riotpiao.com
 
   # ── OIDC via Authentik (server-side env, valid in v2 schema) ────────────────
-  # Use in-cluster URL for config fetch (pod→authentik); browser redirects use
-  # public URLs embedded in the OIDC metadata response (issuer stays public).
   env:
     - name: MINIO_IDENTITY_OPENID_CONFIG_URL
-      # Must use external URL — well-known response contains external issuer/jwks_uri.
-      # MinIO validates issuer in JWT matches well-known issuer. Internal URL = mismatch.
-      # Hairpins through ingress-nginx but stays in-cluster.
       value: "https://authentik.riotpiao.com/application/o/minio/.well-known/openid-configuration"
     - name: MINIO_IDENTITY_OPENID_CLIENT_ID
       value: "minio"

k8s/infra/monitoring/alerts/api-gateway-alerts.yaml

@@ -1,160 +0,0 @@
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: api-gateway-alerts
-  namespace: monitoring
-  labels:
-    release: prometheus
-spec:
-  groups:
-    # ================================================================
-    # SLA Targets (based on canary traffic baselines):
-    #
-    # Availability:  99.9% (43.8 min downtime/month)
-    # LLM Chat:      p95 < 1s (qwen), p95 < 2s (reasoning), p95 < 5s (ornith)
-    # Embeddings:    p95 < 500ms
-    # Rerank:        p95 < 500ms
-    # Models list:   p95 < 300ms
-    # Error rate:    < 1% (5xx), < 5% (4xx excluding auth)
-    #
-    # Baselines from 200-request canary run:
-    #   qwen p99=609ms, reasoning p99=328ms, embeddings p99=287ms,
-    #   rerank p99=218ms, models p99=277ms
-    # SLA set at ~2x p99 for headroom.
-    # ================================================================
-
-    - name: api-gateway.availability
-      rules:
-        # Gateway pods not ready
-        - alert: APIGatewayDown
-          expr: sum(kube_pod_status_ready{namespace="api",condition="true"}) == 0
-          for: 1m
-          labels:
-            severity: critical
-          annotations:
-            summary: "API Gateway has zero ready pods"
-
-        # Gateway pod count below desired
-        - alert: APIGatewayDegraded
-          expr: |
-            sum(kube_pod_status_ready{namespace="api",condition="true"})
-            < kube_deployment_spec_replicas{namespace="api",deployment="api-gateway"}
-          for: 5m
-          labels:
-            severity: warning
-          annotations:
-            summary: "API Gateway {{ $value }} ready pods below desired replica count"
-
-        # Blackbox probe down
-        - alert: APIGatewayProbeDown
-          expr: probe_success{instance=~".*api.riotpiao.com.*"} == 0
-          for: 2m
-          labels:
-            severity: critical
-          annotations:
-            summary: "API Gateway probe failed: {{ $labels.instance }}"
-
-        # LLM serving pods not ready
-        - alert: LLMServingDown
-          expr: sum(kube_pod_status_ready{namespace="llm-serving",condition="true"}) == 0
-          for: 2m
-          labels:
-            severity: critical
-          annotations:
-            summary: "All LLM serving pods down"
-
-        # Individual predictor down
-        - alert: LLMPredictorDown
-          expr: |
-            kube_deployment_status_replicas_ready{namespace="llm-serving"}
-            < kube_deployment_spec_replicas{namespace="llm-serving"}
-          for: 5m
-          labels:
-            severity: warning
-          annotations:
-            summary: "{{ $labels.deployment }} has {{ $value }} ready (below desired)"
-
-    - name: api-gateway.latency
-      # SLA: latency thresholds at ~2x measured p99
-      rules:
-        # Ingress-level latency (all requests through nginx)
-        - alert: APIGatewayLatencyHigh
-          expr: |
-            histogram_quantile(0.95,
-              sum(rate(nginx_ingress_controller_request_duration_seconds_bucket{ingress="api"}[5m])) by (le)
-            ) > 2
-          for: 5m
-          labels:
-            severity: warning
-          annotations:
-            summary: "API Gateway p95 latency {{ $value | printf \"%.1f\" }}s (SLA: <2s)"
-
-        # Extreme latency (p99 > 5s)
-        - alert: APIGatewayLatencyCritical
-          expr: |
-            histogram_quantile(0.99,
-              sum(rate(nginx_ingress_controller_request_duration_seconds_bucket{ingress="api"}[5m])) by (le)
-            ) > 5
-          for: 5m
-          labels:
-            severity: critical
-          annotations:
-            summary: "API Gateway p99 latency {{ $value | printf \"%.1f\" }}s (SLA: <5s)"
-
-    - name: api-gateway.errors
-      rules:
-        # 5xx error rate > 1%
-        - alert: APIGateway5xxErrorRate
-          expr: |
-            sum(rate(nginx_ingress_controller_requests{ingress="api",status=~"5.."}[5m]))
-            / sum(rate(nginx_ingress_controller_requests{ingress="api"}[5m]))
-            > 0.01
-          for: 5m
-          labels:
-            severity: critical
-          annotations:
-            summary: "API Gateway 5xx rate {{ $value | humanizePercentage }} (SLA: <1%)"
-
-        # Total error rate > 10% (including 4xx)
-        - alert: APIGatewayHighErrorRate
-          expr: |
-            sum(rate(nginx_ingress_controller_requests{ingress="api",status=~"[45].."}[5m]))
-            / sum(rate(nginx_ingress_controller_requests{ingress="api"}[5m]))
-            > 0.10
-          for: 10m
-          labels:
-            severity: warning
-          annotations:
-            summary: "API Gateway total error rate {{ $value | humanizePercentage }} (SLA: <10%)"
-
-    - name: api-gateway.resources
-      rules:
-        # Gateway pod restart
-        - alert: APIGatewayRestarted
-          expr: increase(kube_pod_container_status_restarts_total{namespace="api"}[15m]) > 0
-          for: 0m
-          labels:
-            severity: warning
-          annotations:
-            summary: "API Gateway pod {{ $labels.pod }} restarted"
-
-        # LLM predictor restart
-        - alert: LLMPredictorRestarted
-          expr: increase(kube_pod_container_status_restarts_total{namespace="llm-serving"}[15m]) > 0
-          for: 0m
-          labels:
-            severity: warning
-          annotations:
-            summary: "LLM predictor {{ $labels.pod }} restarted"
-
-        # Gateway high memory (>80% of limit)
-        - alert: APIGatewayHighMemory
-          expr: |
-            sum(container_memory_working_set_bytes{namespace="api",container="gateway"}) by (pod)
-            / sum(kube_pod_container_resource_limits{namespace="api",container="gateway",resource="memory"}) by (pod)
-            > 0.8
-          for: 10m
-          labels:
-            severity: warning
-          annotations:
-            summary: "Gateway pod {{ $labels.pod }} memory at {{ $value | humanizePercentage }} of limit"