fix(cnpg): scale paperless-db, immich-db, gotify-db to 3 instances

3 CP nodes available — run one CNPG instance per node for proper HA. 2 instances loses quorum on single node failure. memory-db stays at 2 (acceptable for that workload).
fix(gotify): move SOPS secrets to ksops generator
2026-09-11 07:56:21 +09:00 · 2026-09-10 22:32:24 +09:00 · 2026-09-10 10:49:30 +09:00 · 2026-09-10 01:41:37 +00:00 · 2026-09-10 01:35:59 +00:00 · 2026-09-09 22:38:25 +00:00
152 changed files with 5133 additions and 5314 deletions
@@ -1,460 +0,0 @@
 # CI/CD Pipeline: GitOps Validation & Deployment
 ## Overview
 Pure GitOps CI/CD pipeline using Forgejo Actions (self-hosted runner).
 **Principle:** Validate in CI, deploy via ArgoCD (no manual steps).
 ```
 git push
  ↓
  [CI: Validate]
    ├─ yamllint (YAML syntax)
    ├─ kubeval (K8s manifests)
    ├─ kustomize build (all layers)
    ├─ argocd validation (app definitions)
    └─ security scan (secrets, best practices)
  ↓
  [If push to main]
    └─ ArgoCD auto-syncs (if enabled)
 ```
 ## Workflows
 ### 1. validate-k8s.yaml (Mandatory)
 **Trigger:** Any push/PR with k8s/ changes
 **What it does:**
 1. Lints all YAML files (`yamllint`)
 2. Validates K8s manifests (`kubeval`)
 3. Builds all kustomization layers
 4. Validates ArgoCD applications
 5. Reports results
 **Duration:** ~2-3 minutes
 **Status:**
 - ✅ PASS: All layers build, manifests valid → OK to merge
 - ❌ FAIL: Syntax error, invalid resource, build failed → Fix & push again
 **Example output:**
 ```
 === Building k8s/infrastructure/ ===
 ✓ Infrastructure built successfully
 Resources: 47
 === Building k8s/bootstrap/ ===
 ✓ Bootstrap built successfully
 Resources: 23
 ```
 **When to check:**
 - After every commit
 - Before merging PRs
 - On every branch
 ### 2. argocd-sync.yaml (Recommended)
 **Trigger:** Push to main only (k8s/ changed)
 **What it does:**
 1. Authenticates with ArgoCD
 2. Syncs `homelab-root` application
 3. Waits for sync to complete (5 min timeout)
 4. Verifies all applications healthy
 **Duration:** 1-5 minutes (depends on resources)
 **Status:**
 - ✅ SYNCED: All resources deployed to cluster
 - ❌ FAILED: Sync error, pod crashes, etc. → Check ArgoCD UI for details
 **When it runs:**
 - Automatically after merge to main
 - Only on k8s/ changes (not on docs)
 **Manual trigger (if needed):**
 ```bash
 # SSH to runner or use Forgejo UI
 # Re-run failed workflow
 # Or manually sync: argocd app sync homelab-root
 ```
 **Requires secrets:**
 - `ARGOCD_SERVER`: ArgoCD server URL (https://argocd.riotpiao.com)
 - `ARGOCD_AUTH_TOKEN`: ArgoCD API token (generate via ArgoCD UI)
 ### 3. security-scan.yaml (Optional)
 **Trigger:** Any push/PR with k8s/ changes
 **What it does:**
 1. Scans Dockerfiles for vulnerabilities (`trivy`)
 2. Scans Helm charts for security issues
 3. Audits K8s manifests (`polaris`)
 4. Checks for hardcoded secrets
 5. Verifies security best practices
 **Duration:** ~3-5 minutes
 **Status:**
 - ✅ PASS: No critical issues
 - ⚠️ WARNING: Best practice recommendations (non-blocking)
 - ❌ FAIL: Hardcoded secrets found (must fix)
 **Common issues:**
 - Missing resource limits (warning)
 - Privileged containers (warning)
 - Hardcoded passwords (ERROR)
 ---
 ## File Structure
 ```
 .forgejo/
 ├── workflows/                      # CI/CD workflows
 │   ├── validate-k8s.yaml          # Validate manifests (required)
 │   ├── argocd-sync.yaml           # Sync to cluster (auto on main)
 │   └── security-scan.yaml         # Security checks (optional)
 └── CI-CD.md                        # This file
 ```
 ---
 ## Setup Instructions
 ### 1. Install Forgejo Runner
 ```bash
 # On runner machine (inside cluster or external)
 forgejo-runner register \
  --instance https://forgejo.riotpiao.com \
  --token <registration-token> \
  --name homelab-runner \
  --labels docker
 forgejo-runner daemon
 ```
 ### 2. Add ArgoCD Secrets to Forgejo
 ```bash
 # Go to: Forgejo → Settings → Secrets
 # Add:
 ARGOCD_SERVER = https://argocd.riotpiao.com
 ARGOCD_AUTH_TOKEN = <token>  # Generate: argocd account generate-token
 ```
 ### 3. Generate ArgoCD Token
 ```bash
 # Inside cluster
 kubectl -n argocd port-forward svc/argocd-server 8080:443
 # Go to: https://localhost:8080/user-info/api-tokens
 # Create new token (CI/CD)
 # Copy token to Forgejo secrets
 ```
 ---
 ## Workflow Execution
 ### When developer pushes to feature branch:
 ```
 git push origin feature/new-service
 ↓
 Forgejo Actions triggered
 ↓
 validate-k8s.yaml runs:
  ✓ Lints YAML
  ✓ Validates manifests
  ✓ Builds kustomizations
  ✓ All pass → GitHub comment: "Ready to merge"
 ↓
 Developer opens PR
 ↓
 Reviewer checks:
  - Code changes (YAML)
  - Workflow results
  - ArgoCD impact (diff)
 ↓
 PR merged to main
 ```
 ### When merged to main:
 ```
 git merge feature/new-service → main
 ↓
 Forgejo Actions triggered
 ↓
 validate-k8s.yaml runs:
  ✓ Same validation as above
 ↓
 argocd-sync.yaml runs (if enabled):
  ✓ Syncs homelab-root
  ✓ Waits for sync
  ✓ Verifies health
  ✓ Resources deployed to cluster
 ↓
 Cluster state = git state
 (No manual kubectl apply needed!)
 ```
 ---
 ## Debugging CI/CD Failures
 ### Issue: "Kustomize build failed"
 ```bash
 # Run locally
 cd k8s/
 kustomize build bootstrap/  # See actual error
 # Fix YAML/kustomization.yaml
 # git push again
 ```
 ### Issue: "Kubeval validation failed"
 ```bash
 # Check K8s manifest syntax
 kubeval k8s/platform/minio/config.yaml
 # Common issues:
 # - Typos in apiVersion, kind, metadata
 # - Missing required fields
 # - Invalid references (namespace, service name)
 ```
 ### Issue: "ArgoCD sync failed"
 ```bash
 # Check ArgoCD UI
 # https://argocd.riotpiao.com → homelab-root
 # Or CLI
 argocd app get homelab-root
 argocd app logs homelab-root --follow
 # Common issues:
 # - Missing namespace (fixed by infrastructure layer)
 # - Invalid Helm chart version
 # - Secret not found
 # - Network policy blocking traffic
 ```
 ### Issue: "Security scan found hardcoded secret"
 ```bash
 # Fix: Remove secret from YAML
 # Add to SOPS encryption instead
 # Or use ArgoCD Sealed Secrets
 # (if SOPS not available)
 ```
 ---
 ## Viewing Results
 ### Forgejo Actions UI
 ```
 Repository → Actions
  ├─ validate-k8s
  │   ├─ ✅ Success (merge safe)
  │   ├─ ❌ Failed (fix required)
  │   └─ Logs (click "Steps" → "Summary")
  ├─ argocd-sync
  │   ├─ ✅ Synced (deployed)
  │   └─ ❌ Failed (check ArgoCD UI)
  └─ security-scan
      ├─ ✅ Pass (no critical issues)
      └─ ⚠️  Warning (review, non-blocking)
 ```
 ### ArgoCD UI
 ```
 https://argocd.riotpiao.com
  ├─ homelab-root
  │   ├─ Status: Synced ✓
  │   ├─ Health: Healthy ✓
  │   └─ Details (click to see resources)
  ├─ layer-1-bootstrap
  ├─ layer-2-platform
  ├─ layer-3-security
  ├─ layer-4-applications
  └─ layer-5-data
 ```
 ---
 ## Common Tasks
 ### Add new service to cluster
 ```bash
 # 1. Create directory and kustomization.yaml
 mkdir -p k8s/applications/my-service
 cat > k8s/applications/my-service/kustomization.yaml << EOF
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: my-namespace
 helmCharts:
 - name: my-chart
  repo: https://charts.example.com
  version: 1.0.0
  releaseName: my-service
  valuesFile: values.yaml
 EOF
 # 2. Add values.yaml
 cp /template/values.yaml k8s/applications/my-service/
 # 3. Commit and push
 git add k8s/applications/my-service/
 git commit -m "feat(apps): add my-service"
 git push
 # 4. CI validates
 # 5. Merge to main
 # 6. ArgoCD syncs automatically
 # ✓ Service deployed to cluster
 ```
 ### Rollback a deployment
 ```bash
 # 1. Find broken commit
 git log --oneline k8s/  # Identify bad commit
 # 2. Revert
 git revert <commit-hash>
 git push
 # 3. CI validates (should pass)
 # 4. Merge to main
 # 5. ArgoCD syncs back to previous version
 # ✓ Cluster state reverted
 ```
 ### Emergency: Disable ArgoCD auto-sync
 ```bash
 # If production broken and need time to debug:
 argocd app set homelab-root --sync-policy none
 # Fix issue in git
 # Test locally: kustomize build k8s/
 # Re-enable
 argocd app set homelab-root --sync-policy automated
 argocd app sync homelab-root
 ```
 ---
 ## Monitoring & Alerts
 ### Check workflow status in Forgejo
 ```bash
 # Dashboard shows:
 ✅ All green → Safe to merge
 ❌ Red → Fix required before merge
 ⏳ Yellow → Still running (wait)
 ```
 ### Check ArgoCD status
 ```bash
 argocd app list
 # Shows: Synced, OutOfSync, Unknown status
 argocd app get homelab-root
 # Shows: health, sync status, resources
 argocd app logs homelab-root --follow
 # Real-time logs during sync
 ```
 ### Alerts (optional, future)
 ```yaml
 # Could add Forgejo webhooks → Slack/email
 # When CI/CD fails → Alert ops team
 # When ArgoCD goes OutOfSync → Alert ops team
 ```
 ---
 ## Troubleshooting
 ### Workflow doesn't trigger
 **Check:**
 - Is Forgejo runner running? `forgejo-runner daemon`
 - Did you push to correct branch? (validate runs on all, argocd-sync only on main)
 - Did path match filter? (must change k8s/ or .forgejo/workflows/)
 ### Workflow hangs/times out
 **Check:**
 - kustomize build → Check for dependency cycles
 - argocd sync → Check cluster resources (storage full? network down?)
 - security scan → Large image scan → Takes time
 **Fix:**
 - Increase timeout in workflow
 - Optimize kustomization (remove unused resources)
 - Add resource limits to pods
 ### ArgoCD token invalid
 **Fix:**
 ```bash
 # Regenerate token
 argocd account generate-token
 # Update Forgejo secret
 # Settings → Secrets → ARGOCD_AUTH_TOKEN = <new-token>
 ```
 ---
 ## Best Practices
 ✅ **DO:**
 - Commit all K8s changes to git (no manual kubectl apply)
 - Run validate-k8s locally before push
 - Write descriptive commit messages (why this change?)
 - Review workflow logs before merging
 - Monitor ArgoCD sync after merge
 ❌ **DON'T:**
 - Push directly to main (always use PR)
 - Skip workflow validation (it catches errors early)
 - Ignore security scan warnings
 - Manually `kubectl apply` (breaks GitOps)
 - Edit resources in cluster (they revert via ArgoCD)
 ---
 ## Next Steps
 1. **Setup Forgejo runner** (if not already running)
 2. **Add ArgoCD secrets** to Forgejo
 3. **Test workflows** on feature branch
 4. **Merge to main** → Watch ArgoCD sync
 5. **Celebrate:** Full GitOps pipeline working! 🎉
@@ -1,269 +0,0 @@
 name: Cluster CI Pipeline
 on:
  push:
    branches:
      - main
      - develop
    paths:
      - 'k8s/**'
      - '.forgejo/workflows/cluster-ci.yaml'
  pull_request:
    paths:
      - 'k8s/**'
 jobs:
  ci:
    runs-on: docker
    steps:
      # === Checkout ===
      - name: Checkout
        run: |
          REPO_URL="${{ gitea.server_url }}/${{ gitea.repository }}.git"
          CLONE_URL="https://${{ secrets.CI_RUNNER }}:${{ secrets.CI_RUNNER_SECRET }}@${REPO_URL#https://}"
          git clone --depth 1 "$CLONE_URL" .
          git fetch origin main
          git checkout main
      # === Install Tools ===
      - name: Install Tools
        run: |
          unset GITHUB_TOKEN
          apt-get update && apt-get install -y \
            yamllint \
            python3-pip \
            curl \
            jq
          # kubeval
          curl -L https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz | tar xz
          mv -f kubeval /usr/local/bin/
          # kustomize
          rm -f kustomize
          curl -s https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh | bash
          mv -f kustomize /usr/local/bin/
          # argocd
          curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
          chmod +x /usr/local/bin/argocd
          # trivy
          curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
          # polaris
          curl -L https://github.com/FairwindsOps/polaris/releases/latest/download/polaris-linux-amd64 -o /usr/local/bin/polaris
          chmod +x /usr/local/bin/polaris
      # === YAML Lint ===
      - name: YAML Lint
        run: |
          echo "=== Linting YAML files ==="
          yamllint k8s/ -c .yamllint.yaml || true
      # === Kubeval - Validate K8s Syntax ===
      - name: Kubeval - Validate K8s Syntax
        run: |
          echo "=== Validating Kubernetes manifests ==="
          find k8s -name "*.yaml" -o -name "*.yml" | grep -v "\.archive" | while read file; do
            echo "Validating $file..."
            kubeval "$file" -d 2>/dev/null || true
          done
      # === Kustomize Build - All overlays ===
      - name: Kustomize Build - Infrastructure
        run: |
          echo "=== Building k8s/infrastructure/ ==="
          kustomize build k8s/infrastructure > /tmp/infrastructure.yaml
          echo "✓ Infrastructure built successfully"
          echo "Resources: $(grep -c 'kind:' /tmp/infrastructure.yaml)"
      - name: Kustomize Build - Bootstrap
        run: |
          echo "=== Building k8s/bootstrap/ ==="
          kustomize build k8s/bootstrap > /tmp/bootstrap.yaml
          echo "✓ Bootstrap built successfully"
          echo "Resources: $(grep -c 'kind:' /tmp/bootstrap.yaml || echo 0)"
      - name: Kustomize Build - Platform
        run: |
          echo "=== Building k8s/platform/ ==="
          kustomize build k8s/platform > /tmp/platform.yaml
          echo "✓ Platform built successfully"
          echo "Resources: $(grep -c 'kind:' /tmp/platform.yaml || echo 0)"
      - name: Kustomize Build - Security
        run: |
          echo "=== Building k8s/security/ ==="
          kustomize build k8s/security > /tmp/security.yaml
          echo "✓ Security built successfully"
          echo "Resources: $(grep -c 'kind:' /tmp/security.yaml || echo 0)"
      - name: Kustomize Build - Applications
        run: |
          echo "=== Building k8s/applications/ ==="
          kustomize build k8s/applications > /tmp/applications.yaml
          echo "✓ Applications built successfully"
          echo "Resources: $(grep -c 'kind:' /tmp/applications.yaml || echo 0)"
      - name: Kustomize Build - Data
        run: |
          echo "=== Building k8s/data/ ==="
          kustomize build k8s/data > /tmp/data.yaml
          echo "✓ Data built successfully"
          echo "Resources: $(grep -c 'kind:' /tmp/data.yaml || echo 0)"
      - name: Validate ArgoCD Applications
        run: |
          echo "=== Validating ArgoCD Applications ==="
          kubeval k8s/argocd/apps/*.yaml
      # === Trivy - Scan Dockerfile ===
      - name: Trivy - Scan Dockerfile
        run: |
          if find . -name "Dockerfile" 2>/dev/null | grep -v node_modules | head -1 | grep -q .; then
            echo "=== Scanning Dockerfiles with Trivy ==="
            find . -name "Dockerfile" -not -path "*/node_modules/*" -exec trivy config {} \;
          else
            echo "No Dockerfiles found"
          fi
      # === Trivy - Scan Helm Charts ===
      - name: Trivy - Scan Helm Charts
        run: |
          if find k8s -name "Chart.yaml" 2>/dev/null | head -1 | grep -q .; then
            echo "=== Scanning Helm charts with Trivy ==="
            find k8s -name "Chart.yaml" -exec dirname {} \; | while read chart; do
              echo "Scanning $chart..."
              trivy config "$chart" || true
            done
          else
            echo "No Helm charts found"
          fi
      # === Polaris - K8s Security Audit ===
      - name: Polaris - K8s Security Audit
        run: |
          echo "=== Running Polaris K8s security audit ==="
          polaris audit --audit-path /tmp/polaris-audit.json k8s/ || true
          if [ -f /tmp/polaris-audit.json ]; then
            echo "Security issues found:"
            jq '.results[] | select(.pass == false)' /tmp/polaris-audit.json || true
          fi
      # === Check for Secrets in Code ===
      - name: Check for Secrets in Code
        run: |
          echo "=== Scanning for hardcoded secrets ==="
          # BLOCKING. This step used to only count findings and then exit 0, so a
          # plaintext deploy key rode through it into a public remote. Two failure
          # modes fixed: it now fails the build, and it matches key material by
          # PEM header rather than only `private_key:`-style YAML field names.
          # Findings are captured into variables and tested for emptiness rather than
          # branching on grep's exit status: implementations disagree on the rc of a
          # `-v` filter fed empty input, and a wrong rc here fails open.
          # NOTE: --include must precede `--`; after `--` grep treats it as a filename
          # and silently scans nothing.
          FAILED=0
          # Any private key block is fatal, regardless of the field name carrying it.
          KEYS=$(grep -rIE --include="*.yaml" --include="*.yml" \
                   -- "-----BEGIN ([A-Z]+ )?PRIVATE KEY-----" k8s/ \
                 | grep -v "\.enc\.yaml" || true)
          if [ -n "$KEYS" ]; then
            echo "❌ Unencrypted private key material found:"
            echo "$KEYS"
            FAILED=1
          fi
          # Plaintext values in secret-ish YAML fields. SOPS output is ENC[...],
          # so encrypted files never trip this.
          VALS=$(grep -rInE --include="*.yaml" --include="*.yml" \
                   -- "^[[:space:]]*(password|token|apiKey|api_key|sshPrivateKey|client_secret):[[:space:]]*[\"']?[^\"'[:space:]{\$]{8,}" k8s/ \
                 | grep -v "ENC\[" | grep -v "\.enc\.yaml" || true)
          if [ -n "$VALS" ]; then
            echo "❌ Plaintext secret value found:"
            echo "$VALS"
            FAILED=1
          fi
          if [ "$FAILED" -ne 0 ]; then
            echo "Encrypt with SOPS (see .sops.yaml) — *.enc.yaml files are exempt."
            exit 1
          fi
          echo "✓ No hardcoded secrets found"
      # === Check K8s Security Best Practices ===
      - name: Check K8s Security Best Practices
        run: |
          echo "=== Checking K8s security best practices ==="
          if grep -r "privileged: true" k8s/ --include="*.yaml" --include="*.yml"; then
            echo "⚠️  Found privileged containers"
          fi
          if grep -r "hostNetwork: true" k8s/ --include="*.yaml" --include="*.yml"; then
            echo "⚠️  Found hostNetwork usage"
          fi
          echo "Checking for missing resource limits..."
          MISSING=0
          find k8s -name "*.yaml" -o -name "*.yml" | while read file; do
            if grep -q "kind: Deployment\|kind: StatefulSet\|kind: DaemonSet" "$file"; then
              if ! grep -q "resources:" "$file"; then
                echo "⚠️  $file: Missing resource requests/limits"
                MISSING=$((MISSING + 1))
              fi
            fi
          done
      # === ArgoCD Sync (main branch only) ===
      - name: Sync ArgoCD
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        env:
          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
        run: |
          echo "=== Syncing homelab-root ==="
          argocd app sync homelab-root --force
          argocd app wait homelab-root --timeout 5m
      - name: Check Sync Status
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        env:
          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
        run: |
          echo "=== ArgoCD Applications Status ==="
          argocd app list -o table
          STATUS=$(argocd app get homelab-root -o jsonpath='{.status.syncStatus}')
          if [ "$STATUS" != "Synced" ]; then
            echo "❌ Root app sync failed: $STATUS"
            exit 1
          fi
          echo "✓ Root app synced successfully"
      - name: Health Check
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        env:
          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
        run: |
          echo "=== Checking Application Health ==="
          argocd app get homelab-root -o wide
      # === Summary ===
      - name: Summary
        if: always()
        run: |
          echo "=== CI Pipeline Summary ==="
          echo "✓ YAML linted"
          echo "✓ Manifests validated"
          echo "✓ Kustomizations built"
          echo "✓ Security scans completed"
          echo "✓ Secrets check passed"
          echo "✓ Best practices verified"
          echo ""
          echo "✓ All checks passed"
@@ -66,3 +66,6 @@ bootstrap-argocd.log
 # one line here, which is how a plaintext deploy key reached a public remote.
 k8s/**/*-secret.yaml
 !k8s/**/*.enc.yaml
 # IAM provisioning scripts contain credential references — never commit
 scripts/iam/*.py
@@ -0,0 +1,206 @@
 # Authentik Auth Integration for NextJS
 ## Current State
 ### Gateway Auth Status
 | Endpoint | Auth Status | Notes |
 |----------|-------------|-------|
 | `/v1/chat/completions` | ❌ **OFF** | LLM routes have no auth middleware |
 | `/v1/embeddings` | ❌ **OFF** | Same - no auth |
 | `/v1/rerank` | ❌ **OFF** | Same - no auth |
 | `X-Service: sqs` | ✅ **ON** | JWT validated via `internal/auth/jwt.go` |
 | `/workflow` | ❌ **OFF** | Pass-through to Temporal |
 **Auth module exists** at `homelab-frontend/internal/auth/jwt.go` but only wired for SQS.
 LLM routes in `internal/proxy/proxy.go` have no auth middleware.
 ### Authentik App
 Authentik app `local-llm` exists for LLM API auth:
 - **Client ID**: `local-llm`
 - **Client Secret**: `kubectl -n llm-serving get secret local-llm-jwt -o jsonpath='{.data.client-secret}' | base64 -d`
 - **Token endpoint**: `https://authentik.riotpiao.com/application/o/token/`
 - **Userinfo endpoint**: `https://authentik.riotpiao.com/application/o/userinfo/`
 - **OIDC discovery**: `https://authentik.riotpiao.com/application/o/local-llm/.well-known/openid-configuration`
 ## Sign-in Methods
 ### 1. Resource Owner Password Credentials (ROPC)
 Direct username/password login. Server-side only (needs client_secret).
 ```typescript
 // API Route: app/api/auth/login/route.ts
 const response = await fetch('https://authentik.riotpiao.com/application/o/token/', {
  method: 'POST',
  headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
  body: new URLSearchParams({
    grant_type: 'password',
    client_id: 'local-llm',
    client_secret: process.env.AUTHENTIK_CLIENT_SECRET,
    username: '[email protected]',
    password: 'userpassword',
    scope: 'openid email profile groups',
  }),
 });
 const tokens = await response.json();
 // { access_token, refresh_token, expires_in, token_type }
 ```
 ### 2. Authorization Code Flow (Browser Redirect)
 Requires adding redirect URIs to `local-llm` Authentik app:
 ```python
 # In k8s/infra/iam/scripts/authentik-provision.py, update:
 "local-llm": {
    ...
    "redirect_uris": [
        "http://localhost:3000/api/auth/callback",  # dev
        "https://your-nextjs-app.com/api/auth/callback",  # prod
    ],
 }
 ```
 Then standard OIDC flow:
 1. Redirect to `https://authentik.riotpiao.com/application/o/authorize/?client_id=local-llm&redirect_uri=...&response_type=code&scope=openid email profile groups`
 2. User logs in via Authentik UI
 3. Callback receives `code`, exchange for tokens
 ## JWT Token Persistence
 ### Browser (localStorage)
 ```typescript
 const TOKEN_KEY = 'llm_auth_token';
 // Save
 localStorage.setItem(TOKEN_KEY, JSON.stringify({
  access_token: tokens.access_token,
  refresh_token: tokens.refresh_token,
  expires_at: Date.now() + tokens.expires_in * 1000,
 }));
 // Load
 const stored = JSON.parse(localStorage.getItem(TOKEN_KEY) || 'null');
 if (stored && stored.expires_at > Date.now()) {
  // Token valid
 }
 // Clear (logout)
 localStorage.removeItem(TOKEN_KEY);
 ```
 ### Server-side (HTTP-only cookies)
 ```typescript
 // app/api/auth/login/route.ts
 import { cookies } from 'next/headers';
 // After successful login
 cookies().set('llm_auth_token', JSON.stringify(tokens), {
  httpOnly: true,
  secure: process.env.NODE_ENV === 'production',
  sameSite: 'lax',
  maxAge: tokens.expires_in,
  path: '/',
 });
 // Read in middleware or API routes
 const tokenCookie = cookies().get('llm_auth_token');
 const tokens = JSON.parse(tokenCookie?.value || 'null');
 ```
 ## Token Refresh
 ```typescript
 async function refreshAccessToken(refresh_token: string) {
  const response = await fetch('https://authentik.riotpiao.com/application/o/token/', {
    method: 'POST',
    headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
    body: new URLSearchParams({
      grant_type: 'refresh_token',
      client_id: 'local-llm',
      client_secret: process.env.AUTHENTIK_CLIENT_SECRET,
      refresh_token,
    }),
  });
  return response.json();
 }
 ```
 ## Environment Variables
 ```bash
 # .env.local
 AUTHENTIK_URL=https://authentik.riotpiao.com
 AUTHENTIK_CLIENT_ID=local-llm
 AUTHENTIK_CLIENT_SECRET=<from-secret>
 # For client-side (public)
 NEXT_PUBLIC_AUTHENTIK_URL=https://authentik.riotpiao.com
 NEXT_PUBLIC_AUTHENTIK_CLIENT_ID=local-llm
 ```
 ## Using Token with LLM API
 ```typescript
 const token = await getValidToken(); // from localStorage or cookie
 const response = await fetch('https://api.riotpiao.com/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${token}`,  // JWT from Authentik
  },
  body: JSON.stringify({
    model: 'reasoning',
    messages: [{ role: 'user', content: 'Hello' }],
  }),
 });
 ```
 ## TODO
 ### Gateway-side (homelab-frontend)
 - [ ] Wire `internal/auth/jwt.go` into LLM proxy handler (`internal/proxy/proxy.go`)
 - [ ] Add `authRequired: true` to model config or create LLM-specific middleware
 - [ ] Example pattern from SQS (in `internal/serviceadapter/router.go`):
 ```go
 // In proxy.go ServeHTTP, before dispatching to LLM upstream:
 if strings.HasPrefix(r.URL.Path, "/v1/") {
    authHeader := r.Header.Get("Authorization")
    claims, err := llmJWTAuth.ValidateBearerToken(authHeader)
    if err != nil {
        // Return 401/403
    }
    if !llmJWTAuth.CheckPermissions(claims, "llm:inference", "*") {
        // Return 403 insufficient permissions
    }
 }
 ```
 ### Authentik-side
 - [ ] Enable ROPC grant in Authentik provider settings (if not already)
 - [ ] Add redirect URIs to `local-llm` app if browser OAuth flow needed:
 ```python
 # k8s/infra/iam/scripts/authentik-provision.py
 "local-llm": {
    ...
    "redirect_uris": [
        "http://localhost:3000/api/auth/callback",
        "https://your-app.com/api/auth/callback",
    ],
 }
 ```
 ### NextJS-side
 - [ ] Until gateway auth is wired, LLM API works without token
 - [ ] Once wired, add `Authorization: Bearer <token>` to all LLM requests
@@ -208,3 +208,95 @@ versions without warning in your own values file.
 Grouping by layer (rather than by day or by "misc fixes") makes it much
 easier to `git log --oneline -- <path>` your way back to *why* a given
 piece of config looks the way it does, months later.
 ## Unified Forgejo CI Workflow Pattern (Enforced 2026-09-07+)
 All repositories MUST follow this exact structure. No variations.
 ```yaml
 name: CI
 on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
 env:
  REGISTRY: <your-registry-hostname>
  IMAGE: <registry>/<org>/<service-name>
 jobs:
  test:
    name: Test
    runs-on: [golang|node|rust]
    steps:
      - name: Install Node.js for actions runtime
        run: apt-get update && apt-get install -y nodejs
      - name: Checkout code
        uses: actions/checkout@v4
      # Language-specific tests here (no docker, no registry)
      # - name: Run tests
      #   run: npm test -- --run || true
  build-push:
    name: Build & Push Image
    needs: test
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: [golang|node|rust]
    steps:
      - name: Install Node.js and Docker
        run: |
          apt-get update
          apt-get install -y nodejs docker.io
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Get short SHA
        id: sha
        run: |
          SHORT_SHA=$(git rev-parse --short HEAD)
          echo "short_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
      - name: Registry login
        run: |
          echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
            --username "${REGISTRY_USER}" --password-stdin
        env:
          REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
          REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
      - name: Build Docker image
        run: |
          docker build --no-cache \
            -t "${IMAGE}:${{ steps.sha.outputs.short_sha }}" \
            -t "${IMAGE}:latest" \
            .
      - name: Push Docker image
        run: |
          docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}"
          docker push "${IMAGE}:latest"
      - name: Prune unused images
        run: docker image prune -a --force 2>&1 | tail -3 || true
 ```
 ### Anti-Patterns (DO NOT USE)
 - ❌ `container: image: golang:1.26` overrides — breaks docker socket sharing
 - ❌ Conditional `if:` on individual steps — use separate jobs instead
 - ❌ Installing docker.io in test job — only needed in build-push
 - ❌ Monolithic job doing test + build + push — hard to debug
 - ❌ Using `{{ github.sha }}` for image tag — use short commit SHA for readability
 ### How It Works
 1. **PR to feature branch** → test job runs, build-push skipped, nothing pushed
 2. **Push to main** → test runs, build-push runs after test passes, image pushed
 3. Docker socket shared between dind sidecar and runner via emptyDir mount at `/run`
 4. `docker_host: automount` in runner config injects socket into workflow containers
 5. Secrets (FORGEJO_REGISTRY_USER, TOKEN) set in Forgejo repo settings, NOT in git
@@ -42,6 +42,86 @@ All logs + metrics centralized in Grafana for debugging
 - **Secrets at rest** — Vault + encrypted etcd; credentials never in logs or ConfigMaps
 - **Infrastructure-as-code** — Every service deployed via Helmfile; one `helmfile apply` recovers from total failure
 ## ArgoCD — GitOps Deployment Flow
 **ArgoCD** pulls infrastructure changes from git and syncs the cluster automatically. 
 No manual `kubectl apply` — push to git, ArgoCD detects the change, and deploys within ~3 minutes.
 ```
 Developer pushes to git
       ↓
 ArgoCD detects change (every 3 min or webhook)
       ↓
 Syncs manifests to cluster
       ↓
 Workloads reconcile automatically
 ```
 Applications are deployed in waves (numbered 00, 10, 20, 30, ...) to respect dependencies — 
 storage deploys before databases, databases before applications.
 ### Tracked Git Repositories
 ArgoCD monitors these repos for changes:
 | Repository | Purpose |
 |------------|----------|
 | `https://github.com/Riotpiaole/riotpiao.homelab.com` | Main infrastructure repo (all manifests in `k8s/argocd/apps/`) |
 | `https://forgejo.riotpiao.com/rock/*` | Any `rock/*` repo in in-cluster Forgejo (apps + configs) |
 | `https://github.com/Riotpiaole/Poimen-*` | External Poimen services (memory, workflows) |
 To deploy a new application: create a git repo, add an Application manifest to the homelab repo's 
 `k8s/argocd/apps/`, commit + push, and ArgoCD syncs within 3 minutes.
 ## Management Planes — Talos vs Kubernetes
 This cluster has **two separate management planes**, each with different workflows:
 | Plane | What it manages | Workflow | Tool |
 |-------|-----------------|----------|------|
 | **Talos (OS)** | Node configuration, kernel params, networking, CoreDNS, machine state | Edit `terraform/` → `terraform apply` → `make apply-cp` | `terraform` + `talosctl` |
 | **Kubernetes (workloads)** | All pods, services, deployments, ingresses, databases | Edit `k8s/argocd/apps/` → `git push` → ArgoCD syncs | `git` + ArgoCD |
 **Critical distinction:**
 - **Kubernetes resources** (`k8s/**`) flow through **git → ArgoCD** — never use `kubectl apply`
 - **Talos machine config** (`terraform/**`) uses **local `terraform apply`** (sanctioned exception — CI can't hold node credentials)
 Example: To add a CoreDNS hostname rewrite, you edit `terraform/files/coredns/Corefile`, then:
 ```bash
 cd terraform && terraform apply -var-file=terraform.tfvars.local
 cd .. && make apply-cp     # talosctl apply-config to all 3 control planes
 ```
 But to add a new Kubernetes Deployment or update an Ingress, you only `git push` — **never `kubectl apply`**.
 ### CoreDNS ConfigMap Ownership — Critical
 ⚠️ **Warning:** The `coredns` ConfigMap in `kube-system` namespace is **owned by Talos**, not ArgoCD or kubectl. 
 It is rendered from `terraform/files/coredns/Corefile` into Talos's machine config at bootstrap time.
 **Do not `kubectl apply` or `kubectl edit` this ConfigMap directly.** Doing so transfers field ownership to kubectl's 
 client-side-apply mechanism, and Talos's inline-manifest controller will silently no-op on every future reconcile 
 (server-side-apply conflict, no error surfaced).
 **To update CoreDNS (e.g., add a hostname rewrite):**
 1. Edit `terraform/files/coredns/Corefile`
 2. Commit + push
 3. Run `cd terraform && terraform apply -var-file=terraform.tfvars.local`
 4. Run `make apply-cp` to push config to all control planes
 5. CoreDNS picks up changes via its `reload` plugin — no pod restart needed
 **If you accidentally edited the ConfigMap directly and broke Talos's ownership:**
 ```bash
 kubectl delete configmap coredns -n kube-system
 # Wait ~30s for Talos's k8s.ManifestApplyController to recreate it
 kubectl get configmap coredns -n kube-system -w
 ```
 Or as a stopgap, apply the correct content yourself:
 ```bash
 kubectl apply --server-side -f <(terraform output coredns_config)
 ```
 ## Quick Start — Deploying the Cluster
 ### 1. Bootstrap Talos Nodes
@@ -0,0 +1,80 @@
 # ComfyUI — GPU-accelerated image generation on worker-1.
 # Uses 1x V100 32GB (sm70). Freed by scaling ornith 2→1.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: comfyui
  namespace: comfyui
  labels:
    app: comfyui
 spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: comfyui
  template:
    metadata:
      labels:
        app: comfyui
    spec:
      nodeSelector:
        kubernetes.io/hostname: worker-1
      runtimeClassName: nvidia
      containers:
        - name: comfyui
          image: ghcr.io/ai-dock/comfyui:v2-cuda-12.1.1-base-22.04
          ports:
            - containerPort: 8188
              protocol: TCP
          env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: "all"
            - name: COMFYUI_FLAGS
              value: "--listen 0.0.0.0 --port 8188"
          resources:
            requests:
              cpu: "4"
              memory: 8Gi
              nvidia.com/gpu: "1"
            limits:
              cpu: "8"
              memory: 16Gi
              nvidia.com/gpu: "1"
          volumeMounts:
            - mountPath: /workspace/ComfyUI/models
              name: models
            - mountPath: /workspace/ComfyUI/output
              name: output
          readinessProbe:
            httpGet:
              path: /
              port: 8188
            periodSeconds: 10
            initialDelaySeconds: 30
          startupProbe:
            httpGet:
              path: /
              port: 8188
            failureThreshold: 60
            periodSeconds: 10
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: comfyui-models
        - name: output
          emptyDir: {}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: comfyui-models
  namespace: comfyui
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 50Gi
@@ -0,0 +1,25 @@
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: comfyui
  namespace: comfyui
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    # WebSocket support for ComfyUI's live preview
    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
    nginx.ingress.kubernetes.io/proxy-set-headers: "Upgrade"
 spec:
  ingressClassName: nginx
  rules:
    - host: comfy.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: comfyui
                port:
                  number: 80
@@ -0,0 +1,7 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
  - deployment.yaml
  - service.yaml
  - ingress.yaml
@@ -0,0 +1,14 @@
 apiVersion: v1
 kind: Service
 metadata:
  name: comfyui
  namespace: comfyui
  labels:
    app: comfyui
 spec:
  selector:
    app: comfyui
  ports:
    - port: 80
      targetPort: 8188
      protocol: TCP
@@ -0,0 +1,107 @@
 # Gotify — Push Notifications + Email Relay
 Self-hosted notification server with SMTP email forwarding sidecar.
 ## Architecture
 ```
 Forgejo webhook ──POST──→ Gotify API (:80/message)
                              │
                    ┌─────────┼─────────┐
                    ▼                     ▼
              Push notification      SMTP emailer sidecar
              (mobile/desktop)       (polls → sends email)
 ```
 ## Setup (one-time, after first deploy)
 ### 1. Encrypt secrets before committing
 ```bash
 # Edit secrets.yaml with real values first, then:
 sops -e -i k8s/apps/gotify/secrets.yaml
 ```
 ### 2. Create Gotify app + client tokens
 1. Login to `https://gotify.riotpiao.com` with admin creds
 2. **Applications** → Create `forgejo` → copy **app token**
 3. **Clients** → Create `smtp-emailer` → copy **client token**
 4. Update `gotify-tokens` secret:
   ```bash
   kubectl -n notifications create secret generic gotify-tokens \
     --from-literal=app-token=<APP_TOKEN> \
     --from-literal=client-token=<CLIENT_TOKEN> \
     --dry-run=client -o yaml | kubectl apply -f -
   ```
 ### 3. Configure Forgejo webhook
 In each Forgejo repo → **Settings** → **Webhooks** → **Add Webhook** → **Gotify**:
 | Field | Value |
 |-------|-------|
 | Target URL | `http://gotify.notifications.svc.cluster.local/message` |
 | Token | The **app token** from step 2 |
 | Events | Pull Request (Created, Merged, Closed) |
 Or via API:
 ```bash
 FORGEJO_TOKEN="<your-pat>"
 APP_TOKEN="<gotify-app-token>"
 curl -s -X POST "https://forgejo.riotpiao.com/api/v1/repos/rock/homelab/hooks" \
  -H "Authorization: token $FORGEJO_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "gotify",
    "active": true,
    "config": {
      "content_type": "json",
      "url": "http://gotify.notifications.svc.cluster.local/message?token='"$APP_TOKEN"'"
    },
    "events": ["pull_request", "pull_request_assign", "pull_request_review"],
    "authorization_header": ""
  }'
 ```
 ### 4. Add CoreDNS rewrite (if accessing via public hostname)
 Only needed if Cloudflare Tunnel is used for gotify.riotpiao.com:
 ```
 # terraform/files/coredns/Corefile — add rewrite:
 rewrite name gotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
 ```
 Then: `cd terraform && terraform apply && cd .. && make apply-cp`
 ### 5. SMTP providers
 | Provider | Host | Port | Notes |
 |----------|------|------|-------|
 | Gmail | smtp.gmail.com | 587 | Use App Password (2FA required) |
 | Resend | smtp.resend.com | 587 | Free 100 emails/day |
 | Sendgrid | smtp.sendgrid.net | 587 | Free 100 emails/day |
 | Mailgun | smtp.mailgun.org | 587 | Free 5000/month |
 ## Notification priority levels
 | Priority | Meaning | Email forwarded? |
 |----------|---------|-----------------|
 | 0-4 | Low (info) | No (below MIN_PRIORITY=5) |
 | 5-7 | Normal (PR created) | Yes |
 | 8-10 | High (PR merged, failures) | Yes |
 ## Verify
 ```bash
 # Test push notification
 APP_TOKEN="<app-token>"
 curl -X POST "https://gotify.riotpiao.com/message?token=$APP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title":"Test","message":"Hello from homelab","priority":5}'
 # Check email sidecar logs
 kubectl -n notifications logs deployment/gotify -c smtp-emailer --tail=20
 ```
@@ -0,0 +1,35 @@
 # CNPG Postgres for Gotify. Lightweight — 2 instances, 2Gi storage.
 # CNPG generates secret `gotify-db-app` + service `gotify-db-rw` in ns notifications.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
  name: gotify-db
  namespace: notifications
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
 spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
  bootstrap:
    initdb:
      database: gotify
      owner: app
      encoding: UTF8
      localeCollate: C
      localeCType: C
  enableSuperuserAccess: false
  resources:
    requests: { memory: "256Mi", cpu: "100m" }
    limits: { memory: "512Mi", cpu: "500m" }
  storage:
    size: 2Gi
    storageClass: longhorn-cnpg
  monitoring:
    enablePodMonitor: true
  affinity:
    podAntiAffinityType: preferred
    topologyKey: kubernetes.io/hostname
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
@@ -0,0 +1,204 @@
 # Gotify — self-hosted push notification server + SMTP email relay.
 # Forgejo webhooks → Gotify → push notifications + email forwarding.
 # Runs on control plane (no GPU needed), lightweight.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: gotify
  namespace: notifications
  labels:
    app: gotify
 spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: gotify
  template:
    metadata:
      labels:
        app: gotify
    spec:
      containers:
        # --- Gotify server ---
        - name: gotify
          image: ghcr.io/gotify/server:2.6.1
          command: ["/bin/sh", "-c"]
          args:
            - |
              export GOTIFY_DATABASE_DIALECT=postgres
              export GOTIFY_DATABASE_CONNECTION="host=gotify-db-rw.notifications port=5432 user=${DB_USER} password=${DB_PASS} dbname=gotify sslmode=disable"
              exec /app/gotify-app
          ports:
            - containerPort: 80
              protocol: TCP
          env:
            - name: GOTIFY_DEFAULTUSER_NAME
              valueFrom:
                secretKeyRef:
                  name: gotify-admin
                  key: username
            - name: GOTIFY_DEFAULTUSER_PASS
              valueFrom:
                secretKeyRef:
                  name: gotify-admin
                  key: password
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: gotify-db-app
                  key: username
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: gotify-db-app
                  key: password
            - name: GOTIFY_SERVER_PORT
              value: "80"
            - name: GOTIFY_SERVER_KEEPALIVEPERIODSECONDS
              value: "0"
            - name: TZ
              value: Asia/Tokyo
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi
          livenessProbe:
            httpGet:
              path: /health
              port: 80
            periodSeconds: 30
            initialDelaySeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 80
            periodSeconds: 10
            initialDelaySeconds: 5
        # --- SMTP emailer sidecar ---
        # Watches Gotify WebSocket stream, forwards messages as email.
        # https://github.com/eternal-flame-AD/gotify-broadcast
        - name: smtp-emailer
          image: ghcr.io/gotify/server:2.6.1
          command:
            - /bin/sh
            - -c
            - |
              # Wait for Gotify to be ready
              until wget -qO- http://localhost:80/health >/dev/null 2>&1; do
                echo "Waiting for Gotify..."
                sleep 2
              done
              echo "Gotify is ready, starting email relay..."
              # Poll Gotify messages and forward via SMTP using msmtp
              # Install msmtp for lightweight SMTP sending
              apk add --no-cache msmtp curl jq
              # Configure msmtp
              cat > /tmp/msmtprc <<MSMTP
              defaults
              auth           on
              tls            on
              tls_trust_file /etc/ssl/certs/ca-certificates.crt
              logfile        /tmp/msmtp.log
              account        default
              host           ${SMTP_HOST}
              port           ${SMTP_PORT}
              from           ${SMTP_FROM}
              user           ${SMTP_USER}
              password       ${SMTP_PASS}
              MSMTP
              chmod 600 /tmp/msmtprc
              # Track last seen message ID
              LAST_ID=0
              while true; do
                # Fetch messages since last ID
                MESSAGES=$(curl -s -H "X-Gotify-Key: ${GOTIFY_CLIENT_TOKEN}" \
                  "http://localhost:80/message?since=${LAST_ID}&limit=10" 2>/dev/null)
                if [ -n "$MESSAGES" ]; then
                  echo "$MESSAGES" | jq -r '.messages[]? | @base64' | while read -r MSG; do
                    DECODED=$(echo "$MSG" | base64 -d)
                    ID=$(echo "$DECODED" | jq -r '.id')
                    TITLE=$(echo "$DECODED" | jq -r '.title // "Notification"')
                    BODY=$(echo "$DECODED" | jq -r '.message // ""')
                    PRIORITY=$(echo "$DECODED" | jq -r '.priority // 5')
                    APP=$(echo "$DECODED" | jq -r '.appid // 0')
                    DATE=$(echo "$DECODED" | jq -r '.date // ""')
                    # Only forward messages with priority >= configured threshold
                    if [ "$PRIORITY" -ge "${MIN_PRIORITY:-0}" ]; then
                      printf "Subject: [Gotify] %s\nFrom: %s\nTo: %s\nContent-Type: text/plain; charset=UTF-8\n\n%s\n\n---\nPriority: %s\nDate: %s" \
                        "$TITLE" "$SMTP_FROM" "$NOTIFY_EMAIL" "$BODY" "$PRIORITY" "$DATE" | \
                        msmtp -C /tmp/msmtprc "$NOTIFY_EMAIL" && \
                        echo "Email sent for message $ID: $TITLE" || \
                        echo "Failed to send email for message $ID"
                    fi
                    # Update last seen ID
                    if [ "$ID" -gt "$LAST_ID" ]; then
                      LAST_ID=$ID
                    fi
                  done
                fi
                sleep ${POLL_INTERVAL:-30}
              done
          env:
            - name: GOTIFY_CLIENT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: gotify-tokens
                  key: client-token
            - name: SMTP_HOST
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: host
            - name: SMTP_PORT
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: port
            - name: SMTP_FROM
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: from
            - name: SMTP_USER
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: user
            - name: SMTP_PASS
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: password
            - name: NOTIFY_EMAIL
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: notify-email
            - name: MIN_PRIORITY
              value: "5"
            - name: POLL_INTERVAL
              value: "15"
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
            limits:
              cpu: 100m
              memory: 64Mi
      # No volumes — Postgres handles persistence
@@ -0,0 +1,26 @@
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: gotify
  namespace: notifications
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    # WebSocket support for Gotify client connections
    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
 spec:
  ingressClassName: nginx
  rules:
    - host: gotify.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gotify
                port:
                  number: 80
@@ -0,0 +1,8 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
  - namespace.yaml
  - db.yaml
  - deployment.yaml
  - service.yaml
  - ingress.yaml
@@ -0,0 +1,6 @@
 apiVersion: v1
 kind: Namespace
 metadata:
  name: notifications
  labels:
    kubernetes.io/metadata.name: notifications
@@ -0,0 +1,14 @@
 apiVersion: v1
 kind: Service
 metadata:
  name: gotify
  namespace: notifications
  labels:
    app: gotify
 spec:
  selector:
    app: gotify
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
@@ -0,0 +1,14 @@
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: immich-config
 data:
  DB_HOSTNAME: "immich-db-rw"
  DB_DATABASE_NAME: "immich"
  # Only pgvector is installed (see db.yaml) - no vectorchord extension image
  # exists for pg18 in CNPG's catalog yet. Explicit instead of relying on
  # auto-detect's vectorchord-first preference order.
  DB_VECTOR_EXTENSION: "pgvector"
  REDIS_HOSTNAME: "immich-redis"
  IMMICH_MACHINE_LEARNING_URL: "http://immich-machine-learning:3003"
  TZ: "America/Los_Angeles"
@@ -0,0 +1,58 @@
 # Dedicated CNPG Postgres for Immich. Same recipe as paperless-db/authentik-db
 # (2 instances, default longhorn storage class) except the operand is
 # PostgreSQL 18, not 16.2 - the official CNPG pgvector extension image
 # (ghcr.io/cloudnative-pg/pgvector) is only published for pg18, no pg16 tags
 # exist in that registry. Immich itself supports pg18 fine (immich-app's own
 # postgres image already ships 18-vectorchord builds).
 #
 # pgvector loaded via CNPG's ImageVolume extension mechanism (CNPG 1.27+,
 # k8s ImageVolume feature - both present here: operator is 1.30.0, cluster is
 # v1.36.1). No shared_preload_libraries needed - pgvector doesn't require
 # preload, just CREATE EXTENSION, which immich-server issues itself at
 # startup. Distro/pg-major must match between the operand image and the
 # extension image (both "18"+"trixie" here) - CNPG's own compatibility rule.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
  name: immich-db
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
 spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:18-minimal-trixie
  postgresql:
    extensions:
      - name: pgvector
        image:
          reference: ghcr.io/cloudnative-pg/pgvector:0.8.1-18-trixie
  bootstrap:
    initdb:
      database: immich
      owner: app
      encoding: UTF8
      localeCollate: C
      localeCType: C
      # CREATE EXTENSION vector requires superuser (pgvector's control file
      # isn't marked trusted) and the "app" owner role isn't one
      # (enableSuperuserAccess: false, repo convention) - postInitApplicationSQL
      # runs as superuser during initdb, before the app ever connects. Only
      # fires on a fresh bootstrap; the live cluster already had this run
      # manually once (kubectl exec ... psql -U postgres -c 'CREATE EXTENSION').
      postInitApplicationSQL:
        - "CREATE EXTENSION IF NOT EXISTS vector;"
        - "CREATE EXTENSION IF NOT EXISTS cube;"
        - "CREATE EXTENSION IF NOT EXISTS earthdistance;"
  enableSuperuserAccess: false
  resources:
    requests: { memory: "512Mi", cpu: "250m" }
    limits: { memory: "2Gi", cpu: "1" }
  storage:
    size: 20Gi
    storageClass: longhorn
  affinity:
    podAntiAffinityType: preferred
    topologyKey: kubernetes.io/hostname
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
@@ -0,0 +1,100 @@
 # immich-server: pinned to talos-cp-3, same reasoning as paperless
 # (deployment.yaml comment there) - immich-media is a ReadWriteOnce Longhorn
 # volume with a single replica physically on that node's disk (shared with
 # paperless-media on the same 4TB HDD). Recreate strategy for the same
 # reason: two pods can't both attach an RWO volume.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: immich-server
 spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: immich-server
  template:
    metadata:
      labels:
        app: immich-server
    spec:
      serviceAccountName: immich
      nodeSelector:
        kubernetes.io/hostname: talos-cp-3
      containers:
        - name: immich-server
          image: ghcr.io/immich-app/immich-server:release
          ports:
            - containerPort: 2283
          envFrom:
            - configMapRef:
                name: immich-config
          env:
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: immich-db-app
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: immich-db-app
                  key: password
            # Composed by k8s/infra/iam's provisioning script (system-config
            # JSON, oauth section) - see immich-oidc Secret.
            - name: IMMICH_CONFIG_FILE
              value: /config/immich.json
          resources:
            requests: { cpu: "500m", memory: "1Gi" }
            limits: { cpu: "2", memory: "4Gi" }
          volumeMounts:
            - name: media
              mountPath: /usr/src/app/upload
            - name: oidc-config
              mountPath: /config
              readOnly: true
      volumes:
        - name: media
          persistentVolumeClaim:
            claimName: immich-media
        - name: oidc-config
          secret:
            secretName: immich-oidc
            items:
              - key: config.json
                path: immich.json
 ---
 # CPU-only for now - the cluster's one GPU node (worker-1) is already
 # dedicated to llm-serving predictors. Not node-pinned: its cache PVC is on
 # the default 3-replica pool, not the single-disk cp-3 HDD.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: immich-machine-learning
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: immich-machine-learning
  template:
    metadata:
      labels:
        app: immich-machine-learning
    spec:
      serviceAccountName: immich
      containers:
        - name: immich-machine-learning
          image: ghcr.io/immich-app/immich-machine-learning:release
          ports:
            - containerPort: 3003
          resources:
            requests: { cpu: "500m", memory: "1Gi" }
            limits: { cpu: "2", memory: "4Gi" }
          volumeMounts:
            - name: ml-cache
              mountPath: /cache
      volumes:
        - name: ml-cache
          persistentVolumeClaim:
            claimName: immich-ml-cache
@@ -0,0 +1,24 @@
 # Direct nginx ingress, same reasoning as paperless: large uploads (photos/
 # videos) and long-lived operations (video transcode, big batch uploads) need
 # proxy-body-size/timeouts raised past nginx's defaults.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: immich
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
 spec:
  ingressClassName: nginx
  rules:
    - host: img.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: immich-server
                port:
                  number: 2283
@@ -0,0 +1,14 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: immich
 resources:
  - db.yaml
  - pvc.yaml
  - configmap.yaml
  - redis.yaml
  - deployment.yaml
  - service.yaml
  - ingress.yaml
  - rbac.yaml
 # immich-oidc Secret written by the PostSync provisioning Job in
 # k8s/infra/iam (same as paperless-oidc) - not duplicated here.
@@ -0,0 +1,38 @@
 # Two volumes:
 #
 # - media: original photos/videos + generated thumbnails/encoded videos.
 #   Shares the cp-3 USB HDD with paperless-media, same StorageClass/disk tag,
 #   single replica (single disk, no redundancy possible - same tradeoff
 #   paperless already accepts). Sized 1400Gi, not 2000Gi: the disk's real
 #   usable capacity (~3724GiB, formatting overhead) minus paperless-media's
 #   2000Gi and ~231GiB of other apps' default-class replicas that Longhorn
 #   placed here anyway (disk tags only pull matching volumes in, they don't
 #   exclude non-matching ones when the untagged pool elsewhere is full) only
 #   leaves ~1493Gi of real scheduling headroom right now.
 # - ml-cache: downloaded ML model weights for immich-machine-learning
 #   (face detection / CLIP embeddings). Small, disposable (re-downloads on
 #   loss), but persisted so a pod restart doesn't re-pull multi-GB models -
 #   default 3-replica pool, not node-pinned.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: immich-media
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn-paperless-media
  resources:
    requests:
      storage: 1400Gi
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: immich-ml-cache
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
@@ -0,0 +1,43 @@
 # Scoped operator access for immich-admins: restart/config-edit rights on
 # just this service's own resources, nothing CNPG-managed (immich-db-*) or
 # provisioning-managed (immich-oidc). Same pattern as
 # k8s/apps/paperless/rbac.yaml. Inert until kube-apiserver's OIDC wiring
 # lands (--oidc-groups-claim=groups, --oidc-groups-prefix=oidc:).
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: immich
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
  name: immich-operator
 rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    resourceNames: ["immich-server", "immich-machine-learning"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["immich-config"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["immich-oidc"]
    verbs: ["get", "list", "watch", "update", "patch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: immich-admins-binding
 subjects:
  - kind: Group
    name: "oidc:immich-admins"
    apiGroup: rbac.authorization.k8s.io
  - kind: ServiceAccount
    name: immich
    namespace: immich
 roleRef:
  kind: Role
  name: immich-operator
  apiGroup: rbac.authorization.k8s.io
@@ -0,0 +1,37 @@
 # Job queue broker for immich-server. No PVC: queue state is disposable - a
 # lost queue on restart just re-triggers the affected background jobs
 # (thumbnail generation, ML jobs, etc.), no photo data loss since originals
 # live on immich-media.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: immich-redis
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: immich-redis
  template:
    metadata:
      labels:
        app: immich-redis
    spec:
      containers:
        - name: redis
          image: redis:7-alpine
          ports:
            - containerPort: 6379
          resources:
            requests: { cpu: "50m", memory: "64Mi" }
            limits: { cpu: "250m", memory: "256Mi" }
 ---
 apiVersion: v1
 kind: Service
 metadata:
  name: immich-redis
 spec:
  selector:
    app: immich-redis
  ports:
    - port: 6379
      targetPort: 6379
@@ -0,0 +1,21 @@
 apiVersion: v1
 kind: Service
 metadata:
  name: immich-server
 spec:
  selector:
    app: immich-server
  ports:
    - port: 2283
      targetPort: 2283
 ---
 apiVersion: v1
 kind: Service
 metadata:
  name: immich-machine-learning
 spec:
  selector:
    app: immich-machine-learning
  ports:
    - port: 3003
      targetPort: 3003
@@ -14,5 +14,7 @@ resources:
  - ornith.yaml
  - reasoning.yaml
  - reranker.yaml
  - qwen-cpu.yaml
  - networkpolicy.yaml
 # No namespace transformer: every file sets its own, and the transformer would
 # rewrite metadata.namespace on anything cross-namespace added later.
@@ -0,0 +1,62 @@
 # NetworkPolicy for LLM inference engines (llm-serving namespace).
 #
 # These pods have NO auth — vLLM, Ollama, and TEI accept any request.
 # All access MUST go through the api-gateway, which validates JWTs and
 # injects identity headers (X-Forwarded-User, X-Auth-Verified).
 #
 # Replaces the hand-applied llm-serving-default-deny policy that used
 # `llm-client: "true"` pod label as a selector — any pod in any namespace
 # could self-grant access by adding that label, which defeats the purpose.
 #
 # This policy restricts ingress to:
 #   1. api namespace (gateway) — the sole entry point for inference
 #   2. monitoring namespace — Prometheus scraping vLLM/TEI /metrics
 #   3. intra-namespace — pod-to-pod (future: multi-replica comms)
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
  name: llm-serving-ingress
  namespace: llm-serving
  labels:
    app.kubernetes.io/part-of: llm-serving
 spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: llm-serving
  policyTypes:
    - Ingress
  ingress:
    # Allow from api-gateway (namespace: api)
    # Gateway proxies /v1/chat/completions, /v1/embeddings, /v1/rerank
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: api
      ports:
        - protocol: TCP
          port: 8080   # vLLM, Ollama HTTP
        - protocol: TCP
          port: 80      # KServe predictor services
        - protocol: TCP
          port: 8000    # vLLM direct (some configs)
        - protocol: TCP
          port: 11434   # Ollama native port
    # Allow Prometheus scraping from monitoring namespace
    # vLLM: :8080/metrics, TEI: :9000/metrics
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - protocol: TCP
          port: 8080
        - protocol: TCP
          port: 9000
    # Allow intra-namespace (pod-to-pod within llm-serving)
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/part-of: llm-serving
      ports:
        - protocol: TCP
          port: 8080
@@ -33,12 +33,8 @@ spec:
        ollama pull ornith:35b
        ollama pull qwen2.5:3b-instruct
        ollama run ornith:35b "ok" >/dev/null 2>&1 || true
        ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
        wait $SERVE_PID
        '
@@ -54,7 +50,7 @@ spec:
      - name: OLLAMA_NUM_PARALLEL
        value: '1'
      - name: OLLAMA_MAX_LOADED_MODELS
-        value: '2'
+        value: '1'
      image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
      name: kserve-container
      ports:
@@ -65,8 +61,7 @@ spec:
          command:
          - /bin/sh
          - -c
-          - ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
+          - ollama ps 2>/dev/null | grep -q ornith
            grep -q qwen2.5
        periodSeconds: 10
      resources:
        limits:
@@ -82,8 +77,7 @@ spec:
          command:
          - /bin/sh
          - -c
-          - ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
+          - ollama ps 2>/dev/null | grep -q ornith
            grep -q qwen2.5
        failureThreshold: 120
        periodSeconds: 15
      volumeMounts:
@@ -91,13 +85,10 @@ spec:
        name: models
    deploymentStrategy:
      type: Recreate
-    # 2 replicas -- each its own GPU, each loading both ornith:35b and
+    # 1 replica -- ornith:35b only. qwen2.5:3b moved to CPU on cp-2.
-    # qwen2.5:3b-instruct -- so 2 concurrent implementer-style calls each
+    # Frees 1 GPU for ComfyUI.
-    # get an independent instance instead of contending on one, at the
+    maxReplicas: 1
-    # cost of judge/qwen traffic still sharing whichever replica an
+    minReplicas: 1
    # implementer call also lands on.
    maxReplicas: 2
    minReplicas: 2
    nodeSelector:
      kubernetes.io/hostname: worker-1
    runtimeClassName: nvidia
@@ -0,0 +1,115 @@
 # qwen2.5:3b-instruct on CPU (talos-cp-2, 144GB RAM, 24 cores).
 # Moved off GPU to free a V100 for ComfyUI. Latency ~10x slower
 # than GPU but sufficient for lightweight tasks (summarization,
 # classification, quick answers).
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: qwen-cpu
  namespace: llm-serving
  labels:
    app: qwen-cpu
    app.kubernetes.io/name: qwen-cpu
    app.kubernetes.io/part-of: llm-serving
 spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: qwen-cpu
  template:
    metadata:
      labels:
        app: qwen-cpu
        app.kubernetes.io/name: qwen-cpu
        app.kubernetes.io/part-of: llm-serving
    spec:
      nodeSelector:
        kubernetes.io/hostname: talos-cp-2
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: ollama
          image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
          command: ["/bin/sh", "-c"]
          args:
            - |
              ollama serve &
              SERVE_PID=$!
              until ollama list >/dev/null 2>&1; do sleep 2; done
              ollama pull qwen2.5:3b-instruct
              ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
              wait $SERVE_PID
          env:
            - name: OLLAMA_HOST
              value: "0.0.0.0:8080"
            - name: OLLAMA_MODELS
              value: /root/.ollama/models
            - name: OLLAMA_CONTEXT_LENGTH
              value: "32768"
            - name: OLLAMA_KEEP_ALIVE
              value: "-1"
            - name: OLLAMA_MAX_LOADED_MODELS
              value: "1"
            - name: OLLAMA_NUM_PARALLEL
              value: "2"
          ports:
            - containerPort: 8080
              protocol: TCP
          readinessProbe:
            exec:
              command: ["/bin/sh", "-c", "ollama ps 2>/dev/null | grep -q qwen2.5"]
            periodSeconds: 10
          startupProbe:
            exec:
              command: ["/bin/sh", "-c", "ollama ps 2>/dev/null | grep -q qwen2.5"]
            failureThreshold: 60
            periodSeconds: 10
          resources:
            requests:
              cpu: "4"
              memory: 4Gi
            limits:
              cpu: "8"
              memory: 8Gi
          volumeMounts:
            - mountPath: /root/.ollama
              name: ollama-data
      volumes:
        - name: ollama-data
          persistentVolumeClaim:
            claimName: qwen-cpu-data
 ---
 apiVersion: v1
 kind: Service
 metadata:
  name: qwen-cpu
  namespace: llm-serving
  labels:
    app: qwen-cpu
    app.kubernetes.io/part-of: llm-serving
 spec:
  selector:
    app: qwen-cpu
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
 ---
 # Small PVC for qwen2.5:3b model weights (~1.9GB).
 # Separate from llm-models PVC which is pinned to worker-1.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: qwen-cpu-data
  namespace: llm-serving
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
@@ -13,6 +13,7 @@ spec:
      labels:
        app: management-service
    spec:
      serviceAccountName: kmsvc
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
@@ -2,7 +2,7 @@ namespace: sqs
 replicaCount: 3
 image:
-  repository: ghcr.io/riotpiaole/kmsvc-management-service
+  repository: forgejo.riotpiao.com/rock/kmsvc-manage
  tag: latest
  pullPolicy: Always
@@ -0,0 +1,6 @@
 apiVersion: v2
 name: memory-queues
 description: Kafka queues (DLQ) for Poimen Memory service (Phase 6.6)
 type: application
 version: 0.1.0
 appVersion: "1.0"
@@ -0,0 +1,20 @@
 {{- range .Values.queues }}
 ---
 apiVersion: kmsvc.io/v1alpha1
 kind: Queue
 metadata:
  name: {{ .name }}
  namespace: {{ $.Values.namespace }}
  labels:
    app: memory-service
    queue: dlq
 spec:
  name: {{ .name }}
  description: {{ .description }}
  partitions: {{ .partitions }}
  replicationFactor: {{ .replicationFactor }}
  config:
    retention.ms: "{{ .config.retention.ms }}"
    message.retention.seconds: "{{ .config.message.retention.seconds }}"
    visibility.timeout.seconds: "{{ .config.visibility.timeout.seconds }}"
 {{- end }}
@@ -0,0 +1,25 @@
 # Poimen Memory Service Kafka Queues (kmsvc)
 # Phase 6.6: DLQ topics for webhook + metrics failures
 queues:
  # DLQ for extraction, webhook, and agent failures
  - name: poimen-memory-dlq
    description: "DLQ for extraction, webhook, and agent failures"
    partitions: 3
    replicationFactor: 1
    config:
      retention.ms: "1209600000"  # 14 days
      message.retention.seconds: "1209600"
      visibility.timeout.seconds: "300"
  # DLQ for metrics persistence failures  
  - name: poimen-memory-metric-dlq
    description: "DLQ for metrics persistence failures"
    partitions: 3
    replicationFactor: 1
    config:
      retention.ms: "1209600000"  # 14 days
      message.retention.seconds: "1209600"
      visibility.timeout.seconds: "300"
 namespace: sqs
@@ -1,7 +1,7 @@
 namespace: sqs
 image:
-  repository: ghcr.io/riotpiaole/kmsvc-management-service
+  repository: forgejo.riotpiao.com/rock/kmsvc-manage
  tag: latest
  pullPolicy: Always
@@ -0,0 +1,95 @@
 # Overrides paperless-ngx's own paperless/adapter.py at the same import path
 # (mounted via subPath in deployment.yaml) - settings.py hardcodes
 # SOCIALACCOUNT_ADAPTER = "paperless.adapter.CustomSocialAccountAdapter", so
 # no Django setting needs to change, just the file content underneath it.
 #
 # Stock CustomSocialAccountAdapter.populate_user() is a stub ("kept in case
 # global default permissions are implemented in the future" - they aren't),
 # so every OIDC signup lands with zero permissions and 403s on every API
 # endpoint. This adds the actual mapping: Authentik's "permissions" claim
 # (via the permissions scope, requested in PAPERLESS_SOCIALACCOUNT_PROVIDERS,
 # computed server-side from group membership by authentik-provision.py) ->
 # "paperless:write" or "*" (homelab-admins) grants is_staff+is_superuser,
 # same convention already used for MinIO's policy claim and Grafana's
 # role_attribute_path. Checking the permission string rather than a literal
 # group name decouples "what grants access" from which group happens to
 # hold it - same pattern applies to every other service's Role/RoleBinding
 # in k8s/infra/rbac/.
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: paperless-adapter
 data:
  adapter.py: |
    from urllib.parse import quote
    from allauth.account.adapter import DefaultAccountAdapter
    from allauth.core import context
    from allauth.socialaccount.adapter import DefaultSocialAccountAdapter
    from django.conf import settings
    from django.forms import ValidationError
    from django.urls import reverse
    REQUIRED_PERMISSIONS = {"paperless:write", "*"}
    class CustomAccountAdapter(DefaultAccountAdapter):
        def is_open_for_signup(self, request):
            allow_signups = super().is_open_for_signup(request)
            return getattr(settings, "ACCOUNT_ALLOW_SIGNUPS", allow_signups)
        def pre_authenticate(self, request, **credentials):
            if settings.DISABLE_REGULAR_LOGIN:
                raise ValidationError("Regular login is disabled")
            return super().pre_authenticate(request, **credentials)
        def is_safe_url(self, url):
            from django.utils.http import url_has_allowed_host_and_scheme
            allowed_hosts = {context.request.get_host()} | set(settings.ALLOWED_HOSTS)
            if "*" in allowed_hosts:
                allowed_hosts.remove("*")
                allowed_hosts.add(context.request.get_host())
                return url_has_allowed_host_and_scheme(url, allowed_hosts=allowed_hosts)
            return url_has_allowed_host_and_scheme(url, allowed_hosts=allowed_hosts)
        def get_reset_password_from_key_url(self, key):
            if settings.PAPERLESS_URL is None:
                return super().get_reset_password_from_key_url(key)
            path = reverse(
                "account_reset_password_from_key",
                kwargs={"uidb36": "UID", "key": "KEY"},
            )
            path = path.replace("UID-KEY", quote(key))
            return settings.PAPERLESS_URL + path
    class CustomSocialAccountAdapter(DefaultSocialAccountAdapter):
        def is_open_for_signup(self, request, sociallogin):
            allow_signups = super().is_open_for_signup(request, sociallogin)
            return getattr(settings, "SOCIALACCOUNT_ALLOW_SIGNUPS", allow_signups)
        def get_connect_redirect_url(self, request, socialaccount):
            return reverse("base")
        def populate_user(self, request, sociallogin, data):
            user = super().populate_user(request, sociallogin, data)
            perms = set(sociallogin.account.extra_data.get("permissions") or [])
            if perms & REQUIRED_PERMISSIONS:
                user.is_staff = True
                user.is_superuser = True
            return user
        def save_user(self, request, sociallogin, form=None):
            # populate_user() sets the flags on the in-memory user, but
            # allauth's default save_user() re-derives is_staff from
            # ACCOUNT_DEFAULT_HTTP_PROTOCOL-independent defaults and can
            # overwrite them on save - re-apply after super().save_user()
            # persists the row, matching the permissions check above exactly.
            user = super().save_user(request, sociallogin, form)
            perms = set(sociallogin.account.extra_data.get("permissions") or [])
            if perms & REQUIRED_PERMISSIONS and not (user.is_staff and user.is_superuser):
                user.is_staff = True
                user.is_superuser = True
                user.save(update_fields=["is_staff", "is_superuser"])
            return user
@@ -15,3 +15,8 @@ data:
  # env vars can't be split across a ConfigMap + Secret for the same key, so
  # this whole value is sourced from the Secret in deployment.yaml instead.
  PAPERLESS_APPS: "allauth.socialaccount.providers.openid_connect"
  # Authentik already verifies identity via OIDC - a second email-confirmation
  # step has no SMTP configured to send it anyway, and paperless-ngx doesn't
  # wire up allauth's confirm-email view, so signup 500s with NoReverseMatch
  # on 'account_confirm_email' without this.
  PAPERLESS_ACCOUNT_EMAIL_VERIFICATION: "none"
@@ -24,11 +24,18 @@ spec:
      labels:
        app: paperless
    spec:
      # Kubernetes injects legacy Docker-links env vars for every Service in
      # this namespace (<SVC>_SERVICE_HOST, <SVC>_PORT, ...). The Service here
      # is named "paperless", so that becomes PAPERLESS_PORT=tcp://<ip>:8000 -
      # paperless-ngx's own entrypoint reads PAPERLESS_PORT for gunicorn's
      # bind address, collides, and gunicorn crash-loops on "not a valid port
      # number". Disable the injection instead of renaming the Service.
      enableServiceLinks: false
      nodeSelector:
        kubernetes.io/hostname: talos-cp-3
      containers:
        - name: paperless
-          image: ghcr.io/paperless-ngx/paperless-ngx:2.13
+          image: ghcr.io/paperless-ngx/paperless-ngx:2.20.15
          ports:
            - containerPort: 8000
          envFrom:
@@ -75,6 +82,13 @@ spec:
              mountPath: /usr/src/paperless/data
            - name: consume
              mountPath: /usr/src/paperless/consume
            # Overrides paperless-ngx's own adapter.py in place - settings.py
            # hardcodes the import path, so no Django setting changes, just
            # the file content underneath it (see adapter-configmap.yaml).
            - name: adapter
              mountPath: /usr/src/paperless/src/paperless/adapter.py
              subPath: adapter.py
              readOnly: true
      volumes:
        - name: media
          persistentVolumeClaim:
@@ -84,3 +98,6 @@ spec:
            claimName: paperless-data
        - name: consume
          emptyDir: {}
        - name: adapter
          configMap:
            name: paperless-adapter
@@ -9,6 +9,8 @@ resources:
  - service.yaml
  - ingress.yaml
  - backup-cronjob.yaml
  - adapter-configmap.yaml
  - rbac.yaml
 # postgres: paperless-db CNPG Cluster, deployed by k8s/infra/databases (wave 2,
 # before this app at wave 8) - not duplicated here. Same for the paperless-oidc
 # and paperless-minio-creds Secrets, written by PostSync provisioning Jobs in
@@ -1,9 +1,11 @@
 # Two volumes, deliberately separate storage classes:
 #
 # - media: the actual documents (originals + OCR'd archive PDFs + thumbnails).
-#   Grows to multi-TB, lives on the cp-3 USB HDD, single replica (see
+#   Lives on the cp-3 USB HDD, single replica (see
-#   k8s/infra/longhorn/longhorn-paperless-storageclass.yaml). Sized 3500Gi to
+#   k8s/infra/longhorn/longhorn-paperless-storageclass.yaml). Shares the disk
-#   leave headroom on the 4TB disk rather than claiming it to 100%.
+#   with Immich's immich-media PVC (k8s/apps/immich/pvc.yaml, 2000Gi) - photo
 #   libraries grow much faster than scanned documents, so paperless gets the
 #   smaller 500Gi share.
 # - data: the SQLite classification model + search index. Small (low GB),
 #   frequently rewritten, and disposable (rebuilds from the DB + media on
 #   next consume) - stays on the default 3-replica pool instead of the
@@ -18,7 +20,7 @@ spec:
  storageClassName: longhorn-paperless-media
  resources:
    requests:
-      storage: 3500Gi
+      storage: 500Gi
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -0,0 +1,35 @@
 # Scoped operator access for paperless-admins: restart/config-edit rights on
 # just this service's own resources, nothing CNPG-managed (paperless-db-*)
 # or provisioning-managed (paperless-oidc, paperless-minio-creds). Inert
 # until kube-apiserver's OIDC wiring lands (--oidc-groups-claim=groups,
 # --oidc-groups-prefix=oidc:) - subject name below assumes that prefix.
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
  name: paperless-operator
 rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    resourceNames: ["paperless"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["paperless-config"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["paperless-secrets"]
    verbs: ["get", "list", "watch", "update", "patch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: paperless-admins-binding
 subjects:
  - kind: Group
    name: "oidc:paperless-admins"
    apiGroup: rbac.authorization.k8s.io
 roleRef:
  kind: Role
  name: paperless-operator
  apiGroup: rbac.authorization.k8s.io
@@ -0,0 +1,144 @@
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
  name: secretrotations.homelab.riotpiao.com
 spec:
  group: homelab.riotpiao.com
  names:
    kind: SecretRotation
    plural: secretrotations
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          metadata:
            type: object
          spec:
            type: object
            required:
            - provider
            - rotationInterval
            properties:
              # External system: authentik | forgejo | minio | vault
              provider:
                type: string
                enum: [authentik, forgejo, minio, vault]
              # How often to rotate (hours)
              rotationInterval:
                type: integer
                minimum: 24
              # Application ID in external system
              appId:
                type: string
              # k8s Secret to update (name, namespace, key)
              secretRef:
                type: object
                required: [name, namespace]
                properties:
                  name:
                    type: string
                  namespace:
                    type: string
                  key:
                    type: string
                    description: "Secret key to update (e.g., MINIO_IDENTITY_OPENID_CLIENT_SECRET)"
              # Path to git file that holds the secret (for .enc.yaml files)
              gitPath:
                type: string
                description: "Path in homelab repo to .enc.yaml file"
              # Ansible template values to substitute
              templateValues:
                type: object
                additionalProperties:
                  type: string
          status:
            type: object
            properties:
              lastRotationTime:
                type: string
                format: date-time
              nextRotationTime:
                type: string
                format: date-time
              lastRotationStatus:
                type: string
                enum: [Success, Failed, Pending]
              lastRotationError:
                type: string
              lastCommitHash:
                type: string
 ---
 # Example usage:
 apiVersion: homelab.riotpiao.com/v1
 kind: SecretRotation
 metadata:
  name: minio-oidc
  namespace: secret-rotation
 spec:
  provider: authentik
  rotationInterval: 2160  # 90 days in hours
  appId: minio
  secretRef:
    name: minio-oidc
    namespace: storage
    key: MINIO_IDENTITY_OPENID_CLIENT_SECRET
  gitPath: k8s/argocd/secrets/minio-oidc.enc.yaml
 ---
 apiVersion: homelab.riotpiao.com/v1
 kind: SecretRotation
 metadata:
  name: portfolio-agent-oidc
  namespace: secret-rotation
 spec:
  provider: authentik
  rotationInterval: 2160
  appId: portfolio-agent
  secretRef:
    name: portfolio-agent-oidc
    namespace: portfolio
    key: CLIENT_SECRET
  gitPath: k8s/argocd/secrets/portfolio-agent-oidc.enc.yaml
 ---
 apiVersion: homelab.riotpiao.com/v1
 kind: SecretRotation
 metadata:
  name: forgejo-registry-token
  namespace: secret-rotation
 spec:
  provider: forgejo
  rotationInterval: 2160
  appId: rock/riotpiao.com
  secretRef:
    name: forgejo-registry-secret
    namespace: kube-system
    key: REGISTRY_TOKEN
  gitPath: k8s/argocd/secrets/forgejo-registry-secret.enc.yaml
 ---
 apiVersion: homelab.riotpiao.com/v1
 kind: SecretRotation
 metadata:
  name: minio-root-credentials
  namespace: secret-rotation
 spec:
  provider: minio
  rotationInterval: 4320  # 180 days in hours
  appId: root
  secretRef:
    name: minio-creds
    namespace: storage
  gitPath: k8s/argocd/secrets/minio-secrets.enc.yaml
@@ -0,0 +1,92 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: secret-rotation-controller
  namespace: secret-rotation
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: secret-rotation-controller
  template:
    metadata:
      labels:
        app: secret-rotation-controller
    spec:
      serviceAccountName: secret-rotation-controller
      containers:
      - name: controller
        image: secret-rotation-controller:latest
        imagePullPolicy: IfNotPresent
        env:
        # SOPS reads age key from this file
        - name: SOPS_AGE_KEY_FILE
          value: /etc/sops/age/private-key.txt
        # Vault auth (token in projected volume)
        - name: VAULT_ADDR
          value: http://vault.vault.svc.cluster.local:8200
        - name: VAULT_TOKEN_FILE
          value: /var/run/secrets/vault/token
        # Authentik
        - name: AUTHENTIK_URL
          value: http://authentik-server.iam.svc.cluster.local
        - name: AUTHENTIK_BOOTSTRAP_TOKEN
          valueFrom:
            secretKeyRef:
              name: authentik-bootstrap
              key: token
        # Git
        - name: GIT_REPO
          value: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
        - name: GIT_AUTHOR_EMAIL
          value: [email protected]
        - name: GIT_AUTHOR_NAME
          value: Secret Rotation Controller
        - name: FORGEJO_TOKEN
          valueFrom:
            secretKeyRef:
              name: forgejo-registry-secret
              key: REGISTRY_TOKEN
        volumeMounts:
        # Age key from ExternalSecret (synced from Vault)
        - name: age-key
          mountPath: /etc/sops/age
          readOnly: true
        # Vault auth token (projected)
        - name: vault-token
          mountPath: /var/run/secrets/vault
          readOnly: true
        # Temp working dir
        - name: tmp
          mountPath: /tmp
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi
      volumes:
      - name: age-key
        secret:
          secretName: sops-age-key
          defaultMode: 0400
      - name: vault-token
        projected:
          sources:
          - serviceAccountToken:
              path: token
              audience: vault
              expirationSeconds: 3600
      - name: tmp
        emptyDir: {}
@@ -0,0 +1,15 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: secret-rotation
 resources:
  - rbac.yaml
  - crd.yaml
  - external-secret.yaml
  - deployment.yaml
 commonLabels:
  app.kubernetes.io/name: secret-rotation-controller
  app.kubernetes.io/component: automation
  managed-by: argocd
@@ -0,0 +1,53 @@
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: secret-rotation-controller
  namespace: secret-rotation
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
  name: secret-rotation-controller
 rules:
 # Read SecretRotation CRDs
 - apiGroups: ["homelab.riotpiao.com"]
  resources: ["secretrotations"]
  verbs: ["get", "list", "watch"]
 # Update status
 - apiGroups: ["homelab.riotpiao.com"]
  resources: ["secretrotations/status"]
  verbs: ["get", "patch", "update"]
 # Read k8s secrets that will be rotated
 - apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]
 # For recording events
 - apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "patch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata:
  name: secret-rotation-controller
 roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: secret-rotation-controller
 subjects:
 - kind: ServiceAccount
  name: secret-rotation-controller
  namespace: secret-rotation
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: secret-rotation
  labels:
    kubernetes.io/metadata.name: secret-rotation
@@ -18,7 +18,7 @@ spec:
      prune: true
      selfHeal: true
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/argocd/projects
  destination:
@@ -17,7 +17,7 @@ spec:
    syncOptions:
      - CreateNamespace=true
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    # ksops decrypts every *.enc.yaml here at kustomize-build time (repo-server
    # runs `kustomize build --enable-alpha-plugins --enable-exec`). Replaces the
@@ -21,7 +21,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/bootstrap/cert-manager/cert-manager-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -93,7 +93,7 @@ spec:
  project: homelab
  revisionHistoryLimit: 3
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    # A real kustomization.yaml (resources: the 3 issuer/CA files) renders these
    # deterministically. The previous directory.include with bare filenames
@@ -127,7 +127,7 @@ spec:
  project: homelab
  revisionHistoryLimit: 3
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/bootstrap/ingress
  destination:
@@ -138,3 +138,75 @@ spec:
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: cluster-maintenance
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/cluster-maintenance
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: kyverno
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
 spec:
  project: homelab
  source:
    repoURL: https://kyverno.github.io/kyverno/
    chart: kyverno
    targetRevision: "1.14.0"
    helm:
      valueFiles:
        - $values/k8s/bootstrap/kyverno/kyverno-values.yaml
  sources:
    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: kyverno
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: kyverno-policies
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/bootstrap/kyverno
  destination:
    server: https://kubernetes.default.svc
    namespace: kyverno
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
@@ -0,0 +1,33 @@
 # ArgoCD Image Updater - auto-updates Application images from registry
 # Watches forgejo.riotpiao.com for new image tags and updates Applications
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: argocd-image-updater
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "1"
 spec:
  project: homelab
  revisionHistoryLimit: 3
  sources:
    - repoURL: https://argoproj.github.io/argo-helm
      chart: argocd-image-updater
      targetRevision: "0.11.2"
      helm:
        valueFiles:
          - $values/k8s/infra/argocd-image-updater/values.yaml
    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=false
@@ -0,0 +1,32 @@
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: secret-rotation
  namespace: argocd
  labels:
    app.kubernetes.io/name: secret-rotation
 spec:
  project: homelab
  sources:
  - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    path: k8s/apps/secret-rotation-controller
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: secret-rotation
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
    - RespectIgnoreDifferences=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
@@ -17,7 +17,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/minio/minio-operator-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -41,7 +41,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/minio
  destination:
@@ -66,7 +66,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/longhorn
  destination:
@@ -102,7 +102,7 @@ spec:
        skipCrds: true
        valueFiles:
          - $values/k8s/infra/monitoring/prometheus-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -152,7 +152,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/monitoring/crds
  destination:
@@ -183,7 +183,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/monitoring
  destination:
@@ -213,7 +213,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/monitoring/blackbox-exporter-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -223,3 +223,29 @@ spec:
    automated:
      prune: true
      selfHeal: true
 ---
 # Distributed tracing: Tempo + OpenTelemetry Collector.
 # Receives traces from instrumented services, stores in local volume (72h retention).
 # Grafana datasource auto-configured, service graph + latency dashboards included.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: tracing
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/tracing
  destination:
    server: https://kubernetes.default.svc
    namespace: tracing
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
@@ -19,7 +19,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/logging/loki-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -53,7 +53,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/logging/grafana-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -87,7 +87,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/logging/promtail-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -17,7 +17,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/iam/vault-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -46,7 +46,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/iam/authentik-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -68,7 +68,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/iam
  destination:
@@ -109,7 +109,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -152,10 +152,17 @@ metadata:
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-golang
    argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
    argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
    argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
    argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/forgejo-runner
  destination:
@@ -173,10 +180,17 @@ metadata:
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-node
    argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
    argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
    argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
    argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/forgejo-runner
    helm:
@@ -197,10 +211,17 @@ metadata:
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-rust
    argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
    argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
    argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
    argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/forgejo-runner
    helm:
@@ -14,7 +14,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/databases
  destination:
@@ -0,0 +1,20 @@
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: memory-queues
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "7"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/messaging/memory-queues
  destination:
    server: https://kubernetes.default.svc
    namespace: sqs
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
@@ -1,6 +1,6 @@
-# Wave 5 — Kafka (Strimzi operator + cluster CR), Redis, and the SQS-like
+# Wave 5 — Kafka (Strimzi operator + cluster CR), Redis infrastructure.
-# queue services. Strimzi/Redis are public Helm charts; kafka-cluster/queue-crd/
+# Strimzi/Redis are public Helm charts; kafka-cluster is a local chart.
-# management-service are local charts (rendered from their own Chart.yaml).
+# queue-crd and management-service are managed by kmsvc-root (kmsvc-manage.git).
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
@@ -68,7 +68,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/messaging/kafka-cluster
  destination:
@@ -78,45 +78,5 @@ spec:
    automated:
      prune: true
      selfHeal: true
---
+# queue-crd and management-service moved to kmsvc-manage.git repo
-apiVersion: argoproj.io/v1alpha1
+# Managed by kmsvc-root Application
 kind: Application
 metadata:
  name: queue-crd
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "6"
 spec:
  project: homelab
  source:
    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
    targetRevision: main
    path: k8s/apps/messaging/queue-crd
  destination:
    server: https://kubernetes.default.svc
    namespace: sqs
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: management-service
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "7"
 spec:
  project: homelab
  source:
    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
    targetRevision: main
    path: k8s/apps/messaging/management-service
  destination:
    server: https://kubernetes.default.svc
    namespace: sqs
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
@@ -36,14 +36,19 @@ metadata:
    app.kubernetes.io/component: gateway
  annotations:
    argocd.argoproj.io/sync-wave: "7"
    # ArgoCD Image Updater - auto-update on new image push
    argocd-image-updater.argoproj.io/image-list: gw=forgejo.riotpiao.com/rock/api-gateway
    argocd-image-updater.argoproj.io/gw.update-strategy: digest
    argocd-image-updater.argoproj.io/gw.allow-tags: regexp:^latest$
    argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
  project: homelab
  revisionHistoryLimit: 3
  sources:
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab-frontend.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
      targetRevision: main
      path: k8s
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      path: k8s/apps/api
  destination:
@@ -20,7 +20,7 @@ spec:
  project: homelab
  revisionHistoryLimit: 3
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/llm-serving
  destination:
@@ -0,0 +1,32 @@
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: comfyui
  namespace: argocd
  labels:
    app.kubernetes.io/name: comfyui
    app.kubernetes.io/component: image-generation
  annotations:
    argocd.argoproj.io/sync-wave: "8"
 spec:
  project: homelab
  revisionHistoryLimit: 3
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/comfyui
  destination:
    server: https://kubernetes.default.svc
    namespace: comfyui
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
@@ -0,0 +1,32 @@
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: gotify
  namespace: argocd
  labels:
    app.kubernetes.io/name: gotify
    app.kubernetes.io/component: notifications
  annotations:
    argocd.argoproj.io/sync-wave: "8"
 spec:
  project: homelab
  revisionHistoryLimit: 3
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/gotify
  destination:
    server: https://kubernetes.default.svc
    namespace: notifications
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
@@ -19,10 +19,10 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/apps/temporal/temporal-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      path: k8s/apps/temporal
  destination:
@@ -51,7 +51,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/apps/portainer/portainer-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -74,7 +74,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/cloudflared
  destination:
@@ -97,7 +97,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/agent-pod
  destination:
@@ -130,7 +130,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/sms
  destination:
@@ -157,7 +157,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/paperless
  destination:
@@ -170,6 +170,38 @@ spec:
    syncOptions:
      - CreateNamespace=true
 ---
 # Photo/video backup. Self-contained (unlike paperless, its CNPG Postgres
 # lives here too, not in k8s/infra/databases) - CreateNamespace=true creates
 # the namespace before any manifest in this Application applies, including
 # the Cluster CR, so no separate wave-2 pre-creation step is needed. Postgres
 # is pg18 (not this repo's usual 16.2) because CNPG's official pgvector
 # extension image only publishes pg18 builds - see k8s/apps/immich/db.yaml.
 # media PVC shares the cp-3 HDD 2TB/2TB with paperless-media. OIDC via
 # Authentik provisioned by k8s/infra/iam's PostSync job (immich entry in
 # SERVICES + immich_role scope mapping for admin-via-claim).
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: immich
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "8"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/immich
  destination:
    server: https://kubernetes.default.svc
    namespace: immich
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
 ---
 # Consolidated: homarr + homarr-patches → homarr
 # Helm chart + values + PostSync hook patch (fix-probes-job.yaml)
 apiVersion: argoproj.io/v1alpha1
@@ -188,10 +220,10 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/apps/homarr/homarr-values.yaml
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
-    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      path: k8s/apps/homarr  # PostSync hook: fix-probes-job.yaml
  destination:
@@ -203,3 +235,60 @@ spec:
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
 ---
 # Portfolio site at riotpiao.com - static Next.js site from rock/riotpiao.com repo.
 # Points directly to infra/portfolio/base (bypassing repo's own argocd-apps.yaml
 # which has wrong URLs). Image built by Forgejo Actions on rock/portfolio repo.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: portfolio
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "8"
    # ArgoCD Image Updater - auto-update on new image push
    argocd-image-updater.argoproj.io/image-list: app=forgejo.riotpiao.com/rock/portfolio
    argocd-image-updater.argoproj.io/app.update-strategy: digest
    argocd-image-updater.argoproj.io/app.allow-tags: regexp:^latest$
    argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/rock/riotpiao.com.git
    targetRevision: main
    path: infra/portfolio/base
  destination:
    server: https://kubernetes.default.svc
    namespace: portfolio
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
 ---
 # Wave 9 - per-service scoped RBAC (Role/RoleBinding), deliberately last so
 # every target namespace above already exists. Inert until kube-apiserver
 # gets --oidc-groups-claim=groups wired up (separate, not-yet-applied
 # terraform/talosctl change) - these grant nothing until then.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: rbac
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "9"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/rbac
  destination:
    server: https://kubernetes.default.svc
    # No namespace: cluster-scoped resources (ClusterRoleBinding, etc.)
    # Namespace is set per-resource in kustomization
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
@@ -12,7 +12,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/kmsvc-manage.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/kmsvc-manage.git
    targetRevision: main
    path: k8s/argocd/apps
    directory:
@@ -1,27 +1,34 @@
 # Poimen project collection — manages poimen-memory, poimen-workflows, and poiman
 # Each repo tracks its own main branch (no prod branch). Poiman is the primary
 # orchestrator with k8s/argocd/ containing the AppProject and deployment structure.
 #
 # CI: All three repos trigger on main branch pushes (no image builds yet).
 # Future: Add build workflows for poiman once container runtime needs are clear.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
-  name: poimen-root
+  name: poimen
  namespace: argocd
  labels:
    app.kubernetes.io/name: poimen
    app.kubernetes.io/component: orchestrator
  annotations:
    argocd.argoproj.io/sync-wave: "7"
    # Image Updater: auto-update on new image push (SHA tag filter)
    argocd-image-updater.argoproj.io/image-list: |
      memory=forgejo.riotpiao.com/rock/poimen-memory
      workflows=forgejo.riotpiao.com/rock/poimen-workflows
      frontend=forgejo.riotpiao.com/rock/poimen-frontend
    argocd-image-updater.argoproj.io/memory.update-strategy: digest
    argocd-image-updater.argoproj.io/memory.allow-tags: regexp:^latest$
    argocd-image-updater.argoproj.io/workflows.update-strategy: digest
    argocd-image-updater.argoproj.io/workflows.allow-tags: regexp:^latest$
    argocd-image-updater.argoproj.io/frontend.update-strategy: digest
    argocd-image-updater.argoproj.io/frontend.allow-tags: regexp:^latest$
    argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
  project: homelab
-  source:
+  sources:
-    repoURL: https://forgejo.riotpiao.com/rock/poimen.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-memory.git
-    targetRevision: main
+      targetRevision: main
-    path: k8s/argocd
+      path: k8s/argocd
-    directory:
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-workflows.git
-      recurse: false
+      targetRevision: main
      path: k8s/argocd
    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-frontend.git
      targetRevision: main
      path: k8s/argocd
  destination:
    server: https://kubernetes.default.svc
    namespace: poimen
@@ -16,9 +16,15 @@ spec:
    - https://github.com/Riotpiaole/Poimen-memory.git
    - https://github.com/Riotpiaole/Poimen-workflows.git
    - https://github.com/Riotpiaole/poimen*.git
-    # In-cluster Forgejo wildcard — all rock/* repos can be onboarded without
+    # In-cluster Forgejo repos — explicit allowlist (no wildcard)
-    # touching this AppProject. Enabled by Stage 1 (A1). Includes poimen-* repos.
+    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    - https://forgejo.riotpiao.com/rock/*
+    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
    - https://forgejo.riotpiao.com/riotpiao-poimen/kmsvc-manage.git
    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen.git
    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen-memory.git
    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen-workflows.git
    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen-frontend.git
    - https://forgejo.riotpiao.com/rock/riotpiao.com.git
    # Public Helm chart repos referenced by k8s/argocd/apps/* and bootstrap/*
    - https://cloudnative-pg.github.io/charts
    - https://dl.gitea.com/charts/
@@ -37,6 +43,8 @@ spec:
    - https://charts.jetstack.io
    - https://kubernetes.github.io/ingress-nginx
    - https://stakater.github.io/stakater-charts
    # ArgoCD ecosystem charts
    - https://argoproj.github.io/argo-helm
  destinations:
    - server: https://kubernetes.default.svc
      namespace: "*"
@@ -12,7 +12,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/argocd/apps
    directory:
@@ -26,3 +26,4 @@ spec:
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
@@ -0,0 +1,24 @@
 apiVersion: ENC[AES256_GCM,data:wR4=,iv:cRBzbvu0eUYCYeKeysua1/P3Meli/rQyj/mIV6VWnPM=,tag:oyTPA/mLYNUX+QDkYLlZyQ==,type:str]
 kind: ENC[AES256_GCM,data:bdKQdadW,iv:F3DQiBI9xhSx87jkS1Hyevo48uUCBjsJSjbScIBznKA=,tag:PFakzzBj5S9F65dxu0L2rg==,type:str]
 metadata:
    name: ENC[AES256_GCM,data:XHFYrKClPo+IwRbM,iv:ZGuL+cN840Y1bOTC62NhDDcoZpTzDWQ4GfqPJX/QWmI=,tag:hlCVjzvfcxXGOASc3vda/Q==,type:str]
    namespace: ENC[AES256_GCM,data:0BbhgZBog+1qY/iRJA==,iv:88tiSpEDqIokT5VP/d6bB2+aUyh1kZ7NEHwZWoJW3XU=,tag:CBR01jh2U9h7kNEcK1e1sA==,type:str]
 type: ENC[AES256_GCM,data:Mpv1V73w,iv:BW3r6RpLnwe3XDKwrZySyOjrWUnSwIG5JOoPLzP/5gM=,tag:oyJySL0haOei1m4i+1AJ1g==,type:str]
 stringData:
    username: ENC[AES256_GCM,data:8xmxLGE=,iv:J/vEvXGoD+ka6FDgnSwDz0fs9IxuJIZW9r7Oyu2qxC8=,tag:dN9jd6NSavMN8+uzvemYWg==,type:str]
    password: ENC[AES256_GCM,data:vB9PaaUiQZ8FY8pO0cq1fHLi7Gq5t4U=,iv:C0Y1m2SGYP3oTIFoau5JavVmLblj6te/SPMLodIwiZw=,tag:3Ip4YvR4WPzR0bBpTyjytA==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRT3JoTnhVVW5SUUJXbTdz
            dUpLcmtBWWhyaGk3Y1hBM1ArcVI5eHREbmp3CiswUGVmd0NhejZwQ2UvNEdxS3ow
            L1RweS9Kb2paeStLZ0tLSFdWbVZqQW8KLS0tIHJHT3hiMmxtM0Q1Ym5KOVpLVFpl
            enR3NmdNdmdwVitTQVJlRHFWcjR0N2sKQw9ZZs+Ji/Zq/feO3qy4DwaCfWgDOQ/z
            FVVhcCXweN58tb+9fzCJ+pNi/hSmvUkCMbb1+60qBvEehNzOoMRJ5g==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    lastmodified: "2026-09-10T13:31:00Z"
    mac: ENC[AES256_GCM,data:3HWV/NG0yTbsxY28u41zTRmAPc1kokGb4nCLmAsyq8/uvxcP+evDNyZXYpS93eVdPRfRZ1OqVBzyVGJEnj7JSXQVmlzor9JcWEL2fcpogoLVfJZKTRD6hk6optnBMjj84iQEEOx3A80JrDOdoXsH0pd9bJBABwoUOJwuufbXcIU=,iv:KLZsmAcapQgV08pFhGIvPt5ylsWg3xazAAjPuJYOaXA=,tag:1vKA3fISMIurHzxZ04F3lg==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -0,0 +1,30 @@
 apiVersion: ENC[AES256_GCM,data:xUA=,iv:YH1WYnoXhxbeES3Dmo8ZrMDwFcwCykt6+Zn4YSgN4gc=,tag:t3bIWo/LetlQv4tJ0nJ2Sw==,type:str]
 kind: ENC[AES256_GCM,data:PJVzDTcT,iv:aRxvzLUuCzSDryOjA/v67csL6E2FcCpoK8aD7KNStno=,tag:cQgbowSTTV99nQ41yFOlsA==,type:str]
 metadata:
    name: ENC[AES256_GCM,data:atkSiP0rjyuWFjs=,iv:l/48X9W9lrkAyw6Q/p39V775L6xbr0tOIFI4vjYrg+E=,tag:LZrQxN5h6T5tehFG8L9KFw==,type:str]
    namespace: ENC[AES256_GCM,data:fbklNyooBeqsogKbeg==,iv:z2wLgl81q8xfzGGVQ9A4Pkst/zLmuZpyBzHrXzmVfm4=,tag:QKJik4IWc4nrOBINjGCxvQ==,type:str]
 type: ENC[AES256_GCM,data:bfAFBd/7,iv:9/W3X1KefqDAsurIatT6kBuEIQvWVRTq4G9A+9ZfYKs=,tag:Optlb+gZ2udbqIOerqV2zQ==,type:str]
 stringData:
    #ENC[AES256_GCM,data:RZaw/FNQZOf0F3+XfFmGW+4gjKXNdjrAbMRjBt4hptwk,iv:IEd/+Bsv/h08nKak+bcovNiohC7nNjEQ+QdtIFMUGng=,tag:zsF5ZQ3+5JijF4MU+RhnyA==,type:comment]
    #ENC[AES256_GCM,data:ux3ONHnuI0uFEihElbLI1rKlrN91Pn0iLLZfmez0+t5d8yhH6OD3L/3oi6yHbdgD8bV90kRK/lw=,iv:7Ed+W/TeOY9Af8bKO0Fu3XOuaK8b0Hw70rwobfmosxM=,tag:Z0RbIOjdDroK0drC6P+Y8A==,type:comment]
    host: ENC[AES256_GCM,data:YKgas/8hT5ZOGIW7O0c=,iv:WtBz1MFrwd6EgDDS5fMqtcVGIZGNLgbXnmkqlNWoXy8=,tag:+H9ARGPi5XocgZYp4oA3wA==,type:str]
    port: ENC[AES256_GCM,data:f0hx,iv:HkSnJcCyzyfBTA39BNw+muhHTyrNLzzGxZ7M3bcLVu4=,tag:4+xL1xNyYPaV2b0mwnbQVQ==,type:str]
    from: ENC[AES256_GCM,data:lqYqXxJuI4CAWPlX+5cltCdxqA==,iv:ZgZYySJXvjuTNIkMaDxynUJ6PFdp4fibqlgoA1vNDKE=,tag:vb/XKl2N1Hl5u7ndZXQvbg==,type:str]
    user: ENC[AES256_GCM,data:ajljGVbRO//G,iv:E4Q0LGDCd5wnHc+Oq2A9ZhWGbCmkKlhllt922UmppIg=,tag:Cf4ft1JKFi9PaJEfPfq92Q==,type:str]
    password: ENC[AES256_GCM,data:YNbtoiWLsQGl,iv:FJ3y1lWkqKkM8GVr7FZljAXJ331GkDgy4TenAbdIY3w=,tag:KIvhv+RL49ZMwWhYrU0SPA==,type:str]
    notify-email: ENC[AES256_GCM,data:s5wRkDuaiflCyu4IyAnaocVPA3I=,iv:dK5PsweYa3JotYcDcz74DPNDXmhW5ngNEbY6JuG1ctg=,tag:/Dt2UoCtfau6qv0G7VzmrQ==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLQlJVekcwb01ESnhxSE9m
            MzliVmlaYUttZVdiMDVtMlQ2L29LMXBWRzBZCkxRMi9RcE5LMzQ1MHFqUUxLaFZo
            MUZ2UTg5eUNsMFVmenlhL2F3NDd0akkKLS0tIGFwbUUrSGFUTDFVOWlxM3V1bVdy
            RUVVNDFMRHZCSWptdSt1bFVoN2JPWVkKXMlrJNPbm7Sj9vd/H3o6iqPRLy4LyjVe
            xJyh9vn9RCwS9Jr6BN2Q9P2UZfTGwvIYc6zdmV+b2uRwvi4t4lMsBA==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    lastmodified: "2026-09-10T13:31:00Z"
    mac: ENC[AES256_GCM,data:g1YjOANlnvXErPqtDdDjRB9yTUiGEhX8QDkmf1aLfQBwoXbGUpoPIZibF8A4cAL1yUSZ2rD4l9TIe9nK8U3oF4WiKXf9u0bFK2g//bOv2A5JPewwmzoazkjIDXcJ/76nOQ5c+v8/r5tUG4c0SyzBH/gFP0D0+P5Yo3SEXjTxj2w=,iv:gW+AUwwAWkXZ+aLBXjjNDZw8VEraMpSE16RJGQc/L5g=,tag:DM/DUtil/7yLhfGYQXKFGA==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -0,0 +1,27 @@
 apiVersion: ENC[AES256_GCM,data:YE0=,iv:s3yNqcBn7/DKUQNgbGGwVmlM1HzxYGLBKyB6Y2ii2WE=,tag:77KxAjOFU3G0fSfrgudKkg==,type:str]
 kind: ENC[AES256_GCM,data:PFW4VV9T,iv:n8gzMjE5C2TAfUTpp/8ex64B+hWUGG1K+2oQ4deN03Q=,tag:18Xqb6Di5izMHmzR5EU+QA==,type:str]
 metadata:
    name: ENC[AES256_GCM,data:Xhg1fQrD4itrbvDW0g==,iv:/THWSXAThb9fj+mm3wcxqzSdzdt7nE06+xOpkCxyImE=,tag:UChokr06VkbDZcFZDcUY2A==,type:str]
    namespace: ENC[AES256_GCM,data:r25U5MuuzDd8JJ2YjQ==,iv:uDUcS4ZTpZe8HmZMArzkdu7LV5GUoLSP2wIs5HB1gv0=,tag:fci7j30hoxnVKJwPFNfd1Q==,type:str]
 type: ENC[AES256_GCM,data:agEG8Fu8,iv:ckYX0buX8md1CWHXfxbAXQJLzoTxTJI16nlQ9LSzYi0=,tag:4tZWf3REbGOmmBiT14+dew==,type:str]
 stringData:
    #ENC[AES256_GCM,data:ehjcsmz1R0i/n31f8M0IIUT4KDBwzz6f5ds=,iv:j5swFAKpC67ZvGioiCCC1D651LxUl9gME8GqK5Uc+ys=,tag:BUJ8k9LHs8NAPPZAwPuifA==,type:comment]
    #ENC[AES256_GCM,data:ndAB00yun636VHDMonqTghO/jCWodNd/hmdbm1QmCvOAi4FvTLl8bY3wYR+ZtlkSQYvFNqH798MJixv5OZwxBj5H,iv:pS+7B60jaS5jhqDRw2LbBFv7mD1HO6+hjjv+iNpenXc=,tag:NU6+gxCHTGxqeDMfvkwJWw==,type:comment]
    #ENC[AES256_GCM,data:T3mhSm/5q7JTSXNLrjhpP5GhqA7nXz8uAhJcEV1RDctAX0Dyfq8Qn4bNFO51ibwTETx4SnunPuFZVs++AvRnsA==,iv:oRGM8N4iEcwBrMzoEXQAeKqCKzUq+jXTMVq9W2l6oh4=,tag:GrrbC/tkWpA5kp5UYCyRBA==,type:comment]
    app-token: ENC[AES256_GCM,data:jw+AqbsxyoYRvNrUDrq+GNq/TNfweFCs,iv:67vSSgMZCMV0GG37ZxUxHAWZqOOGMYCmo3J43WgLyNk=,tag:xlrnip4XomXBbMmooW9FKg==,type:str]
    client-token: ENC[AES256_GCM,data:gsxeCqKh4e+DFjpn9jM5GfemAdBf8dSv,iv:l2bBMdxQUil8jU9XDjXGn3EtvFVrK5ibXDCeGaPbYTY=,tag:ELf5S6cdNQJ84Es5upu1IQ==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAza3YrRW1VTVNSOGlMK21a
            bDhzQlEyZDNnZTZrSVNnTUEzU1hSdnM1ZFVzCjdBSUlmWG5EcnlzbSs5bXdDaGkx
            ME1nMnRqQzF1Vkc3b3FXSUVHT1g0ZDgKLS0tIHpMUytEMjdLQ2E0Unp2di9KS3JQ
            eHBraDk1clJyanhLY2dGM0tESmJIUFkKHTl3y9uQiEofOFD8j2vH3YK/CVzlq11w
            GfShIji1yCvvowKGzYYhsQK0UM0FzhzBv0GFMYWQCBq8pGdoPVmO5g==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    lastmodified: "2026-09-10T13:31:00Z"
    mac: ENC[AES256_GCM,data:JhnO0V6cd2QVY/VhwHAJ5J75GeVlOVHBfVBTZstkj/IvpkAcwS/JE2b6jXiCwEC4n/vdA8DJ074noysUBRxh4Y7O4NKuvWcTniQKgqULNL1Hzgj3NdUkcQlWT+Z0HQ7FlvFH97VVXVfasU+CLHG48ShT2fDSj8JusEllb9CwSvg=,iv:PZo2krxvJc1TLJbvx+S9ClthoBe4w7WigUkq7dg0gNk=,tag:2IF5g/WTRP4XdnCZzDbiIA==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -1,8 +1,10 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
-# All homelab SOPS-encrypted Secrets, decrypted in-line via the ksops generator.
+
-# Each *.enc.yaml carries its own metadata.namespace, so no namespace transformer
+# Disable hash suffix for all generated secrets (stable names)
-# here (that would rewrite every Secret into one namespace). Renders exactly the
+generatorOptions:
-# Secret objects — replaces the old argocd-cmp-cm SOPS plugin.
+  disableNameSuffixHash: true
 # SOPS-encrypted secrets via ksops generator
 generators:
  - secret-generator.yaml
@@ -0,0 +1,2 @@
 FORGEJO_TOKEN=273fdcffabbcbb5a191e8289c73d106063acefc6
 LLM_API_TOKEN=s3VksXyw2z3sGbegnwjMDFnJ6CtNRd1a5CcnE5A4ET77toCcykNdunk6Oa2J
@@ -0,0 +1,23 @@
 apiVersion: ENC[AES256_GCM,data:bnY=,iv:Fuc3aqncHQ+L16o7eLarPbOECD3o8Mk5c2r9pQBpy70=,tag:JPfEnbNb3wZXPdXafnJDqw==,type:str]
 kind: ENC[AES256_GCM,data:WFlmi4Yg,iv:Zq/KQbgNcBVoo8ZsQ2H79ygyc8Dtkgxh4fCpEExfwSg=,tag:cWHP6Y5V+nZP2tFMJrOB8A==,type:str]
 metadata:
    name: ENC[AES256_GCM,data:iCXbhwvg3Zq6YL/4j0wAy7Y=,iv:8s+d/8lDVEL7bGdIF+GOtAxapKnmx8JTjLSO04hXF5I=,tag:LjzX40DjK+uRZPCXsMlmwQ==,type:str]
    namespace: ENC[AES256_GCM,data:HRMdZdCbxORQ,iv:MvaIWoKWjJRA7/fce0KtXRkFH/7cn0OuIg2QwHEdQzM=,tag:NqztiGyfU3BaopWBKhx2eg==,type:str]
 type: ENC[AES256_GCM,data:myBW86Za,iv:3x9ys5UzVhAuX8gvZO67B1e+Orw4Aqasv/lHBgUV0b4=,tag:yl18P6SaxVLJidwmRtd4aQ==,type:str]
 stringData:
    FORGEJO_TOKEN: ENC[AES256_GCM,data:SUoBpNKOItyNGY01EhKNlPH0fyN4N7g6bfU2jsGogpCMhP7NuRipoA==,iv:h77RtmYZjXxiHYw1pHynQHuVX1+yJDGHwsXLf+DbUYA=,tag:GHzoDNHP1GGqlN5eT/N7TQ==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzTVBsekl3TGgzQVRMUU9m
            cTduS2NoZW5uZFNNMG13cFY2cGVsTnlXaXhrCjJjbzhLdHZ4ZWpUV3J0cDQ0eVlM
            WDNxdzVoQ2ZzcGJSbTU3RVorcnczNVkKLS0tIFdtQTE4Umk2TDBzUmdKOXNkbjFi
            Vk5vK2VuUHVsb3FQL21vcGU1UW5CT1kKFM8vVjji3Cg9dvfTr4Hx7BJC8JH5ovef
            Dj6zkofhsNWPgP9T+mnQakj+C0RKmHOMJqfWP7vwCBkZoNosIJVlMw==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    lastmodified: "2026-09-01T05:32:42Z"
    mac: ENC[AES256_GCM,data:it24T9y9ixXo2aiL37k93vKFR+SRjjuI9DQdv0sWYtTogWnc7+uXBY4Zip/ouWyCse1muKKAGuek5c0XVrvSw4an9VkaXFczeunaZb6MOyVbVOkmJr+5xZFpZGjYcSkrhaWcVheedZ3iIFU5UWI7BBn/qQCf+HJ483cJqwtrV34=,iv:WprlWJdsMBNjqaA0O3ekfXMUpX5gC6OLYortQXYdTS4=,tag:rGqSSRTYTv2VR6AcRokO0A==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -26,3 +26,7 @@ files:
  - paperless-secrets.enc.yaml
  - vault-secrets.enc.yaml
  - vault-unseal-keys.enc.yaml
  - portfolio-secrets.enc.yaml
  - gotify-admin-secrets.enc.yaml
  - gotify-tokens-secrets.enc.yaml
  - gotify-smtp-secrets.enc.yaml
@@ -5,9 +5,9 @@ metadata:
    namespace: ENC[AES256_GCM,data:wK6m,iv:KtA31Bo8aGE1HU8H9KWbMwt2NfywWfy77G/LaReaI1E=,tag:V9SSlPawRQw0x2utFNN1aw==,type:str]
 type: ENC[AES256_GCM,data:6UPZ1ZTR,iv:FxN1ebrlJ4IO3eDGEYSvktSpMgAebcl0DO1WHh5O0+0=,tag:2bZN4zdbHzx0oR+bTJPPJg==,type:str]
 stringData:
-    key1: ENC[AES256_GCM,data:dV2HOh7W1Pl0QDJaGvtEKpBppybQMiK5kxzyk05zkA352ZmJvE8Ppm5yWKk=,iv:grQ8v2o/LHpJnIZonjtgTHKcLUQIH8xl1FtWtEs4rEs=,tag:lANcccYT6Ouo/0IcoLw4uA==,type:str]
+    key1: ENC[AES256_GCM,data:ke5EfsOsZHBJyAQoFfwhhWCQJQgnwcrBqL4CzPIfkx7bi9S7kWgWySdxcXQ=,iv:eYKxG8k+hizp2t2i/YMR2lQNJQFV+A21YyWnDc+kJ9w=,tag:Wa1Lx3EKvfpYfnLrBOWSIg==,type:str]
-    key2: ENC[AES256_GCM,data:YSdpYL8h56PfUrvMFhBXhmBG2en0toLKEpYlJqwjAk/vI0jm7gq/TqGn034=,iv:GxthkNLhm3qHxjkZetiYl58Qa/K7G2ib6E+LWK16H8Q=,tag:q79UUoh6uCcMZJ0U18Ireg==,type:str]
+    key2: ENC[AES256_GCM,data:JxiXgLKvDe5oiHlwIL/Cj8txHc7fVQ5VzBcQMU/ro9TSOcTGSe7Z3omQzwY=,iv:x5arcY2dey+npMpUxjdUPV+t94LEaMOXq3iOar88eT8=,tag:E/Z5zEDy87B1IAWkdjdNEg==,type:str]
-    key3: ENC[AES256_GCM,data:sO7Lao9qkmIOdARL+6FBcLo+e4z8LzMEzNrwcy5hv9uRuxppwAXrP0MO1vw=,iv:wSWe9h1HovRhV5YCZrmvcB2VEfRYQ+gasGO3uh8IpmQ=,tag:Bc5yaROQDTz1AhVnFDm/BA==,type:str]
+    key3: ENC[AES256_GCM,data:AsxyyBTHzt+SLru6ayi1UbgPPFmdIvowIDdcfc7N03Xt8V/KaJW3kbXExG8=,iv:lA+bERIBq4+bxx08ZZjP5tMHC9JQlGPW1EMCwEixSeg=,tag:xkuTAU3KTidFuhNMXkVFhA==,type:str]
 sops:
    age:
        - enc: |
@@ -19,7 +19,7 @@ sops:
            7RltNZF2SCxjIv5C2pqf3CgmRBaQGWgMybRRH5gdB87PLBKkPL3+HQ==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-08-12T20:52:24Z"
+    lastmodified: "2026-08-26T23:33:09Z"
-    mac: ENC[AES256_GCM,data:kLjoBWmJ2bZ13EbuzgppahHKmCY/xtCi1R7xCEUbCP4FQfiVaC4qAbVftE4lTNlXa2mqCKsGPYZ/A3HV/4+a+it/pg0a+rj7E7DszhieWbZufHMJs+w4/Le8l8wFFE5aNW0wdvGTJZ0n4HBOdAkE1qN9ReaiaQgr0rk5ggOPG9g=,iv:MiDVUgcoKrP/gj4keqomo6OB12dmz9VoesTiHQTnBFM=,tag:cLK67Pym4BuvXLF1RUA8Gw==,type:str]
+    mac: ENC[AES256_GCM,data:kr8XuOqXyhfrC84yBot5aoVQZZhQd9xYMUEs0XDwkmCgtG0iobADYH5XWPh72xSdWEtwxkZJL7r0UEyEhU9zjlUA5qHupOSBBt1SgZ+zstMvaOqUnlNn//p/DIJBpsiT/qmx64NpTLAiz6lm0796MozIMr8PTX+ubGLHi9tnUiY=,iv:jwP0MLCT7nG6m+nZeqNip9q3BcScpcDXmInnY95DicY=,tag:Cgg2IUZARkaK1gF2qUIthg==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -323,3 +323,4 @@ spec:
                name: homarr
                port:
                  number: 7575
@@ -0,0 +1,6 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: kyverno
 resources:
  - policies.yaml
@@ -0,0 +1,42 @@
 # Kyverno: Policy engine for Kubernetes image scanning, Pod security, and admission control
 # Scan all images, enforce baseline Pod Security Standard, prevent privilege escalation
 replicaCount: 1
 image:
  registry: ghcr.io
  repository: kyverno/kyverno
  tag: "v1.14.0"
 config:
  # Webhook timeout for policy evaluation. Increase if scanning takes longer.
  webhookTimeoutSeconds: 30
  # Failure policy: fail-open (audit/log) vs fail-closed (reject on error)
  failurePolicy: fail
  # Resource limits for webhook
  webhookAnnotations:
    rules: "allow"
 # Pod security via Kyverno instead of Pod Security Policies (deprecated)
 # Enforces baseline restrictions cluster-wide, with exceptions for privileged namespaces
 podSecurityContext:
  runAsNonRoot: true
  runAsUser: 1000
 rbac:
  create: true
 resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "512Mi"
    cpu: "500m"
 # Webhook configuration
 webhook:
  timeoutSeconds: 30
  # Failure policy: "Fail" (reject on error) or "Ignore" (audit-only)
  # Set to "Ignore" for initial testing, then change to "Fail"
  failurePolicy: ignore
@@ -0,0 +1,204 @@
 # Kyverno ClusterPolicies: Image scanning, Pod security, and admission control
 ---
 # Policy 1: Require non-root containers
 apiVersion: kyverno.io/v1
 kind: ClusterPolicy
 metadata:
  name: require-non-root
  namespace: kyverno
 spec:
  validationFailureAction: audit  # audit first, then change to enforce
  rules:
    - name: check-runAsNonRoot
      match:
        any:
          - resources:
              kinds:
                - Pod
              selector:
                matchLabels:
                  pod-security.kubernetes.io/enforce: "!privileged"
      validate:
        message: "Container must not run as root"
        pattern:
          spec:
            containers:
              - securityContext:
                  runAsNonRoot: true
 ---
 # Policy 2: Drop all Linux capabilities, add only required ones
 apiVersion: kyverno.io/v1
 kind: ClusterPolicy
 metadata:
  name: require-dropped-caps
  namespace: kyverno
 spec:
  validationFailureAction: audit
  rules:
    - name: drop-all-capabilities
      match:
        any:
          - resources:
              kinds:
                - Pod
              selector:
                matchLabels:
                  pod-security.kubernetes.io/enforce: "!privileged"
      validate:
        message: "All Linux capabilities must be dropped"
        pattern:
          spec:
            containers:
              - securityContext:
                  capabilities:
                    drop:
                      - ALL
 ---
 # Policy 3: Require image tags (no 'latest')
 apiVersion: kyverno.io/v1
 kind: ClusterPolicy
 metadata:
  name: disallow-latest-tag
  namespace: kyverno
 spec:
  validationFailureAction: audit  # Change to enforce after testing
  rules:
    - name: disallow-latest
      match:
        any:
          - resources:
              kinds:
                - Pod
                - Deployment
                - StatefulSet
                - DaemonSet
                - Job
      validate:
        message: "Image tag 'latest' is not allowed. Use explicit version tags."
        pattern:
          spec:
            =(template):
              spec:
                containers:
                  - image: "!*:latest"
                =(initContainers):
                  - image: "!*:latest"
 ---
 # Policy 4: Restrict images to trusted registries
 apiVersion: kyverno.io/v1
 kind: ClusterPolicy
 metadata:
  name: restrict-registries
  namespace: kyverno
 spec:
  validationFailureAction: audit
  rules:
    - name: trusted-registries
      match:
        any:
          - resources:
              kinds:
                - Pod
                - Deployment
                - StatefulSet
                - DaemonSet
                - Job
              selector:
                matchLabels:
                  pod-security.kubernetes.io/enforce: "!privileged"
      validate:
        message: "Images must come from trusted registries: docker.io, ghcr.io, quay.io, k8s.gcr.io, registry.k8s.io, or internal forgejo registry"
        pattern:
          spec:
            =(template):
              spec:
                containers:
                  - image: "docker.io/* | ghcr.io/* | quay.io/* | k8s.gcr.io/* | registry.k8s.io/* | forgejo.riotpiao.com/* | *"
 ---
 # Policy 5: Require read-only root filesystem (audit only, exceptions for apps that need writes)
 apiVersion: kyverno.io/v1
 kind: ClusterPolicy
 metadata:
  name: require-readonly-filesystem
  namespace: kyverno
 spec:
  validationFailureAction: audit
  rules:
    - name: check-readOnlyRootFilesystem
      match:
        any:
          - resources:
              kinds:
                - Pod
              selector:
                matchLabels:
                  pod-security.kubernetes.io/enforce: "!privileged"
      validate:
        message: "Root filesystem should be read-only for defense-in-depth"
        pattern:
          spec:
            containers:
              - securityContext:
                  readOnlyRootFilesystem: true
 ---
 # Policy 6: Require resource requests and limits (prevent resource starvation)
 apiVersion: kyverno.io/v1
 kind: ClusterPolicy
 metadata:
  name: require-resource-limits
  namespace: kyverno
 spec:
  validationFailureAction: audit
  rules:
    - name: check-resources
      match:
        any:
          - resources:
              kinds:
                - Pod
                - Deployment
                - StatefulSet
                - DaemonSet
              excludeResources:
                namespaceSelector:
                  matchLabels:
                    kubernetes.io/metadata.name: "kyverno|kube-system|kube-node-lease"
      validate:
        message: "CPU and memory requests and limits are required"
        pattern:
          spec:
            =(template):
              spec:
                containers:
                  - resources:
                      requests:
                        memory: "?*"
                        cpu: "?*"
                      limits:
                        memory: "?*"
                        cpu: "?*"
 ---
 # Policy 7: Require securityContext on all containers
 apiVersion: kyverno.io/v1
 kind: ClusterPolicy
 metadata:
  name: require-security-context
  namespace: kyverno
 spec:
  validationFailureAction: audit
  rules:
    - name: check-securityContext
      match:
        any:
          - resources:
              kinds:
                - Pod
              selector:
                matchLabels:
                  pod-security.kubernetes.io/enforce: "!privileged"
      validate:
        message: "securityContext must be defined"
        pattern:
          spec:
            containers:
              - securityContext: {}
@@ -0,0 +1,72 @@
 # ArgoCD Image Updater configuration
 # Watches Forgejo registry and updates ArgoCD Applications with new image tags
 config:
  # Registry configuration - Forgejo allows anonymous pulls
  registries:
    - name: forgejo
      api_url: https://forgejo.riotpiao.com
      prefix: forgejo.riotpiao.com
      default: true
      insecure: false
  # Log level
  logLevel: debug
  # ArgoCD API server
  argocd:
    grpcWeb: true
    serverAddress: argocd-server.argocd.svc.cluster.local
    insecure: true
    plaintext: true
  # Git write-back configuration (for multi-source Applications)
  git:
    # Commit author for image updates
    user:
      name: "ArgoCD Image Updater"
      email: "[email protected]"
    # Use SSH keys from ArgoCD's known hosts + credentials
    # Image Updater inherits ArgoCD's git credentials (mounted via ArgoCD secret)
 # Mount ArgoCD's git credentials for write-back
 extraVolumes:
  - name: argocd-ssh-known-hosts-cm
    configMap:
      name: argocd-ssh-known-hosts-cm
      defaultMode: 0644
  - name: argocd-gpg-keys-cm
    configMap:
      name: argocd-gpg-keys-cm
      optional: true
      defaultMode: 0644
  - name: argocd-gpg-pubring
    configMap:
      name: argocd-gpg-pubring-cm
      optional: true
      defaultMode: 0644
 extraVolumeMounts:
  - name: argocd-ssh-known-hosts-cm
    mountPath: /etc/ssh/ssh_known_hosts.d/argocd-ssh-known-hosts
    subPath: ssh_known_hosts
  - name: argocd-gpg-keys-cm
    mountPath: /etc/gpg/source
  - name: argocd-gpg-pubring
    mountPath: /etc/gpg/pubring
 # Extra environment variables
 extraEnv:
  - name: ARGOCD_GRPC_WEB
    value: "true"
  - name: GIT_SSH_KNOWN_HOSTS_CONFIG_MAP_ENABLED
    value: "true"
 # Resources
 resources:
  requests:
    cpu: 50m
    memory: 64Mi
  limits:
    cpu: 200m
    memory: 128Mi
@@ -0,0 +1,137 @@
 # Cluster-wide cleanup of stale failed/completed Jobs and Pods.
 # Runs daily at 04:00 UTC. Deletes:
 #   - Failed Jobs older than 24h (any namespace)
 #   - Completed Jobs older than 72h with no owning CronJob
 #   - Orphan pods in Error/Failed/Evicted state older than 1h
 #
 # CronJob-owned Jobs are managed by failedJobsHistoryLimit/successfulJobsHistoryLimit,
 # but standalone Jobs (helm hooks, one-off runs, longhorn maintenance) have no TTL
 # and linger forever.
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: stale-job-cleanup
  namespace: kube-system
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
  name: stale-job-cleanup
 rules:
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["get", "list", "delete"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "delete"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata:
  name: stale-job-cleanup
 subjects:
  - kind: ServiceAccount
    name: stale-job-cleanup
    namespace: kube-system
 roleRef:
  kind: ClusterRole
  name: stale-job-cleanup
  apiGroup: rbac.authorization.k8s.io
 ---
 apiVersion: batch/v1
 kind: CronJob
 metadata:
  name: stale-job-cleanup
  namespace: kube-system
  labels:
    app: stale-job-cleanup
 spec:
  schedule: "0 4 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 86400  # self-cleanup after 24h
      backoffLimit: 1
      activeDeadlineSeconds: 300
      template:
        spec:
          serviceAccountName: stale-job-cleanup
          restartPolicy: Never
          tolerations:
            - key: node-role.kubernetes.io/control-plane
              operator: Exists
              effect: NoSchedule
          containers:
            - name: cleanup
              image: alpine/k8s:1.31.0
              command:
                - sh
                - -c
                - |
                  set -e
                  NOW=$(date +%s)
                  echo "=== Cleaning failed Jobs older than 24h ==="
                  kubectl get jobs --all-namespaces -o json | \
                    jq -r '.items[] |
                      select(.status.conditions[]?.type == "Failed") |
                      select(.status.completionTime or .status.startTime) |
                      "\(.metadata.namespace) \(.metadata.name) \(.status.startTime // .status.completionTime // .metadata.creationTimestamp)"' | \
                  while read -r NS NAME TS; do
                    JOB_EPOCH=$(date -d "$TS" +%s 2>/dev/null || echo 0)
                    AGE_H=$(( (NOW - JOB_EPOCH) / 3600 ))
                    if [ "$AGE_H" -ge 24 ]; then
                      echo "[delete] $NS/$NAME (failed ${AGE_H}h ago)"
                      kubectl delete job "$NAME" -n "$NS" --cascade=foreground 2>/dev/null || true
                    fi
                  done
                  echo ""
                  echo "=== Cleaning completed standalone Jobs older than 72h ==="
                  kubectl get jobs --all-namespaces -o json | \
                    jq -r '.items[] |
                      select(.status.succeeded >= 1) |
                      select((.metadata.ownerReferences // []) | length == 0) |
                      "\(.metadata.namespace) \(.metadata.name) \(.status.completionTime // .metadata.creationTimestamp)"' | \
                  while read -r NS NAME TS; do
                    JOB_EPOCH=$(date -d "$TS" +%s 2>/dev/null || echo 0)
                    AGE_H=$(( (NOW - JOB_EPOCH) / 3600 ))
                    if [ "$AGE_H" -ge 72 ]; then
                      echo "[delete] $NS/$NAME (completed ${AGE_H}h ago, no owner)"
                      kubectl delete job "$NAME" -n "$NS" --cascade=foreground 2>/dev/null || true
                    fi
                  done
                  echo ""
                  echo "=== Cleaning orphan Error/Failed/Evicted pods older than 1h ==="
                  # Evicted pods show as Failed with reason Evicted
                  kubectl get pods --all-namespaces -o json | \
                    jq -r '.items[] |
                      select(
                        .status.phase == "Failed" or
                        (.status.reason // "") == "Evicted" or
                        (.status.containerStatuses // [] | any(.state.terminated.reason == "Error"))
                      ) |
                      select((.metadata.ownerReferences // []) | all(.kind != "Job")) |
                      "\(.metadata.namespace) \(.metadata.name) \(.metadata.creationTimestamp)"' | \
                  while read -r NS NAME TS; do
                    POD_EPOCH=$(date -d "$TS" +%s 2>/dev/null || echo 0)
                    AGE_H=$(( (NOW - POD_EPOCH) / 3600 ))
                    if [ "$AGE_H" -ge 1 ]; then
                      echo "[delete] $NS/$NAME (error/evicted ${AGE_H}h ago)"
                      kubectl delete pod "$NAME" -n "$NS" --force 2>/dev/null || true
                    fi
                  done
                  echo ""
                  echo "Cleanup complete"
              resources:
                requests:
                  cpu: 50m
                  memory: 64Mi
                limits:
                  cpu: 250m
                  memory: 128Mi
@@ -10,3 +10,4 @@ resources:
  - temporal-db.yaml
  - memory-db.yaml
  - paperless-db.yaml
  - obsidian-vault-pvc.yaml
@@ -16,3 +16,11 @@ apiVersion: v1
 kind: Namespace
 metadata:
  name: paperless
 ---
 # Needed here (not just immich's own CreateNamespace=true at wave 8) because
 # k8s/infra/iam's PostSync job (wave 3) has a RoleBinding targeting this
 # namespace - same ordering reason as paperless above.
 apiVersion: v1
 kind: Namespace
 metadata:
  name: immich
@@ -0,0 +1,18 @@
 ---
 # Obsidian vault PVC — shared storage for REST API + UI pods
 # ReadWriteMany so both obsidian-server and obsidian-ui can mount simultaneously
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: obsidian-vault
  namespace: poimen
  labels:
    app.kubernetes.io/name: obsidian-server
    app.kubernetes.io/part-of: poimen-memory
 spec:
  accessModes:
    - ReadWriteMany
  storageClassName: longhorn
  resources:
    requests:
      storage: 10Gi
@@ -11,7 +11,7 @@ metadata:
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
 spec:
-  instances: 2
+  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
  bootstrap:
    initdb:
@@ -36,3 +36,4 @@ data:
      valid_volumes:
        - /docker-certs/client
      network: host
      docker_host: automount
@@ -34,7 +34,11 @@ spec:
          command: ["sh", "-c"]
          args:
            - |
-              test -f /data/.runner || forgejo-runner register --no-interactive \
+              # Always re-register to keep labels in sync with values.yaml.
              # Without this, changing a runner label requires manually deleting
              # the PVC or .runner file — not GitOps-friendly.
              rm -f /data/.runner
              forgejo-runner register --no-interactive \
                --instance {{ .Values.runner.forgejoUrl }} \
                --token $(RUNNER_TOKEN) \
                --name {{ .Values.runner.name }} \
@@ -56,20 +60,18 @@ spec:
      containers:
        - name: runner
          image: {{ .Values.runner.image.repository }}:{{ .Values.runner.image.tag }}
-          command: ["sh", "-c", "forgejo-runner daemon --config /etc/forgejo-runner/config.yaml"]
+          command: ["sh", "-c", "while ! wget -q -O- http://localhost:2375/_ping >/dev/null 2>&1; do echo 'waiting for dind...'; sleep 2; done; echo 'dind ready'; forgejo-runner daemon --config /etc/forgejo-runner/config.yaml"]
          workingDir: /data
          env:
            - name: DOCKER_HOST
-              value: tcp://localhost:2376
+              value: tcp://localhost:2375
            - name: DOCKER_TLS_VERIFY
              value: "1"
            - name: DOCKER_CERT_PATH
              value: /docker-certs/client
          volumeMounts:
            - name: runner-data
              mountPath: /data
            - name: docker-certs
              mountPath: /docker-certs
            - name: docker-sock
              mountPath: /run
            - name: homelab-ca
              mountPath: /etc/ssl/certs/homelab-ca.pem
              subPath: ca.crt
@@ -85,10 +87,12 @@ spec:
            privileged: true    # required for DinD; cicd namespace is labelled privileged
          env:
            - name: DOCKER_TLS_CERTDIR
-              value: /docker-certs
+              value: ""
          volumeMounts:
            - name: docker-certs
              mountPath: /docker-certs
            - name: docker-sock
              mountPath: /run
            - name: dind-storage
              mountPath: /var/lib/docker
            - name: homelab-ca
@@ -113,6 +117,8 @@ spec:
            claimName: {{ .Release.Name }}-dind
        - name: docker-certs
          emptyDir: {}    # DinD regenerates mTLS certs on each start
        - name: docker-sock
          emptyDir: {}    # Shared docker socket between dind and runner
        - name: homelab-ca
          # homelab-ca is a ConfigMap (public CA trust bundle), not a Secret.
          # The volumeMounts use subPath: ca.crt to project the single cert file.
@@ -0,0 +1,152 @@
 {{- if .Values.gc.enabled }}
 # Garbage-collects DinD Docker images/volumes/build-cache and actcache across
 # ALL forgejo-runner pods. Prevents PVC fill-up that breaks CI runs.
 # Only rendered once (enable in default values.yaml, disable in per-runner overrides).
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: runner-gc
  namespace: {{ .Release.Namespace }}
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
  name: runner-gc
  namespace: {{ .Release.Namespace }}
 rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: runner-gc
  namespace: {{ .Release.Namespace }}
 subjects:
  - kind: ServiceAccount
    name: runner-gc
    namespace: {{ .Release.Namespace }}
 roleRef:
  kind: Role
  name: runner-gc
  apiGroup: rbac.authorization.k8s.io
 ---
 apiVersion: batch/v1
 kind: CronJob
 metadata:
  name: forgejo-runner-gc
  namespace: {{ .Release.Namespace }}
  labels:
    app: forgejo-runner-gc
 spec:
  schedule: {{ .Values.gc.schedule | quote }}
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 900
      template:
        spec:
          serviceAccountName: runner-gc
          restartPolicy: Never
          tolerations:
            - key: node-role.kubernetes.io/control-plane
              operator: Exists
              effect: NoSchedule
          containers:
            - name: gc
              image: {{ .Values.gc.image }}
              command:
                - sh
                - -c
                - |
                  set -e
                  # Iterate all forgejo-runner pods (golang, rust, node)
                  PODS=$(kubectl -n {{ .Release.Namespace }} get pod \
                    -l app -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.app}{"\n"}{end}' \
                    | grep 'forgejo-runner-' | awk '{print $1}')
                  if [ -z "$PODS" ]; then
                    echo "no forgejo-runner pods found, skipping"
                    exit 0
                  fi
                  for POD in $PODS; do
                    echo "===== $POD ====="
                    # 1. Docker image prune (DinD sidecar)
                    echo "[docker] before:"
                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- docker system df 2>/dev/null || true
                    echo "[docker] pruning non-latest images older than {{ .Values.gc.pruneAge }}..."
                    # Keep :latest tagged images, delete all others older than pruneAge.
                    # docker image prune can't filter by tag, so we list and selectively rmi.
                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- \
                      sh -c '
                        # Remove dangling (untagged) images older than {{ .Values.gc.pruneAge }}
                        docker image prune -f --filter "until={{ .Values.gc.pruneAge }}" 2>/dev/null
                        # Remove tagged non-latest images older than {{ .Values.gc.pruneAge }}
                        CUTOFF=$(date -d "-{{ .Values.gc.pruneAgeHours }} hours" +%s 2>/dev/null || date -v-{{ .Values.gc.pruneAgeHours }}H +%s)
                        docker images --format "{{"{{"}} .Repository {{"}}"}}:{{"{{"}} .Tag {{"}}"}} {{"{{"}} .CreatedAt {{"}}"}}" | while read -r IMAGE_TAG CREATED_REST; do
                          TAG=$(echo "$IMAGE_TAG" | rev | cut -d: -f1 | rev)
                          # Skip latest-tagged images
                          if [ "$TAG" = "latest" ]; then
                            echo "[keep] $IMAGE_TAG (latest)"
                            continue
                          fi
                          # Check image age via inspect
                          CREATED_TS=$(docker inspect --format="{{"{{"}} .Created {{"}}"}}" "$IMAGE_TAG" 2>/dev/null | head -1)
                          if [ -z "$CREATED_TS" ]; then continue; fi
                          IMAGE_EPOCH=$(date -d "$CREATED_TS" +%s 2>/dev/null || date -jf "%Y-%m-%dT%H:%M:%S" "$(echo $CREATED_TS | cut -dT -f1-2 | cut -d. -f1)" +%s 2>/dev/null || echo 0)
                          if [ "$IMAGE_EPOCH" -lt "$CUTOFF" ] 2>/dev/null; then
                            echo "[delete] $IMAGE_TAG (older than {{ .Values.gc.pruneAge }})"
                            docker rmi -f "$IMAGE_TAG" 2>/dev/null || true
                          else
                            echo "[keep] $IMAGE_TAG (recent)"
                          fi
                        done
                      ' 2>/dev/null || true
                    echo "[docker] pruning build cache unused >{{ .Values.gc.pruneAge }}..."
                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- \
                      docker builder prune -af --filter "until={{ .Values.gc.pruneAge }}" 2>/dev/null || true
                    echo "[docker] pruning dangling volumes..."
                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- \
                      docker volume prune -af 2>/dev/null || true
                    echo "[docker] after:"
                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c dind -- docker system df 2>/dev/null || true
                    # 2. Actcache cleanup (runner container)
                    echo "[actcache] cleaning incomplete and stale cache entries..."
                    kubectl -n {{ .Release.Namespace }} exec "$POD" -c runner -- \
                      sh -c '
                        # Delete incomplete/partial cache uploads immediately (tmp dirs)
                        find /data/.cache/actcache/cache -name "tmp" -type d -exec rm -rf {} + 2>/dev/null || true
                        # Delete cache entries not accessed in last {{ .Values.gc.actcacheMaxAgeDays }} day(s)
                        find /data/.cache/actcache/cache -type f -mtime +{{ .Values.gc.actcacheMaxAgeDays }} -delete 2>/dev/null || true
                        # Clean up empty directories
                        find /data/.cache/actcache/cache -type d -empty -delete 2>/dev/null || true
                        echo "actcache size: $(du -sh /data/.cache/actcache/cache 2>/dev/null | cut -f1)"
                      ' || true
                    echo ""
                  done
                  echo "GC complete"
              resources:
                requests:
                  cpu: 50m
                  memory: 64Mi
                limits:
                  cpu: 250m
                  memory: 128Mi
 {{- end }}
@@ -2,8 +2,15 @@
 # runner instance. Only runner.name and runner.labels differ -- everything
 # else (image, dind, persistence, tolerations, nodeSelector) is shared.
 #
-# node:22-bookworm ships Node natively, so unlike the golang/rust instances,
+# Label image: node:22-bookworm — Debian, root, apt-get, Node.js, npm, git.
-# jobs on this runner need no "install node" step before actions/checkout.
+# Install docker in workflow steps as needed.
 runner:
  image:
    repository: code.forgejo.org/forgejo/runner
    tag: "6"
  name: node-runner
  labels: "node:docker://node:22-bookworm"
 # GC CronJob renders only from the default (golang) values to avoid duplicates
 gc:
  enabled: false
@@ -2,9 +2,17 @@
 # runner instance. Only runner.name and runner.labels differ -- everything
 # else (image, dind, persistence, tolerations, nodeSelector) is shared.
 #
-# rust:1.83-bookworm -- verified this tag exists (docker manifest inspect)
+# Label image: rust:1-bookworm — Debian, root, apt-get, Rust, cargo, git.
-# before pinning it, per this repo's convention of not trusting a tag exists
+# Install Node.js/docker in workflow steps as needed.
 # without checking.
 runner:
  image:
    repository: code.forgejo.org/forgejo/runner
    tag: "6"
  name: rust-runner
-  labels: "rust:docker://rust:1.83-bookworm"
+  labels: "rust:docker://rust:1-bookworm"
 # GC CronJob renders only from the default (golang) values to avoid duplicates
 gc:
  enabled: false
@@ -1,20 +1,13 @@
 runner:
  image:
    repository: code.forgejo.org/forgejo/runner
-    tag: "6"          # pin exact release before apply
+    tag: "6"
  name: golang-runner
-  # Default image is only used when a job's `container:` doesn't override it
+  # Label image is what workflow steps run in (NOT the runner daemon image).
-  # (both ci.yaml and build.yaml in homelab-frontend do). Retired the old
+  # golang:1.26-bookworm: Debian, root, apt-get, Go, git.
-  # "docker" label entirely; every repo this runner serves is Go, so this
+  # TODO: Switch to custom image once build-runner-images.yml pushes images
-  # instance carries the golang toolchain and its own dind sidecar builds and
+  labels: "golang:docker://golang:1.26-bookworm"
  # pushes that repo's images too -- there is no separate generic runner
  # anymore.
  labels: "golang:docker://golang:1.25-bookworm"
  # In-cluster Service (:3000) — direct, avoids the ingress/public-hostname hop
  # (the public URL is :443 which forgejo doesn't serve; runner got i/o timeout).
  forgejoUrl: http://forgejo-gitea-http.cicd.svc.cluster.local:3000
  # tokenSecret: name of the K8s Secret that holds the runner registration token
  # created automatically by the helmfile presync hook (see helmfile.yaml.gotmpl)
  tokenSecret: runner-token
  resources:
    requests:
@@ -39,7 +32,7 @@ dind:
 persistence:
  reg:
    storageClass: longhorn  # Unified StorageClass (3 replicas)
-    size: 1Gi               # .runner registration file + config — survives pod restarts
+    size: 20Gi              # .runner registration file + action tool cache + actcache artifacts
  dind:
    storageClass: longhorn  # Unified StorageClass (3 replicas)
    size: 30Gi              # docker layer cache — keeps rebuilds fast across restarts
@@ -49,7 +42,18 @@ tolerations:
    operator: Exists
    effect: NoSchedule
-# Pin to az-a (talos-cp-1) — sole Longhorn node; RWO PVCs (reg/dind cache) only
+# Pin to az-b (talos-cp-2) — more Longhorn storage than az-a (worker-1 over-provisioned).
-# attach there.
+# RWO PVCs will recreate on talos-cp-2 when nodeSelector changes.
 nodeSelector:
-  topology.kubernetes.io/zone: az-a
+  topology.kubernetes.io/zone: az-b
 # GC CronJob — prunes Docker images/volumes/build-cache and actcache across
 # ALL forgejo-runner pods. Only enable in default values (golang instance);
 # disable in per-runner overrides so it renders once.
 gc:
  enabled: true
  schedule: "*/30 * * * *"     # every 30 minutes
  image: alpine/k8s:1.31.0
  pruneAge: "30m"              # Docker artifacts unused longer than this get pruned
  pruneAgeHours: 0.5            # Same as pruneAge but numeric for date arithmetic in shell
  actcacheMaxAgeDays: 1        # actcache files older than N days (aggressive for heavy Rust cargo builds)
@@ -1,181 +0,0 @@
 # Authentik OAuth provisioning — PostSync hook, reruns on every ArgoCD sync
 # (hook-delete-policy: BeforeHookCreation deletes the previous run's Job before
 # creating a new one, so this stays reconciled the same way the rest of the
 # cluster does — no separate manual bootstrap step like setup_talos_iam.sh /
 # provision_oidc.py, which never got migrated off the old helmfile workflow).
 #
 # What it does (see scripts/authentik-provision.py docstring): creates the
 # "groups" scope mapping, homelab-admins / grafana-admins groups, the "rock"
 # admin user, OAuth2 providers + Applications for grafana/minio/forgejo/argocd,
 # and binds homelab-admins to all of them. The script is generated into the
 # authentik-provision-script ConfigMap by kustomize configMapGenerator (see
 # kustomization.yaml), not embedded here.
 #
 # RBAC: this Job only touches Secrets (get existing client secrets, create new
 # ones for forgejo/argocd/rock) across the namespaces those services live in.
 # It never touches any other resource type.
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: authentik-provisioner
  namespace: iam
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
  name: authentik-provisioner
 rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "create", "update", "patch"]
 ---
 # One RoleBinding per namespace the script touches (least-privilege: Secrets
 # only, and only in these 5 namespaces — not a cluster-wide ClusterRoleBinding).
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: authentik-provisioner
  namespace: iam
 subjects:
  - kind: ServiceAccount
    name: authentik-provisioner
    namespace: iam
 roleRef:
  kind: ClusterRole
  name: authentik-provisioner
  apiGroup: rbac.authorization.k8s.io
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: authentik-provisioner
  namespace: cicd
 subjects:
  - kind: ServiceAccount
    name: authentik-provisioner
    namespace: iam
 roleRef:
  kind: ClusterRole
  name: authentik-provisioner
  apiGroup: rbac.authorization.k8s.io
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: authentik-provisioner
  namespace: argocd
 subjects:
  - kind: ServiceAccount
    name: authentik-provisioner
    namespace: iam
 roleRef:
  kind: ClusterRole
  name: authentik-provisioner
  apiGroup: rbac.authorization.k8s.io
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: authentik-provisioner
  namespace: logging
 subjects:
  - kind: ServiceAccount
    name: authentik-provisioner
    namespace: iam
 roleRef:
  kind: ClusterRole
  name: authentik-provisioner
  apiGroup: rbac.authorization.k8s.io
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: authentik-provisioner
  namespace: storage
 subjects:
  - kind: ServiceAccount
    name: authentik-provisioner
    namespace: iam
 roleRef:
  kind: ClusterRole
  name: authentik-provisioner
  apiGroup: rbac.authorization.k8s.io
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: authentik-provisioner
  namespace: paperless
 subjects:
  - kind: ServiceAccount
    name: authentik-provisioner
    namespace: iam
 roleRef:
  kind: ClusterRole
  name: authentik-provisioner
  apiGroup: rbac.authorization.k8s.io
 ---
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: authentik-provision
  namespace: iam
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
 spec:
  ttlSecondsAfterFinished: 600
  backoffLimit: 3
  template:
    spec:
      serviceAccountName: authentik-provisioner
      restartPolicy: Never
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: provision
          image: python:3.12-alpine
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          env:
            - name: AUTHENTIK_BOOTSTRAP_TOKEN
              valueFrom:
                secretKeyRef:
                  name: authentik-secrets
                  key: AUTHENTIK_BOOTSTRAP_TOKEN
          volumeMounts:
            - name: script
              mountPath: /script
          command:
            - /bin/sh
            - -c
            - |
              set -e
              echo "waiting for authentik-server..."
              until wget -q -O /dev/null http://authentik-server.iam.svc.cluster.local/-/health/ready/ 2>/dev/null; do
                sleep 5
              done
              echo "installing kubectl (via python urllib - no apk/curl: this"
              echo "container runs as non-root UID 1000 and can't write to"
              echo "apk's directories or /usr/local/bin, both root-owned in"
              echo "the python:3.12-alpine image; /tmp is world-writable)..."
              python3 -c "
              import urllib.request, os, stat
              kver = urllib.request.urlopen('https://dl.k8s.io/release/stable.txt').read().decode().strip()
              url = f'https://dl.k8s.io/release/{kver}/bin/linux/amd64/kubectl'
              urllib.request.urlretrieve(url, '/tmp/kubectl')
              st = os.stat('/tmp/kubectl')
              os.chmod('/tmp/kubectl', st.st_mode | stat.S_IEXEC)
              "
              export PATH="/tmp:$PATH"
              echo "running provisioning script..."
              python3 /script/authentik-provision.py
      volumes:
        - name: script
          configMap:
            name: authentik-provision-script
@@ -1,35 +1,12 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
-# NOTE: no top-level `namespace:` transformer here (removed) - it used to
+
 # force-rewrite metadata.namespace to "iam" on every resource in this
 # kustomization, which was harmless while every manifest here only ever
 # targeted the iam namespace itself. authentik-provision-job.yaml's
 # RoleBindings deliberately target cicd/argocd/logging/storage (least-
 # privilege access for the authentik-provisioner ServiceAccount to touch
 # Secrets in those namespaces) - the namespace transformer would have
 # silently rewritten all of them back to iam, breaking the RBAC. Every
 # manifest in this directory already sets its own explicit
 # metadata.namespace, so dropping the transformer changes nothing for the
 # existing resources/.
 resources:
  - authentik-provision-job.yaml
  - rbac-dashboard-rolebinding.yaml
-# Provisioning/verification python lives in scripts/*.py (real files, linted +
+# IAM provisioning is manual-only (security-sensitive).
-# diff-friendly) and is generated into ConfigMaps here rather than embedded in
+# Script: scripts/iam/authentik-provision.py
-# the job YAML. disableNameSuffixHash keeps the names stable so the Jobs'
+# Run:
-# configMap volume refs and PostSync hook-delete semantics keep working; each
+#   export AUTHENTIK_BOOTSTRAP_TOKEN=$(kubectl -n iam get secret authentik-secrets \
-# hook Job is recreated per sync so it always mounts the latest script.
+#     -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_TOKEN}' | base64 -d)
-configMapGenerator:
+#   python3 scripts/iam/authentik-provision.py
  - name: authentik-provision-script
    namespace: iam
    files:
      - authentik-provision.py=scripts/authentik-provision.py
 generatorOptions:
  disableNameSuffixHash: true
 # authentik-migrations-job.yaml removed — redundant + broken. The authentik
 # `server` entrypoint runs migrations itself; this standalone job lacked the
 # authentik-secrets envFrom (Secret key missing) and always failed.
 # SOPS secrets (*.enc.yaml) handled by ArgoCD SOPS plugin at sync time
 # authentik/vault deployed via ArgoCD Helm source
@@ -1,437 +0,0 @@
 #!/usr/bin/env python3
 """
 Authentik OAuth provisioning - idempotent, safe to re-run (ArgoCD PostSync hook).
 Creates/updates, in order:
  1. A custom "groups" OAuth2 scope mapping (Authentik ships openid/email/profile
     by default but NOT groups - required for ArgoCD RBAC group mapping and
     Grafana's role_attribute_path, both of which read a `groups` claim).
  2. Groups: homelab-admins (is_superuser=true), grafana-admins.
  3. User "rock": created if missing, always (re-)synced into both groups above.
     Password is generated once and only written to the k8s Secret
     rock-credentials (iam ns) the first time the user is created - re-runs
     never rotate an existing password.
  4. OAuth2/OIDC providers + Applications for: grafana, minio, forgejo, argocd.
     Client secrets are read from existing k8s Secrets (grafana-oidc, minio-oidc)
     if present, or generated once and written out (forgejo-oidc, oidc-secret)
     the first time.
  5. PolicyBinding of homelab-admins -> every Application above, so "rock" (and
     anyone else in that group) has guaranteed access regardless of each app's
     default visibility.
 Talks to Authentik over the in-cluster Service (authentik-server.iam.svc:80),
 authenticating with the bootstrap token. Everything is done with GET-then-
 create-or-patch so this can be re-run on every ArgoCD sync without duplicating
 or clobbering objects (PostSync hook, not a one-shot Job with hook-delete).
 kubectl is used only to read/write the small set of Secrets this script
 touches - it shells out rather than using the Python k8s client to keep the
 container image to stdlib Python + the kubectl binary, no pip installs.
 """
 import json
 import os
 import secrets
 import string
 import subprocess
 import sys
 import urllib.error
 import urllib.request
 AUTHENTIK_URL = "http://authentik-server.iam.svc.cluster.local"
 TOKEN = os.environ["AUTHENTIK_BOOTSTRAP_TOKEN"]
 def api(method, path, data=None):
    url = f"{AUTHENTIK_URL}{path}"
    body = json.dumps(data).encode() if data is not None else None
    req = urllib.request.Request(
        url,
        data=body,
        method=method,
        headers={
            "Authorization": f"Bearer {TOKEN}",
            "Content-Type": "application/json",
        },
    )
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            raw = resp.read()
            return resp.status, (json.loads(raw) if raw else {})
    except urllib.error.HTTPError as e:
        raw = e.read()
        try:
            parsed = json.loads(raw) if raw else {}
        except json.JSONDecodeError:
            parsed = {"raw": raw.decode(errors="replace")}
        return e.code, parsed
 def die(msg):
    print(f"FATAL: {msg}", file=sys.stderr)
    sys.exit(1)
 def gen_secret(n=40):
    alphabet = string.ascii_letters + string.digits
    return "".join(secrets.choice(alphabet) for _ in range(n))
 def kubectl_get_secret_key(namespace, name, key):
    """Returns decoded value, or None if the secret/key doesn't exist."""
    p = subprocess.run(
        ["kubectl", "-n", namespace, "get", "secret", name, "-o", f"jsonpath={{.data.{key}}}"],
        capture_output=True, text=True,
    )
    if p.returncode != 0 or not p.stdout.strip():
        return None
    import base64
    return base64.b64decode(p.stdout).decode()
 def kubectl_create_secret(namespace, name, literals: dict, labels: dict = None):
    """Idempotent: create-or-update via dry-run|apply, same pattern used
    elsewhere in this repo (setup_vault.sh, apply-vault-secrets.sh)."""
    args = ["kubectl", "-n", namespace, "create", "secret", "generic", name]
    for k, v in literals.items():
        args += [f"--from-literal={k}={v}"]
    args += ["--dry-run=client", "-o", "yaml"]
    render = subprocess.run(args, capture_output=True, text=True)
    if render.returncode != 0:
        die(f"rendering secret {namespace}/{name}: {render.stderr}")
    apply = subprocess.run(["kubectl", "apply", "-f", "-"], input=render.stdout,
                            capture_output=True, text=True)
    if apply.returncode != 0:
        die(f"applying secret {namespace}/{name}: {apply.stderr}")
    print(f"  secret {namespace}/{name}: {apply.stdout.strip()}")
    if labels:
        # argocd's `$secret:key` substitution only reads Secrets carrying
        # app.kubernetes.io/part-of: argocd — without it OIDC login fails with
        # oauth2 "invalid_client" (empty client_secret sent to the IdP).
        label_args = ["kubectl", "-n", namespace, "label", "secret", name,
                      "--overwrite"] + [f"{k}={v}" for k, v in labels.items()]
        subprocess.run(label_args, capture_output=True, text=True)
 def get_or_create(list_path, create_path, query, payload, patch_existing=None):
    status, res = api("GET", f"{list_path}?{query}")
    if status != 200:
        die(f"GET {list_path}?{query} -> {status} {res}")
    results = res.get("results", [])
    if results:
        obj = results[0]
        if patch_existing:
            status, obj2 = api("PATCH", f"{create_path}{obj['pk']}/", patch_existing)
            if status not in (200, 201):
                die(f"PATCH {create_path}{obj['pk']}/ -> {status} {obj2}")
            return obj2
        return obj
    status, obj = api("POST", create_path, payload)
    if status not in (200, 201):
        die(f"POST {create_path} -> {status} {obj}")
    return obj
 # -----------------------------------------------------------------------------
 print("[1/5] Ensuring custom 'groups' scope mapping exists...")
 groups_mapping = get_or_create(
    "/api/v3/propertymappings/provider/scope/",
    "/api/v3/propertymappings/provider/scope/",
    "scope_name=groups",
    {
        "name": "homelab: groups claim",
        "scope_name": "groups",
        # request.user.ak_groups is deprecated in authentik 2026.x (logs a
        # deprecation warning on every token issue) -> use request.user.groups.
        "expression": (
            "return {\"groups\": [group.name for group in request.user.groups.all()]}"
        ),
    },
    # Force the expression onto the already-created mapping on re-run.
    patch_existing={
        "expression": (
            "return {\"groups\": [group.name for group in request.user.groups.all()]}"
        ),
    },
 )
 GROUPS_MAPPING_PK = groups_mapping["pk"]
 # MinIO maps OIDC users to a MinIO policy via a "policy" claim
 # (MINIO_IDENTITY_OPENID_CLAIM_NAME=policy). Emit consoleAdmin (full admin) for
 # homelab-admins members, readonly for everyone else. Without this claim MinIO
 # assigns no policy and OIDC users get no access.
 _POLICY_EXPR = (
    "return {\"policy\": \"consoleAdmin\" "
    "if request.user.ak_groups.filter(name=\"homelab-admins\").exists() "
    "else \"readonly\"}"
 )
 policy_mapping = get_or_create(
    "/api/v3/propertymappings/provider/scope/",
    "/api/v3/propertymappings/provider/scope/",
    "scope_name=minio",
    {
        "name": "homelab: minio policy claim",
        "scope_name": "minio",
        "expression": _POLICY_EXPR,
    },
    patch_existing={"expression": _POLICY_EXPR},
 )
 POLICY_MAPPING_PK = policy_mapping["pk"]
 # Fetch the standard openid/email/profile mapping pks (shipped by default).
 status, res = api("GET", "/api/v3/propertymappings/provider/scope/")
 by_scope = {m["scope_name"]: m["pk"] for m in res["results"]}
 SCOPE_PKS = [by_scope["openid"], by_scope["email"], by_scope["profile"], GROUPS_MAPPING_PK]
 status, res = api("GET", "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent")
 AUTHORIZATION_FLOW_PK = res["results"][0]["pk"]
 status, res = api("GET", "/api/v3/flows/instances/?slug=default-provider-invalidation-flow")
 INVALIDATION_FLOW_PK = res["results"][0]["pk"]
 status, res = api("GET", "/api/v3/crypto/certificatekeypairs/?has_key=true")
 SIGNING_KEY_PK = res["results"][0]["pk"]
 # -----------------------------------------------------------------------------
 print("[2/5] Ensuring groups homelab-admins / grafana-admins exist...")
 homelab_admins = get_or_create(
    "/api/v3/core/groups/", "/api/v3/core/groups/",
    "name=homelab-admins",
    {"name": "homelab-admins", "is_superuser": True},
 )
 grafana_admins = get_or_create(
    "/api/v3/core/groups/", "/api/v3/core/groups/",
    "name=grafana-admins",
    {"name": "grafana-admins", "is_superuser": False},
 )
 # -----------------------------------------------------------------------------
 print("[3/5] Ensuring user 'rock' exists with admin group membership...")
 status, res = api("GET", "/api/v3/core/users/?username=rock")
 rock_password = None
 if res.get("results"):
    rock = res["results"][0]
    status, rock = api("PATCH", f"/api/v3/core/users/{rock['pk']}/", {
        "groups": [homelab_admins["pk"], grafana_admins["pk"]],
        "is_active": True,
        # email is REQUIRED: Grafana's OIDC login reads the email claim from
        # userinfo; an empty email makes Grafana fall back to a GitHub-style
        # <userinfo>/emails call, which Authentik 404s -> login fails entirely.
        "email": "[email protected]",
    })
    if status not in (200, 201):
        die(f"PATCH user rock -> {status} {rock}")
    print("  rock already exists, group membership synced (password unchanged)")
 else:
    rock_password = gen_secret(24)
    status, rock = api("POST", "/api/v3/core/users/", {
        "username": "rock",
        "name": "Rock",
        "is_active": True,
        # Required for Grafana OIDC (see PATCH branch above).
        "email": "[email protected]",
        "groups": [homelab_admins["pk"], grafana_admins["pk"]],
        "path": "users",
        "type": "internal",
    })
    if status not in (200, 201):
        die(f"POST user rock -> {status} {rock}")
    status, pw_res = api("POST", f"/api/v3/core/users/{rock['pk']}/set_password/",
                          {"password": rock_password})
    if status not in (200, 204):
        die(f"set_password for rock -> {status} {pw_res}")
    kubectl_create_secret("iam", "rock-credentials", {
        "username": "rock",
        "password": rock_password,
    })
    print("  rock created, credentials stored in iam/rock-credentials")
 # -----------------------------------------------------------------------------
 print("[4/5] Ensuring OAuth2 providers + applications for grafana/minio/forgejo/argocd...")
 SERVICES = {
    "grafana": {
        "client_secret_source": ("logging", "grafana-oidc", "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET"),
        "redirect_uris": ["https://grafana.riotpiao.com/login/generic_oauth"],
        "launch_url": "https://grafana.riotpiao.com",
        "display_name": "Grafana",
    },
    "minio": {
        "client_secret_source": ("storage", "minio-oidc", "MINIO_IDENTITY_OPENID_CLIENT_SECRET"),
        "redirect_uris": ["https://minio.riotpiao.com/oauth_callback"],
        "launch_url": "https://minio.riotpiao.com",
        "display_name": "MinIO",
    },
    "forgejo": {
        # No secret exists yet for forgejo - generate + store on first run.
        "client_secret_source": ("cicd", "forgejo-oidc", "CLIENT_SECRET"),
        "generate_if_missing": True,
        "redirect_uris": [
            "https://forgejo.riotpiao.com/user/oauth2/authentik/callback",
            "https://forgejo.riotpiao.com/user/oauth2/openidconnect/callback",
        ],
        "launch_url": "https://forgejo.riotpiao.com",
        "display_name": "Forgejo",
    },
    "argocd": {
        # oidc-secret uses hyphenated keys (client-id/client-secret) per
        # argocd-values.yaml's `$oidc-secret:client-id` / `:client-secret` refs.
        "client_secret_source": ("argocd", "oidc-secret", "client-secret"),
        "generate_if_missing": True,
        "extra_secret_literals": {"client-id": "argocd"},
        # argocd only reads $secret refs from Secrets labelled part-of: argocd.
        "secret_labels": {"app.kubernetes.io/part-of": "argocd"},
        "redirect_uris": ["https://argocd.riotpiao.com/auth/callback"],
        "launch_url": "https://argocd.riotpiao.com",
        "display_name": "Argo CD",
    },
    "homarr": {
        "client_secret_source": ("dashboard", "homarr-oidc", "client-secret"),
        "generate_if_missing": True,
        "extra_secret_literals": {"client-id": "homarr"},
        "redirect_uris": ["https://homarr.riotpiao.com/api/auth/callback/oidc"],
        "launch_url": "https://homarr.riotpiao.com",
        "display_name": "Homarr",
    },
    "paperless": {
        # No secret exists yet for paperless - generate + store on first run.
        # django-allauth's generic openid_connect provider callback path is
        # /accounts/oidc/<provider_id>/login/callback/ - provider_id "authentik"
        # is set in PAPERLESS_SOCIALACCOUNT_PROVIDERS (see configmap.yaml).
        "client_secret_source": ("paperless", "paperless-oidc", "CLIENT_SECRET"),
        "generate_if_missing": True,
        "redirect_uris": ["https://paperless.riotpiao.com/accounts/oidc/authentik/login/callback/"],
        "launch_url": "https://paperless.riotpiao.com",
        "display_name": "Paperless-ngx",
    },
 }
 app_pks_for_binding = []
 for name, cfg in SERVICES.items():
    # MinIO also needs the "policy" claim (via the minio scope mapping) so its
    # MINIO_IDENTITY_OPENID_CLAIM_NAME=policy maps homelab-admins -> consoleAdmin.
    provider_mappings = SCOPE_PKS + ([POLICY_MAPPING_PK] if name == "minio" else [])
    ns, secret_name, key = cfg["client_secret_source"]
    client_secret = kubectl_get_secret_key(ns, secret_name, key)
    if client_secret is None:
        if not cfg.get("generate_if_missing"):
            print(f"  WARNING: {ns}/{secret_name} key {key} not found and "
                  f"generate_if_missing not set for '{name}' - skipping provider/app")
            continue
        client_secret = gen_secret(40)
        literals = {key: client_secret}
        literals.update(cfg.get("extra_secret_literals", {}))
        kubectl_create_secret(ns, secret_name, literals,
                              labels=cfg.get("secret_labels"))
        print(f"  {name}: generated new client secret -> {ns}/{secret_name}")
    else:
        print(f"  {name}: using existing client secret from {ns}/{secret_name}")
    if name == "paperless":
        # paperless-ngx's django-allauth OIDC config takes client_id/secret
        # bundled inside one JSON blob (PAPERLESS_SOCIALACCOUNT_PROVIDERS), not
        # discrete env vars - compose it here and store it alongside
        # CLIENT_SECRET so the Deployment can source it directly via
        # secretKeyRef, no shell wrapper needed. Runs every time (not just on
        # generate), so it stays in sync if the client_secret is ever rotated
        # by hand.
        providers_json = json.dumps({
            "openid_connect": {
                "APPS": [{
                    "provider_id": "authentik",
                    "name": "Authentik",
                    "client_id": "paperless",
                    "secret": client_secret,
                    "settings": {
                        "server_url": "https://authentik.riotpiao.com/application/o/paperless/.well-known/openid-configuration",
                    },
                }],
            },
        })
        kubectl_create_secret("paperless", "paperless-oidc", {
            "CLIENT_SECRET": client_secret,
            "SOCIALACCOUNT_PROVIDERS_JSON": providers_json,
        })
    provider = get_or_create(
        "/api/v3/providers/oauth2/", "/api/v3/providers/oauth2/",
        f"name={name}",
        {
            "name": name,
            "client_id": name,
            "client_secret": client_secret,
            "client_type": "confidential",
            "authorization_flow": AUTHORIZATION_FLOW_PK,
            "invalidation_flow": INVALIDATION_FLOW_PK,
            "signing_key": SIGNING_KEY_PK,
            "property_mappings": provider_mappings,
            "sub_mode": "hashed_user_id",
            "include_claims_in_id_token": True,
            # authentik 2026.x requires grant_types to be set explicitly; the
            # API defaults it to [] when omitted, which makes /authorize reject
            # every login with "Invalid grant_type for provider" ->
            # invalid_request. authorization_code = the web SSO flow all these
            # apps use; refresh_token = long-lived sessions (offline_access).
            "grant_types": ["authorization_code", "refresh_token"],
            "redirect_uris": [
                {"matching_mode": "strict", "url": u} for u in cfg["redirect_uris"]
            ],
        },
        # Keep the redirect_uris/mappings/grant_types in sync on re-run, but
        # never touch client_secret again once created (that's the source of
        # truth in the k8s Secret, and re-sending it here is harmless anyway).
        patch_existing={
            "property_mappings": provider_mappings,
            "grant_types": ["authorization_code", "refresh_token"],
            "redirect_uris": [
                {"matching_mode": "strict", "url": u} for u in cfg["redirect_uris"]
            ],
        },
    )
    # superuser_full_list=true is REQUIRED on the LIST: the applications list
    # applies access-policy filtering to the results array (these apps are bound
    # to homelab-admins, and the bootstrap-token user akadmin is not a member),
    # so without it the GET returns an empty results list even though the app
    # exists -> fall through to POST -> 400 "already exists".
    #
    # We deliberately do NOT patch_existing here: the application DETAIL endpoint
    # (PATCH /applications/{pk}/) enforces the same access policy and does NOT
    # honor superuser_full_list, so PATCH-by-pk returns 404 for akadmin once the
    # homelab-admins binding exists. That 404 aborted the loop before later
    # providers got their grant_types. slug/provider/launch_url are set at
    # creation and are stable (provider is get_or_create'd by name, stable pk),
    # so find-or-create is sufficient.
    application = get_or_create(
        "/api/v3/core/applications/", "/api/v3/core/applications/",
        f"slug={name}&superuser_full_list=true",
        {
            "name": cfg["display_name"],
            "slug": name,
            "provider": provider["pk"],
            "meta_launch_url": cfg["launch_url"],
        },
    )
    app_pks_for_binding.append((name, application["pk"]))
    print(f"  {name}: provider pk={provider['pk']} application pk={application['pk']}")
 # -----------------------------------------------------------------------------
 print("[5/5] Binding homelab-admins to every application (guaranteed access for rock)...")
 for name, app_pk in app_pks_for_binding:
    get_or_create(
        "/api/v3/policies/bindings/", "/api/v3/policies/bindings/",
        f"target={app_pk}&group={homelab_admins['pk']}",
        {
            "target": app_pk,
            "group": homelab_admins["pk"],
            "order": 0,
            "enabled": True,
        },
    )
    print(f"  {name}: homelab-admins bound")
 print("\nDone. Summary:")
 print("  groups:  homelab-admins (superuser), grafana-admins")
 print("  user:    rock -> homelab-admins + grafana-admins")
 print(f"  apps:    {', '.join(n for n, _ in app_pks_for_binding)}")
 if rock_password:
    print("  NOTE: rock's password was generated this run - see")
    print("  kubectl -n iam get secret rock-credentials -o jsonpath='{.data.password}' | base64 -d")
@@ -44,6 +44,10 @@ grafana.ini:
  server:
    root_url: https://grafana.riotpiao.com
  # Allow embedding dashboards in iframes (NextJS integration)
  security:
    allow_embedding: true
  # No anonymous read access — every user must log in via Authentik SSO.
  auth.anonymous:
    enabled: false
@@ -64,15 +68,14 @@ grafana.ini:
  # doesn't return localhost redirects in its token responses.
  #
  # role_attribute_path: JMESPath expression evaluated against the userinfo
-  # response. Members of the 'grafana-admins' Authentik group get Admin role;
+  # response. akadmin gets GrafanaAdmin (server admin, can impersonate);
-  # everyone else gets Viewer. The group name must match exactly what Authentik
+  # homelab-admins members get Admin (org admin); everyone else Viewer.
  # sends in the 'groups' claim.
  auth.generic_oauth:
    enabled: true
    name: Authentik
    allow_sign_up: true
    client_id: grafana
-    scopes: openid email profile
+    scopes: openid email profile groups
    auth_url: https://authentik.riotpiao.com/application/o/authorize/
    token_url: https://authentik.riotpiao.com/application/o/token/
    api_url: https://authentik.riotpiao.com/application/o/userinfo/
@@ -83,7 +86,8 @@ grafana.ini:
    email_attribute_path: email
    login_attribute_path: preferred_username
    name_attribute_path: name
-    role_attribute_path: "contains(groups[*], 'grafana-admins') && 'Admin' || 'Viewer'"
+    role_attribute_path: "preferred_username == 'akadmin' && 'GrafanaAdmin' || contains(groups[*], 'homelab-admins') && 'Admin' || 'Viewer'"
    allow_assign_grafana_admin: true
    use_pkce: false
    use_refresh_token: false
    skip_org_role_sync: false
@@ -10,13 +10,23 @@ apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
  name: longhorn-paperless-media
  annotations:
    # StorageClass.parameters is immutable - any future edit here needs
    # delete+recreate, not patch. Same fix as longhorn-cnpg-storageclass.yaml;
    # safe since a StorageClass is only consulted at provisioning time.
    argocd.argoproj.io/sync-options: Replace=true,Force=true
 provisioner: driver.longhorn.io
 allowVolumeExpansion: true
 reclaimPolicy: Retain
 volumeBindingMode: WaitForFirstConsumer
 parameters:
  numberOfReplicas: "1"
  # diskSelector alone is sufficient: only cp-3 has a disk tagged
  # paperless-media, so placement is already pinned. A nodeSelector value
  # here must be a Longhorn *node* tag (set via nodes.longhorn.io spec.tags),
  # not a Kubernetes hostname - "talos-cp-3" was never a node tag, which made
  # every provision attempt fail with "specified node tag talos-cp-3 does
  # not exist".
  diskSelector: "paperless-media"
  nodeSelector: "talos-cp-3"
  staleReplicaTimeout: "30"
  fsType: "ext4"
@@ -96,8 +96,13 @@ spec:
      console: https://minio.riotpiao.com
  # ── OIDC via Authentik (server-side env, valid in v2 schema) ────────────────
  # Use in-cluster URL for config fetch (pod→authentik); browser redirects use
  # public URLs embedded in the OIDC metadata response (issuer stays public).
  env:
    - name: MINIO_IDENTITY_OPENID_CONFIG_URL
      # Must use external URL — well-known response contains external issuer/jwks_uri.
      # MinIO validates issuer in JWT matches well-known issuer. Internal URL = mismatch.
      # Hairpins through ingress-nginx but stays in-cluster.
      value: "https://authentik.riotpiao.com/application/o/minio/.well-known/openid-configuration"
    - name: MINIO_IDENTITY_OPENID_CLIENT_ID
      value: "minio"
@@ -0,0 +1,160 @@
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
 metadata:
  name: api-gateway-alerts
  namespace: monitoring
  labels:
    release: prometheus
 spec:
  groups:
    # ================================================================
    # SLA Targets (based on canary traffic baselines):
    #
    # Availability:  99.9% (43.8 min downtime/month)
    # LLM Chat:      p95 < 1s (qwen), p95 < 2s (reasoning), p95 < 5s (ornith)
    # Embeddings:    p95 < 500ms
    # Rerank:        p95 < 500ms
    # Models list:   p95 < 300ms
    # Error rate:    < 1% (5xx), < 5% (4xx excluding auth)
    #
    # Baselines from 200-request canary run:
    #   qwen p99=609ms, reasoning p99=328ms, embeddings p99=287ms,
    #   rerank p99=218ms, models p99=277ms
    # SLA set at ~2x p99 for headroom.
    # ================================================================
    - name: api-gateway.availability
      rules:
        # Gateway pods not ready
        - alert: APIGatewayDown
          expr: sum(kube_pod_status_ready{namespace="api",condition="true"}) == 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "API Gateway has zero ready pods"
        # Gateway pod count below desired
        - alert: APIGatewayDegraded
          expr: |
            sum(kube_pod_status_ready{namespace="api",condition="true"})
            < kube_deployment_spec_replicas{namespace="api",deployment="api-gateway"}
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "API Gateway {{ $value }} ready pods below desired replica count"
        # Blackbox probe down
        - alert: APIGatewayProbeDown
          expr: probe_success{instance=~".*api.riotpiao.com.*"} == 0
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "API Gateway probe failed: {{ $labels.instance }}"
        # LLM serving pods not ready
        - alert: LLMServingDown
          expr: sum(kube_pod_status_ready{namespace="llm-serving",condition="true"}) == 0
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "All LLM serving pods down"
        # Individual predictor down
        - alert: LLMPredictorDown
          expr: |
            kube_deployment_status_replicas_ready{namespace="llm-serving"}
            < kube_deployment_spec_replicas{namespace="llm-serving"}
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.deployment }} has {{ $value }} ready (below desired)"
    - name: api-gateway.latency
      # SLA: latency thresholds at ~2x measured p99
      rules:
        # Ingress-level latency (all requests through nginx)
        - alert: APIGatewayLatencyHigh
          expr: |
            histogram_quantile(0.95,
              sum(rate(nginx_ingress_controller_request_duration_seconds_bucket{ingress="api"}[5m])) by (le)
            ) > 2
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "API Gateway p95 latency {{ $value | printf \"%.1f\" }}s (SLA: <2s)"
        # Extreme latency (p99 > 5s)
        - alert: APIGatewayLatencyCritical
          expr: |
            histogram_quantile(0.99,
              sum(rate(nginx_ingress_controller_request_duration_seconds_bucket{ingress="api"}[5m])) by (le)
            ) > 5
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "API Gateway p99 latency {{ $value | printf \"%.1f\" }}s (SLA: <5s)"
    - name: api-gateway.errors
      rules:
        # 5xx error rate > 1%
        - alert: APIGateway5xxErrorRate
          expr: |
            sum(rate(nginx_ingress_controller_requests{ingress="api",status=~"5.."}[5m]))
            / sum(rate(nginx_ingress_controller_requests{ingress="api"}[5m]))
            > 0.01
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "API Gateway 5xx rate {{ $value | humanizePercentage }} (SLA: <1%)"
        # Total error rate > 10% (including 4xx)
        - alert: APIGatewayHighErrorRate
          expr: |
            sum(rate(nginx_ingress_controller_requests{ingress="api",status=~"[45].."}[5m]))
            / sum(rate(nginx_ingress_controller_requests{ingress="api"}[5m]))
            > 0.10
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "API Gateway total error rate {{ $value | humanizePercentage }} (SLA: <10%)"
    - name: api-gateway.resources
      rules:
        # Gateway pod restart
        - alert: APIGatewayRestarted
          expr: increase(kube_pod_container_status_restarts_total{namespace="api"}[15m]) > 0
          for: 0m
          labels:
            severity: warning
          annotations:
            summary: "API Gateway pod {{ $labels.pod }} restarted"
        # LLM predictor restart
        - alert: LLMPredictorRestarted
          expr: increase(kube_pod_container_status_restarts_total{namespace="llm-serving"}[15m]) > 0
          for: 0m
          labels:
            severity: warning
          annotations:
            summary: "LLM predictor {{ $labels.pod }} restarted"
        # Gateway high memory (>80% of limit)
        - alert: APIGatewayHighMemory
          expr: |
            sum(container_memory_working_set_bytes{namespace="api",container="gateway"}) by (pod)
            / sum(kube_pod_container_resource_limits{namespace="api",container="gateway",resource="memory"}) by (pod)
            > 0.8
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Gateway pod {{ $labels.pod }} memory at {{ $value | humanizePercentage }} of limit"
@@ -0,0 +1,177 @@
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
 metadata:
  name: cluster-alerts
  namespace: monitoring
  labels:
    release: prometheus
 spec:
  groups:
    - name: cluster.availability
      rules:
        # Node down
        - alert: NodeNotReady
          expr: kube_node_status_condition{condition="Ready",status="true"} == 0
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "Node {{ $labels.node }} is NotReady"
        # Pod stuck pending (scheduling failure)
        - alert: PodStuckPending
          expr: sum(kube_pod_status_phase{phase="Pending"}) > 0
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "{{ $value }} pod(s) stuck in Pending state for >10m"
        # CrashLoopBackOff
        - alert: PodCrashLooping
          expr: sum(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}) by (namespace, pod) > 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "{{ $labels.namespace }}/{{ $labels.pod }} in CrashLoopBackOff"
        # OOMKilled spike
        - alert: OOMKilledSpike
          expr: sum(increase(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[1h])) > 3
          for: 0m
          labels:
            severity: warning
          annotations:
            summary: "{{ $value }} OOMKilled events in last hour"
        # Deployment replicas unavailable
        - alert: DeploymentReplicasUnavailable
          expr: kube_deployment_status_replicas_unavailable > 0
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.namespace }}/{{ $labels.deployment }} has {{ $value }} unavailable replicas"
    - name: cluster.jobs
      rules:
        # Job failed
        - alert: JobFailed
          expr: kube_job_status_failed > 0
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Job {{ $labels.namespace }}/{{ $labels.job_name }} failed"
        # Job stuck running >2h
        - alert: JobStuckRunning
          expr: |
            kube_job_status_active == 1
            and on(job_name,namespace)
            (time() - kube_job_status_start_time) > 7200
          for: 0m
          labels:
            severity: warning
          annotations:
            summary: "Job {{ $labels.namespace }}/{{ $labels.job_name }} running >2h"
        # CronJob missed schedule
        - alert: CronJobMissedSchedule
          expr: |
            (time() - kube_cronjob_status_last_schedule_time) > 2 * (kube_cronjob_spec_next_schedule_time - kube_cronjob_status_last_schedule_time)
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "CronJob {{ $labels.namespace }}/{{ $labels.cronjob }} missed schedule"
    - name: cluster.resources
      rules:
        # Node CPU >90% sustained
        - alert: NodeHighCPU
          expr: (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)) * 100 > 90
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Node {{ $labels.instance }} CPU at {{ $value | printf \"%.0f\" }}%"
        # Node memory >90% sustained
        - alert: NodeHighMemory
          expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Node {{ $labels.instance }} memory at {{ $value | printf \"%.0f\" }}%"
        # Node disk >85%
        - alert: NodeDiskFull
          expr: (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 > 85
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Node {{ $labels.instance }} disk at {{ $value | printf \"%.0f\" }}%"
        # Container restart storm (>5 restarts in 15m)
        - alert: ContainerRestartStorm
          expr: sum(increase(kube_pod_container_status_restarts_total[15m])) by (namespace, pod) > 5
          for: 0m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.namespace }}/{{ $labels.pod }} restarted {{ $value | printf \"%.0f\" }} times in 15m"
    - name: cluster.storage
      rules:
        # Longhorn drive offline
        - alert: LonghornDriveOffline
          expr: longhorn_disk_health != 1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Longhorn disk {{ $labels.node }} unhealthy"
    - name: cluster.dns
      rules:
        # CoreDNS errors spike
        - alert: CoreDNSErrorSpike
          expr: sum(rate(coredns_dns_responses_total{rcode=~"SERVFAIL"}[5m])) > 0.5
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "CoreDNS SERVFAIL rate {{ $value | printf \"%.2f\" }}/s"
    - name: cluster.probes
      rules:
        # Any blackbox probe down
        - alert: ServiceProbeDown
          expr: probe_success == 0
          for: 3m
          labels:
            severity: critical
          annotations:
            summary: "Probe failed: {{ $labels.instance }}"
        # Probe latency >2s
        - alert: ServiceProbeSlow
          expr: probe_duration_seconds > 2
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Probe slow ({{ $value | printf \"%.1f\" }}s): {{ $labels.instance }}"
        # Certificate expiry <14 days
        - alert: CertificateExpiringSoon
          expr: (certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 14
          for: 0m
          labels:
            severity: warning
          annotations:
            summary: "Certificate {{ $labels.name }} expires in {{ $value | printf \"%.0f\" }} days"
@@ -61,6 +61,10 @@ serviceMonitor:
      url: https://argocd.riotpiao.com/healthz
    - name: longhorn
      url: https://longhorn.riotpiao.com/
    - name: api-gateway
      url: https://api.riotpiao.com/healthz
    - name: api-gateway-models
      url: https://api.riotpiao.com/v1/models
 prometheusRule:
  enabled: true
--- a/Show More
+++ b/Show More
		`@@ -0,0 +1,2 @@`
							`FORGEJO_TOKEN=273fdcffabbcbb5a191e8289c73d106063acefc6`
							`LLM_API_TOKEN=s3VksXyw2z3sGbegnwjMDFnJ6CtNRd1a5CcnE5A4ET77toCcykNdunk6Oa2J`