fix(argocd): update repoURL from rock/homelab to riotpiao-poimen/homelab

Repo transferred to riotpiao-poimen org. Old URL returns 301 redirect which ArgoCD doesn't follow — all 26 apps stopped syncing. Live cluster already patched via kubectl. This commit persists the fix.
feat(gotify): push notification server + SMTP email relay (#19 )
2026-09-10 10:41:04 +09:00 · 2026-09-10 01:35:59 +00:00 · 2026-09-09 22:38:25 +00:00 · 2026-09-09 02:11:28 +00:00 · 2026-09-08 15:29:42 -07:00 · 2026-09-08 16:56:36 +00:00
255 changed files with 11008 additions and 5618 deletions
@@ -1,460 +0,0 @@
 # CI/CD Pipeline: GitOps Validation & Deployment
 ## Overview
 Pure GitOps CI/CD pipeline using Forgejo Actions (self-hosted runner).
 **Principle:** Validate in CI, deploy via ArgoCD (no manual steps).
 ```
 git push
  ↓
  [CI: Validate]
    ├─ yamllint (YAML syntax)
    ├─ kubeval (K8s manifests)
    ├─ kustomize build (all layers)
    ├─ argocd validation (app definitions)
    └─ security scan (secrets, best practices)
  ↓
  [If push to main]
    └─ ArgoCD auto-syncs (if enabled)
 ```
 ## Workflows
 ### 1. validate-k8s.yaml (Mandatory)
 **Trigger:** Any push/PR with k8s/ changes
 **What it does:**
 1. Lints all YAML files (`yamllint`)
 2. Validates K8s manifests (`kubeval`)
 3. Builds all kustomization layers
 4. Validates ArgoCD applications
 5. Reports results
 **Duration:** ~2-3 minutes
 **Status:**
 - ✅ PASS: All layers build, manifests valid → OK to merge
 - ❌ FAIL: Syntax error, invalid resource, build failed → Fix & push again
 **Example output:**
 ```
 === Building k8s/infrastructure/ ===
 ✓ Infrastructure built successfully
 Resources: 47
 === Building k8s/bootstrap/ ===
 ✓ Bootstrap built successfully
 Resources: 23
 ```
 **When to check:**
 - After every commit
 - Before merging PRs
 - On every branch
 ### 2. argocd-sync.yaml (Recommended)
 **Trigger:** Push to main only (k8s/ changed)
 **What it does:**
 1. Authenticates with ArgoCD
 2. Syncs `homelab-root` application
 3. Waits for sync to complete (5 min timeout)
 4. Verifies all applications healthy
 **Duration:** 1-5 minutes (depends on resources)
 **Status:**
 - ✅ SYNCED: All resources deployed to cluster
 - ❌ FAILED: Sync error, pod crashes, etc. → Check ArgoCD UI for details
 **When it runs:**
 - Automatically after merge to main
 - Only on k8s/ changes (not on docs)
 **Manual trigger (if needed):**
 ```bash
 # SSH to runner or use Forgejo UI
 # Re-run failed workflow
 # Or manually sync: argocd app sync homelab-root
 ```
 **Requires secrets:**
 - `ARGOCD_SERVER`: ArgoCD server URL (https://argocd.riotpiao.com)
 - `ARGOCD_AUTH_TOKEN`: ArgoCD API token (generate via ArgoCD UI)
 ### 3. security-scan.yaml (Optional)
 **Trigger:** Any push/PR with k8s/ changes
 **What it does:**
 1. Scans Dockerfiles for vulnerabilities (`trivy`)
 2. Scans Helm charts for security issues
 3. Audits K8s manifests (`polaris`)
 4. Checks for hardcoded secrets
 5. Verifies security best practices
 **Duration:** ~3-5 minutes
 **Status:**
 - ✅ PASS: No critical issues
 - ⚠️ WARNING: Best practice recommendations (non-blocking)
 - ❌ FAIL: Hardcoded secrets found (must fix)
 **Common issues:**
 - Missing resource limits (warning)
 - Privileged containers (warning)
 - Hardcoded passwords (ERROR)
 ---
 ## File Structure
 ```
 .forgejo/
 ├── workflows/                      # CI/CD workflows
 │   ├── validate-k8s.yaml          # Validate manifests (required)
 │   ├── argocd-sync.yaml           # Sync to cluster (auto on main)
 │   └── security-scan.yaml         # Security checks (optional)
 └── CI-CD.md                        # This file
 ```
 ---
 ## Setup Instructions
 ### 1. Install Forgejo Runner
 ```bash
 # On runner machine (inside cluster or external)
 forgejo-runner register \
  --instance https://forgejo.riotpiao.com \
  --token <registration-token> \
  --name homelab-runner \
  --labels docker
 forgejo-runner daemon
 ```
 ### 2. Add ArgoCD Secrets to Forgejo
 ```bash
 # Go to: Forgejo → Settings → Secrets
 # Add:
 ARGOCD_SERVER = https://argocd.riotpiao.com
 ARGOCD_AUTH_TOKEN = <token>  # Generate: argocd account generate-token
 ```
 ### 3. Generate ArgoCD Token
 ```bash
 # Inside cluster
 kubectl -n argocd port-forward svc/argocd-server 8080:443
 # Go to: https://localhost:8080/user-info/api-tokens
 # Create new token (CI/CD)
 # Copy token to Forgejo secrets
 ```
 ---
 ## Workflow Execution
 ### When developer pushes to feature branch:
 ```
 git push origin feature/new-service
 ↓
 Forgejo Actions triggered
 ↓
 validate-k8s.yaml runs:
  ✓ Lints YAML
  ✓ Validates manifests
  ✓ Builds kustomizations
  ✓ All pass → GitHub comment: "Ready to merge"
 ↓
 Developer opens PR
 ↓
 Reviewer checks:
  - Code changes (YAML)
  - Workflow results
  - ArgoCD impact (diff)
 ↓
 PR merged to main
 ```
 ### When merged to main:
 ```
 git merge feature/new-service → main
 ↓
 Forgejo Actions triggered
 ↓
 validate-k8s.yaml runs:
  ✓ Same validation as above
 ↓
 argocd-sync.yaml runs (if enabled):
  ✓ Syncs homelab-root
  ✓ Waits for sync
  ✓ Verifies health
  ✓ Resources deployed to cluster
 ↓
 Cluster state = git state
 (No manual kubectl apply needed!)
 ```
 ---
 ## Debugging CI/CD Failures
 ### Issue: "Kustomize build failed"
 ```bash
 # Run locally
 cd k8s/
 kustomize build bootstrap/  # See actual error
 # Fix YAML/kustomization.yaml
 # git push again
 ```
 ### Issue: "Kubeval validation failed"
 ```bash
 # Check K8s manifest syntax
 kubeval k8s/platform/minio/config.yaml
 # Common issues:
 # - Typos in apiVersion, kind, metadata
 # - Missing required fields
 # - Invalid references (namespace, service name)
 ```
 ### Issue: "ArgoCD sync failed"
 ```bash
 # Check ArgoCD UI
 # https://argocd.riotpiao.com → homelab-root
 # Or CLI
 argocd app get homelab-root
 argocd app logs homelab-root --follow
 # Common issues:
 # - Missing namespace (fixed by infrastructure layer)
 # - Invalid Helm chart version
 # - Secret not found
 # - Network policy blocking traffic
 ```
 ### Issue: "Security scan found hardcoded secret"
 ```bash
 # Fix: Remove secret from YAML
 # Add to SOPS encryption instead
 # Or use ArgoCD Sealed Secrets
 # (if SOPS not available)
 ```
 ---
 ## Viewing Results
 ### Forgejo Actions UI
 ```
 Repository → Actions
  ├─ validate-k8s
  │   ├─ ✅ Success (merge safe)
  │   ├─ ❌ Failed (fix required)
  │   └─ Logs (click "Steps" → "Summary")
  ├─ argocd-sync
  │   ├─ ✅ Synced (deployed)
  │   └─ ❌ Failed (check ArgoCD UI)
  └─ security-scan
      ├─ ✅ Pass (no critical issues)
      └─ ⚠️  Warning (review, non-blocking)
 ```
 ### ArgoCD UI
 ```
 https://argocd.riotpiao.com
  ├─ homelab-root
  │   ├─ Status: Synced ✓
  │   ├─ Health: Healthy ✓
  │   └─ Details (click to see resources)
  ├─ layer-1-bootstrap
  ├─ layer-2-platform
  ├─ layer-3-security
  ├─ layer-4-applications
  └─ layer-5-data
 ```
 ---
 ## Common Tasks
 ### Add new service to cluster
 ```bash
 # 1. Create directory and kustomization.yaml
 mkdir -p k8s/applications/my-service
 cat > k8s/applications/my-service/kustomization.yaml << EOF
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: my-namespace
 helmCharts:
 - name: my-chart
  repo: https://charts.example.com
  version: 1.0.0
  releaseName: my-service
  valuesFile: values.yaml
 EOF
 # 2. Add values.yaml
 cp /template/values.yaml k8s/applications/my-service/
 # 3. Commit and push
 git add k8s/applications/my-service/
 git commit -m "feat(apps): add my-service"
 git push
 # 4. CI validates
 # 5. Merge to main
 # 6. ArgoCD syncs automatically
 # ✓ Service deployed to cluster
 ```
 ### Rollback a deployment
 ```bash
 # 1. Find broken commit
 git log --oneline k8s/  # Identify bad commit
 # 2. Revert
 git revert <commit-hash>
 git push
 # 3. CI validates (should pass)
 # 4. Merge to main
 # 5. ArgoCD syncs back to previous version
 # ✓ Cluster state reverted
 ```
 ### Emergency: Disable ArgoCD auto-sync
 ```bash
 # If production broken and need time to debug:
 argocd app set homelab-root --sync-policy none
 # Fix issue in git
 # Test locally: kustomize build k8s/
 # Re-enable
 argocd app set homelab-root --sync-policy automated
 argocd app sync homelab-root
 ```
 ---
 ## Monitoring & Alerts
 ### Check workflow status in Forgejo
 ```bash
 # Dashboard shows:
 ✅ All green → Safe to merge
 ❌ Red → Fix required before merge
 ⏳ Yellow → Still running (wait)
 ```
 ### Check ArgoCD status
 ```bash
 argocd app list
 # Shows: Synced, OutOfSync, Unknown status
 argocd app get homelab-root
 # Shows: health, sync status, resources
 argocd app logs homelab-root --follow
 # Real-time logs during sync
 ```
 ### Alerts (optional, future)
 ```yaml
 # Could add Forgejo webhooks → Slack/email
 # When CI/CD fails → Alert ops team
 # When ArgoCD goes OutOfSync → Alert ops team
 ```
 ---
 ## Troubleshooting
 ### Workflow doesn't trigger
 **Check:**
 - Is Forgejo runner running? `forgejo-runner daemon`
 - Did you push to correct branch? (validate runs on all, argocd-sync only on main)
 - Did path match filter? (must change k8s/ or .forgejo/workflows/)
 ### Workflow hangs/times out
 **Check:**
 - kustomize build → Check for dependency cycles
 - argocd sync → Check cluster resources (storage full? network down?)
 - security scan → Large image scan → Takes time
 **Fix:**
 - Increase timeout in workflow
 - Optimize kustomization (remove unused resources)
 - Add resource limits to pods
 ### ArgoCD token invalid
 **Fix:**
 ```bash
 # Regenerate token
 argocd account generate-token
 # Update Forgejo secret
 # Settings → Secrets → ARGOCD_AUTH_TOKEN = <new-token>
 ```
 ---
 ## Best Practices
 ✅ **DO:**
 - Commit all K8s changes to git (no manual kubectl apply)
 - Run validate-k8s locally before push
 - Write descriptive commit messages (why this change?)
 - Review workflow logs before merging
 - Monitor ArgoCD sync after merge
 ❌ **DON'T:**
 - Push directly to main (always use PR)
 - Skip workflow validation (it catches errors early)
 - Ignore security scan warnings
 - Manually `kubectl apply` (breaks GitOps)
 - Edit resources in cluster (they revert via ArgoCD)
 ---
 ## Next Steps
 1. **Setup Forgejo runner** (if not already running)
 2. **Add ArgoCD secrets** to Forgejo
 3. **Test workflows** on feature branch
 4. **Merge to main** → Watch ArgoCD sync
 5. **Celebrate:** Full GitOps pipeline working! 🎉
@@ -1,247 +0,0 @@
 name: Cluster CI Pipeline
 on:
  push:
    branches:
      - main
      - develop
    paths:
      - 'k8s/**'
      - '.forgejo/workflows/cluster-ci.yaml'
  pull_request:
    paths:
      - 'k8s/**'
 jobs:
  ci:
    runs-on: docker
    steps:
      # === Checkout ===
      - name: Checkout
        run: |
          REPO_URL="${{ gitea.server_url }}/${{ gitea.repository }}.git"
          CLONE_URL="https://${{ secrets.CI_RUNNER }}:${{ secrets.CI_RUNNER_SECRET }}@${REPO_URL#https://}"
          git clone --depth 1 "$CLONE_URL" .
          git fetch origin main
          git checkout main
      # === Install Tools ===
      - name: Install Tools
        run: |
          unset GITHUB_TOKEN
          apt-get update && apt-get install -y \
            yamllint \
            python3-pip \
            curl \
            jq
          # kubeval
          curl -L https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz | tar xz
          mv -f kubeval /usr/local/bin/
          # kustomize
          rm -f kustomize
          curl -s https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh | bash
          mv -f kustomize /usr/local/bin/
          # argocd
          curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
          chmod +x /usr/local/bin/argocd
          # trivy
          curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
          # polaris
          curl -L https://github.com/FairwindsOps/polaris/releases/latest/download/polaris-linux-amd64 -o /usr/local/bin/polaris
          chmod +x /usr/local/bin/polaris
      # === YAML Lint ===
      - name: YAML Lint
        run: |
          echo "=== Linting YAML files ==="
          yamllint k8s/ -c .yamllint.yaml || true
      # === Kubeval - Validate K8s Syntax ===
      - name: Kubeval - Validate K8s Syntax
        run: |
          echo "=== Validating Kubernetes manifests ==="
          find k8s -name "*.yaml" -o -name "*.yml" | grep -v "\.archive" | while read file; do
            echo "Validating $file..."
            kubeval "$file" -d 2>/dev/null || true
          done
      # === Kustomize Build - All overlays ===
      - name: Kustomize Build - Infrastructure
        run: |
          echo "=== Building k8s/infrastructure/ ==="
          kustomize build k8s/infrastructure > /tmp/infrastructure.yaml
          echo "✓ Infrastructure built successfully"
          echo "Resources: $(grep -c 'kind:' /tmp/infrastructure.yaml)"
      - name: Kustomize Build - Bootstrap
        run: |
          echo "=== Building k8s/bootstrap/ ==="
          kustomize build k8s/bootstrap > /tmp/bootstrap.yaml
          echo "✓ Bootstrap built successfully"
          echo "Resources: $(grep -c 'kind:' /tmp/bootstrap.yaml || echo 0)"
      - name: Kustomize Build - Platform
        run: |
          echo "=== Building k8s/platform/ ==="
          kustomize build k8s/platform > /tmp/platform.yaml
          echo "✓ Platform built successfully"
          echo "Resources: $(grep -c 'kind:' /tmp/platform.yaml || echo 0)"
      - name: Kustomize Build - Security
        run: |
          echo "=== Building k8s/security/ ==="
          kustomize build k8s/security > /tmp/security.yaml
          echo "✓ Security built successfully"
          echo "Resources: $(grep -c 'kind:' /tmp/security.yaml || echo 0)"
      - name: Kustomize Build - Applications
        run: |
          echo "=== Building k8s/applications/ ==="
          kustomize build k8s/applications > /tmp/applications.yaml
          echo "✓ Applications built successfully"
          echo "Resources: $(grep -c 'kind:' /tmp/applications.yaml || echo 0)"
      - name: Kustomize Build - Data
        run: |
          echo "=== Building k8s/data/ ==="
          kustomize build k8s/data > /tmp/data.yaml
          echo "✓ Data built successfully"
          echo "Resources: $(grep -c 'kind:' /tmp/data.yaml || echo 0)"
      - name: Validate ArgoCD Applications
        run: |
          echo "=== Validating ArgoCD Applications ==="
          kubeval k8s/argocd/apps/*.yaml
      # === Trivy - Scan Dockerfile ===
      - name: Trivy - Scan Dockerfile
        run: |
          if find . -name "Dockerfile" 2>/dev/null | grep -v node_modules | head -1 | grep -q .; then
            echo "=== Scanning Dockerfiles with Trivy ==="
            find . -name "Dockerfile" -not -path "*/node_modules/*" -exec trivy config {} \;
          else
            echo "No Dockerfiles found"
          fi
      # === Trivy - Scan Helm Charts ===
      - name: Trivy - Scan Helm Charts
        run: |
          if find k8s -name "Chart.yaml" 2>/dev/null | head -1 | grep -q .; then
            echo "=== Scanning Helm charts with Trivy ==="
            find k8s -name "Chart.yaml" -exec dirname {} \; | while read chart; do
              echo "Scanning $chart..."
              trivy config "$chart" || true
            done
          else
            echo "No Helm charts found"
          fi
      # === Polaris - K8s Security Audit ===
      - name: Polaris - K8s Security Audit
        run: |
          echo "=== Running Polaris K8s security audit ==="
          polaris audit --audit-path /tmp/polaris-audit.json k8s/ || true
          if [ -f /tmp/polaris-audit.json ]; then
            echo "Security issues found:"
            jq '.results[] | select(.pass == false)' /tmp/polaris-audit.json || true
          fi
      # === Check for Secrets in Code ===
      - name: Check for Secrets in Code
        run: |
          echo "=== Scanning for hardcoded secrets ==="
          SECRETS_FOUND=0
          for pattern in "password:" "secret:" "token:" "api_key:" "apikey:" "private_key:" "privatekey:"; do
            if grep -r "$pattern" k8s/ --include="*.yaml" --include="*.yml" | grep -v "^Binary"; then
              echo "⚠️  Found potential secret pattern: $pattern"
              SECRETS_FOUND=$((SECRETS_FOUND + 1))
            fi
          done
          if [ $SECRETS_FOUND -gt 0 ]; then
            echo "⚠️  Warning: Found $SECRETS_FOUND potential secrets"
            echo "Secrets should be encrypted with SOPS or stored in ArgoCD Sealed Secrets"
          else
            echo "✓ No hardcoded secrets found"
          fi
      # === Check K8s Security Best Practices ===
      - name: Check K8s Security Best Practices
        run: |
          echo "=== Checking K8s security best practices ==="
          if grep -r "privileged: true" k8s/ --include="*.yaml" --include="*.yml"; then
            echo "⚠️  Found privileged containers"
          fi
          if grep -r "hostNetwork: true" k8s/ --include="*.yaml" --include="*.yml"; then
            echo "⚠️  Found hostNetwork usage"
          fi
          echo "Checking for missing resource limits..."
          MISSING=0
          find k8s -name "*.yaml" -o -name "*.yml" | while read file; do
            if grep -q "kind: Deployment\|kind: StatefulSet\|kind: DaemonSet" "$file"; then
              if ! grep -q "resources:" "$file"; then
                echo "⚠️  $file: Missing resource requests/limits"
                MISSING=$((MISSING + 1))
              fi
            fi
          done
      # === ArgoCD Sync (main branch only) ===
      - name: Sync ArgoCD
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        env:
          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
        run: |
          echo "=== Syncing homelab-root ==="
          argocd app sync homelab-root --force
          argocd app wait homelab-root --timeout 5m
      - name: Check Sync Status
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        env:
          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
        run: |
          echo "=== ArgoCD Applications Status ==="
          argocd app list -o table
          STATUS=$(argocd app get homelab-root -o jsonpath='{.status.syncStatus}')
          if [ "$STATUS" != "Synced" ]; then
            echo "❌ Root app sync failed: $STATUS"
            exit 1
          fi
          echo "✓ Root app synced successfully"
      - name: Health Check
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        env:
          ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
          ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
        run: |
          echo "=== Checking Application Health ==="
          argocd app get homelab-root -o wide
      # === Summary ===
      - name: Summary
        if: always()
        run: |
          echo "=== CI Pipeline Summary ==="
          echo "✓ YAML linted"
          echo "✓ Manifests validated"
          echo "✓ Kustomizations built"
          echo "✓ Security scans completed"
          echo "✓ Secrets check passed"
          echo "✓ Best practices verified"
          echo ""
          echo "✓ All checks passed"
@@ -50,10 +50,22 @@ terraform/*.tfstate.*
 terraform.tfvars.local
 skills-lock.json
 secrets-plaintext.yaml
-skills-lock.json
+
 # Saved plan files — binary, environment-specific, may embed resource attributes
 terraform/tfplan
 terraform/tfplan-*
 .DS_Store
 CLAUDE.md
 docs/
-k8s/argocd/seed-repo-secret.yamlbootstrap-argocd.log
+bootstrap-argocd.log
 # Any plaintext (non-SOPS) secret manifest. Encrypted ones are *.enc.yaml and
 # ARE committed — see .sops.yaml. A missing newline once merged two patterns on
 # one line here, which is how a plaintext deploy key reached a public remote.
 k8s/**/*-secret.yaml
 !k8s/**/*.enc.yaml
 # IAM provisioning scripts contain credential references — never commit
 scripts/iam/*.py
@@ -1,3 +1,5 @@
 creation_rules:
-  - path_regex: k8s/.*secrets.*\.ya?ml
+  # `secrets?` — singular too. A `seed-repo-secret.yaml` once slipped this regex
-    age: age1smu533f803gmd0jq60s2zaj9zlznajy0ca6rtewd4r37mr2hs3uqsrldfh
+  # and was committed in plaintext to a public remote.
  - path_regex: k8s/.*secrets?.*\.ya?ml
    age: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
@@ -0,0 +1,206 @@
 # Authentik Auth Integration for NextJS
 ## Current State
 ### Gateway Auth Status
 | Endpoint | Auth Status | Notes |
 |----------|-------------|-------|
 | `/v1/chat/completions` | ❌ **OFF** | LLM routes have no auth middleware |
 | `/v1/embeddings` | ❌ **OFF** | Same - no auth |
 | `/v1/rerank` | ❌ **OFF** | Same - no auth |
 | `X-Service: sqs` | ✅ **ON** | JWT validated via `internal/auth/jwt.go` |
 | `/workflow` | ❌ **OFF** | Pass-through to Temporal |
 **Auth module exists** at `homelab-frontend/internal/auth/jwt.go` but only wired for SQS.
 LLM routes in `internal/proxy/proxy.go` have no auth middleware.
 ### Authentik App
 Authentik app `local-llm` exists for LLM API auth:
 - **Client ID**: `local-llm`
 - **Client Secret**: `kubectl -n llm-serving get secret local-llm-jwt -o jsonpath='{.data.client-secret}' | base64 -d`
 - **Token endpoint**: `https://authentik.riotpiao.com/application/o/token/`
 - **Userinfo endpoint**: `https://authentik.riotpiao.com/application/o/userinfo/`
 - **OIDC discovery**: `https://authentik.riotpiao.com/application/o/local-llm/.well-known/openid-configuration`
 ## Sign-in Methods
 ### 1. Resource Owner Password Credentials (ROPC)
 Direct username/password login. Server-side only (needs client_secret).
 ```typescript
 // API Route: app/api/auth/login/route.ts
 const response = await fetch('https://authentik.riotpiao.com/application/o/token/', {
  method: 'POST',
  headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
  body: new URLSearchParams({
    grant_type: 'password',
    client_id: 'local-llm',
    client_secret: process.env.AUTHENTIK_CLIENT_SECRET,
    username: '[email protected]',
    password: 'userpassword',
    scope: 'openid email profile groups',
  }),
 });
 const tokens = await response.json();
 // { access_token, refresh_token, expires_in, token_type }
 ```
 ### 2. Authorization Code Flow (Browser Redirect)
 Requires adding redirect URIs to `local-llm` Authentik app:
 ```python
 # In k8s/infra/iam/scripts/authentik-provision.py, update:
 "local-llm": {
    ...
    "redirect_uris": [
        "http://localhost:3000/api/auth/callback",  # dev
        "https://your-nextjs-app.com/api/auth/callback",  # prod
    ],
 }
 ```
 Then standard OIDC flow:
 1. Redirect to `https://authentik.riotpiao.com/application/o/authorize/?client_id=local-llm&redirect_uri=...&response_type=code&scope=openid email profile groups`
 2. User logs in via Authentik UI
 3. Callback receives `code`, exchange for tokens
 ## JWT Token Persistence
 ### Browser (localStorage)
 ```typescript
 const TOKEN_KEY = 'llm_auth_token';
 // Save
 localStorage.setItem(TOKEN_KEY, JSON.stringify({
  access_token: tokens.access_token,
  refresh_token: tokens.refresh_token,
  expires_at: Date.now() + tokens.expires_in * 1000,
 }));
 // Load
 const stored = JSON.parse(localStorage.getItem(TOKEN_KEY) || 'null');
 if (stored && stored.expires_at > Date.now()) {
  // Token valid
 }
 // Clear (logout)
 localStorage.removeItem(TOKEN_KEY);
 ```
 ### Server-side (HTTP-only cookies)
 ```typescript
 // app/api/auth/login/route.ts
 import { cookies } from 'next/headers';
 // After successful login
 cookies().set('llm_auth_token', JSON.stringify(tokens), {
  httpOnly: true,
  secure: process.env.NODE_ENV === 'production',
  sameSite: 'lax',
  maxAge: tokens.expires_in,
  path: '/',
 });
 // Read in middleware or API routes
 const tokenCookie = cookies().get('llm_auth_token');
 const tokens = JSON.parse(tokenCookie?.value || 'null');
 ```
 ## Token Refresh
 ```typescript
 async function refreshAccessToken(refresh_token: string) {
  const response = await fetch('https://authentik.riotpiao.com/application/o/token/', {
    method: 'POST',
    headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
    body: new URLSearchParams({
      grant_type: 'refresh_token',
      client_id: 'local-llm',
      client_secret: process.env.AUTHENTIK_CLIENT_SECRET,
      refresh_token,
    }),
  });
  return response.json();
 }
 ```
 ## Environment Variables
 ```bash
 # .env.local
 AUTHENTIK_URL=https://authentik.riotpiao.com
 AUTHENTIK_CLIENT_ID=local-llm
 AUTHENTIK_CLIENT_SECRET=<from-secret>
 # For client-side (public)
 NEXT_PUBLIC_AUTHENTIK_URL=https://authentik.riotpiao.com
 NEXT_PUBLIC_AUTHENTIK_CLIENT_ID=local-llm
 ```
 ## Using Token with LLM API
 ```typescript
 const token = await getValidToken(); // from localStorage or cookie
 const response = await fetch('https://api.riotpiao.com/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${token}`,  // JWT from Authentik
  },
  body: JSON.stringify({
    model: 'reasoning',
    messages: [{ role: 'user', content: 'Hello' }],
  }),
 });
 ```
 ## TODO
 ### Gateway-side (homelab-frontend)
 - [ ] Wire `internal/auth/jwt.go` into LLM proxy handler (`internal/proxy/proxy.go`)
 - [ ] Add `authRequired: true` to model config or create LLM-specific middleware
 - [ ] Example pattern from SQS (in `internal/serviceadapter/router.go`):
 ```go
 // In proxy.go ServeHTTP, before dispatching to LLM upstream:
 if strings.HasPrefix(r.URL.Path, "/v1/") {
    authHeader := r.Header.Get("Authorization")
    claims, err := llmJWTAuth.ValidateBearerToken(authHeader)
    if err != nil {
        // Return 401/403
    }
    if !llmJWTAuth.CheckPermissions(claims, "llm:inference", "*") {
        // Return 403 insufficient permissions
    }
 }
 ```
 ### Authentik-side
 - [ ] Enable ROPC grant in Authentik provider settings (if not already)
 - [ ] Add redirect URIs to `local-llm` app if browser OAuth flow needed:
 ```python
 # k8s/infra/iam/scripts/authentik-provision.py
 "local-llm": {
    ...
    "redirect_uris": [
        "http://localhost:3000/api/auth/callback",
        "https://your-app.com/api/auth/callback",
    ],
 }
 ```
 ### NextJS-side
 - [ ] Until gateway auth is wired, LLM API works without token
 - [ ] Once wired, add `Authorization: Bearer <token>` to all LLM requests
@@ -208,3 +208,95 @@ versions without warning in your own values file.
 Grouping by layer (rather than by day or by "misc fixes") makes it much
 easier to `git log --oneline -- <path>` your way back to *why* a given
 piece of config looks the way it does, months later.
 ## Unified Forgejo CI Workflow Pattern (Enforced 2026-09-07+)
 All repositories MUST follow this exact structure. No variations.
 ```yaml
 name: CI
 on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
 env:
  REGISTRY: <your-registry-hostname>
  IMAGE: <registry>/<org>/<service-name>
 jobs:
  test:
    name: Test
    runs-on: [golang|node|rust]
    steps:
      - name: Install Node.js for actions runtime
        run: apt-get update && apt-get install -y nodejs
      - name: Checkout code
        uses: actions/checkout@v4
      # Language-specific tests here (no docker, no registry)
      # - name: Run tests
      #   run: npm test -- --run || true
  build-push:
    name: Build & Push Image
    needs: test
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: [golang|node|rust]
    steps:
      - name: Install Node.js and Docker
        run: |
          apt-get update
          apt-get install -y nodejs docker.io
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Get short SHA
        id: sha
        run: |
          SHORT_SHA=$(git rev-parse --short HEAD)
          echo "short_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
      - name: Registry login
        run: |
          echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
            --username "${REGISTRY_USER}" --password-stdin
        env:
          REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
          REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
      - name: Build Docker image
        run: |
          docker build --no-cache \
            -t "${IMAGE}:${{ steps.sha.outputs.short_sha }}" \
            -t "${IMAGE}:latest" \
            .
      - name: Push Docker image
        run: |
          docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}"
          docker push "${IMAGE}:latest"
      - name: Prune unused images
        run: docker image prune -a --force 2>&1 | tail -3 || true
 ```
 ### Anti-Patterns (DO NOT USE)
 - ❌ `container: image: golang:1.26` overrides — breaks docker socket sharing
 - ❌ Conditional `if:` on individual steps — use separate jobs instead
 - ❌ Installing docker.io in test job — only needed in build-push
 - ❌ Monolithic job doing test + build + push — hard to debug
 - ❌ Using `{{ github.sha }}` for image tag — use short commit SHA for readability
 ### How It Works
 1. **PR to feature branch** → test job runs, build-push skipped, nothing pushed
 2. **Push to main** → test runs, build-push runs after test passes, image pushed
 3. Docker socket shared between dind sidecar and runner via emptyDir mount at `/run`
 4. `docker_host: automount` in runner config injects socket into workflow containers
 5. Secrets (FORGEJO_REGISTRY_USER, TOKEN) set in Forgejo repo settings, NOT in git
@@ -1,6 +1,6 @@
 # ── Node IPs (3-CP HA topology) ───────────────────────────────────────────────
 CP1_IP  := 192.168.1.166  # talos-cp-1
-CP2_IP  := 192.168.1.213  # talos-cp-2 (storage: 3 disks)
+CP2_IP  := 192.168.1.214  # talos-cp-2 (storage: 3 disks)
 CP3_IP  := 192.168.1.162  # talos-cp-3
 CP_VIP  := 192.168.1.166  # controlplane VIP (currently .166)
@@ -28,7 +28,7 @@ W_CONFIG = cluster-config/worker-$(N).yaml
 # ── Help ──────────────────────────────────────────────────────────────────────
 .PHONY: help
 help:
-	@echo "Homelab cluster (3-CP HA: .166/.213/.163) — available targets"
+	@echo "Homelab cluster (3-CP HA: .166/.214/.162) — available targets"
 	@echo ""
 	@echo "  Status & Services"
 	@echo "    nodes             kubectl get nodes"
@@ -42,6 +42,86 @@ All logs + metrics centralized in Grafana for debugging
 - **Secrets at rest** — Vault + encrypted etcd; credentials never in logs or ConfigMaps
 - **Infrastructure-as-code** — Every service deployed via Helmfile; one `helmfile apply` recovers from total failure
 ## ArgoCD — GitOps Deployment Flow
 **ArgoCD** pulls infrastructure changes from git and syncs the cluster automatically. 
 No manual `kubectl apply` — push to git, ArgoCD detects the change, and deploys within ~3 minutes.
 ```
 Developer pushes to git
       ↓
 ArgoCD detects change (every 3 min or webhook)
       ↓
 Syncs manifests to cluster
       ↓
 Workloads reconcile automatically
 ```
 Applications are deployed in waves (numbered 00, 10, 20, 30, ...) to respect dependencies — 
 storage deploys before databases, databases before applications.
 ### Tracked Git Repositories
 ArgoCD monitors these repos for changes:
 | Repository | Purpose |
 |------------|----------|
 | `https://github.com/Riotpiaole/riotpiao.homelab.com` | Main infrastructure repo (all manifests in `k8s/argocd/apps/`) |
 | `https://forgejo.riotpiao.com/rock/*` | Any `rock/*` repo in in-cluster Forgejo (apps + configs) |
 | `https://github.com/Riotpiaole/Poimen-*` | External Poimen services (memory, workflows) |
 To deploy a new application: create a git repo, add an Application manifest to the homelab repo's 
 `k8s/argocd/apps/`, commit + push, and ArgoCD syncs within 3 minutes.
 ## Management Planes — Talos vs Kubernetes
 This cluster has **two separate management planes**, each with different workflows:
 | Plane | What it manages | Workflow | Tool |
 |-------|-----------------|----------|------|
 | **Talos (OS)** | Node configuration, kernel params, networking, CoreDNS, machine state | Edit `terraform/` → `terraform apply` → `make apply-cp` | `terraform` + `talosctl` |
 | **Kubernetes (workloads)** | All pods, services, deployments, ingresses, databases | Edit `k8s/argocd/apps/` → `git push` → ArgoCD syncs | `git` + ArgoCD |
 **Critical distinction:**
 - **Kubernetes resources** (`k8s/**`) flow through **git → ArgoCD** — never use `kubectl apply`
 - **Talos machine config** (`terraform/**`) uses **local `terraform apply`** (sanctioned exception — CI can't hold node credentials)
 Example: To add a CoreDNS hostname rewrite, you edit `terraform/files/coredns/Corefile`, then:
 ```bash
 cd terraform && terraform apply -var-file=terraform.tfvars.local
 cd .. && make apply-cp     # talosctl apply-config to all 3 control planes
 ```
 But to add a new Kubernetes Deployment or update an Ingress, you only `git push` — **never `kubectl apply`**.
 ### CoreDNS ConfigMap Ownership — Critical
 ⚠️ **Warning:** The `coredns` ConfigMap in `kube-system` namespace is **owned by Talos**, not ArgoCD or kubectl. 
 It is rendered from `terraform/files/coredns/Corefile` into Talos's machine config at bootstrap time.
 **Do not `kubectl apply` or `kubectl edit` this ConfigMap directly.** Doing so transfers field ownership to kubectl's 
 client-side-apply mechanism, and Talos's inline-manifest controller will silently no-op on every future reconcile 
 (server-side-apply conflict, no error surfaced).
 **To update CoreDNS (e.g., add a hostname rewrite):**
 1. Edit `terraform/files/coredns/Corefile`
 2. Commit + push
 3. Run `cd terraform && terraform apply -var-file=terraform.tfvars.local`
 4. Run `make apply-cp` to push config to all control planes
 5. CoreDNS picks up changes via its `reload` plugin — no pod restart needed
 **If you accidentally edited the ConfigMap directly and broke Talos's ownership:**
 ```bash
 kubectl delete configmap coredns -n kube-system
 # Wait ~30s for Talos's k8s.ManifestApplyController to recreate it
 kubectl get configmap coredns -n kube-system -w
 ```
 Or as a stopgap, apply the correct content yourself:
 ```bash
 kubectl apply --server-side -f <(terraform output coredns_config)
 ```
 ## Quick Start — Deploying the Cluster
 ### 1. Bootstrap Talos Nodes
@@ -12,16 +12,15 @@
 #   - Talos cluster up; kubectl context points at it
 #   - helm 3, kubectl
 #   - SOPS age key at $SOPS_KEY (for the ArgoCD SOPS CMP plugin)
-#   - GitHub read-only deploy key private half at $DEPLOY_KEY (public half added
+#
-#     to the GitHub repo's Deploy keys)
+# The GitHub seed repo is public, so it is cloned anonymously over HTTPS — no
 # deploy key, no repository Secret, one less thing to bootstrap before ArgoCD.
 #
 set -euo pipefail
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 BOOT="$SCRIPT_DIR/k8s/bootstrap"
 SOPS_KEY="${SOPS_KEY:-$HOME/.sops/key.txt}"
 DEPLOY_KEY="${DEPLOY_KEY:-$HOME/.ssh/argocd_seed}"
 GITHUB_SSH="[email protected]:Riotpiaole/riotpiao.homelab.com.git"
 log()  { echo "[$(date +%H:%M:%S)] $*"; }
 die()  { echo "ERROR: $*" >&2; exit 1; }
@@ -39,7 +38,6 @@ preflight() {
  kubectl cluster-info >/dev/null || die "kubectl not configured / cluster unreachable"
  command -v helm >/dev/null || die "helm 3 not found"
  [[ -f "$SOPS_KEY" ]]   || die "SOPS age key missing at $SOPS_KEY"
  [[ -f "$DEPLOY_KEY" ]] || die "GitHub deploy key missing at $DEPLOY_KEY (see phase4-argocd/seed-repo-secret.example.yaml)"
  log "✅ preflight ok"
 }
@@ -194,13 +192,10 @@ p3_forgejo() {
 p4_argocd() {
  phase "PHASE 4: ArgoCD (seeded from GitHub)"
-  # Always ensure namespace + repository secret (idempotent)
+  # Always ensure namespace (idempotent). The seed repo is public — ArgoCD clones
  # it anonymously over HTTPS, so there is no repository Secret to create.
  kubectl create ns argocd --dry-run=client -o yaml | kubectl apply -f -
-  kubectl -n argocd create secret generic seed-github-repo \
+
    --from-literal=type=git --from-literal=url="$GITHUB_SSH" \
    --from-file=sshPrivateKey="$DEPLOY_KEY" --dry-run=client -o yaml | kubectl apply -f -
  kubectl -n argocd label secret seed-github-repo argocd.argoproj.io/secret-type=repository --overwrite 2>/dev/null || true
  # Decrypt and apply any encrypted secrets from bootstrap dir (local SOPS)
  if command -v sops &> /dev/null; then
    export SOPS_AGE_KEY_FILE="$SOPS_KEY"
@@ -0,0 +1,18 @@
 apiVersion: v1
 data:
  settings.json: |
    {
      "defaultProvider": "homelab-ornith",
      "defaultModel": "ornith:35b",
      "defaultThinkingLevel": "medium",
      "theme": "light",
      "compaction": {
        "enabled": true,
        "reserveTokens": 16000,
        "keepRecentTokens": 6000
      }
    }
 kind: ConfigMap
 metadata:
  name: pi-config
  namespace: agent-pod
@@ -0,0 +1,49 @@
 # Exposes agent-hub at api.riotpiao.com/console (WebSocket) and /run
 # (trigger a new session) -- both are routes on the same hub.js service.
 #
 # Was ingressClassName: kong until Kong was retired on 2026-08-19. Pointed
 # straight at nginx rather than through the replacement Go gateway because that
 # gateway has no WebSocket upgrade support yet -- routing /console through it
 # would break the console outright. nginx handles the upgrade natively.
 #
 # Path precedence: the nginx Ingress api/api catch-alls `/` on this same host
 # to the gateway. nginx matches longest prefix first, so these three paths win
 # over `/` and the rest of the host still reaches the gateway.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: console
  namespace: agent-pod
  annotations:
    # A console WebSocket stays open across a whole agent session; nginx's 60s
    # default read timeout would drop it mid-run.
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
 spec:
  ingressClassName: nginx
  rules:
    - host: api.riotpiao.com
      http:
        paths:
          - path: /console
            pathType: Prefix
            backend:
              service:
                name: agent-hub
                port:
                  number: 9090
          - path: /run
            pathType: Prefix
            backend:
              service:
                name: agent-hub
                port:
                  number: 9090
          - path: /sessions
            pathType: Prefix
            backend:
              service:
                name: agent-hub
                port:
                  number: 9090
@@ -0,0 +1,982 @@
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: coordinator-src
  namespace: agent-pod
 data:
  coordinator.js: |
    #!/usr/bin/env node
    // coordinator: local CLI that drives a multi-phase, multi-task pipeline of
    // planner/investigator/implementer/judge stages, same state machine as
    // hub.js's old /pipeline handler. Every interactive stage runs through the
    // patched agent-manager fork's `spawn` subcommand, so it is a real tracked
    // session (tmux pane on agent-manager's private socket + a state.db row)
    // from the moment it exists -- attachable and visible in agent-manager's
    // own TUI the whole time it runs.
    //
    // Per repo, each role (planner/investigator/implementer/judge) is ONE
    // persistent agent-manager session, not a fresh spawn per task: the first
    // task to need a role spawns it, every later task for that role reuses the
    // same tmux pane via `tmux send-keys` (see runOnPool) -- the same nudge
    // mechanism that used to only fire on a stall now doubles as "give this
    // agent its next task." Every role reads everything it needs fresh off disk
    // each call, so every pane gets a `/new` before every reuse instead of
    // accumulating history that degrades and eventually errors out task after
    // task -- same pane, same agent-manager session, zero memory of the last
    // task it handled.
    // Because only one implementer/judge/etc. exists per repo, tasks within a
    // phase run strictly sequentially against the pool -- no per-task worktree,
    // no per-task branch, no merge-back step; every task commits straight onto
    // the phase branch in the repo's one shared clone.
    //
    // The unit of concurrency is now the REPO, not the task: runCoordinator
    // takes a list of repos and runs up to REPO_CONCURRENCY of them at once,
    // each with its own clone (under WORK_DIR/<repoId>) and its own 4-agent
    // pool. The coordinator never kills a role's session; it rests at an idle
    // prompt between tasks, and agent-manager's session list becomes the audit
    // trail of everything every repo's pipeline ran. Completion is signaled by
    // sentinel files under the repo's clone (unchanged convention), waited on
    // with fs.watch instead of polling.
    const fs = require("node:fs");
    const path = require("node:path");
    const { spawn } = require("node:child_process");
    const WORK_DIR = process.env.HUB_WORK_DIR || path.join(require("node:os").tmpdir(), "agent-harness-work");
    // Never rely on a bare `pi`/`agent-manager` on $PATH -- see PI_BIN's own
    // comment below; the same collision risk applies to any CLI name. Always
    // invoke explicit pinned paths.
    const PI_BIN =
      process.env.PI_BIN ||
      path.join(__dirname, "..", ".pi-cli", "node_modules", "@earendil-works", "pi-coding-agent", "dist", "cli.js");
    const AGENT_MANAGER_BIN = process.env.AGENT_MANAGER_BIN || path.join(__dirname, "..", ".bin", "agent-manager-fork");
    // agent-manager's own session-state DB -- used to detect a session that has
    // actually died (process crashed/exited, status flips to "errored"/"dead")
    // instead of one that's merely slow. Read-only introspection plus the one
    // UPDATE in killDeadSession below, same class of operation as the tmux
    // nudges already done directly against agent-manager's internals.
    const AGENT_MANAGER_DB =
      process.env.AGENT_MANAGER_DB || path.join(require("node:os").homedir(), ".config", "agent-manager", "state.db");
    // Empty means "let pi fall back to ~/.pi/agent/settings.json's default"
    // (currently anthropic/claude-sonnet-4-5, real paid usage). Set both to
    // route every stage -- headless (spawnPi) and interactive (runOnPool) --
    // at the homelab model instead, e.g. AGENT_PROVIDER=homelab-ornith
    // AGENT_MODEL=ornith:35b.
    const AGENT_PROVIDER = process.env.AGENT_PROVIDER || "";
    const AGENT_MODEL = process.env.AGENT_MODEL || "";
    // judge can run a different model than the rest of the chain, e.g.
    // homelab-reasoning instead of homelab-ornith now that verifier/PRM is
    // retired. Falls back to AGENT_PROVIDER/AGENT_MODEL when unset, so a run
    // that doesn't care keeps one uniform model everywhere.
    const JUDGE_PROVIDER = process.env.JUDGE_PROVIDER || AGENT_PROVIDER;
    const JUDGE_MODEL = process.env.JUDGE_MODEL || AGENT_MODEL;
    function providerModelFor(role) {
      return role === "judge" ? { provider: JUDGE_PROVIDER, model: JUDGE_MODEL } : { provider: AGENT_PROVIDER, model: AGENT_MODEL };
    }
    // agent-manager's private tmux server and session-naming scheme
    // (internal/tmux/tmux.go: defaultSocket = "agentmgr", sessionName(id) =
    // "am_"+id) -- stable, documented internals of the fork, used here only
    // for read-only introspection (pane capture) and role nudges, exactly the
    // class of operation hub.js already ran directly against its own sessions
    // rather than asking a model to do it.
    const AM_SOCKET = "agentmgr";
    function amSessionName(id) {
      return `am_${id}`;
    }
    function runAmTmux(args) {
      return runCmd("tmux", ["-L", AM_SOCKET, ...args]);
    }
    const ROLE_SKILLS = new Set(["planner", "investigator", "info-collector", "implementer", "judge", "resolver"]);
    // Every role reads everything it needs fresh off disk each call -- PLAN.md,
    // the task spec, judge's verdict file, `git diff` against baseBranch --
    // nothing depends on remembering earlier tasks. Left to accumulate, a
    // pooled session's conversation grows without bound across every task in a
    // repo and both correctness and reliability degrade hard once it does
    // (observed: a planner session at ~1.5M cumulative tokens started erroring
    // out every call, an investigator session that far gone started narrating a
    // different codebase entirely). So every role gets reset to a clean
    // conversation before every reuse instead of just being nudged with the
    // next prompt -- same pane, same agent-manager session (still
    // visible/attachable), zero history carried between tasks.
    function sleep(ms) {
      return new Promise((resolve) => setTimeout(resolve, ms));
    }
    const HARD_RULES =
      "Read and follow ~/.pi/agent/skills/karpathy-guidelines/SKILL.md and " +
      "~/.pi/agent/skills/caveman/SKILL.md as hard rules for this entire task, before anything else. ";
    // judge (routed to homelab-reasoning) has been observed narrating an
    // entire review in prose -- "I should run git diff, then check X..." --
    // and then writing a verdict based on that narration without ever calling
    // a real tool. Live example: a phase-judge call produced a page of
    // "I would check..." reasoning, declared VERDICT: PASS, and showed the
    // touch command as a fenced code block IN ITS OWN TEXT rather than
    // executing it. Coordinator just timed out waiting on a sentinel that was
    // never going to appear, since nothing was ever actually run. Spelled out
    // explicitly since "use the judge skill" alone apparently isn't enough to
    // rule this out.
    const REQUIRE_REAL_TOOL_CALLS =
      "Do not narrate what you would check -- actually run the commands via a real tool call and read their real " +
      "output before writing anything. A verdict based on describing checks instead of executing them is invalid. " +
      "Writing the verdict file and touching the sentinel are themselves tool calls you must execute, not text to " +
      "display in your response. ";
    function parseVerdictLine(text, label) {
      if (!text) return null;
      const re = new RegExp(`${label}:\\s*(\\w+)`, "i");
      const m = text.match(re);
      return m ? m[1].toUpperCase() : null;
    }
    function runCmd(bin, args, cwd) {
      return new Promise((resolve) => {
        const child = spawn(bin, args, { cwd, stdio: ["ignore", "pipe", "pipe"] });
        let out = "";
        child.stdout.on("data", (c) => (out += c));
        child.stderr.on("data", (c) => (out += c));
        child.on("close", (code) => resolve({ code, out: out.trim() }));
      });
    }
    function runGit(cwd, args) {
      return runCmd("git", args, cwd);
    }
    // A stage saying "commit" in its prompt is a request, not a guarantee -- seen
    // in practice: a stage writes a real file and simply never runs `git add`/
    // `git commit`, leaving it untracked and invisible to every later `git diff`.
    // Sweep and commit anything left dirty after every stage, deterministically.
    async function commitPending(cwd, message) {
      await runGit(cwd, ["add", "-A"]);
      const status = await runGit(cwd, ["status", "--porcelain"]);
      if (!status.out) return { committed: false };
      const commit = await runGit(cwd, ["commit", "-m", message]);
      return { committed: commit.code === 0, error: commit.code !== 0 ? commit.out : undefined };
    }
    // Headless one-shot pi call (`pi -p --mode json <prompt>`), used only for
    // quick diagnostic/mechanical calls that don't need to be a watchable
    // session: resolver's crash/stall diagnosis, and the initial clone. Kept
    // exactly as before -- only the interactive per-role stages (runOnPool,
    // below) go through agent-manager.
    //
    // Bounded by SPAWN_PI_TIMEOUT_MS -- unlike runOnPool's pooled sessions
    // (which now have status polling to catch a dead session fast, see
    // waitForSentinel/killDeadSession), this is a raw child_process with no
    // equivalent escape hatch. Observed live: a resolver call shared the
    // default backend with a concurrently-busy repo's implementer and sat for
    // 6+ minutes producing nothing -- with no timeout here, that blocks the
    // entire calling repo's pipeline forever, since askResolver is always
    // awaited before the next stage can run.
    const SPAWN_PI_TIMEOUT_MS = 5 * 60 * 1000;
    function spawnPi({ agent, prompt, cwd }) {
      const finalPrompt = ROLE_SKILLS.has(agent) ? `/skill:${agent} ${HARD_RULES}${prompt}` : prompt;
      const args = ["-p", "--mode", "json"];
      if (AGENT_PROVIDER) args.push("--provider", AGENT_PROVIDER);
      if (AGENT_MODEL) args.push("--model", AGENT_MODEL);
      args.push(finalPrompt);
      const child = spawn(PI_BIN, args, { stdio: ["ignore", "pipe", "pipe"], cwd });
      let lastText = "";
      let stderrTail = "";
      let buf = "";
      child.stdout.on("data", (chunk) => {
        buf += chunk;
        let idx;
        while ((idx = buf.indexOf("\n")) !== -1) {
          const line = buf.slice(0, idx);
          buf = buf.slice(idx + 1);
          if (!line.trim()) continue;
          try {
            const event = JSON.parse(line);
            if (event.type === "message_end" && event.message && Array.isArray(event.message.content)) {
              const text = event.message.content
                .filter((c) => c.type === "text")
                .map((c) => c.text)
                .join("\n");
              if (text) lastText = text;
            }
          } catch {
            // non-JSON stdout noise, ignore
          }
        }
      });
      child.stderr.on("data", (chunk) => {
        process.stderr.write(chunk);
        stderrTail = (stderrTail + chunk.toString()).slice(-4000);
      });
      return new Promise((resolve) => {
        let settled = false;
        const timer = setTimeout(() => {
          if (settled) return;
          settled = true;
          child.kill("SIGKILL");
          resolve({ code: null, lastText, stderrTail, timedOut: true });
        }, SPAWN_PI_TIMEOUT_MS);
        child.on("close", (code) => {
          if (settled) return;
          settled = true;
          clearTimeout(timer);
          resolve({ code, lastText, stderrTail });
        });
      });
    }
    async function askResolver(cwd, repoId, diagnosticPrompt) {
      const result = await spawnPi({ agent: "resolver", prompt: diagnosticPrompt, cwd });
      return parseVerdictLine(result.lastText, "RESOLUTION");
    }
    const STAGE_TIMEOUT_MS = 10 * 60 * 1000;
    const NUDGE_TIMEOUT_MS = 5 * 60 * 1000;
    // Resolves as soon as filePath appears (fs.watch on its directory, same as
    // before), as soon as target's agent-manager status flips to "errored" or
    // "dead" (polled -- state.db has no watch mechanism), or after limitMs with
    // neither. A session that has actually crashed will never touch the
    // sentinel, so without the status poll this just burns the full STAGE_
    // TIMEOUT_MS waiting on a file that was never coming, same as a genuine
    // stall -- polling status catches that in ~pollMs instead.
    function waitForSentinel(filePath, target, limitMs, pollMs = 5000) {
      return new Promise((resolve) => {
        if (fs.existsSync(filePath)) return resolve({ ok: true });
        const dir = path.dirname(filePath);
        const id = target.replace(/^am_/, "");
        let settled = false;
        let watcher;
        let poller;
        let timer;
        const finish = (result) => {
          if (settled) return;
          settled = true;
          clearTimeout(timer);
          clearInterval(poller);
          if (watcher) {
            try {
              watcher.close();
            } catch {
              // already closed
            }
          }
          resolve(result);
        };
        try {
          watcher = fs.watch(dir, () => {
            if (fs.existsSync(filePath)) finish({ ok: true });
          });
        } catch {
          // dir missing at watch time is a real bug elsewhere (cwd should
          // already exist); surface it as a timeout rather than hang forever.
          return finish({ timedOut: true });
        }
        // Closes the race between the existsSync check above and the watcher
        // actually being attached.
        if (fs.existsSync(filePath)) return finish({ ok: true });
        poller = setInterval(async () => {
          const { out } = await runCmd("sqlite3", [AGENT_MANAGER_DB, `SELECT status FROM sessions WHERE id='${id}'`]);
          const status = out.trim();
          if (status === "errored" || status === "dead") finish({ dead: true, status });
        }, pollMs);
        timer = setTimeout(() => finish({ timedOut: true }), limitMs);
      });
    }
    // Kills a session that's actually crashed (not just slow) and archives it
    // in agent-manager's own DB so it stops showing up as a live, unattended
    // pane -- otherwise every crash leaves an orphaned tmux session + state.db
    // row behind permanently, identical to the manually-cleaned-up poiman-
    // planner ghost session found earlier this same run.
    async function killDeadSession(target) {
      await runAmTmux(["kill-session", "-t", target]);
      const id = target.replace(/^am_/, "");
      await runCmd("sqlite3", [AGENT_MANAGER_DB, `UPDATE sessions SET archived=1 WHERE id='${id}'`]);
    }
    // Runs one task's worth of work on a persistent per-role agent: spawns the
    // role's session the first time it's ever needed for this repo, sends every
    // later prompt into that same tmux pane via send-keys -- prefixed with a
    // `/new` first, so the pane and agent-manager session stay the same but the
    // model starts that prompt with a clean conversation, no history carried
    // over from whatever task this role last handled. pool is a plain object
    // keyed by role name ("planner"/"investigator"/"implementer"/"judge"),
    // shared across every task in a repo's pipeline (see runRepoPipeline) -- it
    // IS the 4-agent pool, one entry per role, filled in lazily as each role
    // gets its first task.
    // A dead/errored session gets one respawn-and-retry (same prompt, fresh
    // session) before this stage is abandoned -- matches resolver-SKILL.md's
    // own documented contract of retrying a failed stage at most once.
    const DEAD_SESSION_RETRIES = 1;
    async function runOnPool(pool, cwd, repoId, role, prompt, sentinelFile) {
      fs.rmSync(sentinelFile, { force: true });
      const label = `${repoId}-${role}`;
      // A pooled session's shell cwd drifts as it explores the repo (e.g. cd
      // into a Rust workspace subdirectory to read source) and nothing resets
      // it back between turns. Seen in practice: a repo whose own internal
      // workspace folder is one letter off from the repo's own directory name
      // ("poiman" the repo vs. "poimen" the crate workspace inside it) was
      // enough for the agent to touch its sentinel one level off from where
      // this function is watching for it -- coordinator waits out the full
      // STAGE_TIMEOUT_MS for a file that already exists, just in the wrong
      // place. State the absolute target directory and use absolute paths for
      // every filesystem instruction, so there's nothing for the agent to get
      // wrong by reasoning about a relative "current directory."
      const cwdReminder = `Your working directory for this task is ${cwd} -- if your shell isn't already there, run: cd ${cwd}\n\n`;
      const spawnFresh = async () => {
        const spawnArgs = ["spawn", "--tool", "pi", "--cwd", cwd, "--name", label, "--group", repoId, "--prompt", cwdReminder + HARD_RULES + prompt];
        const { provider, model } = providerModelFor(role);
        if (provider) spawnArgs.push("--provider", provider);
        if (model) spawnArgs.push("--model", model);
        const spawned = await runCmd(AGENT_MANAGER_BIN, spawnArgs);
        return spawned.code === 0 ? amSessionName(spawned.out) : null;
      };
      let target = pool[role];
      if (!target) {
        target = await spawnFresh();
        if (!target) return { ok: false, crashed: true, error: "spawn failed", sessionName: label };
        pool[role] = target;
      } else {
        await runAmTmux(["send-keys", "-t", target, "/new", "Enter"]);
        await sleep(1000);
        await runAmTmux(["send-keys", "-t", target, cwdReminder + HARD_RULES + prompt, "Enter"]);
      }
      for (let deadRetries = 0; ; deadRetries++) {
        const outcome = await waitForSentinel(sentinelFile, target, STAGE_TIMEOUT_MS);
        if (outcome.ok) return { ok: true, sessionName: label };
        if (outcome.dead) {
          await killDeadSession(target);
          if (pool[role] === target) delete pool[role];
          if (deadRetries >= DEAD_SESSION_RETRIES) {
            return { ok: false, crashed: true, error: `session died (status: ${outcome.status})`, sessionName: label };
          }
          target = await spawnFresh();
          if (!target) return { ok: false, crashed: true, error: "respawn after death failed", sessionName: label };
          pool[role] = target;
          continue;
        }
        // Plain stall -- session still alive, just slow. Ask resolver once,
        // nudge if it says worth it, and stop here either way (this is not
        // the death path, so no respawn/retry loop).
        const pane = await runAmTmux(["capture-pane", "-t", target, "-p", "-S", "-200"]);
        const resolution = await askResolver(
          cwd,
          repoId,
          `Repo ${repoId}'s "${role}" agent hasn't finished its current task after 10 minutes. Its pane tail:\n${pane.out.slice(-3000)}\n\n` +
            `Decide: is it still making real progress and worth nudging to wrap up, or stuck and worth abandoning?`
        );
        let ok = false;
        if (resolution === "RETRY") {
          await runAmTmux(["send-keys", "-t", target, `Please wrap up now and run: touch ${sentinelFile}`, "Enter"]);
          const nudged = await waitForSentinel(sentinelFile, target, NUDGE_TIMEOUT_MS);
          ok = nudged.ok === true;
          if (nudged.dead) {
            await killDeadSession(target);
            if (pool[role] === target) delete pool[role];
          }
        }
        return { ok, sessionName: label };
      }
    }
    function plannerPrompt(task, specHint, judgeOnly, cwd) {
      // judgeOnly (auto-discovered tasks only, see parseTaskBoard): planner
      // itself decides whether the task is already done before planning it,
      // reading tasks/INDEX.md's own status notes plus git log/current code --
      // replaces what used to be a separate judge pre-check call. One LLM round
      // trip instead of two, and the same agent that's about to plan the task
      // is the one deciding whether planning it is even necessary.
      const resultFile = path.join(cwd, `.task-result-${task}`);
      const decideStep = judgeOnly
        ? `First, decide whether task ${task} is already fully implemented on this branch: check ` +
          `\`git log --oneline --grep '${task}'\`, tasks/INDEX.md's own status notes for this task, and the current ` +
          `code directly against its spec (${specHint})'s acceptance criteria. Write your decision to ` +
          `${resultFile} as a single "VERDICT: PASS" (already done, no further work needed) or ` +
          `"VERDICT: FAIL" (needs work) line plus one line of rationale. If VERDICT is FAIL, continue below and ` +
          `draft the plan in this same turn; if VERDICT is PASS, skip the rest and go straight to the touch step.\n\n`
        : "";
      return (
        `${decideStep}Use the planner skill to draft PLAN.md for task ${task}, reading its spec (${specHint}). ` +
        `PLAN.md is scratch state for this harness, not a deliverable -- do NOT commit it or add it to git. ` +
        `Then run: touch ${path.join(cwd, `.stage-done-${task}-planner`)}`
      );
    }
    function investigatorPrompt(task, cwd) {
      return (
        `Use the investigator skill to confirm PLAN.md against real sources for task ${task}, append findings. ` +
        `PLAN.md is scratch state for this harness, not a deliverable -- do NOT commit it or add it to git. ` +
        `Then run: touch ${path.join(cwd, `.stage-done-${task}-investigator`)}`
      );
    }
    function implementerPrompt(task, attempt, feedbackHint, cwd) {
      return (
        `Use the implementer skill to implement what the current PLAN.md specifies for task ${task} (commit as you go). ` +
        `${feedbackHint} Then run: touch ${path.join(cwd, `.stage-done-${task}-implementer-${attempt}`)}`
      );
    }
    function judgePrompt(task, baseBranch, attempt, cwd) {
      return (
        `${REQUIRE_REAL_TOOL_CALLS}Use the judge skill to review the diff against ${baseBranch}...HEAD for task ${task}. ` +
        `Write your verdict to ${path.join(cwd, `.task-result-${task}`)} as a single "VERDICT: PASS" or "VERDICT: FAIL" ` +
        `line plus one line of rationale, then run: touch ${path.join(cwd, `.stage-done-${task}-judge-${attempt}`)}`
      );
    }
    const MAX_IMPLEMENT_ATTEMPTS = 5;
    const MAX_PLAN_REVISIONS = 3;
    // Runs one task against the repo's shared role pool: planner drafts
    // PLAN.md (for auto-discovered tasks, first deciding off tasks/INDEX.md and
    // the repo's own state whether the task is already done -- see
    // plannerPrompt's judgeOnly branch; judge never does this pre-check),
    // investigator confirms it, then implementer and judge go back and
    // forth -- judge's FAIL rationale lands in .task-result-<task>, which the
    // next implementer attempt is told to read and address. After
    // MAX_IMPLEMENT_ATTEMPTS straight fails, the planner role is asked to judge
    // whether the plan itself is wrong -- fresh conversation, same as any other
    // planner call, reading PLAN.md/the judge feedback/the
    // diff off disk rather than remembering having drafted the original plan.
    // If it decides the approach is wrong it revises PLAN.md and the implementer
    // gets a fresh attempt budget.
    // MAX_PLAN_REVISIONS caps this from looping forever on a task that's
    // genuinely stuck. All work happens directly in cwd (the repo's one shared
    // clone, currently checked out to the phase branch) -- no worktree, since
    // only one implementer/judge exist per repo and tasks run strictly one at a
    // time (see runPhase).
    async function runTaskOnPool(cwd, baseBranch, task, pool, repoId, pipelineSession, judgeOnly) {
      const resultFile = path.join(cwd, `.task-result-${task}`);
      fs.rmSync(resultFile, { force: true });
      const specHint = `the file under tasks/ starting with "${task}-"`;
      const stage = async (role, prompt, sentinel, displayLabel) => {
        const label = displayLabel || role;
        pipelineSession.activeTasks[task] = { stage: label, startedAt: new Date().toISOString() };
        logProgress(pipelineSession);
        const result = await runOnPool(pool, cwd, repoId, role, prompt, sentinel);
        await commitPending(cwd, `task: ${task} (${label})`);
        return result;
      };
      const abandon = (stageLabel, result, attempt) => {
        delete pipelineSession.activeTasks[task];
        logProgress(pipelineSession);
        return {
          task,
          status: result.crashed ? "spawn-crashed" : "timed-out",
          error: result.error,
          stoppedAt: stageLabel,
          ...(attempt !== undefined ? { attempt } : {}),
        };
      };
      // PLAN.md is scratch state for this one task, not a deliverable (see
      // plannerPrompt/investigatorPrompt -- it's gitignored too, as a backstop
      // in case an agent commits it anyway). Discard it once the task is done,
      // whatever the outcome, so it never bleeds into the next task's planner
      // call or sits around as stale harness clutter in the shared clone.
      try {
        let result = await stage("planner", plannerPrompt(task, specHint, judgeOnly, cwd), path.join(cwd, `.stage-done-${task}-planner`));
        if (!result.ok) return abandon("planner", result);
        if (judgeOnly) {
          const quickText = fs.existsSync(resultFile) ? fs.readFileSync(resultFile, "utf8") : "";
          if (parseVerdictLine(quickText, "VERDICT") === "PASS") {
            delete pipelineSession.activeTasks[task];
            logProgress(pipelineSession);
            return { task, status: "done", judgeRationale: quickText, judgeOnlyPass: true };
          }
        }
        result = await stage("investigator", investigatorPrompt(task, cwd), path.join(cwd, `.stage-done-${task}-investigator`));
        if (!result.ok) return abandon("investigator", result);
        let planRevisions = 0;
        let implementAttempt = 0;
        let verdict = null;
        let resultText = "";
        let justRevisedPlan = false;
        while (true) {
          implementAttempt++;
          const feedbackHint = fs.existsSync(resultFile)
            ? justRevisedPlan
              ? `${resultFile} holds the judge's feedback against the OLD plan, which prompted a plan revision -- ` +
                `PLAN.md has since changed. Read the current PLAN.md as the source of truth, not the old feedback verbatim.`
              : `A previous judge review exists at ${resultFile} -- read it and address every issue it raises.`
            : "";
          justRevisedPlan = false;
          result = await stage(
            "implementer",
            implementerPrompt(task, implementAttempt, feedbackHint, cwd),
            path.join(cwd, `.stage-done-${task}-implementer-${implementAttempt}`)
          );
          if (!result.ok) return abandon("implementer", result, implementAttempt);
          result = await stage("judge", judgePrompt(task, baseBranch, implementAttempt, cwd), path.join(cwd, `.stage-done-${task}-judge-${implementAttempt}`));
          if (!result.ok) return abandon("judge", result, implementAttempt);
          resultText = fs.existsSync(resultFile) ? fs.readFileSync(resultFile, "utf8") : "";
          verdict = parseVerdictLine(resultText, "VERDICT");
          if (verdict === "PASS") break;
          if (implementAttempt >= MAX_IMPLEMENT_ATTEMPTS) {
            if (planRevisions >= MAX_PLAN_REVISIONS) break;
            planRevisions++;
            result = await stage(
              "planner",
              `Implementer failed judge review ${MAX_IMPLEMENT_ATTEMPTS} times in a row for task ${task}. Read PLAN.md, ` +
                `the judge's feedback in ${resultFile}, and the current diff against ${baseBranch}...HEAD. Decide ` +
                `whether the plan's approach itself is wrong, not just the implementation -- if so, revise PLAN.md. If ` +
                `you change the approach, also use the investigator skill to confirm the new approach against real ` +
                `sources. If the plan is sound, note why in PLAN.md and leave it as-is. PLAN.md is scratch state for ` +
                `this harness, not a deliverable -- do NOT commit it or add it to git. Then run: ` +
                `touch ${path.join(cwd, `.stage-done-${task}-planner-revise-${planRevisions}`)}`,
              path.join(cwd, `.stage-done-${task}-planner-revise-${planRevisions}`),
              "planner-revise"
            );
            if (!result.ok) return abandon("planner-revise", result, planRevisions);
            implementAttempt = 0;
            justRevisedPlan = true;
          }
        }
        delete pipelineSession.activeTasks[task];
        logProgress(pipelineSession);
        if (verdict !== "PASS" && planRevisions >= MAX_PLAN_REVISIONS) {
          return { task, status: "unresolved", judgeRationale: resultText, implementAttempts: implementAttempt, planRevisions };
        }
        return { task, status: verdict === "PASS" ? "done" : "done-with-concerns", judgeRationale: resultText };
      } finally {
        fs.rmSync(path.join(cwd, "PLAN.md"), { force: true });
      }
    }
    // Committed (never gitignored) so it survives a resumed phase branch --
    // one task id per line, appended as each task resolves. This is what lets
    // a resumed run skip straight past already-resolved tasks instead of
    // re-running planner's judgeOnly decision on every one of them again:
    // resuming the git branch alone only recovers the CODE, not "which tasks
    // are already settled," and re-deciding that from scratch for every task
    // burns a full LLM call per already-done task before ever reaching the
    // first one that actually needs work.
    function progressLedgerPath(cwd) {
      return path.join(cwd, ".agent-progress");
    }
    function readCompletedTasks(cwd) {
      const file = progressLedgerPath(cwd);
      if (!fs.existsSync(file)) return new Set();
      return new Set(
        fs
          .readFileSync(file, "utf8")
          .split("\n")
          .map((line) => line.trim())
          .filter(Boolean)
      );
    }
    async function recordTaskComplete(cwd, task) {
      fs.appendFileSync(progressLedgerPath(cwd), `${task}\n`);
      await runGit(cwd, ["add", path.basename(progressLedgerPath(cwd))]);
      await runGit(cwd, ["commit", "-m", `chore: mark ${task} complete in progress ledger`]);
    }
    // Runs every task in a phase (no declared dependency between them) strictly
    // one at a time against the repo's shared role pool -- only one implementer/
    // judge/etc. exists per repo, so there is no per-task concurrency to have
    // here anymore (see REPO_CONCURRENCY below for where concurrency now
    // lives). No worktrees: every task commits directly onto phaseBranch in the
    // one shared cwd. baseBranch here is the TRUE base (e.g. "main") -- judge
    // reviews `git diff baseBranch...HEAD`, not phaseBranch...HEAD, which would
    // always be empty since HEAD *is* phaseBranch while it's checked out.
    //
    // Pushes phaseBranch after every task, not just once at full-phase-end: the
    // pod is ephemeral and every restart re-clones baseBranch fresh (see
    // runRepoPipeline) -- without this, a redeploy mid-phase silently discards
    // every task committed so far, and the next run re-decides "is this done?"
    // from a clone that never saw any of that work.
    async function runPhase(cwd, baseBranch, phaseBranch, phaseTasks, pool, repoId, pipelineSession) {
      const entries = phaseTasks.map((t) => (typeof t === "string" ? { id: t, judgeOnly: false } : t));
      const completed = readCompletedTasks(cwd);
      for (const entry of entries) {
        if (completed.has(entry.id)) {
          const result = { task: entry.id, status: "done", resumed: true };
          pipelineSession.taskResults.push(result);
          logProgress(pipelineSession);
          continue;
        }
        const result = await runTaskOnPool(cwd, baseBranch, entry.id, pool, repoId, pipelineSession, entry.judgeOnly);
        pipelineSession.taskResults.push(result);
        if (result.status === "done" || result.status === "done-with-concerns") {
          await recordTaskComplete(cwd, entry.id);
        }
        await runGit(cwd, ["push", "-u", "origin", phaseBranch]);
        logProgress(pipelineSession);
      }
    }
    // Discovers phases/tasks from the repo's own tasks/INDEX.md instead of
    // requiring the caller to pass --tasks. Matches this convention's board
    // shape (see e.g. Poimen/agent-rust's tasks/INDEX.md): a numbered phase
    // heading ("## 1 — Foundations · T0.x"), followed by a markdown table
    // whose rows link to each task's own spec file ("| [T0.1](T0.1-....md) |
    // ... |"). Headings that aren't a numbered phase (prose sections like
    // "## Ordering — declared, never derived", "## Progress") are skipped --
    // only "## <digits> — ..." starts a new phase. Returns null if
    // tasks/INDEX.md doesn't exist; an empty array if it exists but no phase
    // yielded any task rows.
    function parseTaskBoard(cwd) {
      const indexPath = path.join(cwd, "tasks", "INDEX.md");
      if (!fs.existsSync(indexPath)) return null;
      const phaseHeaderRe = /^##\s+\d+\s+—/;
      const taskRowRe = /^\|\s*\[([A-Za-z0-9.]+)\]\(/;
      const phases = [];
      let current = null;
      for (const line of fs.readFileSync(indexPath, "utf8").split("\n")) {
        if (phaseHeaderRe.test(line)) {
          current = [];
          phases.push(current);
          continue;
        }
        const m = line.match(taskRowRe);
        if (m && current) current.push(m[1]);
      }
      return phases.filter((phase) => phase.length > 0);
    }
    function phaseLabelFor(phaseTasks, index) {
      const first = phaseTasks[0];
      const id = typeof first === "string" ? first : first.id;
      const dot = id.indexOf(".");
      return dot === -1 ? `phase-${index}` : id.slice(0, dot);
    }
    function logProgress(pipelineSession) {
      console.log(`[repo ${pipelineSession.id}] ${JSON.stringify(pipelineSession)}`);
    }
    // Runs one repo's full pipeline: clone, then phases strictly sequentially.
    // tasks: array of phases, each phase an array of task ids with no declared
    // dependency on each other (e.g. [["T0.1","T0.2"], ["T1.1","T1.2","T1.3"]]).
    // A flat array of ids is also accepted and treated as one single phase. If
    // omitted, phases are discovered from the repo's own tasks/INDEX.md and run
    // judgeOnly first (a cheap "is this already done" check against the
    // board's possibly-stale checkmarks). Each phase gets its own branch
    // (agent-run/<repoId>/<phaseLabel>, e.g. .../T1); once every task in that
    // phase lands "done" or "done-with-concerns" AND the phase judge (the same
    // pooled judge agent that reviewed each task) passes the integration
    // review, the phase branch is squash-merged into baseBranch and pushed,
    // then the next phase branches off that updated base. Any failure halts
    // this repo's pipeline before merging -- it does not affect other repos
    // running concurrently (see runCoordinator).
    async function runRepoPipeline({ repoId, repo, baseBranch, tasks, branchName }, pipelineSession) {
      const cwd = path.join(WORK_DIR, repoId);
      // repoId is a slug derived from the repo URL now (see slugFor), not a
      // fresh UUID -- reusable across separate `runCoordinator` invocations
      // against the same repo, so a stale clone from a prior run has to be
      // wiped before this one starts, not merged into.
      fs.rmSync(cwd, { recursive: true, force: true });
      fs.mkdirSync(cwd, { recursive: true });
      const pool = {};
      const finish = (status) => {
        pipelineSession.status = status;
        pipelineSession.endedAt = new Date().toISOString();
        logProgress(pipelineSession);
        return pipelineSession;
      };
      // Deterministic, not routed through an LLM -- clone is 100% mechanical
      // (same reasoning as commitPending/the squash-merge sequence below), and
      // was the one place left that broke that pattern: a headless spawnPi
      // call here meant a crash gave zero diagnostic output, just a silent
      // exit code with nothing to debug from.
      const clone = await runGit(cwd, ["clone", "--branch", baseBranch, repo, "."]);
      if (clone.code !== 0) {
        pipelineSession.gitError = clone.out;
        return finish("clone-crashed");
      }
      if (!fs.existsSync(path.join(cwd, ".git"))) return finish("clone-missing");
      let phases = tasks ? (Array.isArray(tasks[0]) ? tasks : [tasks]) : parseTaskBoard(cwd);
      if (!phases || phases.length === 0) {
        pipelineSession.gitError = "no tasks given and tasks/INDEX.md not found or empty";
        return finish("no-tasks-found");
      }
      if (!tasks) {
        phases = phases.map((phase) => phase.map((id) => ({ id, judgeOnly: true })));
      }
      pipelineSession.totalTasks = phases.flat().length;
      for (let i = 0; i < phases.length; i++) {
        const phaseTasks = phases[i];
        const phaseLabel = phaseLabelFor(phaseTasks, i);
        const phaseBranch = branchName ? `${branchName}/${phaseLabel}` : `agent-run/${repoId}/${phaseLabel}`;
        // Resume a phase branch a prior (since-restarted) run already pushed,
        // instead of always branching fresh off baseBranch -- otherwise every
        // redeploy silently discards whatever tasks that prior run already
        // committed and pushed (see runPhase's per-task push below).
        const fetchExisting = await runGit(cwd, ["fetch", "origin", phaseBranch]);
        const resuming = fetchExisting.code === 0;
        const branchResult = resuming
          ? await runGit(cwd, ["checkout", "-b", phaseBranch, "FETCH_HEAD"])
          : await runGit(cwd, ["checkout", "-b", phaseBranch]);
        if (branchResult.code !== 0) {
          pipelineSession.gitError = branchResult.out;
          return finish("branch-crashed");
        }
        // Idempotent and run every phase, NOT gated on a fresh (non-resumed)
        // start -- every run this session was a resume, so the old i===0 &&
        // !resuming gate meant this setup permanently never ran on poiman's
        // branch, and portfolio's PLAN.md stayed tracked from before this rule
        // ever existed (gitignore has no effect on an already-tracked file --
        // observed live: it kept getting swept back in by every `git add -A`
        // regardless of the ignore rule). Check-and-fix on every phase instead
        // of once-at-genesis so a repo that's missing either self-heals on its
        // very next run rather than carrying the gap forever.
        const gitignorePath = path.join(cwd, ".gitignore");
        const currentGitignore = fs.existsSync(gitignorePath) ? fs.readFileSync(gitignorePath, "utf8").split("\n") : [];
        const requiredGitignoreLines = [
          "*.tar.gz",
          "*.tgz",
          "*.crate",
          "*.zip",
          "*.bin",
          "*.whl",
          "vendor/",
          "node_modules/",
          ".task-result-*",
          ".phase-result-*",
          ".stage-done-*",
          "PLAN.md",
        ];
        const missingGitignoreLines = requiredGitignoreLines.filter((line) => !currentGitignore.includes(line));
        if (missingGitignoreLines.length > 0) {
          fs.appendFileSync(
            gitignorePath,
            "\n# agent-harness: build artifacts, vendored archives, and harness bookkeeping never belong in source control\n" +
              missingGitignoreLines.join("\n") +
              "\n"
          );
          await runGit(cwd, ["add", ".gitignore"]);
          await runGit(cwd, ["commit", "-m", "chore: broaden .gitignore for agent-run artifacts"]);
        }
        const trackedFiles = await runGit(cwd, ["ls-tree", "-r", "HEAD", "--name-only"]);
        if (trackedFiles.out.split("\n").includes("PLAN.md")) {
          await runGit(cwd, ["rm", "--cached", "PLAN.md"]);
          await runGit(cwd, ["commit", "-m", "chore: untrack PLAN.md (already gitignored, was committed pre-rule)"]);
        }
        await runPhase(cwd, baseBranch, phaseBranch, phaseTasks, pool, repoId, pipelineSession);
        const phaseTaskIds = new Set(phaseTasks.map((t) => (typeof t === "string" ? t : t.id)));
        const phaseResults = pipelineSession.taskResults.filter((r) => phaseTaskIds.has(r.task));
        const phaseClean =
          phaseResults.length === phaseTaskIds.size && phaseResults.every((r) => r.status === "done" || r.status === "done-with-concerns");
        if (!phaseClean) {
          pipelineSession.haltedAt = phaseLabel;
          return finish("halted-phase-failed");
        }
        const phaseResultFile = path.join(cwd, `.phase-result-${phaseLabel}`);
        fs.rmSync(phaseResultFile, { force: true });
        const phaseJudge = await runOnPool(
          pool,
          cwd,
          repoId,
          "judge",
          `${REQUIRE_REAL_TOOL_CALLS}Use the judge skill to review the full phase diff for phase ${phaseLabel} against ` +
            `${baseBranch}...HEAD (covers every task in this phase: ${[...phaseTaskIds].join(", ")}). Every ` +
            `individual task already passed its own judge review -- your job here is different: confirm the ` +
            `tasks integrate correctly as one coherent narrative, and that real integration tests (not just ` +
            `each task's isolated unit checks) exist and actually exercise the phase's intended use case end ` +
            `to end. Write your verdict to ${phaseResultFile} as a single "VERDICT: PASS" or ` +
            `"VERDICT: FAIL" line plus rationale, then run: touch ${path.join(cwd, `.stage-done-phase-${phaseLabel}-judge`)}`,
          path.join(cwd, `.stage-done-phase-${phaseLabel}-judge`)
        );
        await commitPending(cwd, `phase: ${phaseLabel} integration review`);
        if (!phaseJudge.ok) {
          pipelineSession.haltedAt = phaseLabel;
          pipelineSession.gitError = phaseJudge.error;
          return finish("phase-judge-crashed");
        }
        const phaseJudgeText = fs.existsSync(phaseResultFile) ? fs.readFileSync(phaseResultFile, "utf8") : "";
        if (parseVerdictLine(phaseJudgeText, "VERDICT") !== "PASS") {
          pipelineSession.haltedAt = phaseLabel;
          pipelineSession.phaseJudgeRationale = phaseJudgeText;
          return finish("halted-phase-judge-failed");
        }
        const checkoutBase = await runGit(cwd, ["checkout", baseBranch]);
        if (checkoutBase.code !== 0) {
          pipelineSession.gitError = checkoutBase.out;
          return finish("squash-crashed");
        }
        const squash = await runGit(cwd, ["merge", "--squash", phaseBranch]);
        if (squash.code !== 0) {
          await runGit(cwd, ["merge", "--abort"]);
          pipelineSession.gitError = squash.out;
          return finish("squash-crashed");
        }
        const commit = await runGit(cwd, ["commit", "-m", `feat: ${phaseLabel} (${[...phaseTaskIds].join(", ")})`]);
        if (commit.code !== 0) {
          pipelineSession.gitError = commit.out;
          return finish("squash-crashed");
        }
        const push = await runGit(cwd, ["push", "origin", baseBranch]);
        if (push.code !== 0) {
          pipelineSession.gitError = push.out;
          return finish("squash-push-crashed");
        }
        // Milestone's content now lives in baseBranch as one squashed commit --
        // the phase branch (and whatever a prior restart already pushed of it)
        // has no further reason to exist. Delete it both places so a future run
        // never tries to resume a phase that's already done, and so origin
        // doesn't accumulate one dangling branch per completed phase forever.
        await runGit(cwd, ["branch", "-D", phaseBranch]);
        await runGit(cwd, ["push", "origin", "--delete", phaseBranch]);
        logProgress(pipelineSession);
      }
      return finish("completed");
    }
    async function runConcurrent(items, limit, worker) {
      const results = new Array(items.length);
      let i = 0;
      async function next() {
        while (i < items.length) {
          const idx = i++;
          results[idx] = await worker(items[idx], idx);
        }
      }
      await Promise.all(Array.from({ length: Math.min(limit, items.length) }, next));
      return results;
    }
    // How many repos can be mid-flight at once. Each repo gets its own clone
    // and its own 4-agent pool (planner/investigator/implementer/judge), so
    // this is now the real concurrency knob -- tasks within one repo are
    // already serialized against that repo's pool (see runPhase). The backend
    // (homelab-ornith) actually runs 2 GPU replicas behind one Kubernetes
    // Service, each with its own copy of the model loaded (see homelab's
    // k8s/apps/llm-serving/ornith.yaml) -- so up to 2 concurrent LLM calls get
    // real independent instances; a 3rd+ concurrent call queues inside
    // whichever replica the Service's own load-balancing lands it on (each
    // replica runs OLLAMA_NUM_PARALLEL=1). REPO_CONCURRENCY above 2 is still
    // useful (more repos in flight overlaps git/file work, not just LLM calls)
    // but past 2 simultaneous LLM calls, extra concurrency mostly means queueing
    // rather than added throughput -- bump the backend's replica count to
    // change that, not this constant.
    const REPO_CONCURRENCY = Number(process.env.REPO_CONCURRENCY) || 3;
    // repoId is the repo's own name, not a random id -- it's what every role
    // session's --name is built from (see runOnPool: `${repoId}-${role}`), so
    // agent-manager's own session list groups naturally by repo ("portfolio-
    // planner", "portfolio-judge", "poiman-planner", ...) instead of by opaque
    // UUID. Takes the last path segment of the URL, strips a trailing `.git`,
    // and sanitizes anything that isn't safe in a tmux session name / directory
    // name / git branch name. Two different repos that happen to share a
    // basename (e.g. two orgs' "portfolio") would collide -- not handled, since
    // nothing about this harness's usage has needed more than one org per run.
    function slugFor(repoUrl) {
      const last = repoUrl.replace(/\/+$/, "").split("/").pop() || repoUrl;
      return last.replace(/\.git$/, "").replace(/[^a-zA-Z0-9._-]/g, "-");
    }
    // Top-level entry point: runs every repo in `repos` to completion, up to
    // REPO_CONCURRENCY at a time. Returns a map of repoId -> final
    // pipelineSession, one per repo, independent of how the others fared.
    async function runCoordinator({ repos, base, tasks, branchName }) {
      const sessions = {};
      await runConcurrent(repos, REPO_CONCURRENCY, async (repoUrl) => {
        const repoId = slugFor(repoUrl);
        const pipelineSession = {
          id: repoId,
          repo: repoUrl,
          status: "running",
          taskResults: [],
          activeTasks: {},
          totalTasks: 0,
          startedAt: new Date().toISOString(),
        };
        sessions[repoId] = pipelineSession;
        await runRepoPipeline({ repoId, repo: repoUrl, baseBranch: base, tasks, branchName }, pipelineSession);
      });
      return sessions;
    }
    function parseArgs(argv) {
      const opts = { base: "main" };
      for (let i = 0; i < argv.length; i++) {
        const a = argv[i];
        if (a === "--repo") opts.repo = argv[++i];
        else if (a === "--repos") opts.repos = argv[++i];
        else if (a === "--base") opts.base = argv[++i];
        else if (a === "--tasks") opts.tasks = argv[++i];
        else if (a === "--branch") opts.branch = argv[++i];
      }
      return opts;
    }
    async function main() {
      const opts = parseArgs(process.argv.slice(2));
      const repos = opts.repos ? opts.repos.split(",") : opts.repo ? [opts.repo] : null;
      if (!repos || repos.length === 0) {
        console.error(
          "usage: coordinator.js --repos <url1,url2,...> [--tasks T0.1,T0.2;T1.1,T1.2,...] [--base main] [--branch <name>]\n" +
            "  --repo <url> also accepted for a single repo\n" +
            "  --tasks applies to every repo listed; omitted: each repo discovers its own phases from tasks/INDEX.md\n" +
            "  REPO_CONCURRENCY env var (default 3): how many repos run at once"
        );
        // process.exitCode + natural exit, not process.exit() -- stdout piped
        // through kubectl exec (not a TTY) can drop buffered console.log/
        // console.error output if the process exits before it flushes. Setting
        // exitCode and letting the event loop drain naturally is the
        // documented-safe way to exit with a specific code without racing it.
        process.exitCode = 1;
        return;
      }
      const phases = opts.tasks ? opts.tasks.split(";").map((phase) => phase.split(",")) : null;
      const sessions = await runCoordinator({ repos, base: opts.base, tasks: phases, branchName: opts.branch });
      process.exitCode = Object.values(sessions).every((s) => s.status === "completed") ? 0 : 1;
    }
    if (require.main === module) {
      main();
    }
    module.exports = { runCoordinator, runRepoPipeline, runOnPool, parseTaskBoard };
@@ -0,0 +1,168 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: agent-pod
  namespace: agent-pod
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: agent-pod
  template:
    metadata:
      labels:
        app: agent-pod
    spec:
      # api.riotpiao.com has no in-cluster DNS record (only resolves from the
      # home network's own resolver) -- pin it to ingress-nginx-controller's
      # ClusterIP so pi's models.json baseUrl works unchanged. TLS still
      # terminates correctly since SNI/Host still say api.riotpiao.com.
      hostAliases:
        - ip: "10.101.128.185"
          hostnames:
            - "api.riotpiao.com"
      containers:
        # hub.js runs in the same container as pi (not a sidecar) so it can
        # spawn `pi -p --mode json` directly via child_process -- a separate
        # container can't exec into another container's filesystem/PATH.
        # It IS the container's long-running process now; no more `sleep
        # infinity` placeholder.
        #
        # Also builds the agent-manager fork (github.com/Riotpiaole/
        # agent-manager, add-headless-spawn branch) from source and drops
        # coordinator.js in beside hub.js -- neither is the container's
        # foreground process. hub.js keeps that role unchanged; coordinator.js
        # itself now owns multi-repo concurrency (REPO_CONCURRENCY env,
        # default 3), so one invocation handles every repo:
        # `kubectl exec <pod> -- node /root/coordinator.js --repos
        # repoA,repoB,... --tasks ...`. Each repo gets its own clone and its
        # own persistent 4-agent pool (planner/investigator/implementer/
        # judge, one agent-manager session per role, reused across every
        # task in that repo) on the container's local tmux server --
        # `kubectl exec -it <pod> -- agent-manager` attaches its TUI live
        # against those same sessions, no cross-machine visibility problem
        # since spawner, tmux server, and viewer are all colocated here.
        #
        # No prebuilt Linux binary is shipped for agent-manager: the local
        # .bin/ build is macOS arm64 (wrong OS/arch for this container
        # anyway) and it's 27MB, well over a ConfigMap's ~1MiB cap. Debian's
        # `apt-get golang-go` is far too old for this fork's go 1.26.5
        # requirement, so the real Go toolchain is fetched directly from
        # go.dev instead.
        - name: pi
          image: node:22-slim
          command:
            - sh
            - -c
            - |
              set -e
              apt-get update && apt-get install -y git curl jq openssh-client tmux python3 sqlite3 gcc build-essential
              ssh-keygen -y -f /root/.ssh/id_forgejo > /root/.ssh/id_forgejo.pub
              eval "$(ssh-agent -s)"
              ssh-add /root/.ssh/id_forgejo
              npm install -g @earendil-works/[email protected]
              npm install --prefix /root ws
              curl -fsSL "https://go.dev/dl/go1.26.5.linux-$(dpkg --print-architecture).tar.gz" | tar -C /usr/local -xz
              export PATH="$PATH:/usr/local/go/bin"
              git clone --branch add-headless-spawn --depth 1 \
                https://github.com/Riotpiaole/agent-manager.git /root/agent-manager-src
              (cd /root/agent-manager-src && go build -o /usr/local/bin/agent-manager .)
              # Language toolchains for whatever repos the implementer/investigator/
              # judge roles actually build and test -- go was already fetched above
              # only for building agent-manager itself, and its PATH export above is
              # local to this script, invisible to `kubectl exec` sessions into the
              # already-running container. Symlinking both into /usr/local/bin (on
              # PATH for every exec session, interactive or not) instead of relying
              # on shell rc sourcing, which pi's non-interactive tool calls don't do.
              ln -sf /usr/local/go/bin/go /usr/local/bin/go
              curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable
              ln -sf /root/.cargo/bin/cargo /usr/local/bin/cargo
              ln -sf /root/.cargo/bin/rustc /usr/local/bin/rustc
              ln -sf /root/.cargo/bin/rustup /usr/local/bin/rustup
              node /root/hub.js
          env:
            - name: PI_BIN
              value: pi
            - name: AGENT_MANAGER_BIN
              value: /usr/local/bin/agent-manager
            - name: HUB_WORK_DIR
              value: /root/agent-harness-work
            # planner/investigator/implementer stay on the default
            # (homelab-ornith/ornith:35b, pi's settings.json default). Judge
            # moves to the separate homelab-reasoning backend (DeepSeek-R1,
            # its own 2 GPU replicas) so judge calls stop contending with the
            # other 3 roles for the 2 ornith pods -- an entire role's worth
            # of traffic moves onto otherwise-idle capacity instead.
            - name: JUDGE_PROVIDER
              value: homelab-reasoning
            - name: JUDGE_MODEL
              value: reasoning
          ports:
            - containerPort: 9090
          resources:
            requests:
              cpu: "4"
              memory: 8Gi
            limits:
              cpu: "8"
              memory: 16Gi
          volumeMounts:
            - name: pi-config
              mountPath: /root/.pi/agent/settings.json
              subPath: settings.json
            - name: pi-models
              mountPath: /root/.pi/agent/models.json
              subPath: models.json
            - name: pi-skills
              mountPath: /root/.pi/agent/skills
            - name: hub-src
              mountPath: /root/hub.js
              subPath: hub.js
            - name: coordinator-src
              mountPath: /root/coordinator.js
              subPath: coordinator.js
            - name: ssh-key
              mountPath: /root/.ssh/id_forgejo
              subPath: id_forgejo
            - name: ssh-config
              mountPath: /root/.ssh/config
              subPath: config
      volumes:
        - name: pi-config
          configMap:
            name: pi-config
        - name: pi-models
          secret:
            secretName: pi-models
        - name: pi-skills
          configMap:
            name: pi-skills
            items:
              - key: planner-SKILL.md
                path: planner/SKILL.md
              - key: investigator-SKILL.md
                path: investigator/SKILL.md
              - key: info-collector-SKILL.md
                path: info-collector/SKILL.md
              - key: implementer-SKILL.md
                path: implementer/SKILL.md
              - key: judge-SKILL.md
                path: judge/SKILL.md
              - key: resolver-SKILL.md
                path: resolver/SKILL.md
        - name: hub-src
          configMap:
            name: hub-src
        - name: coordinator-src
          configMap:
            name: coordinator-src
        - name: ssh-key
          secret:
            secretName: agent-pod-ssh-key
            defaultMode: 0600
        - name: ssh-config
          configMap:
            name: agent-pod-ssh-config
@@ -0,0 +1,700 @@
 apiVersion: v1
 data:
  hub.js: |
    #!/usr/bin/env node
    // agent-hub: lives inside the pi container (not a sidecar) so it can spawn
    // `pi` directly, and control the pod's own tmux server. One persistent
    // in-cluster service -- POST /run to trigger a single ad-hoc headless agent
    // run, POST /pipeline to run an ordered list of task phases against a
    // repo/branch (phases run sequentially, up to PHASE_CONCURRENCY tasks within
    // a phase run concurrently, each in its own git worktree). Within one task,
    // planner/investigator/implementer/judge are separate agents in separate
    // named tmux sessions (task-<id>-<role>, attachable via `kubectl exec -it --
    // tmux attach -t <name>` while running), coordinating only through what's on
    // disk in that task's worktree -- not one shared conversation. GET /console
    // (WebSocket) watches every concurrent headless run live, relaying pi's own
    // session protocol verbatim (same event shape Claude Code sessions use).
    const http = require("node:http");
    const crypto = require("node:crypto");
    const fs = require("node:fs");
    const path = require("node:path");
    const { spawn } = require("node:child_process");
    const readline = require("node:readline");
    const { WebSocketServer } = require("ws");
    const PORT = process.env.HUB_PORT || 9090;
    const WORK_DIR = process.env.HUB_WORK_DIR || path.join(require("node:os").tmpdir(), "agent-harness-work");
    // Never rely on a bare `pi` on $PATH -- both `pi` and `agent-console` collide
    // with unrelated tools on this machine (a Rust CLI and a Datadog TUI,
    // respectively, discovered the hard way this session). Always invoke the
    // exact pinned @earendil-works/[email protected] installed locally under
    // .pi-cli/, by explicit path.
    const PI_BIN =
      process.env.PI_BIN ||
      path.join(
        __dirname,
        "..",
        ".pi-cli",
        "node_modules",
        "@earendil-works",
        "pi-coding-agent",
        "dist",
        "cli.js"
      );
    // Job-type skills under pi/skills/<name>/SKILL.md (mounted at
    // ~/.pi/agent/skills/<name>/ in agent-pod). When `agent` matches one of
    // these, the prompt is forced through pi's `/skill:<name> <args>` mechanism
    // instead of being sent bare -- see pi's skills.md docs on single-shot skill
    // loading. `resolver` is never dispatched directly by a caller; only the
    // pipeline driver invokes it, on stage crashes.
    const ROLE_SKILLS = new Set([
      "planner",
      "investigator",
      "info-collector",
      "implementer",
      "judge",
      "resolver",
    ]);
    const sessions = new Map(); // id -> {id, agent, status, events, startedAt, endedAt, pipelineId?, stage?}
    const viewers = new Set(); // WebSocket connections watching /console
    function broadcast(type, session) {
      const msg = JSON.stringify({ type, session });
      for (const ws of viewers) {
        if (ws.readyState === ws.OPEN) ws.send(msg);
      }
    }
    function startSession(agent, extra = {}) {
      const id = extra.id || crypto.randomUUID();
      const session = {
        ...extra,
        id,
        agent,
        status: "running",
        events: [],
        startedAt: new Date().toISOString(),
      };
      sessions.set(id, session);
      broadcast("start", session);
      return session;
    }
    function addEvent(session, rawLine) {
      const event = JSON.parse(rawLine);
      session.events.push(event);
      broadcast("event", session);
      return event;
    }
    function endSession(session, status) {
      session.status = status;
      session.endedAt = new Date().toISOString();
      broadcast("end", session);
    }
    // Extracts the plain-text content of a message_end event, if any -- used to
    // find the VERDICT:/RESOLUTION: line judge/resolver skills are required to
    // end their final message with.
    function textOf(event) {
      if (event.type !== "message_end" || !event.message || !Array.isArray(event.message.content)) {
        return null;
      }
      return event.message.content
        .filter((c) => c.type === "text")
        .map((c) => c.text)
        .join("\n");
    }
    // Core spawn primitive. Spawns `pi -p --mode json <extraArgs> <prompt>`,
    // relays every line as a session event exactly like before. Returns
    // { session, done } -- `session` is available synchronously (so an HTTP
    // handler can respond with its id right away, same as the old runAgent),
    // `done` is a Promise resolving once the process exits, for callers that
    // need to wait on a stage (the pipeline driver) rather than fire-and-forget.
    function spawnPi({ agent, prompt, provider, model, cwd, sessionExtra = {} }) {
      const session = startSession(agent, sessionExtra);
      const args = ["-p", "--mode", "json"];
      if (provider) args.push("--provider", provider);
      if (model) args.push("--model", model);
      const finalPrompt = ROLE_SKILLS.has(agent) ? `/skill:${agent} ${prompt}` : prompt;
      args.push(finalPrompt);
      const child = spawn(PI_BIN, args, {
        stdio: ["ignore", "pipe", "pipe"],
        cwd,
      });
      const rl = readline.createInterface({ input: child.stdout });
      let lastText = "";
      let stderrTail = "";
      rl.on("line", (line) => {
        if (!line.trim()) return;
        try {
          const event = addEvent(session, line);
          const text = textOf(event);
          if (text) lastText = text;
        } catch {
          // non-JSON stdout noise, ignore
        }
      });
      child.stderr.on("data", (chunk) => {
        process.stderr.write(chunk);
        stderrTail = (stderrTail + chunk.toString()).slice(-4000);
      });
      const done = new Promise((resolve) => {
        child.on("close", (code) => {
          endSession(session, code === 0 ? "done" : "error");
          resolve({ code, session, lastText, stderrTail });
        });
      });
      return { session, done };
    }
    function runAgent(agent, prompt, extraArgs = {}) {
      // Fire-and-forget: caller (the /run handler) doesn't await `done`.
      return spawnPi({ agent, prompt, ...extraArgs }).session;
    }
    function parseVerdictLine(text, label) {
      if (!text) return null;
      const re = new RegExp(`${label}:\\s*(\\w+)`, "i");
      const m = text.match(re);
      return m ? m[1].toUpperCase() : null;
    }
    // Deterministic git operations, run directly by hub.js rather than left to
    // the model -- branch creation and pushing after each task are mechanical,
    // not judgment calls, and need to happen reliably every time regardless of
    // what a task's stages did or didn't remember to do.
    function runCmd(bin, args, cwd) {
      return new Promise((resolve) => {
        const child = spawn(bin, args, { cwd, stdio: ["ignore", "pipe", "pipe"] });
        let out = "";
        child.stdout.on("data", (c) => (out += c));
        child.stderr.on("data", (c) => (out += c));
        child.on("close", (code) => resolve({ code, out: out.trim() }));
      });
    }
    function runGit(cwd, args) {
      return runCmd("git", args, cwd);
    }
    // A stage saying "commit" in its prompt is a request, not a guarantee -- seen
    // in practice: a stage writes a real file and simply never runs `git add`/
    // `git commit`, leaving it untracked and invisible to every later `git diff`.
    // Sweep and commit anything left dirty after every stage, deterministically.
    async function commitPending(cwd, message) {
      await runGit(cwd, ["add", "-A"]);
      const status = await runGit(cwd, ["status", "--porcelain"]);
      if (!status.out) return { committed: false };
      const commit = await runGit(cwd, ["commit", "-m", message]);
      return { committed: commit.code === 0, error: commit.code !== 0 ? commit.out : undefined };
    }
    // Invokes the `resolver` skill to diagnose a stuck/crashed stage and decide
    // RETRY vs ABORT. Shared by both crash-recovery paths below (headless
    // exit-code failures and interactive sentinel-file timeouts) -- the
    // diagnostic prompt differs per caller, but "ask resolver, parse the
    // RESOLUTION: line" is identical either way.
    async function askResolver(pipelineId, cwd, task, diagnosticPrompt) {
      const resolverResult = await spawnPi({
        agent: "resolver",
        prompt: diagnosticPrompt,
        cwd,
        sessionExtra: { pipelineId, stage: "resolver", task },
      }).done;
      return parseVerdictLine(resolverResult.lastText, "RESOLUTION");
    }
    // Runs one pipeline stage, and if it crashes (nonzero exit -- not a semantic
    // judge FAIL, which is handled separately), asks the resolver to diagnose
    // and decide RETRY vs ABORT. Retries the failed stage at most once,
    // regardless of what resolver recommends a second time -- a hard cap, not
    // indefinite trust in the model's judgment.
    async function runStageWithResolver(pipelineId, cwd, stage, prompt, task) {
      let result = await spawnPi({
        agent: stage,
        prompt,
        cwd,
        sessionExtra: { pipelineId, stage, task },
      }).done;
      if (result.code === 0) return result;
      const resolution = await askResolver(
        pipelineId,
        cwd,
        task,
        `Stage "${stage}" exited with code ${result.code}. Its stderr tail:\n${result.stderrTail}`
      );
      if (resolution === "RETRY") {
        result = await spawnPi({
          agent: stage,
          prompt,
          cwd,
          sessionExtra: { pipelineId, stage, task },
        }).done;
      }
      return result;
    }
    // Deterministic tmux operations -- same rationale as runGit: mechanical,
    // not a judgment call, run directly rather than trusted to a prompt.
    function runTmux(args) {
      return runCmd("tmux", args);
    }
    function tmuxSessionName(task) {
      return `task-${task.replace(/[^a-zA-Z0-9]/g, "-")}`;
    }
    // Bounded-concurrency pool -- runs `worker` over `items`, at most `limit` in
    // flight at once. No external dep; a plain in-order index cursor shared by
    // `limit` runner loops.
    async function runConcurrent(items, limit, worker) {
      const results = new Array(items.length);
      let i = 0;
      async function next() {
        while (i < items.length) {
          const idx = i++;
          results[idx] = await worker(items[idx], idx);
        }
      }
      await Promise.all(Array.from({ length: Math.min(limit, items.length) }, next));
      return results;
    }
    const STAGE_TIMEOUT_MS = 10 * 60 * 1000;
    const NUDGE_TIMEOUT_MS = 5 * 60 * 1000;
    const POLL_MS = 10 * 1000;
    async function waitForFile(filePath, limitMs) {
      const start = Date.now();
      while (!fs.existsSync(filePath)) {
        if (Date.now() - start > limitMs) return false;
        await new Promise((r) => setTimeout(r, POLL_MS));
      }
      return true;
    }
    const MAX_IMPLEMENT_ATTEMPTS = 5;
    const MAX_PLAN_REVISIONS = 3;
    // Runs one role as its own fresh interactive pi session in its own named
    // tmux session -- planner, investigator, implementer, and judge are
    // separate agents with separate context, not turns in one shared
    // conversation. They coordinate only through what's on disk in the task's
    // worktree: PLAN.md, committed code, judge's result file. Each session is
    // attachable while it runs (kubectl exec -it -- tmux attach -t <name>) and
    // killed once its sentinel file lands or it's abandoned after resolver
    // escalation.
    async function runStage(pipelineId, cwd, task, stageLabel, stagePrompt, sentinelFile) {
      const sessionName = `${tmuxSessionName(task)}-${stageLabel}`;
      fs.rmSync(sentinelFile, { force: true });
      const spawned = await runTmux(["new-session", "-d", "-s", sessionName, "-c", cwd, PI_BIN, stagePrompt]);
      if (spawned.code !== 0) return { ok: false, crashed: true, error: spawned.out, sessionName };
      let ok = await waitForFile(sentinelFile, STAGE_TIMEOUT_MS);
      if (!ok) {
        const pane = await runTmux(["capture-pane", "-t", sessionName, "-p", "-S", "-200"]);
        const resolution = await askResolver(
          pipelineId,
          cwd,
          task,
          `Task ${task}'s "${stageLabel}" stage hasn't finished after 10 minutes. ` +
            `Its pane tail:\n${pane.out.slice(-3000)}\n\nDecide: is it still making ` +
            `real progress and worth nudging to wrap up, or stuck and worth abandoning?`
        );
        if (resolution === "RETRY") {
          await runTmux([
            "send-keys",
            "-t",
            sessionName,
            `Please wrap up the "${stageLabel}" stage now and touch ${path.basename(sentinelFile)} when done.`,
            "Enter",
          ]);
          ok = await waitForFile(sentinelFile, NUDGE_TIMEOUT_MS);
        }
      }
      await runTmux(["kill-session", "-t", sessionName]);
      return { ok, sessionName };
    }
    // Runs one task in its own git worktree (see runPhase): planner drafts
    // PLAN.md, investigator confirms it, then implementer and judge go back and
    // forth -- judge's FAIL rationale lands in .task-result-<task>, which the
    // next implementer attempt is told to read and address. After
    // MAX_IMPLEMENT_ATTEMPTS straight fails, planner is brought back in to
    // judge whether the *plan* itself is wrong, not just the implementation; if
    // so it revises PLAN.md and the implementer gets a fresh attempt budget
    // against the new plan. MAX_PLAN_REVISIONS caps this from looping forever
    // on a task that's genuinely stuck.
    async function runTaskInteractive(pipelineId, cwd, baseBranch, task, pipelineSession, judgeOnly) {
      const resultFile = path.join(cwd, `.task-result-${task}`);
      fs.rmSync(resultFile, { force: true });
      const specHint = `the file under tasks/ starting with "${task}-"`;
      const runRole = async (stageLabel, prompt, sentinel) => {
        pipelineSession.activeTasks[task] = {
          stage: stageLabel,
          sessionName: `${tmuxSessionName(task)}-${stageLabel}`,
          startedAt: new Date().toISOString(),
        };
        broadcast("event", pipelineSession);
        const result = await runStage(pipelineId, cwd, task, stageLabel, prompt, sentinel);
        await commitPending(cwd, `task: ${task} (${stageLabel})`);
        return result;
      };
      const abandon = (stageLabel, result, attempt) => {
        delete pipelineSession.activeTasks[task];
        broadcast("event", pipelineSession);
        return {
          task,
          status: result.crashed ? "spawn-crashed" : "timed-out",
          error: result.error,
          stoppedAt: stageLabel,
          ...(attempt !== undefined ? { attempt } : {}),
        };
      };
      // Task's implementation is inherited already-committed (e.g. from a base
      // branch of prior work) -- try one judge pass against the spec directly
      // (no PLAN.md exists yet) before paying for a full planner/investigator
      // redo. PASS ends the task here; FAIL falls through into the normal flow
      // below, so planner/implementer pick up with the judge's real feedback.
      if (judgeOnly) {
        const quick = await runRole(
          "judge",
          `Task ${task} may already be implemented on this branch -- check ` +
            `\`git log --oneline --grep '${task}'\` and the current code directly against its spec ` +
            `(${specHint})'s acceptance criteria (no PLAN.md exists for this task yet). Write your ` +
            `verdict to .task-result-${task} as a single "VERDICT: PASS" or "VERDICT: FAIL" line plus ` +
            `one line of rationale, then run: touch .stage-done-${task}-judge-0`,
          path.join(cwd, `.stage-done-${task}-judge-0`)
        );
        if (!quick.ok) return abandon("judge", quick, 0);
        const quickText = fs.existsSync(resultFile) ? fs.readFileSync(resultFile, "utf8") : "";
        if (parseVerdictLine(quickText, "VERDICT") === "PASS") {
          delete pipelineSession.activeTasks[task];
          broadcast("event", pipelineSession);
          return { task, status: "done", judgeRationale: quickText, judgeOnlyPass: true };
        }
      }
      let result = await runRole(
        "planner",
        `Use the planner skill to draft PLAN.md for task ${task}, reading its spec (${specHint}). Commit PLAN.md, then run: touch .stage-done-${task}-planner`,
        path.join(cwd, `.stage-done-${task}-planner`)
      );
      if (!result.ok) return abandon("planner", result);
      result = await runRole(
        "investigator",
        `Use the investigator skill to confirm PLAN.md against real sources, append findings, commit. Then run: touch .stage-done-${task}-investigator`,
        path.join(cwd, `.stage-done-${task}-investigator`)
      );
      if (!result.ok) return abandon("investigator", result);
      let planRevisions = 0;
      let implementAttempt = 0;
      let verdict = null;
      let resultText = "";
      let justRevisedPlan = false;
      while (true) {
        implementAttempt++;
        const feedbackHint = fs.existsSync(resultFile)
          ? justRevisedPlan
            ? `.task-result-${task} holds the judge's feedback against the OLD plan, which prompted a plan revision -- ` +
              `PLAN.md has since changed. Read the current PLAN.md as the source of truth, not the old feedback verbatim.`
            : `A previous judge review exists at .task-result-${task} -- read it and address every issue it raises.`
          : "";
        justRevisedPlan = false;
        result = await runRole(
          "implementer",
          `Use the implementer skill to implement what the current PLAN.md specifies (commit as you go). ${feedbackHint} Then run: touch .stage-done-${task}-implementer-${implementAttempt}`,
          path.join(cwd, `.stage-done-${task}-implementer-${implementAttempt}`)
        );
        if (!result.ok) return abandon("implementer", result, implementAttempt);
        result = await runRole(
          "judge",
          `Use the judge skill to review the diff against ${baseBranch}...HEAD. Write your verdict to ` +
            `.task-result-${task} as a single "VERDICT: PASS" or "VERDICT: FAIL" line plus one line of ` +
            `rationale, then run: touch .stage-done-${task}-judge-${implementAttempt}`,
          path.join(cwd, `.stage-done-${task}-judge-${implementAttempt}`)
        );
        if (!result.ok) return abandon("judge", result, implementAttempt);
        resultText = fs.existsSync(resultFile) ? fs.readFileSync(resultFile, "utf8") : "";
        verdict = parseVerdictLine(resultText, "VERDICT");
        if (verdict === "PASS") break;
        if (implementAttempt >= MAX_IMPLEMENT_ATTEMPTS) {
          if (planRevisions >= MAX_PLAN_REVISIONS) break;
          planRevisions++;
          result = await runRole(
            "planner-revise",
            `Implementer failed judge review ${MAX_IMPLEMENT_ATTEMPTS} times in a row for task ${task}. Read PLAN.md, ` +
              `the judge's feedback in .task-result-${task}, and the current diff against ${baseBranch}...HEAD. Decide ` +
              `whether the plan's approach itself is wrong, not just the implementation -- if so, revise PLAN.md and ` +
              `commit. If you change the approach, also use the investigator skill to confirm the new approach against ` +
              `real sources before committing. If the plan is sound, note why in PLAN.md and leave it as-is. Then run: ` +
              `touch .stage-done-${task}-planner-revise-${planRevisions}`,
            path.join(cwd, `.stage-done-${task}-planner-revise-${planRevisions}`)
          );
          if (!result.ok) return abandon("planner-revise", result, planRevisions);
          implementAttempt = 0;
          justRevisedPlan = true;
        }
      }
      delete pipelineSession.activeTasks[task];
      broadcast("event", pipelineSession);
      if (verdict !== "PASS" && planRevisions >= MAX_PLAN_REVISIONS) {
        return { task, status: "unresolved", judgeRationale: resultText, implementAttempts: implementAttempt, planRevisions };
      }
      return {
        task,
        status: verdict === "PASS" ? "done" : "done-with-concerns",
        judgeRationale: resultText,
      };
    }
    const PHASE_CONCURRENCY = 3;
    // Runs one phase (a batch of tasks with no declared dependency on each
    // other) with up to PHASE_CONCURRENCY tasks in flight at once. Each task
    // gets its own git worktree off workBranch -- concurrent pi sessions writing
    // into one shared working tree would corrupt the index; worktrees share the
    // same object database but give each task an isolated checkout. After a
    // task's session ends, its branch is merged back into workBranch and pushed,
    // one merge at a time (git ref updates aren't safe to run concurrently even
    // though the worktrees themselves are isolated).
    async function runPhase(pipelineId, cwd, workBranch, phaseTasks, pipelineSession) {
      // Each entry is either a plain task id, or { id, judgeOnly: true } when
      // the task's implementation already exists (e.g. inherited from a base
      // branch) and just needs a real judge pass rather than a full
      // planner/investigator/implementer redo.
      const entries = phaseTasks.map((t) => (typeof t === "string" ? { id: t, judgeOnly: false } : t));
      const worktrees = {};
      for (const entry of entries) {
        const task = entry.id;
        const wtDir = path.join(WORK_DIR, pipelineId, `wt-${task.replace(/[^a-zA-Z0-9]/g, "-")}`);
        const taskBranch = `task/${task}`;
        const add = await runGit(cwd, ["worktree", "add", "-b", taskBranch, wtDir, workBranch]);
        if (add.code !== 0) {
          pipelineSession.taskResults.push({ task, status: "worktree-crashed", error: add.out });
          continue;
        }
        worktrees[task] = { wtDir, taskBranch };
      }
      const runnable = entries.filter((e) => worktrees[e.id]);
      await runConcurrent(runnable, PHASE_CONCURRENCY, async (entry) => {
        const { wtDir } = worktrees[entry.id];
        const result = await runTaskInteractive(pipelineId, wtDir, workBranch, entry.id, pipelineSession, entry.judgeOnly);
        pipelineSession.taskResults.push(result);
        return result;
      });
      // Merge + push sequentially -- ref updates on the shared repo, one at a
      // time, in the declared task order for this phase.
      for (const entry of runnable) {
        const task = entry.id;
        const { wtDir, taskBranch } = worktrees[task];
        const result = pipelineSession.taskResults.find((r) => r.task === task);
        const merge = await runGit(cwd, ["merge", "--no-ff", taskBranch, "-m", `merge: ${task}`]);
        if (merge.code !== 0) {
          await runGit(cwd, ["merge", "--abort"]);
          if (result) {
            result.status = "merge-conflict";
            result.mergeError = merge.out;
          }
        } else {
          const push = await runGit(cwd, ["push", "-u", "origin", workBranch]);
          if (result) {
            result.pushed = push.code === 0;
            if (!result.pushed) result.pushError = push.out;
          }
        }
        await runGit(cwd, ["worktree", "remove", wtDir, "--force"]);
        await runGit(cwd, ["branch", "-D", taskBranch]);
        broadcast("event", pipelineSession);
      }
    }
    // tasks: array of phases, each phase an array of task ids with no declared
    // dependency on each other (e.g. [["T0.1","T0.2"], ["T1.1","T1.2","T1.3"]]) --
    // caller's responsibility to supply real phase grouping (see tasks/INDEX.md;
    // filename/numeric sort does NOT match execution order on boards like this).
    // A flat array of ids is also accepted and treated as one single phase.
    // Phases run strictly sequentially (a phase boundary is a real dependency
    // gate); tasks within a phase run concurrently, each in its own worktree --
    // see runPhase.
    function runPipeline({ pipelineId, repo, baseBranch, tasks, branchName }) {
      const cwd = path.join(WORK_DIR, pipelineId);
      fs.mkdirSync(cwd, { recursive: true });
      const phases = Array.isArray(tasks[0]) ? tasks : [tasks];
      const pipelineSession = startSession("pipeline", {
        id: pipelineId,
        pipelineId,
        stage: "pipeline",
        taskResults: [],
        activeTasks: {},
        totalTasks: phases.flat().length,
      });
      (async () => {
        const clone = await runStageWithResolver(
          pipelineId,
          cwd,
          "planner",
          `Run exactly this command, verbatim, no variation: git clone --branch ${baseBranch} ${repo} . -- the trailing dot is required, it clones directly into the current directory instead of creating a subdirectory. Do not cd anywhere first or after. Do nothing else.`,
          "clone"
        );
        if (clone.code !== 0) return endSession(pipelineSession, "clone-crashed");
        if (!fs.existsSync(path.join(cwd, ".git"))) {
          // The model deciding to `cd` elsewhere before cloning (instead of
          // cloning into the assigned cwd) is a real failure mode seen in
          // practice, not a hypothetical -- exit code 0 doesn't mean the clone
          // landed where every later stage's cwd assumes it did.
          return endSession(pipelineSession, "clone-missing");
        }
        // Dedicated branch, never main -- and pushed after every single task
        // (not just at the end) so a pod restart mid-run loses at most the
        // in-progress task's work, not everything since the start.
        const workBranch = branchName || `agent-run/${pipelineId}`;
        const branchResult = await runGit(cwd, ["checkout", "-b", workBranch]);
        if (branchResult.code !== 0) {
          pipelineSession.gitError = branchResult.out;
          return endSession(pipelineSession, "branch-crashed");
        }
        // Seen in practice: a task manually downloads a dependency tarball
        // (crates.io registry access isn't guaranteed from every sandboxed
        // checkout) and it lands at repo root, outside whatever .gitignore
        // already covers -- then git add -A (ours or the model's own) commits
        // it. Append broad build-artifact/archive patterns before any task
        // runs, so it's excluded regardless of who stages files later.
        const gitignoreAdditions = [
          "",
          "# agent-harness: build artifacts and vendored archives never belong in source control",
          "*.tar.gz",
          "*.tgz",
          "*.crate",
          "*.zip",
          "*.bin",
          "*.whl",
          "vendor/",
          "node_modules/",
          "",
          "# agent-harness: task completion sentinel files, harness bookkeeping only",
          ".task-result-*",
          ".stage-done-*",
        ].join("\n");
        fs.appendFileSync(path.join(cwd, ".gitignore"), gitignoreAdditions + "\n");
        await runGit(cwd, ["add", ".gitignore"]);
        await runGit(cwd, ["commit", "-m", "chore: broaden .gitignore for agent-run artifacts"]);
        for (const phaseTasks of phases) {
          await runPhase(pipelineId, cwd, workBranch, phaseTasks, pipelineSession);
        }
        const crashed = pipelineSession.taskResults.filter((r) => r.status.endsWith("-crashed"));
        endSession(pipelineSession, crashed.length > 0 ? "completed-with-crashes" : "completed");
      })();
      return pipelineSession;
    }
    const server = http.createServer((req, res) => {
      const url = new URL(req.url, "http://localhost");
      if (url.pathname === "/healthz") {
        res.writeHead(200).end();
        return;
      }
      if (url.pathname === "/sessions" && req.method === "GET") {
        res.writeHead(200, { "Content-Type": "application/json" });
        res.end(JSON.stringify([...sessions.values()]));
        return;
      }
      if (url.pathname === "/run" && req.method === "POST") {
        let body = "";
        req.on("data", (chunk) => (body += chunk));
        req.on("end", () => {
          try {
            const { agent, prompt, provider, model } = JSON.parse(body);
            if (!agent || !prompt) throw new Error("agent and prompt are required");
            const session = runAgent(agent, prompt, { provider, model });
            res.writeHead(200, { "Content-Type": "application/json" });
            res.end(JSON.stringify({ id: session.id }));
          } catch (err) {
            res.writeHead(400, { "Content-Type": "application/json" });
            res.end(JSON.stringify({ error: err.message }));
          }
        });
        return;
      }
      if (url.pathname === "/pipeline" && req.method === "POST") {
        let body = "";
        req.on("data", (chunk) => (body += chunk));
        req.on("end", () => {
          try {
            const { repo, baseBranch, tasks, branchName } = JSON.parse(body);
            if (!repo || !baseBranch || !Array.isArray(tasks) || tasks.length === 0) {
              throw new Error("repo, baseBranch, and a non-empty tasks array are required");
            }
            const pipelineId = crypto.randomUUID();
            runPipeline({ pipelineId, repo, baseBranch, tasks, branchName });
            res.writeHead(200, { "Content-Type": "application/json" });
            res.end(JSON.stringify({ id: pipelineId }));
          } catch (err) {
            res.writeHead(400, { "Content-Type": "application/json" });
            res.end(JSON.stringify({ error: err.message }));
          }
        });
        return;
      }
      res.writeHead(404).end();
    });
    const wss = new WebSocketServer({ server, path: "/console" });
    wss.on("connection", (ws) => {
      for (const session of sessions.values()) {
        ws.send(JSON.stringify({ type: "snapshot", session }));
      }
      viewers.add(ws);
      ws.on("close", () => viewers.delete(ws));
    });
    server.listen(PORT, () => console.log(`agent-hub listening on :${PORT}`));
 kind: ConfigMap
 metadata:
  name: hub-src
  namespace: agent-pod
@@ -0,0 +1,11 @@
 apiVersion: v1
 kind: Service
 metadata:
  name: agent-hub
  namespace: agent-pod
 spec:
  selector:
    app: agent-pod
  ports:
    - port: 9090
      targetPort: 9090
@@ -0,0 +1,12 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: agent-pod
 resources:
  - deployment.yaml
  - configmap.yaml
  - hub-configmap.yaml
  - coordinator-configmap.yaml
  - pi-skills-configmap.yaml
  - ssh-configmap.yaml
  - hub-service.yaml
  - console-ingress.yaml
@@ -0,0 +1,122 @@
 apiVersion: v1
 data:
  implementer-SKILL.md: |
    ---
    name: implementer
    description: Turns a confirmed PLAN.md into real code changes in the current checkout, committing incrementally. Use as the implementation stage of a spec-to-push pipeline, after planner and investigator have run.
    allowed-tools: Read Grep Find Ls Write Edit Bash
    ---
    Execute the already-agreed plan; don't re-litigate it. `PLAN.md` (plus any `## Investigation` flags) is the source of truth for *what*; use judgment only for *how*, within the codebase's existing conventions.
    - Read `PLAN.md` top to bottom. Treat flagged/unconfirmed steps conservatively (safer, more literal reading; note it in the commit). Work steps in order. Commit after each meaningful step (`git add -A && git commit -m "..."`), not one giant commit — the judge stage needs real diff history.
    - Push only if the task explicitly asks for it.
    Match existing style. Don't refactor or "improve" code the plan didn't ask you to touch.
    **UI/frontend changes:** don't trust that the code compiles as proof it works. Start the app (or its dev server) and use `npx playwright` via Bash to actually load the page and look — screenshot the affected view before and after your change, and click through the golden path the plan describes. `npx playwright screenshot <url> out.png` for a quick visual check; for interaction (clicks, form fills, navigation), write a small throwaway script under a scratch path (e.g. `/tmp/`, never committed) using `playwright` the library, run it with `node`, then delete it. This doesn't apply to non-UI work (a Rust library, a CLI, a backend-only change) — use judgment.
    **Hard rules:**
    - Follow DRY and SOLID. Don't duplicate logic that already exists elsewhere in the codebase you're touching — reuse or extract instead. Keep each unit responsible for one thing.
    - Never commit anything that doesn't belong in source control: build artifacts, downloaded/vendored dependencies, secrets, scratch/debug files. `.gitignore` already blocks common patterns; if you create something outside those patterns, delete it before committing rather than relying on `.gitignore` to catch it.
    **Never vendor a dependency by downloading/extracting it into the repo.** Use the language's real package manager (`cargo add`, `npm install`, etc.) so the dependency is declared in the manifest and lockfile, not a tarball or extracted source tree sitting in the checkout. If the package manager can't reach its registry from here, say so in your commit message rather than working around it — a later commit sweep (`git add -A`) commits whatever's in the checkout, including anything downloaded for a workaround, even if you never intended to keep it.
  info-collector-SKILL.md: |
    ---
    name: info-collector
    description: Gathers and summarizes information on a topic from the web without judging or confirming any particular approach. Use standalone when you need raw research/context on a subject, not a verdict on a specific plan (that's the investigator skill).
    allowed-tools: Read Bash
    ---
    **Persona:** You are a research assistant. Your job is to gather relevant information on a topic and summarize it neutrally — you are not asked to approve, reject, or recommend anything, just to collect and organize what's out there.
    **Thinking mode:** Medium — breadth of coverage matters more than deep verification here (that's the investigator skill's job).
    **Modes:**
    - **Collect mode** (default) — search the web (`curl` against `https://api.search.brave.com/res/v1/web/search`, header `X-Subscription-Token: $BRAVE_API_KEY`, `--data-urlencode "q=<query>"`) with varied queries to cover the topic from multiple angles, then produce a structured summary: topic areas found, key facts, and links to sources for each. Do not editorialize about which approach is "right" — that's out of scope for this skill.
    - If asked to write the summary to a file, write it and report the path; otherwise return it directly in your response.
  investigator-SKILL.md: |
    ---
    name: investigator
    description: Reads an existing PLAN.md and confirms its approach against real, current sources via web search, appending findings and flags. Use to sanity-check a plan before implementation, or standalone to verify a claimed approach is actually correct.
    allowed-tools: Read Bash
    ---
    Check whether the plan's claims about the real world are actually true right now. Cite sources; don't assert without one.
    - Read `PLAN.md` and the spec docs on disk. For each claim depending on external facts (a library's current API, a service's behavior), search the web (`curl` against `https://api.search.brave.com/res/v1/web/search`, header `X-Subscription-Token: $BRAVE_API_KEY`) to confirm or refute it. Append a `## Investigation` section to `PLAN.md`: each claim, its source(s), PASS/FLAG. Commit: `git add PLAN.md && git commit -m "investigate: confirm plan against sources"`.
    - No `PLAN.md`? Just answer the question asked, citing sources.
    Flag unconfirmed/contradicted claims rather than silently fixing them — that decision belongs to whoever reads the flag next.
    **Never download anything into the repo checkout.** Need to inspect a dependency's real source/docs? Fetch into `/tmp/`, not the repo tree — a later `git add -A` sweep commits whatever's sitting in the checkout, staged or not.
  judge-SKILL.md: |
    ---
    name: judge
    description: LLM-as-judge. Reviews a git diff against PLAN.md and the original spec, and returns a PASS/FAIL verdict with rationale. Use as the final review/report stage of a spec-to-push pipeline (the implementer stage already pushed; this reports on what shipped), or standalone to review any diff against stated criteria.
    allowed-tools: Read Bash
    ---
    Independent reviewer. Judge whether the implementation satisfies the plan and spec, on the evidence in front of you — not on how confident the commit messages sound. Don't rubber-stamp.
    - Run `git diff <base-branch>...HEAD` to see exactly what changed. Compare against `PLAN.md`'s steps and the spec docs. Does every step have a corresponding change? Does the diff contradict any investigator flag? Anything obviously broken on inspection?
    - **UI/frontend changes:** a diff that reads correctly can still render broken. Start the app and use `npx playwright` via Bash to actually look — screenshot the affected view, click through the golden path the plan/spec describes. FAIL on a visual defect the diff alone wouldn't show (broken layout, a control that doesn't do what its code claims, a state the plan promised that never renders). Doesn't apply to non-UI work — use judgment.
    - FAIL on DRY/SOLID violations (duplicated logic that should reuse existing code, mixed-responsibility units) and on anything committed that doesn't belong in source control (build artifacts, vendored dependencies, secrets, scratch files) — name the specific file/lines in your rationale.
    - No `PLAN.md`/base given? Review whatever diff/criteria are in the task directly.
    You MUST end your final message with a literal verdict line, exactly one of:
    ```
    VERDICT: PASS
    ```
    ```
    VERDICT: FAIL
    ```
    followed by your rationale. The pipeline driver parses this exact line mechanically to record the outcome — omitting it or rephrasing it breaks the pipeline.
  planner-SKILL.md: |
    ---
    name: planner
    description: Clones a target repo/branch, reads its markdown specs, and writes a verifiable step-by-step implementation plan (PLAN.md). Use as the first stage of a spec-to-PR pipeline, or standalone when asked to plan out a task before implementing it.
    allowed-tools: Read Grep Find Ls Write Bash
    ---
    Plan a concrete, verifiable implementation. Don't implement — that's `implementer`'s job, after `investigator` confirms.
    - If cwd is empty: `git clone` the given repo/branch first (only at pipeline start).
    - Read every markdown spec file for the task. Decompose into a numbered list of concrete steps, each naming the files/areas it touches and how to verify it's done. State assumptions explicitly; if ambiguous, pick the literal reading and note the ambiguity.
    - Write the plan to exactly `./PLAN.md` in the repo root — not `tasks/PLAN.md`, not `plans/<name>.md`, not any other name or location. Every later stage looks for the plan at that exact path. Commit: `git add PLAN.md && git commit -m "plan: <summary>"`.
    Don't touch any other file.
  resolver-SKILL.md: |
    ---
    name: resolver
    description: Diagnoses why a pipeline stage crashed (nonzero exit, not a semantic pass/fail) and decides whether it's safe to retry. Invoked by the pipeline driver when a planner/investigator/implementer/judge stage process fails to run to completion.
    allowed-tools: Read Bash
    ---
    **Persona:** You are an incident triager, not a fixer. A pipeline stage stopped running — your job is to look at what's on disk and what the failed stage's own output said, figure out why, and decide whether re-running that stage is likely to succeed or would just fail the same way again.
    **Thinking mode:** Medium — this is triage (root cause + retry/no-retry judgment), not deep design work.
    **Modes:**
    - **Triage mode** (default) — read the failed stage's name and its last stdout/stderr tail (given in the task). Check the working directory's current state (`git status`, `git log -1`) to see what, if anything, that stage managed to do before stopping. Distinguish transient causes (network blip, a flaky command, an interrupted git operation left in a bad-but-fixable state) from structural ones (the plan itself is broken, a required tool/credential is missing, the repo is in a state no retry will fix).
    You MUST end your final message with a literal resolution line, exactly one of:
    ```
    RESOLUTION: RETRY
    ```
    ```
    RESOLUTION: ABORT
    ```
    followed by your rationale. The pipeline driver parses this exact line mechanically and retries the failed stage **at most once** regardless of what you recommend a second time — don't assume unlimited retries. If the working directory is left in a broken state that a retry needs cleaned up first (e.g. a half-finished `git` operation), say so and do that cleanup yourself (via `bash`) before recommending `RETRY`.
 kind: ConfigMap
 metadata:
  name: pi-skills
  namespace: agent-pod
@@ -0,0 +1,12 @@
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: agent-pod-ssh-config
  namespace: agent-pod
 data:
  config: |
    Host git.riotpiao.com
        IdentityFile /root/.ssh/id_forgejo
        Port 2222
        User git
        StrictHostKeyChecking accept-new
@@ -0,0 +1,46 @@
 # Edge route for the API gateway.
 #
 # Lives here rather than in the central k8s/bootstrap/ingress/ingress.yaml
 # because that Application syncs in wave 1, before namespace `api` exists.
 #
 # nginx terminates TLS with the wildcard *.riotpiao.com cert (served as its
 # default-ssl-certificate, so no per-rule `tls:` block is needed) and forwards
 # plain HTTP to the gateway.
 #
 # Backend was kong-proxy:80 until Kong was retired on 2026-08-19; it is now the
 # Go gateway's Service, api-gateway:8080, deployed from rock/homelab-frontend.
 # Reverting the cutover is a change to these two lines and nothing else.
 #
 # Catch-all `/` on purpose: everything under this host belongs to the gateway.
 # Listing per-API paths here would duplicate the gateway's routing table inside
 # nginx, and the two copies would drift.
 #
 # In-cluster callers should prefer http://api-gateway.api.svc.cluster.local:8080
 # directly. Resolving api.riotpiao.com sends them out to nginx and back in,
 # which is a pointless hairpin unless they need TLS or the public hostname.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: api
  namespace: api
  annotations:
    # An API gateway carries streaming responses (SSE, gRPC-web, LLM token
    # streams). nginx's 60s default read timeout and its response buffering
    # would truncate or stall those.
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
 spec:
  ingressClassName: nginx
  rules:
    - host: api.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-gateway
                port:
                  number: 8080
@@ -0,0 +1,14 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 # Explicit allowlist. Anything added to this directory and not listed here is
 # silently dropped — no error, no drift shown.
 #
 # Down to a single Ingress since Kong was retired (2026-08-19). The Kong Helm
 # values, the KongClusterPlugin for Prometheus, the six KongPlugin CRs behind
 # the path-per-model LLM surface, the KongConsumer and the key-auth plan all
 # went with it.
 resources:
  - ingress.yaml
 # No top-level `namespace:` transformer on purpose: ingress.yaml sets its own
 # namespace, and the transformer rewrites metadata.namespace on every resource
 # it builds, which is a trap for anything cross-namespace added later.
@@ -1,17 +0,0 @@
 cloudflared:
    tunnelToken: ENC[AES256_GCM,data:GrZNC75M1T304m+MBbgcL9Wa6VD3Cc+joERjUWD7PsI5NlqFpVuorzub4OTaiE/F0sqDXtXdSwQP9jGTkgUnNbviUVEIBDb+zuiyzCfzuhc53oMvoUYRkRNeH5DR9TBEszspj8+mjQMDAKQFKtTmyDodgh8DdPg8LE8YTCqyx6CcpkGZ8yWY06VoKpZfFSOan/gHwRwG2500P8U+rzI676EKVUhWjBFP0iTbwDQR7spI7oz8Gon/0Q==,iv:O3i0v+M5L3i9O7SbBtDAJe5IsQDgw+alI0Y9arZNojs=,tag:/T63FivzvbLFaXEaBsn5FA==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpU2tjOGpJNk9leE9LSnlx
            b2tVbFFMMkZ3OHZuU0VhcUp5Tjl4ZDh2R1U4CjFFMURDWFJtN1lETUNpSmEwbU9u
            aEIxUW5qWTJGTnlobjV0emlpY00rM28KLS0tIFd0cllZeXFodFhJNTJMNkRLNyt1
            cXVyNnM4Y2pXeUFTYzU1OXlaOXR2RWcKp7/M/YFfpJg4L6a0AcYTV3Rza+bzaOeD
            OUIwyns8ZsPtU8ILbRYUUdD2EJFiOPnWP4yX70Ak10v12gfB7vRJ6A==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1smu533f803gmd0jq60s2zaj9zlznajy0ca6rtewd4r37mr2hs3uqsrldfh
    lastmodified: "2026-07-20T17:49:35Z"
    mac: ENC[AES256_GCM,data:MhuuuInJjGbzoftxVqMZUov1tJpOt5Vb8GwTr0R12hRCVHwdV+cuC1YeXTQidYaduO7neGnYe0p8ESLEpyY06j4nSpCvGRPPaxTeSeA8IcIk71xNtA1X0FVPv51s59rpvVSjiMDCqrcOv1aJybfQnBZoc/9bfmCYGqhDkD6sg1U=,iv:ThTjUGYE9GiyIgXS+0KDLYS20RPJjMG9DKyMSoKGk/4=,tag:KqZYps5qSuKPUI4U3Iabmw==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -0,0 +1,80 @@
 # ComfyUI — GPU-accelerated image generation on worker-1.
 # Uses 1x V100 32GB (sm70). Freed by scaling ornith 2→1.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: comfyui
  namespace: comfyui
  labels:
    app: comfyui
 spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: comfyui
  template:
    metadata:
      labels:
        app: comfyui
    spec:
      nodeSelector:
        kubernetes.io/hostname: worker-1
      runtimeClassName: nvidia
      containers:
        - name: comfyui
          image: ghcr.io/ai-dock/comfyui:v2-cuda-12.1.1-base-22.04
          ports:
            - containerPort: 8188
              protocol: TCP
          env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: "all"
            - name: COMFYUI_FLAGS
              value: "--listen 0.0.0.0 --port 8188"
          resources:
            requests:
              cpu: "4"
              memory: 8Gi
              nvidia.com/gpu: "1"
            limits:
              cpu: "8"
              memory: 16Gi
              nvidia.com/gpu: "1"
          volumeMounts:
            - mountPath: /workspace/ComfyUI/models
              name: models
            - mountPath: /workspace/ComfyUI/output
              name: output
          readinessProbe:
            httpGet:
              path: /
              port: 8188
            periodSeconds: 10
            initialDelaySeconds: 30
          startupProbe:
            httpGet:
              path: /
              port: 8188
            failureThreshold: 60
            periodSeconds: 10
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: comfyui-models
        - name: output
          emptyDir: {}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: comfyui-models
  namespace: comfyui
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 50Gi
@@ -0,0 +1,25 @@
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: comfyui
  namespace: comfyui
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    # WebSocket support for ComfyUI's live preview
    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
    nginx.ingress.kubernetes.io/proxy-set-headers: "Upgrade"
 spec:
  ingressClassName: nginx
  rules:
    - host: comfy.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: comfyui
                port:
                  number: 80
@@ -0,0 +1,7 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
  - deployment.yaml
  - service.yaml
  - ingress.yaml
@@ -0,0 +1,14 @@
 apiVersion: v1
 kind: Service
 metadata:
  name: comfyui
  namespace: comfyui
  labels:
    app: comfyui
 spec:
  selector:
    app: comfyui
  ports:
    - port: 80
      targetPort: 8188
      protocol: TCP
@@ -1,40 +0,0 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: duckdns-updater
  namespace: kube-system
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: duckdns-updater
  template:
    metadata:
      labels:
        app: duckdns-updater
    spec:
      containers:
        - name: updater
          image: curlimages/curl:latest
          command:
            - sh
            - -c
            - |
              while true; do
                curl -fsS "https://www.duckdns.org/update?domains=riotpiao&token=${DUCKDNS_TOKEN}&ip="
                sleep 300
              done
          env:
            - name: DUCKDNS_TOKEN
              valueFrom:
                secretKeyRef:
                  name: duckdns-token
                  key: token
          resources:
            requests:
              cpu: 5m
              memory: 16Mi
            limits:
              cpu: 50m
              memory: 32Mi
      restartPolicy: Always
@@ -0,0 +1,107 @@
 # Gotify — Push Notifications + Email Relay
 Self-hosted notification server with SMTP email forwarding sidecar.
 ## Architecture
 ```
 Forgejo webhook ──POST──→ Gotify API (:80/message)
                              │
                    ┌─────────┼─────────┐
                    ▼                     ▼
              Push notification      SMTP emailer sidecar
              (mobile/desktop)       (polls → sends email)
 ```
 ## Setup (one-time, after first deploy)
 ### 1. Encrypt secrets before committing
 ```bash
 # Edit secrets.yaml with real values first, then:
 sops -e -i k8s/apps/gotify/secrets.yaml
 ```
 ### 2. Create Gotify app + client tokens
 1. Login to `https://gotify.riotpiao.com` with admin creds
 2. **Applications** → Create `forgejo` → copy **app token**
 3. **Clients** → Create `smtp-emailer` → copy **client token**
 4. Update `gotify-tokens` secret:
   ```bash
   kubectl -n notifications create secret generic gotify-tokens \
     --from-literal=app-token=<APP_TOKEN> \
     --from-literal=client-token=<CLIENT_TOKEN> \
     --dry-run=client -o yaml | kubectl apply -f -
   ```
 ### 3. Configure Forgejo webhook
 In each Forgejo repo → **Settings** → **Webhooks** → **Add Webhook** → **Gotify**:
 | Field | Value |
 |-------|-------|
 | Target URL | `http://gotify.notifications.svc.cluster.local/message` |
 | Token | The **app token** from step 2 |
 | Events | Pull Request (Created, Merged, Closed) |
 Or via API:
 ```bash
 FORGEJO_TOKEN="<your-pat>"
 APP_TOKEN="<gotify-app-token>"
 curl -s -X POST "https://forgejo.riotpiao.com/api/v1/repos/rock/homelab/hooks" \
  -H "Authorization: token $FORGEJO_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "gotify",
    "active": true,
    "config": {
      "content_type": "json",
      "url": "http://gotify.notifications.svc.cluster.local/message?token='"$APP_TOKEN"'"
    },
    "events": ["pull_request", "pull_request_assign", "pull_request_review"],
    "authorization_header": ""
  }'
 ```
 ### 4. Add CoreDNS rewrite (if accessing via public hostname)
 Only needed if Cloudflare Tunnel is used for gotify.riotpiao.com:
 ```
 # terraform/files/coredns/Corefile — add rewrite:
 rewrite name gotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
 ```
 Then: `cd terraform && terraform apply && cd .. && make apply-cp`
 ### 5. SMTP providers
 | Provider | Host | Port | Notes |
 |----------|------|------|-------|
 | Gmail | smtp.gmail.com | 587 | Use App Password (2FA required) |
 | Resend | smtp.resend.com | 587 | Free 100 emails/day |
 | Sendgrid | smtp.sendgrid.net | 587 | Free 100 emails/day |
 | Mailgun | smtp.mailgun.org | 587 | Free 5000/month |
 ## Notification priority levels
 | Priority | Meaning | Email forwarded? |
 |----------|---------|-----------------|
 | 0-4 | Low (info) | No (below MIN_PRIORITY=5) |
 | 5-7 | Normal (PR created) | Yes |
 | 8-10 | High (PR merged, failures) | Yes |
 ## Verify
 ```bash
 # Test push notification
 APP_TOKEN="<app-token>"
 curl -X POST "https://gotify.riotpiao.com/message?token=$APP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title":"Test","message":"Hello from homelab","priority":5}'
 # Check email sidecar logs
 kubectl -n notifications logs deployment/gotify -c smtp-emailer --tail=20
 ```
@@ -0,0 +1,35 @@
 # CNPG Postgres for Gotify. Lightweight — 2 instances, 2Gi storage.
 # CNPG generates secret `gotify-db-app` + service `gotify-db-rw` in ns notifications.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
  name: gotify-db
  namespace: notifications
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
 spec:
  instances: 2
  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
  bootstrap:
    initdb:
      database: gotify
      owner: app
      encoding: UTF8
      localeCollate: C
      localeCType: C
  enableSuperuserAccess: false
  resources:
    requests: { memory: "256Mi", cpu: "100m" }
    limits: { memory: "512Mi", cpu: "500m" }
  storage:
    size: 2Gi
    storageClass: longhorn-cnpg
  monitoring:
    enablePodMonitor: true
  affinity:
    podAntiAffinityType: preferred
    topologyKey: kubernetes.io/hostname
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
@@ -0,0 +1,204 @@
 # Gotify — self-hosted push notification server + SMTP email relay.
 # Forgejo webhooks → Gotify → push notifications + email forwarding.
 # Runs on control plane (no GPU needed), lightweight.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: gotify
  namespace: notifications
  labels:
    app: gotify
 spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: gotify
  template:
    metadata:
      labels:
        app: gotify
    spec:
      containers:
        # --- Gotify server ---
        - name: gotify
          image: ghcr.io/gotify/server:2.6.1
          command: ["/bin/sh", "-c"]
          args:
            - |
              export GOTIFY_DATABASE_DIALECT=postgres
              export GOTIFY_DATABASE_CONNECTION="host=gotify-db-rw.notifications port=5432 user=${DB_USER} password=${DB_PASS} dbname=gotify sslmode=disable"
              exec /app/gotify-app
          ports:
            - containerPort: 80
              protocol: TCP
          env:
            - name: GOTIFY_DEFAULTUSER_NAME
              valueFrom:
                secretKeyRef:
                  name: gotify-admin
                  key: username
            - name: GOTIFY_DEFAULTUSER_PASS
              valueFrom:
                secretKeyRef:
                  name: gotify-admin
                  key: password
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: gotify-db-app
                  key: username
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: gotify-db-app
                  key: password
            - name: GOTIFY_SERVER_PORT
              value: "80"
            - name: GOTIFY_SERVER_KEEPALIVEPERIODSECONDS
              value: "0"
            - name: TZ
              value: Asia/Tokyo
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi
          livenessProbe:
            httpGet:
              path: /health
              port: 80
            periodSeconds: 30
            initialDelaySeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 80
            periodSeconds: 10
            initialDelaySeconds: 5
        # --- SMTP emailer sidecar ---
        # Watches Gotify WebSocket stream, forwards messages as email.
        # https://github.com/eternal-flame-AD/gotify-broadcast
        - name: smtp-emailer
          image: ghcr.io/gotify/server:2.6.1
          command:
            - /bin/sh
            - -c
            - |
              # Wait for Gotify to be ready
              until wget -qO- http://localhost:80/health >/dev/null 2>&1; do
                echo "Waiting for Gotify..."
                sleep 2
              done
              echo "Gotify is ready, starting email relay..."
              # Poll Gotify messages and forward via SMTP using msmtp
              # Install msmtp for lightweight SMTP sending
              apk add --no-cache msmtp curl jq
              # Configure msmtp
              cat > /tmp/msmtprc <<MSMTP
              defaults
              auth           on
              tls            on
              tls_trust_file /etc/ssl/certs/ca-certificates.crt
              logfile        /tmp/msmtp.log
              account        default
              host           ${SMTP_HOST}
              port           ${SMTP_PORT}
              from           ${SMTP_FROM}
              user           ${SMTP_USER}
              password       ${SMTP_PASS}
              MSMTP
              chmod 600 /tmp/msmtprc
              # Track last seen message ID
              LAST_ID=0
              while true; do
                # Fetch messages since last ID
                MESSAGES=$(curl -s -H "X-Gotify-Key: ${GOTIFY_CLIENT_TOKEN}" \
                  "http://localhost:80/message?since=${LAST_ID}&limit=10" 2>/dev/null)
                if [ -n "$MESSAGES" ]; then
                  echo "$MESSAGES" | jq -r '.messages[]? | @base64' | while read -r MSG; do
                    DECODED=$(echo "$MSG" | base64 -d)
                    ID=$(echo "$DECODED" | jq -r '.id')
                    TITLE=$(echo "$DECODED" | jq -r '.title // "Notification"')
                    BODY=$(echo "$DECODED" | jq -r '.message // ""')
                    PRIORITY=$(echo "$DECODED" | jq -r '.priority // 5')
                    APP=$(echo "$DECODED" | jq -r '.appid // 0')
                    DATE=$(echo "$DECODED" | jq -r '.date // ""')
                    # Only forward messages with priority >= configured threshold
                    if [ "$PRIORITY" -ge "${MIN_PRIORITY:-0}" ]; then
                      printf "Subject: [Gotify] %s\nFrom: %s\nTo: %s\nContent-Type: text/plain; charset=UTF-8\n\n%s\n\n---\nPriority: %s\nDate: %s" \
                        "$TITLE" "$SMTP_FROM" "$NOTIFY_EMAIL" "$BODY" "$PRIORITY" "$DATE" | \
                        msmtp -C /tmp/msmtprc "$NOTIFY_EMAIL" && \
                        echo "Email sent for message $ID: $TITLE" || \
                        echo "Failed to send email for message $ID"
                    fi
                    # Update last seen ID
                    if [ "$ID" -gt "$LAST_ID" ]; then
                      LAST_ID=$ID
                    fi
                  done
                fi
                sleep ${POLL_INTERVAL:-30}
              done
          env:
            - name: GOTIFY_CLIENT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: gotify-tokens
                  key: client-token
            - name: SMTP_HOST
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: host
            - name: SMTP_PORT
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: port
            - name: SMTP_FROM
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: from
            - name: SMTP_USER
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: user
            - name: SMTP_PASS
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: password
            - name: NOTIFY_EMAIL
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: notify-email
            - name: MIN_PRIORITY
              value: "5"
            - name: POLL_INTERVAL
              value: "15"
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
            limits:
              cpu: 100m
              memory: 64Mi
      # No volumes — Postgres handles persistence
@@ -0,0 +1,26 @@
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: gotify
  namespace: notifications
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    # WebSocket support for Gotify client connections
    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
 spec:
  ingressClassName: nginx
  rules:
    - host: gotify.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gotify
                port:
                  number: 80
@@ -0,0 +1,9 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
  - namespace.yaml
  - db.yaml
  - secrets.yaml
  - deployment.yaml
  - service.yaml
  - ingress.yaml
@@ -0,0 +1,6 @@
 apiVersion: v1
 kind: Namespace
 metadata:
  name: notifications
  labels:
    kubernetes.io/metadata.name: notifications
@@ -0,0 +1,92 @@
 #ENC[AES256_GCM,data:zdor03HqWha5KnsrNNfM+NT96bZafKhVJ1vfARlqbXzd56QTMq6tScippCbJWCtqLhK5,iv:shItz1uc3RYNBo9Xmq4M+0/Yjw6PbpF4x1tfZZQOSUM=,tag:8Bcwz5XkgLtjuMrDxFXOZQ==,type:comment]
 #ENC[AES256_GCM,data:QuQeqD21UgcVOUjlJTzQOAXioYWmnc4b38UTPAClL3WIf7/MqDZ2SQ==,iv:PHw7QplRXB4XBQh3fEolZpdvMMxLucaZx/aY8K5LE20=,tag:Vp3BUb2JPOBffwZW0jPfOg==,type:comment]
 #
 #ENC[AES256_GCM,data:3JOwvqkN0IsF4ogdR6SK9xZKCs7I,iv:uB3qIR9r2AZ7pewPzV+5anulFHnnmUmlBhfhUdM8A2U=,tag:w4g3k9bBcqO6o0pGfCRjuA==,type:comment]
 #ENC[AES256_GCM,data:mMQ0VWb11cOPA5QgdUqNGgeeLyzbFRKU7YOhFHmUx7Zb3NmePRXZ2gT6Y9vuieiMxCL/GeNXeg==,iv:URCI0VXAlcT5QI01L4yBB+P3+VT7lEfbmQiMQhVTUYk=,tag:CiyVNUzFZBmydrK9I861Og==,type:comment]
 #ENC[AES256_GCM,data:AX/p8bnjaddZFwR3hkvZNsm8op3nx6b8jnWEj6mNvPryq/MGpoS9SkNvlyACQ5KYMIYG6QtltG9buv4n5CVJzyYX,iv:YKdhnl+YtvvOFjpNXp3Ap3mnDPaDa0YoWupviRwPjwE=,tag:etGle/aGv7RemlmUeKEkCg==,type:comment]
 #ENC[AES256_GCM,data:Rs33VKbpeUCDrUnblHfBXiVjZa7tlxFV+QWiMYMv/uLzmXBS7NCpU7fIgNKG,iv:ZrZug41B6AysyVylQ6T6Z+Txc2nBUvblX8pRP2r12iQ=,tag:SvrNa0N0mprMq/Wqf77tZg==,type:comment]
 #ENC[AES256_GCM,data:sBzMx0RT9FAdtE4pEyWUkNX/nKDgHoEHk4IITSKpAl0/3i2uWgWs9NO4Sp1USAQaTtIa+Os=,iv:Zqj2GAtfw3UsIDgrTZ8erug+KgjP2ckNcas7NaZei+E=,tag:6xRTvzspIVLVhQJMQinPOQ==,type:comment]
 #ENC[AES256_GCM,data:bD+GzYKonJecM6Pym1powOYISWJDPIEVT8V1Hu8AM1U/wCdmq70TNq6DRu1FDSG+,iv:a/p1nM+rkie+mZ7E9A1Zg3qQloGTdtRWSIUJPiFNk24=,tag:ymyVypKvIUWmcZeHc5BQFQ==,type:comment]
 apiVersion: ENC[AES256_GCM,data:4vs=,iv:JAbEScwusLVWdsz4AuuYsBQDuusynhsbstrEyihrZ/k=,tag:b1IY2bxT6TL7djcFCiyMlg==,type:str]
 kind: ENC[AES256_GCM,data:1+fCn5DR,iv:HvI/Id59qQC//Uh3Q0P3oJo8ZSWZnhTXv3LTox3BuEU=,tag:i9aPbsqch9YKekzbf5G4cg==,type:str]
 metadata:
    name: ENC[AES256_GCM,data:4528BwgvzAtwiHyK,iv:/vz1jcRbLJmAoqDvgTmDbxvE5gYxpInSEGBP7IUFvlI=,tag:yZPEJNrHht9IVrlzsAUqFA==,type:str]
    namespace: ENC[AES256_GCM,data:FZMvrE3BV3iD3As59A==,iv:PfOFjLpZspP6JeTdNwa1IwjRl4JE1XKeAOab+8S8imk=,tag:yg3bWjkPngilwirv1CscDw==,type:str]
 type: ENC[AES256_GCM,data:QnoC8qce,iv:8paBbJZtmEbKSKlkwhy+3lLiRLoy1I+lB0xMBPdcNMg=,tag:p4/aUaSZHBzeU1CRqJ5rLw==,type:str]
 stringData:
    username: ENC[AES256_GCM,data:ANiWzQM=,iv:I3LrmBV5HEtjO+aysXuZEOmbvhyOfuOJCqZHx1ykWyU=,tag:zEV4gu2125L0bHGUkgGdNQ==,type:str]
    password: ENC[AES256_GCM,data:E5OW/ydSmskcnR8UYZsi9SZM5kpbiAE=,iv:NTgtzAK8P0jeczb8Egrrkmlf1Ejulabxh2/rLWzwNTk=,tag:kF2luernkCvnNl8o3EEvfA==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3b3VUNi9oRFB3ZjFyeVQr
            a2dTR3AvWnFDNi9YZ0QvaVNLU3dUSnlEMVdZCklhZjAzNURsZDBNWEI1MzZ1LzFl
            cUNhUWJUcVhNSjZqSG43MWdudStaRUkKLS0tIEpuc2kwR2hpUzRiSXZyb1I0V3lE
            YWJPcHFZU1lBOFFrQ3ErdzF4RzRqYzgK7q0N+ZcF/plIHR7HeRTF2qRmE4Za+eO+
            mMCo01fxd3ybf8gXnC9lrYmIK4oCkS8cg/B8mfIIBqts7XDzHiCaGA==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    lastmodified: "2026-09-10T01:34:10Z"
    mac: ENC[AES256_GCM,data:SURFYfHkesBFpdzeoMSdASSduGefetZ9sPY+0KgeUP4lMRX70kH6vnZdt5UyXe4hMgRCDA38sF+UgITW1pxhngqp8tL8QK5GBj+wTX9/zGmbtYl0zPvjvvXLx1vXGvUtfcEyxHW90EfDWae3JxQHCNlUcEH61hNj1ccZX+ceufM=,iv:mCQp3110nD3tg1eaf6P4e9XiMPbO+bhchC0zZ5+vqX8=,tag:vWGl3iaEi3KEoW87JBl3/g==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
 ---
 apiVersion: ENC[AES256_GCM,data:too=,iv:DFFOApoqaZDGmJe7NcGBrvE69kp85YpLwJUsImKSxmk=,tag:sKWVbV1U3oI656Sb2+iWew==,type:str]
 kind: ENC[AES256_GCM,data:Lp1+y5tL,iv:pHbNnGCX6qsAIMSjkdHNyuHZct63jcDatnaek6Z5rjg=,tag:k/0Y7noIi/nAAVLpCv9GTw==,type:str]
 metadata:
    name: ENC[AES256_GCM,data:QHAhhTHIxeyBXCfE2g==,iv:Q/2HuRPlbEC5AMONhi6POcCDdCJv3vHWMvmm6ZXGrr0=,tag:B+3Jhut9I7NzfpGtF969dQ==,type:str]
    namespace: ENC[AES256_GCM,data:VnXAy9fR8ixXanheUw==,iv:Uvkcm7zqgLCWlzcx9fO2/ccytzM7MlD0Zl1X8WNaz7Q=,tag:JsWelai4T+aNg1UabZPMTg==,type:str]
 type: ENC[AES256_GCM,data:I3UeAwER,iv:mtIMz/kuzbHx0HMSPVaqzHMLvhCGzvkdNLRfr7xNKBM=,tag:QCgh1anTe7oN2j3kkUM2/g==,type:str]
 stringData:
    #ENC[AES256_GCM,data:1XpJNypHreS+gUaRgEZe+CaJykvp3JJgI6Q=,iv:g/loGxxl6hvVvODmqwtip6om92df0tWVVg/msZvDOrg=,tag:ATCy0nG4PpGNoPQDRkXRcA==,type:comment]
    #ENC[AES256_GCM,data:EmitK30ZlOZz7MruQ/ItDim2LjU3uUZlOQhfr2KO65t2lcAw3xcGZAdaQpQfyZ0RDUQucUgnl+g8ckhH6rZ7+/N9,iv:4655+CO98ak8cG/dZI0H2RCykLS29AWAZ4LjrH3Tg2M=,tag:eqhcnuY6rbyy98ABZrtPrw==,type:comment]
    #ENC[AES256_GCM,data:iBIgf2Ru/cwE8D1L7uTdCmcii3lJxny1JE84jntuhROkCK1bqZU2cyf8eC0rq1zONiWwPuTokqFv18v+pdmbzQ==,iv:cnOCtS1qGy/w7ZkG2zRjqpgSeLU0FgvuV4tuBJ2SQ5U=,tag:3uxrBDiH/zMpLvFxZVfn1Q==,type:comment]
    app-token: ENC[AES256_GCM,data:NNpIl+DOZ9hjP5TY0bmHWRfSC88GTy1O,iv:NDnz4i/rYbbUxpxUSHFyXc31iI4PQSasjEcHV3u9E70=,tag:8mBEOwhXxVPmoX85fAVKdg==,type:str]
    client-token: ENC[AES256_GCM,data:nMSuoalWILpFijzVcB7sfA+d4EDb7HzL,iv:cgqkBnBUDqYMBXpQVCMzIPvpVvMu3BKl1CDqtY04xrk=,tag:Eql+ZAqPdvzm2M19HOoOHA==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3b3VUNi9oRFB3ZjFyeVQr
            a2dTR3AvWnFDNi9YZ0QvaVNLU3dUSnlEMVdZCklhZjAzNURsZDBNWEI1MzZ1LzFl
            cUNhUWJUcVhNSjZqSG43MWdudStaRUkKLS0tIEpuc2kwR2hpUzRiSXZyb1I0V3lE
            YWJPcHFZU1lBOFFrQ3ErdzF4RzRqYzgK7q0N+ZcF/plIHR7HeRTF2qRmE4Za+eO+
            mMCo01fxd3ybf8gXnC9lrYmIK4oCkS8cg/B8mfIIBqts7XDzHiCaGA==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    lastmodified: "2026-09-10T01:34:10Z"
    mac: ENC[AES256_GCM,data:SURFYfHkesBFpdzeoMSdASSduGefetZ9sPY+0KgeUP4lMRX70kH6vnZdt5UyXe4hMgRCDA38sF+UgITW1pxhngqp8tL8QK5GBj+wTX9/zGmbtYl0zPvjvvXLx1vXGvUtfcEyxHW90EfDWae3JxQHCNlUcEH61hNj1ccZX+ceufM=,iv:mCQp3110nD3tg1eaf6P4e9XiMPbO+bhchC0zZ5+vqX8=,tag:vWGl3iaEi3KEoW87JBl3/g==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
 ---
 apiVersion: ENC[AES256_GCM,data:Ub4=,iv:t0/4neL1XNR9sc2a0rZ7c4fOnZjRw0ACOY2Pp2A61IY=,tag:X8W/L0wfQLw4JF5aQ1dteg==,type:str]
 kind: ENC[AES256_GCM,data:54r8Je+L,iv:feX1TVQNZ7VrYIFiYKvopB0hYEIl9iw0ly2A6u/j+1Y=,tag:gD7iCoOCxIUT5uj1bGfikQ==,type:str]
 metadata:
    name: ENC[AES256_GCM,data:ub9FC+uwioKPMXg=,iv:1YbPJQle7t60Onnh77R8ffc8M87DNU1cmBQbVA72B28=,tag:3DsCpXMHcTIYN3uBh6Ztug==,type:str]
    namespace: ENC[AES256_GCM,data:dLIZlOoIo9YF2VyHew==,iv:YxQFWoMhVMG3jUJqmAtAUwzRr7MkSRjiWlq92rnymIE=,tag:iw5AI7kKlp+wKcgg2GGMxw==,type:str]
 type: ENC[AES256_GCM,data:4+CN/8f0,iv:Gmnh4TP0oCAiJ2VTrBT/Q1PWkPZks6XnzWu3io32hvQ=,tag:m5SNbovIU7pjwGACc4Q50g==,type:str]
 stringData:
    #ENC[AES256_GCM,data:bGrL1ia7o4ADA4J7soyOLCMVq3Y7kUU0BHhdz9ID45/q,iv:fwMHTJ68cYgBrzeAQ/CNdpfH14bAv7Exu7H6/QUk1ok=,tag:Lwc+FnbCd48oErLp5qEmcw==,type:comment]
    #ENC[AES256_GCM,data:bQdj5OgHL60gffBB5DZPOL6+RFWxrrBxa8lPkXbYR7viOf0v0xb9iBqjS/o5kIOjjcdeJibgGhQ=,iv:DJQBppj4w3KqFOC1MTEYwoc1LT0Qz17fKb611hgH+AE=,tag:D+l7BE+FEgxZFTIDLpKsiA==,type:comment]
    host: ENC[AES256_GCM,data:kaPQN225/tgQLLmejLQ=,iv:Xn69DoAa7Zop6EjZwIRgO/zVPdxvARgKkoacIE4Fcwc=,tag:hlCapggLwTsXr4DeejAYkQ==,type:str]
    port: ENC[AES256_GCM,data:vkBH,iv:uCA8fhnMZDg/qiSQYRYmhuc7dnteoavcuJxfSCE8GU0=,tag:ujosmW6GjrEa30mBCEFXgw==,type:str]
    from: ENC[AES256_GCM,data:kiWkZ7zmIT6gOHCH6hHI8fiJcQ==,iv:tqpYbLohLYbQ6kotOC5vDw6cZEFvNK1UEiKIa+nFzuQ=,tag:POZAxMpiGItr1lRY1yhW6w==,type:str]
    user: ENC[AES256_GCM,data:Vdew4lqPsOD2,iv:1zJE/hEDbchie4crugcjCfvACklXoeSzqopXbPulD/E=,tag:QdJvWQRi9Tc7pgMBigKsBg==,type:str]
    password: ENC[AES256_GCM,data:JcjnnXFkAXmI,iv:VgpYDrTn6o3l7tAk0lGMB2rpu+kB1IuAFu2VZMyJkX4=,tag:QR+En9nv/30yyt09GTcyqg==,type:str]
    notify-email: ENC[AES256_GCM,data:zMvA9KXy6rK2HFRrjAmZd5c/ay8=,iv:BtCXUsHneVc1QzvPKRwBjmbYc39I3Xy/zFoMg4l22Ks=,tag:NTZKDoPaLARJoTIX9u6oGQ==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3b3VUNi9oRFB3ZjFyeVQr
            a2dTR3AvWnFDNi9YZ0QvaVNLU3dUSnlEMVdZCklhZjAzNURsZDBNWEI1MzZ1LzFl
            cUNhUWJUcVhNSjZqSG43MWdudStaRUkKLS0tIEpuc2kwR2hpUzRiSXZyb1I0V3lE
            YWJPcHFZU1lBOFFrQ3ErdzF4RzRqYzgK7q0N+ZcF/plIHR7HeRTF2qRmE4Za+eO+
            mMCo01fxd3ybf8gXnC9lrYmIK4oCkS8cg/B8mfIIBqts7XDzHiCaGA==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    lastmodified: "2026-09-10T01:34:10Z"
    mac: ENC[AES256_GCM,data:SURFYfHkesBFpdzeoMSdASSduGefetZ9sPY+0KgeUP4lMRX70kH6vnZdt5UyXe4hMgRCDA38sF+UgITW1pxhngqp8tL8QK5GBj+wTX9/zGmbtYl0zPvjvvXLx1vXGvUtfcEyxHW90EfDWae3JxQHCNlUcEH61hNj1ccZX+ceufM=,iv:mCQp3110nD3tg1eaf6P4e9XiMPbO+bhchC0zZ5+vqX8=,tag:vWGl3iaEi3KEoW87JBl3/g==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -0,0 +1,14 @@
 apiVersion: v1
 kind: Service
 metadata:
  name: gotify
  namespace: notifications
  labels:
    app: gotify
 spec:
  selector:
    app: gotify
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
@@ -1,95 +0,0 @@
 # PostSync hook to patch Homarr deployment probes
 # Chart v8.23.0 doesn't support probe customization via values
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: homarr-fix-probes
  namespace: dashboard
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
 spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: homarr-probe-patcher
      containers:
      - name: patch
        image: python:3.12-alpine
        command:
        - /bin/sh
        - -c
        - |
          set -e
          echo "Installing kubectl..."
          wget -q -O /tmp/kubectl https://dl.k8s.io/release/v1.28.0/bin/linux/amd64/kubectl
          chmod +x /tmp/kubectl
          echo "Patching Homarr deployment probes..."
          /tmp/kubectl -n dashboard patch deployment homarr --type=json -p='[
            {
              "op": "replace",
              "path": "/spec/template/spec/containers/0/livenessProbe/initialDelaySeconds",
              "value": 60
            },
            {
              "op": "replace",
              "path": "/spec/template/spec/containers/0/livenessProbe/periodSeconds",
              "value": 30
            },
            {
              "op": "replace",
              "path": "/spec/template/spec/containers/0/livenessProbe/timeoutSeconds",
              "value": 5
            },
            {
              "op": "replace",
              "path": "/spec/template/spec/containers/0/readinessProbe/initialDelaySeconds",
              "value": 45
            },
            {
              "op": "replace",
              "path": "/spec/template/spec/containers/0/readinessProbe/periodSeconds",
              "value": 15
            },
            {
              "op": "replace",
              "path": "/spec/template/spec/containers/0/readinessProbe/timeoutSeconds",
              "value": 5
            }
          ]'
          echo "✅ Probes patched successfully"
          echo "  Liveness: 60s initial, 30s period, 5s timeout"
          echo "  Readiness: 45s initial, 15s period, 5s timeout"
 ---
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: homarr-probe-patcher
  namespace: dashboard
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
  name: homarr-probe-patcher
  namespace: dashboard
 rules:
 - apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "patch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: homarr-probe-patcher
  namespace: dashboard
 roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: homarr-probe-patcher
 subjects:
 - kind: ServiceAccount
  name: homarr-probe-patcher
  namespace: dashboard
@@ -1,23 +0,0 @@
 apiVersion: ENC[AES256_GCM,data:XZc=,iv:0Lgui0+X2oTlgCX5HEIIihEsGJD0C2+pDIOTCNi4r+g=,tag:U2DPsVJ1gCoZw1YcgmT7UA==,type:str]
 kind: ENC[AES256_GCM,data:SRkjGwsv,iv:KRBfS7COsTs/l3Fhyk5FvH7IVc+r/6M6FVOo+kPDe1Q=,tag:113xdFELRoJiU4Kn8BmvPA==,type:str]
 metadata:
    name: ENC[AES256_GCM,data:stIxzQqOUvpbEANL4aQ=,iv:Q+DCLGtw3wihhdAgEbzYHyQhFTPWFyE2iJKl9mYaA2E=,tag:2XRGRbPQLNUGbA1dZI9xEQ==,type:str]
    namespace: ENC[AES256_GCM,data:wIhhcrQi3cDF,iv:gnCCkV7GKXA2HWOOeNUGnpAycf8U7pl6F8Vfo6DhLR4=,tag:xnqvCtUdFPB5D9meJb36GA==,type:str]
 type: ENC[AES256_GCM,data:1qUDnlHH,iv:gBaC+6MXvZY/bIpy/cEG/zWwLBaUFMVfxAHa1OnwfnQ=,tag:u9PcCfvA2BydFNQQKws1Og==,type:str]
 stringData:
    SECRET_ENCRYPTION_KEY: ENC[AES256_GCM,data:b0h8ekf7RIbQ9YTpaXPmR3aeNonk60lgltPmnRe7lVk9wlAdaXkNChINcAsarw9YhfYvfCZfS8U33/QxLGjaYA==,iv:uOmx8w3XPKnoxfasf7gqi/bljxigpLwaAC30c2TroKg=,tag:JphkeY49bQLm6cPjT6g8LA==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiUW1BVmEvTnJqa3RuQ3NG
            UFg3SE1SclpWd01TZnFOK0trQ1c2YThUQmpRCkdGc3FJSmRCcFIycWcvZXJCbDdj
            T1M2cFFUakpOa2FST1k2UHc0a1hCRWMKLS0tIDk0UVdYRUkvTEM4TEVWRElrY3dN
            SFU2bkZxUGlDL0JNSnY3UU54Z3BhbFUKnJYwwQkUveNjTF9CveVUBF8RO7QjDnYA
            1+COKzvXuCIupvPcS87AtkrOfbfTu3IK78pfxZn04FmMOuAyj0ynzA==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1smu533f803gmd0jq60s2zaj9zlznajy0ca6rtewd4r37mr2hs3uqsrldfh
    lastmodified: "2026-07-22T15:59:29Z"
    mac: ENC[AES256_GCM,data:uMmKD4fvu39KZc2lDoDD25g/p/eIGW/CYFIwYXcRFPdbLLLi+iW9ks/2l5tHqzlwHEtlL1RH4HLOQr25k3GAJA2FhVnyVmyZ4N2EnJfic/9XTfef9E8tKVo2cXEgYMCPbpH8sVu2OoaQ9BFw8dTM9ZxuTdGKksGOtGp56I4R8hA=,iv:QFhVAZ1jykTNLPDzqjYf9jkg5PuqYA8+s0Hn57ND3KA=,tag:Zz2qYANQP33izwg+3Ap16A==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -1,5 +1,22 @@
 # Homarr landing page with Authentik SSO
-# Probes patched via PostSync hook (chart doesn't support customization)
+
 # Probe tuning (chart DOES expose these — the old PostSync patch-job was
 # unnecessary and fragile: it only patched one Deployment revision, so any later
 # rollout reverted to the chart's aggressive defaults). Homarr's first-boot icon
 # updater blocks the event loop for ~50s ("icons updater took 49553ms"), during
 # which /api/health/live can't answer within the default 10s×3 window → kubelet
 # SIGTERMs the pod → CrashLoopBackOff (247 restarts, 503 at the ingress). Give
 # liveness a wide window so the icon import can finish without a kill.
 livenessProbe:
  initialDelaySeconds: 60
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 10
 readinessProbe:
  initialDelaySeconds: 30
  periodSeconds: 15
  timeoutSeconds: 5
  failureThreshold: 6
 image:
  repository: ghcr.io/homarr-labs/homarr
@@ -13,10 +30,19 @@ replicaCount: 1
 env:
  AUTH_PROVIDERS: "oidc,credentials"
  AUTH_OIDC_ISSUER: "https://authentik.riotpiao.com/application/o/homarr/"
  # AUTH_OIDC_URI (authorize endpoint) is REQUIRED in addition to ISSUER — homarr
  # hides the "Sign in with Authentik" button entirely when it's absent (per the
  # authentik Homarr integration + homarr SSO docs). This was the missing var.
  AUTH_OIDC_URI: "https://authentik.riotpiao.com/application/o/authorize/"
  AUTH_OIDC_CLIENT_NAME: "Authentik"
  AUTH_OIDC_GROUPS_ATTRIBUTE: "groups"
  AUTH_OIDC_SCOPE_OVERWRITE: "openid email profile groups"
  AUTH_OIDC_AUTO_LOGIN: "false"
  # Link the OIDC identity to an existing homarr account with the same email.
  OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING: "true"
  # The analytics cron blocked the (single-threaded) Next.js event loop for ~16s
  # per run ("callback took longer than expected"), compounding CPU pressure.
  DISABLE_ANALYTICS: "true"
  BASE_URL: "https://homarr.riotpiao.com"
  NEXTAUTH_URL: "https://homarr.riotpiao.com"
@@ -41,8 +67,12 @@ tolerations:
 resources:
  requests:
-    cpu: 100m
+    cpu: 250m
-    memory: 256Mi
+    memory: 384Mi
  limits:
-    cpu: 500m
+    # Next.js 16 + bundled redis + the icon-updater (28k icons) saturated the old
-    memory: 512Mi
+    # 500m limit; CPU throttling made Next.js abort with exit 134 (SIGABRT) and
    # self-restart in a loop, so nginx saw no upstream and returned 502. Give it
    # real CPU headroom.
    cpu: "2"
    memory: 1Gi
@@ -1,7 +1,7 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: dashboard
-resources:
+# Probes are now tuned via homarr-values.yaml (chart-native); the old
-  - fix-probes-job.yaml
+# fix-probes-job PostSync hook is removed. homarr-secrets/auth-oidc/db-encryption
-# PostSync hook to patch Homarr deployment probes
+# Secrets are delivered by the sops-secrets (ksops) Application.
-# (homarr-secrets.enc.yaml managed by sops-secrets Application)
+resources: []
@@ -0,0 +1,14 @@
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: immich-config
 data:
  DB_HOSTNAME: "immich-db-rw"
  DB_DATABASE_NAME: "immich"
  # Only pgvector is installed (see db.yaml) - no vectorchord extension image
  # exists for pg18 in CNPG's catalog yet. Explicit instead of relying on
  # auto-detect's vectorchord-first preference order.
  DB_VECTOR_EXTENSION: "pgvector"
  REDIS_HOSTNAME: "immich-redis"
  IMMICH_MACHINE_LEARNING_URL: "http://immich-machine-learning:3003"
  TZ: "America/Los_Angeles"
@@ -0,0 +1,58 @@
 # Dedicated CNPG Postgres for Immich. Same recipe as paperless-db/authentik-db
 # (2 instances, default longhorn storage class) except the operand is
 # PostgreSQL 18, not 16.2 - the official CNPG pgvector extension image
 # (ghcr.io/cloudnative-pg/pgvector) is only published for pg18, no pg16 tags
 # exist in that registry. Immich itself supports pg18 fine (immich-app's own
 # postgres image already ships 18-vectorchord builds).
 #
 # pgvector loaded via CNPG's ImageVolume extension mechanism (CNPG 1.27+,
 # k8s ImageVolume feature - both present here: operator is 1.30.0, cluster is
 # v1.36.1). No shared_preload_libraries needed - pgvector doesn't require
 # preload, just CREATE EXTENSION, which immich-server issues itself at
 # startup. Distro/pg-major must match between the operand image and the
 # extension image (both "18"+"trixie" here) - CNPG's own compatibility rule.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
  name: immich-db
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
 spec:
  instances: 2
  imageName: ghcr.io/cloudnative-pg/postgresql:18-minimal-trixie
  postgresql:
    extensions:
      - name: pgvector
        image:
          reference: ghcr.io/cloudnative-pg/pgvector:0.8.1-18-trixie
  bootstrap:
    initdb:
      database: immich
      owner: app
      encoding: UTF8
      localeCollate: C
      localeCType: C
      # CREATE EXTENSION vector requires superuser (pgvector's control file
      # isn't marked trusted) and the "app" owner role isn't one
      # (enableSuperuserAccess: false, repo convention) - postInitApplicationSQL
      # runs as superuser during initdb, before the app ever connects. Only
      # fires on a fresh bootstrap; the live cluster already had this run
      # manually once (kubectl exec ... psql -U postgres -c 'CREATE EXTENSION').
      postInitApplicationSQL:
        - "CREATE EXTENSION IF NOT EXISTS vector;"
        - "CREATE EXTENSION IF NOT EXISTS cube;"
        - "CREATE EXTENSION IF NOT EXISTS earthdistance;"
  enableSuperuserAccess: false
  resources:
    requests: { memory: "512Mi", cpu: "250m" }
    limits: { memory: "2Gi", cpu: "1" }
  storage:
    size: 20Gi
    storageClass: longhorn
  affinity:
    podAntiAffinityType: preferred
    topologyKey: kubernetes.io/hostname
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
@@ -0,0 +1,100 @@
 # immich-server: pinned to talos-cp-3, same reasoning as paperless
 # (deployment.yaml comment there) - immich-media is a ReadWriteOnce Longhorn
 # volume with a single replica physically on that node's disk (shared with
 # paperless-media on the same 4TB HDD). Recreate strategy for the same
 # reason: two pods can't both attach an RWO volume.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: immich-server
 spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: immich-server
  template:
    metadata:
      labels:
        app: immich-server
    spec:
      serviceAccountName: immich
      nodeSelector:
        kubernetes.io/hostname: talos-cp-3
      containers:
        - name: immich-server
          image: ghcr.io/immich-app/immich-server:release
          ports:
            - containerPort: 2283
          envFrom:
            - configMapRef:
                name: immich-config
          env:
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: immich-db-app
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: immich-db-app
                  key: password
            # Composed by k8s/infra/iam's provisioning script (system-config
            # JSON, oauth section) - see immich-oidc Secret.
            - name: IMMICH_CONFIG_FILE
              value: /config/immich.json
          resources:
            requests: { cpu: "500m", memory: "1Gi" }
            limits: { cpu: "2", memory: "4Gi" }
          volumeMounts:
            - name: media
              mountPath: /usr/src/app/upload
            - name: oidc-config
              mountPath: /config
              readOnly: true
      volumes:
        - name: media
          persistentVolumeClaim:
            claimName: immich-media
        - name: oidc-config
          secret:
            secretName: immich-oidc
            items:
              - key: config.json
                path: immich.json
 ---
 # CPU-only for now - the cluster's one GPU node (worker-1) is already
 # dedicated to llm-serving predictors. Not node-pinned: its cache PVC is on
 # the default 3-replica pool, not the single-disk cp-3 HDD.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: immich-machine-learning
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: immich-machine-learning
  template:
    metadata:
      labels:
        app: immich-machine-learning
    spec:
      serviceAccountName: immich
      containers:
        - name: immich-machine-learning
          image: ghcr.io/immich-app/immich-machine-learning:release
          ports:
            - containerPort: 3003
          resources:
            requests: { cpu: "500m", memory: "1Gi" }
            limits: { cpu: "2", memory: "4Gi" }
          volumeMounts:
            - name: ml-cache
              mountPath: /cache
      volumes:
        - name: ml-cache
          persistentVolumeClaim:
            claimName: immich-ml-cache
@@ -0,0 +1,24 @@
 # Direct nginx ingress, same reasoning as paperless: large uploads (photos/
 # videos) and long-lived operations (video transcode, big batch uploads) need
 # proxy-body-size/timeouts raised past nginx's defaults.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: immich
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
 spec:
  ingressClassName: nginx
  rules:
    - host: img.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: immich-server
                port:
                  number: 2283
@@ -0,0 +1,14 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: immich
 resources:
  - db.yaml
  - pvc.yaml
  - configmap.yaml
  - redis.yaml
  - deployment.yaml
  - service.yaml
  - ingress.yaml
  - rbac.yaml
 # immich-oidc Secret written by the PostSync provisioning Job in
 # k8s/infra/iam (same as paperless-oidc) - not duplicated here.
@@ -0,0 +1,38 @@
 # Two volumes:
 #
 # - media: original photos/videos + generated thumbnails/encoded videos.
 #   Shares the cp-3 USB HDD with paperless-media, same StorageClass/disk tag,
 #   single replica (single disk, no redundancy possible - same tradeoff
 #   paperless already accepts). Sized 1400Gi, not 2000Gi: the disk's real
 #   usable capacity (~3724GiB, formatting overhead) minus paperless-media's
 #   2000Gi and ~231GiB of other apps' default-class replicas that Longhorn
 #   placed here anyway (disk tags only pull matching volumes in, they don't
 #   exclude non-matching ones when the untagged pool elsewhere is full) only
 #   leaves ~1493Gi of real scheduling headroom right now.
 # - ml-cache: downloaded ML model weights for immich-machine-learning
 #   (face detection / CLIP embeddings). Small, disposable (re-downloads on
 #   loss), but persisted so a pod restart doesn't re-pull multi-GB models -
 #   default 3-replica pool, not node-pinned.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: immich-media
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn-paperless-media
  resources:
    requests:
      storage: 1400Gi
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: immich-ml-cache
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
@@ -0,0 +1,43 @@
 # Scoped operator access for immich-admins: restart/config-edit rights on
 # just this service's own resources, nothing CNPG-managed (immich-db-*) or
 # provisioning-managed (immich-oidc). Same pattern as
 # k8s/apps/paperless/rbac.yaml. Inert until kube-apiserver's OIDC wiring
 # lands (--oidc-groups-claim=groups, --oidc-groups-prefix=oidc:).
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: immich
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
  name: immich-operator
 rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    resourceNames: ["immich-server", "immich-machine-learning"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["immich-config"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["immich-oidc"]
    verbs: ["get", "list", "watch", "update", "patch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: immich-admins-binding
 subjects:
  - kind: Group
    name: "oidc:immich-admins"
    apiGroup: rbac.authorization.k8s.io
  - kind: ServiceAccount
    name: immich
    namespace: immich
 roleRef:
  kind: Role
  name: immich-operator
  apiGroup: rbac.authorization.k8s.io
@@ -0,0 +1,37 @@
 # Job queue broker for immich-server. No PVC: queue state is disposable - a
 # lost queue on restart just re-triggers the affected background jobs
 # (thumbnail generation, ML jobs, etc.), no photo data loss since originals
 # live on immich-media.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: immich-redis
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: immich-redis
  template:
    metadata:
      labels:
        app: immich-redis
    spec:
      containers:
        - name: redis
          image: redis:7-alpine
          ports:
            - containerPort: 6379
          resources:
            requests: { cpu: "50m", memory: "64Mi" }
            limits: { cpu: "250m", memory: "256Mi" }
 ---
 apiVersion: v1
 kind: Service
 metadata:
  name: immich-redis
 spec:
  selector:
    app: immich-redis
  ports:
    - port: 6379
      targetPort: 6379
@@ -0,0 +1,21 @@
 apiVersion: v1
 kind: Service
 metadata:
  name: immich-server
 spec:
  selector:
    app: immich-server
  ports:
    - port: 2283
      targetPort: 2283
 ---
 apiVersion: v1
 kind: Service
 metadata:
  name: immich-machine-learning
 spec:
  selector:
    app: immich-machine-learning
  ports:
    - port: 3003
      targetPort: 3003
@@ -0,0 +1,56 @@
 apiVersion: serving.kserve.io/v1beta1
 kind: InferenceService
 metadata:
  annotations:
    serving.kserve.io/deploymentMode: RawDeployment
  labels:
    app.kubernetes.io/name: llm-embeddings
    app.kubernetes.io/part-of: llm-serving
  name: embeddings
  namespace: llm-serving
 spec:
  predictor:
    containers:
    - args:
      - --model-id=nomic-ai/nomic-embed-text-v2-moe
      - --port=8080
      - --hostname=0.0.0.0
      - --auto-truncate
      env:
      - name: HUGGINGFACE_HUB_CACHE
        value: /mnt/models
      image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.2@sha256:4d632b76bd14cb57044a1ffb0ad48ab0ba4939e705a9a615ccc740658575c26e
      name: kserve-container
      ports:
      - containerPort: 8080
        protocol: TCP
      readinessProbe:
        httpGet:
          path: /health
          port: 8080
        periodSeconds: 10
      resources:
        limits:
          cpu: '16'
          memory: 8Gi
        requests:
          cpu: '8'
          memory: 4Gi
      startupProbe:
        failureThreshold: 60
        httpGet:
          path: /health
          port: 8080
        periodSeconds: 10
      volumeMounts:
      - mountPath: /mnt/models
        name: models
    maxReplicas: 1
    minReplicas: 1
    nodeSelector:
      kubernetes.io/hostname: worker-1
    volumes:
    - name: models
      persistentVolumeClaim:
        claimName: llm-models
@@ -0,0 +1,20 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 # Explicit allowlist, matching k8s/apps/api. Anything added to this directory
 # and not listed here is silently dropped — no error, no drift shown.
 #
 # These five were adopted from live state on 2026-08-15; they had been applied
 # by hand and carried no ArgoCD ownership. Each was exported and verified with
 # `kubectl diff -f <file>` returning empty before the Application below was
 # created, so the first sync was a no-op rather than a redeploy. Re-verify that
 # way after any edit here: a GPU predictor restart is a weights reload measured
 # in tens of seconds, not a rolling update.
 resources:
  - embeddings.yaml
  - ornith.yaml
  - reasoning.yaml
  - reranker.yaml
  - qwen-cpu.yaml
  - networkpolicy.yaml
 # No namespace transformer: every file sets its own, and the transformer would
 # rewrite metadata.namespace on anything cross-namespace added later.
@@ -0,0 +1,62 @@
 # NetworkPolicy for LLM inference engines (llm-serving namespace).
 #
 # These pods have NO auth — vLLM, Ollama, and TEI accept any request.
 # All access MUST go through the api-gateway, which validates JWTs and
 # injects identity headers (X-Forwarded-User, X-Auth-Verified).
 #
 # Replaces the hand-applied llm-serving-default-deny policy that used
 # `llm-client: "true"` pod label as a selector — any pod in any namespace
 # could self-grant access by adding that label, which defeats the purpose.
 #
 # This policy restricts ingress to:
 #   1. api namespace (gateway) — the sole entry point for inference
 #   2. monitoring namespace — Prometheus scraping vLLM/TEI /metrics
 #   3. intra-namespace — pod-to-pod (future: multi-replica comms)
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
  name: llm-serving-ingress
  namespace: llm-serving
  labels:
    app.kubernetes.io/part-of: llm-serving
 spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: llm-serving
  policyTypes:
    - Ingress
  ingress:
    # Allow from api-gateway (namespace: api)
    # Gateway proxies /v1/chat/completions, /v1/embeddings, /v1/rerank
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: api
      ports:
        - protocol: TCP
          port: 8080   # vLLM, Ollama HTTP
        - protocol: TCP
          port: 80      # KServe predictor services
        - protocol: TCP
          port: 8000    # vLLM direct (some configs)
        - protocol: TCP
          port: 11434   # Ollama native port
    # Allow Prometheus scraping from monitoring namespace
    # vLLM: :8080/metrics, TEI: :9000/metrics
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - protocol: TCP
          port: 8080
        - protocol: TCP
          port: 9000
    # Allow intra-namespace (pod-to-pod within llm-serving)
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/part-of: llm-serving
      ports:
        - protocol: TCP
          port: 8080
@@ -0,0 +1,99 @@
 apiVersion: serving.kserve.io/v1beta1
 kind: InferenceService
 metadata:
  annotations:
    serving.kserve.io/deploymentMode: RawDeployment
    # The konghq.com/{connect,read,write}-timeout annotations that used to live
    # here went with Kong (retired 2026-08-19). They existed because Kong read
    # its upstream timeouts off the Kubernetes Service, and its 60s default cut
    # off the first request after any pod restart — a restart flushes VRAM and
    # reloading ornith:35b takes longer than that. OLLAMA_KEEP_ALIVE=-1 hid the
    # problem in steady state.
    #
    # The equivalent budget now belongs to the Go gateway's per-route timeout
    # config in rock/homelab-frontend, not to an annotation on this object.
  labels:
    app.kubernetes.io/name: llm-ornith
    app.kubernetes.io/part-of: llm-serving
  name: ornith
  namespace: llm-serving
 spec:
  predictor:
    containers:
    - command:
      - /bin/sh
      - -c
      - 'set -e
        ollama serve &
        SERVE_PID=$!
        until ollama list >/dev/null 2>&1; do sleep 2; done
        ollama pull ornith:35b
        ollama run ornith:35b "ok" >/dev/null 2>&1 || true
        wait $SERVE_PID
        '
      env:
      - name: OLLAMA_HOST
        value: 0.0.0.0:8080
      - name: OLLAMA_MODELS
        value: /mnt/models/ollama
      - name: OLLAMA_CONTEXT_LENGTH
        value: '32768'
      - name: OLLAMA_KEEP_ALIVE
        value: '-1'
      - name: OLLAMA_NUM_PARALLEL
        value: '1'
      - name: OLLAMA_MAX_LOADED_MODELS
        value: '1'
      image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
      name: kserve-container
      ports:
      - containerPort: 8080
        protocol: TCP
      readinessProbe:
        exec:
          command:
          - /bin/sh
          - -c
          - ollama ps 2>/dev/null | grep -q ornith
        periodSeconds: 10
      resources:
        limits:
          cpu: '16'
          memory: 16Gi
          nvidia.com/gpu: '1'
        requests:
          cpu: '8'
          memory: 8Gi
          nvidia.com/gpu: '1'
      startupProbe:
        exec:
          command:
          - /bin/sh
          - -c
          - ollama ps 2>/dev/null | grep -q ornith
        failureThreshold: 120
        periodSeconds: 15
      volumeMounts:
      - mountPath: /mnt/models
        name: models
    deploymentStrategy:
      type: Recreate
    # 1 replica -- ornith:35b only. qwen2.5:3b moved to CPU on cp-2.
    # Frees 1 GPU for ComfyUI.
    maxReplicas: 1
    minReplicas: 1
    nodeSelector:
      kubernetes.io/hostname: worker-1
    runtimeClassName: nvidia
    volumes:
    - name: models
      persistentVolumeClaim:
        claimName: llm-models
@@ -0,0 +1,115 @@
 # qwen2.5:3b-instruct on CPU (talos-cp-2, 144GB RAM, 24 cores).
 # Moved off GPU to free a V100 for ComfyUI. Latency ~10x slower
 # than GPU but sufficient for lightweight tasks (summarization,
 # classification, quick answers).
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: qwen-cpu
  namespace: llm-serving
  labels:
    app: qwen-cpu
    app.kubernetes.io/name: qwen-cpu
    app.kubernetes.io/part-of: llm-serving
 spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: qwen-cpu
  template:
    metadata:
      labels:
        app: qwen-cpu
        app.kubernetes.io/name: qwen-cpu
        app.kubernetes.io/part-of: llm-serving
    spec:
      nodeSelector:
        kubernetes.io/hostname: talos-cp-2
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: ollama
          image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
          command: ["/bin/sh", "-c"]
          args:
            - |
              ollama serve &
              SERVE_PID=$!
              until ollama list >/dev/null 2>&1; do sleep 2; done
              ollama pull qwen2.5:3b-instruct
              ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
              wait $SERVE_PID
          env:
            - name: OLLAMA_HOST
              value: "0.0.0.0:8080"
            - name: OLLAMA_MODELS
              value: /root/.ollama/models
            - name: OLLAMA_CONTEXT_LENGTH
              value: "32768"
            - name: OLLAMA_KEEP_ALIVE
              value: "-1"
            - name: OLLAMA_MAX_LOADED_MODELS
              value: "1"
            - name: OLLAMA_NUM_PARALLEL
              value: "2"
          ports:
            - containerPort: 8080
              protocol: TCP
          readinessProbe:
            exec:
              command: ["/bin/sh", "-c", "ollama ps 2>/dev/null | grep -q qwen2.5"]
            periodSeconds: 10
          startupProbe:
            exec:
              command: ["/bin/sh", "-c", "ollama ps 2>/dev/null | grep -q qwen2.5"]
            failureThreshold: 60
            periodSeconds: 10
          resources:
            requests:
              cpu: "4"
              memory: 4Gi
            limits:
              cpu: "8"
              memory: 8Gi
          volumeMounts:
            - mountPath: /root/.ollama
              name: ollama-data
      volumes:
        - name: ollama-data
          persistentVolumeClaim:
            claimName: qwen-cpu-data
 ---
 apiVersion: v1
 kind: Service
 metadata:
  name: qwen-cpu
  namespace: llm-serving
  labels:
    app: qwen-cpu
    app.kubernetes.io/part-of: llm-serving
 spec:
  selector:
    app: qwen-cpu
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
 ---
 # Small PVC for qwen2.5:3b model weights (~1.9GB).
 # Separate from llm-models PVC which is pinned to worker-1.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: qwen-cpu-data
  namespace: llm-serving
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
@@ -0,0 +1,122 @@
 apiVersion: serving.kserve.io/v1beta1
 kind: InferenceService
 metadata:
  annotations:
    serving.kserve.io/deploymentMode: RawDeployment
  labels:
    app.kubernetes.io/name: llm-reasoning
    app.kubernetes.io/part-of: llm-serving
  name: reasoning
  namespace: llm-serving
 spec:
  predictor:
    containers:
    - args:
      # bnb-4bit retired: no int4 tensor cores on sm70/V100, dequant-then-
      # matmul is two slow kernel launches instead of one fused int4 GEMM,
      # decode crawled at 2.5-10 tok/s regardless of TP/PP. Switched to
      # JunHowie/Qwen3-32B-GPTQ-Int4 -- same dense Qwen3-32B weights, same
      # hermes/qwen3 parser stack (no narration-bug risk, same as before),
      # only the quant format changes. Plain (non-Marlin) GPTQ kernel is
      # confirmed Volta-compatible; Marlin needs sm80+ and vLLM would try
      # to auto-upgrade to it, so --quantization is pinned explicitly to
      # `gptq` to force the plain kernel. Verified checkpoint size: 19.34GB
      # (summed from the real safetensors index, not bits-per-param math).
      # max-model-len=131072 is Qwen3-32B's real ceiling (config.json YaRN:
      # factor=4.0, original_max_position_embeddings=32768) -- 200k was
      # asked for but exceeds this architecturally regardless of VRAM.
      # KV cache math: 256KB/token total (64 layers, 8 KV heads, 128
      # head_dim, fp16), PP=2 splits both weights and KV load ~evenly, so
      # each GPU carries ~9.67GB weights + ~128KB/token KV. At
      # gpu-memory-utilization=0.90 (28.8GB/GPU usable), that leaves
      # ~19.1GB/GPU for KV cache -> ~156k tokens/GPU capacity, comfortably
      # above the 131072 target with room to spare -- the old
      # OffloadingConnector CPU-DRAM spillover (tuned for the previous
      # model's much smaller 16384 context) is no longer needed and is
      # dropped. Staying on PP=2 and vLLM 0.11.0 (no version bump needed,
      # this checkpoint only requires vllm>=0.9.2) -- plain GPTQ has no
      # TP>1 restriction unlike bnb, so tensor-parallel-size=2 is worth
      # trying later, but not risking a parallelism-strategy change in the
      # same rollout as the quant+context-length change.
      # This GPTQ requant's own config.json ships max_position_embeddings=
      # 40960 and rope_scaling=None -- confirmed directly (curl'd the raw
      # config.json), the base Qwen3-32B repo's YaRN block did NOT carry
      # over during quantization. Re-applying it explicitly here restores
      # the same math the base model documents (32768 * 4.0 = 131072);
      # without this, --max-model-len=131072 fails ModelConfig validation
      # against the checkpoint's own (unscaled) 40960 ceiling.
      - --model=JunHowie/Qwen3-32B-GPTQ-Int4
      - --served-model-name=reasoning
      - --quantization=gptq
      - --dtype=float16
      - --kv-cache-dtype=auto
      - --rope-scaling={"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}
      - --tensor-parallel-size=1
      - --pipeline-parallel-size=2
      - --max-model-len=131072
      - --gpu-memory-utilization=0.90
      - --max-num-seqs=4
      - --enable-chunked-prefill
      - --enable-prefix-caching
      # qwen3 is vLLM's dedicated reasoning parser for this family's <think>
      # blocks.
      - --reasoning-parser=qwen3
      # hermes is the documented tool-call parser for general (non-Coder)
      # Qwen3 models -- native chat template support, not narrated text.
      - --enable-auto-tool-choice
      - --tool-call-parser=hermes
      - --host=0.0.0.0
      - --port=8080
      env:
      - name: VLLM_USE_FLASHINFER_SAMPLER
        value: '0'
      - name: VLLM_ATTENTION_BACKEND
        value: TRITON_ATTN
      - name: HF_HOME
        value: /mnt/models
      image: vllm/vllm-openai:v0.11.0@sha256:014a95f21c9edf6abe0aea6b07353f96baa4ec291c427bb1176dc7c93a85845c
      name: kserve-container
      ports:
      - containerPort: 8080
        protocol: TCP
      readinessProbe:
        httpGet:
          path: /health
          port: 8080
        periodSeconds: 10
      resources:
        limits:
          cpu: '16'
          memory: 36Gi
          nvidia.com/gpu: '2'
        requests:
          cpu: '8'
          memory: 12Gi
          nvidia.com/gpu: '2'
      startupProbe:
        failureThreshold: 80
        httpGet:
          path: /health
          port: 8080
        periodSeconds: 15
      volumeMounts:
      - mountPath: /mnt/models
        name: models
      - mountPath: /dev/shm
        name: shm
    deploymentStrategy:
      type: Recreate
    maxReplicas: 1
    minReplicas: 1
    nodeSelector:
      kubernetes.io/hostname: worker-1
    runtimeClassName: nvidia
    volumes:
    - name: models
      persistentVolumeClaim:
        claimName: llm-models
    - emptyDir:
        medium: Memory
        sizeLimit: 2Gi
      name: shm
@@ -0,0 +1,56 @@
 apiVersion: serving.kserve.io/v1beta1
 kind: InferenceService
 metadata:
  annotations:
    serving.kserve.io/deploymentMode: RawDeployment
  labels:
    app.kubernetes.io/name: llm-reranker
    app.kubernetes.io/part-of: llm-serving
  name: reranker
  namespace: llm-serving
 spec:
  predictor:
    containers:
    - args:
      - --model-id=BAAI/bge-reranker-base
      - --port=8080
      - --hostname=0.0.0.0
      - --auto-truncate
      env:
      - name: HUGGINGFACE_HUB_CACHE
        value: /mnt/models
      image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.2@sha256:4d632b76bd14cb57044a1ffb0ad48ab0ba4939e705a9a615ccc740658575c26e
      name: kserve-container
      ports:
      - containerPort: 8080
        protocol: TCP
      readinessProbe:
        httpGet:
          path: /health
          port: 8080
        periodSeconds: 10
      resources:
        limits:
          cpu: '16'
          memory: 8Gi
        requests:
          cpu: '8'
          memory: 4Gi
      startupProbe:
        failureThreshold: 60
        httpGet:
          path: /health
          port: 8080
        periodSeconds: 10
      volumeMounts:
      - mountPath: /mnt/models
        name: models
    maxReplicas: 1
    minReplicas: 1
    nodeSelector:
      kubernetes.io/hostname: worker-1
    volumes:
    - name: models
      persistentVolumeClaim:
        claimName: llm-models
@@ -13,6 +13,7 @@ spec:
      labels:
        app: management-service
    spec:
      serviceAccountName: kmsvc
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
@@ -2,7 +2,7 @@ namespace: sqs
 replicaCount: 3
 image:
-  repository: ghcr.io/riotpiaole/kmsvc-management-service
+  repository: forgejo.riotpiao.com/rock/kmsvc-manage
  tag: latest
  pullPolicy: Always
@@ -0,0 +1,6 @@
 apiVersion: v2
 name: memory-queues
 description: Kafka queues (DLQ) for Poimen Memory service (Phase 6.6)
 type: application
 version: 0.1.0
 appVersion: "1.0"
@@ -0,0 +1,20 @@
 {{- range .Values.queues }}
 ---
 apiVersion: kmsvc.io/v1alpha1
 kind: Queue
 metadata:
  name: {{ .name }}
  namespace: {{ $.Values.namespace }}
  labels:
    app: memory-service
    queue: dlq
 spec:
  name: {{ .name }}
  description: {{ .description }}
  partitions: {{ .partitions }}
  replicationFactor: {{ .replicationFactor }}
  config:
    retention.ms: "{{ .config.retention.ms }}"
    message.retention.seconds: "{{ .config.message.retention.seconds }}"
    visibility.timeout.seconds: "{{ .config.visibility.timeout.seconds }}"
 {{- end }}
@@ -0,0 +1,25 @@
 # Poimen Memory Service Kafka Queues (kmsvc)
 # Phase 6.6: DLQ topics for webhook + metrics failures
 queues:
  # DLQ for extraction, webhook, and agent failures
  - name: poimen-memory-dlq
    description: "DLQ for extraction, webhook, and agent failures"
    partitions: 3
    replicationFactor: 1
    config:
      retention.ms: "1209600000"  # 14 days
      message.retention.seconds: "1209600"
      visibility.timeout.seconds: "300"
  # DLQ for metrics persistence failures  
  - name: poimen-memory-metric-dlq
    description: "DLQ for metrics persistence failures"
    partitions: 3
    replicationFactor: 1
    config:
      retention.ms: "1209600000"  # 14 days
      message.retention.seconds: "1209600"
      visibility.timeout.seconds: "300"
 namespace: sqs
@@ -10,13 +10,13 @@ metadata:
  name: queue-operator
 rules:
  - apiGroups: ["kmsvc.io"]
-    resources: ["queues", "temporalworkers"]
+    resources: ["queues"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["kmsvc.io"]
-    resources: ["queues/status", "temporalworkers/status"]
+    resources: ["queues/status"]
    verbs: ["get", "update", "patch"]
  - apiGroups: ["kmsvc.io"]
-    resources: ["queues/finalizers", "temporalworkers/finalizers"]
+    resources: ["queues/finalizers"]
    verbs: ["update"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
@@ -27,9 +27,6 @@ rules:
  - apiGroups: [""]
    resources: ["pods", "nodes"]
    verbs: ["get"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
@@ -1,62 +0,0 @@
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
  name: temporalworkers.kmsvc.io
 spec:
  group: kmsvc.io
  names:
    kind: TemporalWorker
    plural: temporalworkers
    singular: temporalworker
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          required:
            - spec
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              type: object
              description: Temporal worker specification
              properties:
                namespace:
                  type: string
                  description: Temporal namespace
                taskQueue:
                  type: string
                  description: Task queue name
                workflowTypes:
                  type: array
                  items:
                    type: string
                  description: List of workflow types to execute
                activityTypes:
                  type: array
                  items:
                    type: string
                  description: List of activity types to execute
                concurrency:
                  type: integer
                  minimum: 1
                  description: Worker concurrency level
            status:
              type: object
              description: Temporal worker status
              properties:
                ready:
                  type: boolean
                lastHeartbeat:
                  type: string
                  format: date-time
                error:
                  type: string
@@ -1,7 +1,7 @@
 namespace: sqs
 image:
-  repository: ghcr.io/riotpiaole/kmsvc-management-service
+  repository: forgejo.riotpiao.com/rock/kmsvc-manage
  tag: latest
  pullPolicy: Always
@@ -0,0 +1,95 @@
 # Overrides paperless-ngx's own paperless/adapter.py at the same import path
 # (mounted via subPath in deployment.yaml) - settings.py hardcodes
 # SOCIALACCOUNT_ADAPTER = "paperless.adapter.CustomSocialAccountAdapter", so
 # no Django setting needs to change, just the file content underneath it.
 #
 # Stock CustomSocialAccountAdapter.populate_user() is a stub ("kept in case
 # global default permissions are implemented in the future" - they aren't),
 # so every OIDC signup lands with zero permissions and 403s on every API
 # endpoint. This adds the actual mapping: Authentik's "permissions" claim
 # (via the permissions scope, requested in PAPERLESS_SOCIALACCOUNT_PROVIDERS,
 # computed server-side from group membership by authentik-provision.py) ->
 # "paperless:write" or "*" (homelab-admins) grants is_staff+is_superuser,
 # same convention already used for MinIO's policy claim and Grafana's
 # role_attribute_path. Checking the permission string rather than a literal
 # group name decouples "what grants access" from which group happens to
 # hold it - same pattern applies to every other service's Role/RoleBinding
 # in k8s/infra/rbac/.
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: paperless-adapter
 data:
  adapter.py: |
    from urllib.parse import quote
    from allauth.account.adapter import DefaultAccountAdapter
    from allauth.core import context
    from allauth.socialaccount.adapter import DefaultSocialAccountAdapter
    from django.conf import settings
    from django.forms import ValidationError
    from django.urls import reverse
    REQUIRED_PERMISSIONS = {"paperless:write", "*"}
    class CustomAccountAdapter(DefaultAccountAdapter):
        def is_open_for_signup(self, request):
            allow_signups = super().is_open_for_signup(request)
            return getattr(settings, "ACCOUNT_ALLOW_SIGNUPS", allow_signups)
        def pre_authenticate(self, request, **credentials):
            if settings.DISABLE_REGULAR_LOGIN:
                raise ValidationError("Regular login is disabled")
            return super().pre_authenticate(request, **credentials)
        def is_safe_url(self, url):
            from django.utils.http import url_has_allowed_host_and_scheme
            allowed_hosts = {context.request.get_host()} | set(settings.ALLOWED_HOSTS)
            if "*" in allowed_hosts:
                allowed_hosts.remove("*")
                allowed_hosts.add(context.request.get_host())
                return url_has_allowed_host_and_scheme(url, allowed_hosts=allowed_hosts)
            return url_has_allowed_host_and_scheme(url, allowed_hosts=allowed_hosts)
        def get_reset_password_from_key_url(self, key):
            if settings.PAPERLESS_URL is None:
                return super().get_reset_password_from_key_url(key)
            path = reverse(
                "account_reset_password_from_key",
                kwargs={"uidb36": "UID", "key": "KEY"},
            )
            path = path.replace("UID-KEY", quote(key))
            return settings.PAPERLESS_URL + path
    class CustomSocialAccountAdapter(DefaultSocialAccountAdapter):
        def is_open_for_signup(self, request, sociallogin):
            allow_signups = super().is_open_for_signup(request, sociallogin)
            return getattr(settings, "SOCIALACCOUNT_ALLOW_SIGNUPS", allow_signups)
        def get_connect_redirect_url(self, request, socialaccount):
            return reverse("base")
        def populate_user(self, request, sociallogin, data):
            user = super().populate_user(request, sociallogin, data)
            perms = set(sociallogin.account.extra_data.get("permissions") or [])
            if perms & REQUIRED_PERMISSIONS:
                user.is_staff = True
                user.is_superuser = True
            return user
        def save_user(self, request, sociallogin, form=None):
            # populate_user() sets the flags on the in-memory user, but
            # allauth's default save_user() re-derives is_staff from
            # ACCOUNT_DEFAULT_HTTP_PROTOCOL-independent defaults and can
            # overwrite them on save - re-apply after super().save_user()
            # persists the row, matching the permissions check above exactly.
            user = super().save_user(request, sociallogin, form)
            perms = set(sociallogin.account.extra_data.get("permissions") or [])
            if perms & REQUIRED_PERMISSIONS and not (user.is_staff and user.is_superuser):
                user.is_staff = True
                user.is_superuser = True
                user.save(update_fields=["is_staff", "is_superuser"])
            return user
@@ -0,0 +1,94 @@
 # Nightly: pg_dump the paperless DB + mirror the media PVC into the scoped
 # `paperless` MinIO bucket (see minio-provision-paperless-job.yaml). This is a
 # BACKUP target, not live storage - paperless-ngx has no native S3 backend, it
 # only ever reads/writes the local media PVC directly.
 #
 # Pinned to talos-cp-3, same as deployment.yaml: media is a ReadWriteOnce
 # Longhorn volume with a single replica physically on that node's disk -
 # mounting it read-only here from a different node would conflict with the
 # live webserver's attachment.
 apiVersion: batch/v1
 kind: CronJob
 metadata:
  name: paperless-backup
 spec:
  schedule: "0 3 * * *"  # 03:00 daily, low-traffic window
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          nodeSelector:
            kubernetes.io/hostname: talos-cp-3
          initContainers:
            - name: pg-dump
              image: postgres:16-alpine
              env:
                - name: PGHOST
                  value: paperless-db-rw
                - name: PGDATABASE
                  value: paperless
                - name: PGUSER
                  valueFrom:
                    secretKeyRef:
                      name: paperless-db-app
                      key: username
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: paperless-db-app
                      key: password
              command:
                - sh
                - -c
                - pg_dump --format=custom --file=/backup/paperless-db.dump
              volumeMounts:
                - name: backup
                  mountPath: /backup
          containers:
            - name: mc-mirror
              image: minio/mc:latest
              env:
                - name: ACCESS_KEY
                  valueFrom:
                    secretKeyRef:
                      name: paperless-minio-creds
                      key: ACCESS_KEY
                - name: SECRET_KEY
                  valueFrom:
                    secretKeyRef:
                      name: paperless-minio-creds
                      key: SECRET_KEY
                - name: BUCKET
                  valueFrom:
                    secretKeyRef:
                      name: paperless-minio-creds
                      key: BUCKET
                - name: ENDPOINT
                  valueFrom:
                    secretKeyRef:
                      name: paperless-minio-creds
                      key: ENDPOINT
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  mc alias set b "$ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY"
                  mc cp /backup/paperless-db.dump "b/$BUCKET/db/paperless-db-$(date +%Y%m%d).dump"
                  mc mirror --overwrite /media "b/$BUCKET/media"
                  echo "Backup done."
              volumeMounts:
                - name: backup
                  mountPath: /backup
                - name: media
                  mountPath: /media
                  readOnly: true
          volumes:
            - name: backup
              emptyDir: {}
            - name: media
              persistentVolumeClaim:
                claimName: paperless-media
                readOnly: true
@@ -0,0 +1,22 @@
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: paperless-config
 data:
  PAPERLESS_URL: "https://paperless.riotpiao.com"
  PAPERLESS_TIME_ZONE: "America/Los_Angeles"
  PAPERLESS_OCR_LANGUAGE: "eng"
  PAPERLESS_DBHOST: "paperless-db-rw"
  PAPERLESS_DBNAME: "paperless"
  PAPERLESS_REDIS: "redis://paperless-redis:6379"
  # django-allauth generic OIDC provider. The client_id/secret/server_url
  # bundle itself lives in the paperless-oidc Secret
  # (SOCIALACCOUNT_PROVIDERS_JSON key, composed by authentik-provision.py) -
  # env vars can't be split across a ConfigMap + Secret for the same key, so
  # this whole value is sourced from the Secret in deployment.yaml instead.
  PAPERLESS_APPS: "allauth.socialaccount.providers.openid_connect"
  # Authentik already verifies identity via OIDC - a second email-confirmation
  # step has no SMTP configured to send it anyway, and paperless-ngx doesn't
  # wire up allauth's confirm-email view, so signup 500s with NoReverseMatch
  # on 'account_confirm_email' without this.
  PAPERLESS_ACCOUNT_EMAIL_VERIFICATION: "none"
@@ -0,0 +1,103 @@
 # Single container runs webserver + consumer + scheduler (paperless-ngx's
 # stock entrypoint does this internally) - no need to split into separate
 # Deployments. replicas: 1 only: paperless-media is ReadWriteOnce, and the
 # consumer polling the media dir doesn't benefit from horizontal scaling here.
 #
 # Pinned to talos-cp-3: paperless-media's disk physically lives there. Longhorn
 # RWO volumes can only be attached from one node at a time, and the nightly
 # backup-cronjob.yaml also mounts this same PVC (read-only) to mirror it into
 # MinIO - pinning both to the same node avoids a cross-node attach conflict,
 # and keeps the 3.5Ti read/write path off the network entirely.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: paperless
 spec:
  replicas: 1
  strategy:
    type: Recreate  # ReadWriteOnce media PVC - avoid two pods fighting over it
  selector:
    matchLabels:
      app: paperless
  template:
    metadata:
      labels:
        app: paperless
    spec:
      # Kubernetes injects legacy Docker-links env vars for every Service in
      # this namespace (<SVC>_SERVICE_HOST, <SVC>_PORT, ...). The Service here
      # is named "paperless", so that becomes PAPERLESS_PORT=tcp://<ip>:8000 -
      # paperless-ngx's own entrypoint reads PAPERLESS_PORT for gunicorn's
      # bind address, collides, and gunicorn crash-loops on "not a valid port
      # number". Disable the injection instead of renaming the Service.
      enableServiceLinks: false
      nodeSelector:
        kubernetes.io/hostname: talos-cp-3
      containers:
        - name: paperless
          image: ghcr.io/paperless-ngx/paperless-ngx:2.20.15
          ports:
            - containerPort: 8000
          envFrom:
            - configMapRef:
                name: paperless-config
          env:
            - name: PAPERLESS_DBUSER
              valueFrom:
                secretKeyRef:
                  name: paperless-db-app
                  key: username
            - name: PAPERLESS_DBPASS
              valueFrom:
                secretKeyRef:
                  name: paperless-db-app
                  key: password
            - name: PAPERLESS_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: paperless-secrets
                  key: PAPERLESS_SECRET_KEY
            - name: PAPERLESS_ADMIN_USER
              valueFrom:
                secretKeyRef:
                  name: paperless-secrets
                  key: PAPERLESS_ADMIN_USER
            - name: PAPERLESS_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: paperless-secrets
                  key: PAPERLESS_ADMIN_PASSWORD
            - name: PAPERLESS_SOCIALACCOUNT_PROVIDERS
              valueFrom:
                secretKeyRef:
                  name: paperless-oidc
                  key: SOCIALACCOUNT_PROVIDERS_JSON
          resources:
            requests: { cpu: "500m", memory: "1Gi" }
            limits: { cpu: "2", memory: "4Gi" }
          volumeMounts:
            - name: media
              mountPath: /usr/src/paperless/media
            - name: data
              mountPath: /usr/src/paperless/data
            - name: consume
              mountPath: /usr/src/paperless/consume
            # Overrides paperless-ngx's own adapter.py in place - settings.py
            # hardcodes the import path, so no Django setting changes, just
            # the file content underneath it (see adapter-configmap.yaml).
            - name: adapter
              mountPath: /usr/src/paperless/src/paperless/adapter.py
              subPath: adapter.py
              readOnly: true
      volumes:
        - name: media
          persistentVolumeClaim:
            claimName: paperless-media
        - name: data
          persistentVolumeClaim:
            claimName: paperless-data
        - name: consume
          emptyDir: {}
        - name: adapter
          configMap:
            name: paperless-adapter
@@ -0,0 +1,24 @@
 # Direct nginx ingress to the paperless Service - not routed via the Go
 # api-gateway (api.riotpiao.com), which has no WebSocket upgrade support and
 # paperless-ngx keeps a long-lived /ws/ connection open for live task status.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: paperless
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"  # large scanned PDF uploads
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
 spec:
  ingressClassName: nginx
  rules:
    - host: paperless.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: paperless
                port:
                  number: 8000
@@ -0,0 +1,17 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: paperless
 resources:
  - pvc.yaml
  - configmap.yaml
  - redis.yaml
  - deployment.yaml
  - service.yaml
  - ingress.yaml
  - backup-cronjob.yaml
  - adapter-configmap.yaml
  - rbac.yaml
 # postgres: paperless-db CNPG Cluster, deployed by k8s/infra/databases (wave 2,
 # before this app at wave 8) - not duplicated here. Same for the paperless-oidc
 # and paperless-minio-creds Secrets, written by PostSync provisioning Jobs in
 # k8s/infra/iam and k8s/infra/minio respectively.
@@ -0,0 +1,35 @@
 # Two volumes, deliberately separate storage classes:
 #
 # - media: the actual documents (originals + OCR'd archive PDFs + thumbnails).
 #   Lives on the cp-3 USB HDD, single replica (see
 #   k8s/infra/longhorn/longhorn-paperless-storageclass.yaml). Shares the disk
 #   with Immich's immich-media PVC (k8s/apps/immich/pvc.yaml, 2000Gi) - photo
 #   libraries grow much faster than scanned documents, so paperless gets the
 #   smaller 500Gi share.
 # - data: the SQLite classification model + search index. Small (low GB),
 #   frequently rewritten, and disposable (rebuilds from the DB + media on
 #   next consume) - stays on the default 3-replica pool instead of the
 #   single-disk HDD.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: paperless-media
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn-paperless-media
  resources:
    requests:
      storage: 500Gi
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: paperless-data
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
@@ -0,0 +1,35 @@
 # Scoped operator access for paperless-admins: restart/config-edit rights on
 # just this service's own resources, nothing CNPG-managed (paperless-db-*)
 # or provisioning-managed (paperless-oidc, paperless-minio-creds). Inert
 # until kube-apiserver's OIDC wiring lands (--oidc-groups-claim=groups,
 # --oidc-groups-prefix=oidc:) - subject name below assumes that prefix.
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
  name: paperless-operator
 rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    resourceNames: ["paperless"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["paperless-config"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["paperless-secrets"]
    verbs: ["get", "list", "watch", "update", "patch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: paperless-admins-binding
 subjects:
  - kind: Group
    name: "oidc:paperless-admins"
    apiGroup: rbac.authorization.k8s.io
 roleRef:
  kind: Role
  name: paperless-operator
  apiGroup: rbac.authorization.k8s.io
@@ -0,0 +1,37 @@
 # Task queue broker + websocket channel layer for paperless-ngx. No PVC:
 # queued/scheduled task state is disposable - a lost queue on restart just
 # means re-triggering consumption, not data loss (documents themselves live
 # on paperless-media).
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: paperless-redis
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: paperless-redis
  template:
    metadata:
      labels:
        app: paperless-redis
    spec:
      containers:
        - name: redis
          image: redis:7-alpine
          ports:
            - containerPort: 6379
          resources:
            requests: { cpu: "50m", memory: "64Mi" }
            limits: { cpu: "250m", memory: "256Mi" }
 ---
 apiVersion: v1
 kind: Service
 metadata:
  name: paperless-redis
 spec:
  selector:
    app: paperless-redis
  ports:
    - port: 6379
      targetPort: 6379
@@ -0,0 +1,10 @@
 apiVersion: v1
 kind: Service
 metadata:
  name: paperless
 spec:
  selector:
    app: paperless
  ports:
    - port: 8000
      targetPort: 8000
@@ -33,7 +33,7 @@ tls:
 # RWO block volume. 10Gi is generous for config data but cheap on Longhorn.
 persistence:
  enabled: true
-  storageClass: "longhorn-wffc"
+  storageClass: "longhorn"
  size: 10Gi
 resources:
@@ -52,7 +52,8 @@ tolerations:
    operator: Exists
    effect: NoSchedule
-# Pin to az-a (talos-cp-1) — sole Longhorn storage node. Its RWO PVC can only
+# Pin to az-b (talos-cp-2) — sole Longhorn storage node (dedicated disks).
-# attach there; without this the pod may land on cp-2/cp-3 and fail to mount.
+# Its RWO PVC can only attach there; without this the pod may land on
 # cp-1/cp-3 and fail to mount.
 nodeSelector:
-  topology.kubernetes.io/zone: az-a
+  topology.kubernetes.io/zone: az-b
@@ -0,0 +1,144 @@
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
  name: secretrotations.homelab.riotpiao.com
 spec:
  group: homelab.riotpiao.com
  names:
    kind: SecretRotation
    plural: secretrotations
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          metadata:
            type: object
          spec:
            type: object
            required:
            - provider
            - rotationInterval
            properties:
              # External system: authentik | forgejo | minio | vault
              provider:
                type: string
                enum: [authentik, forgejo, minio, vault]
              # How often to rotate (hours)
              rotationInterval:
                type: integer
                minimum: 24
              # Application ID in external system
              appId:
                type: string
              # k8s Secret to update (name, namespace, key)
              secretRef:
                type: object
                required: [name, namespace]
                properties:
                  name:
                    type: string
                  namespace:
                    type: string
                  key:
                    type: string
                    description: "Secret key to update (e.g., MINIO_IDENTITY_OPENID_CLIENT_SECRET)"
              # Path to git file that holds the secret (for .enc.yaml files)
              gitPath:
                type: string
                description: "Path in homelab repo to .enc.yaml file"
              # Ansible template values to substitute
              templateValues:
                type: object
                additionalProperties:
                  type: string
          status:
            type: object
            properties:
              lastRotationTime:
                type: string
                format: date-time
              nextRotationTime:
                type: string
                format: date-time
              lastRotationStatus:
                type: string
                enum: [Success, Failed, Pending]
              lastRotationError:
                type: string
              lastCommitHash:
                type: string
 ---
 # Example usage:
 apiVersion: homelab.riotpiao.com/v1
 kind: SecretRotation
 metadata:
  name: minio-oidc
  namespace: secret-rotation
 spec:
  provider: authentik
  rotationInterval: 2160  # 90 days in hours
  appId: minio
  secretRef:
    name: minio-oidc
    namespace: storage
    key: MINIO_IDENTITY_OPENID_CLIENT_SECRET
  gitPath: k8s/argocd/secrets/minio-oidc.enc.yaml
 ---
 apiVersion: homelab.riotpiao.com/v1
 kind: SecretRotation
 metadata:
  name: portfolio-agent-oidc
  namespace: secret-rotation
 spec:
  provider: authentik
  rotationInterval: 2160
  appId: portfolio-agent
  secretRef:
    name: portfolio-agent-oidc
    namespace: portfolio
    key: CLIENT_SECRET
  gitPath: k8s/argocd/secrets/portfolio-agent-oidc.enc.yaml
 ---
 apiVersion: homelab.riotpiao.com/v1
 kind: SecretRotation
 metadata:
  name: forgejo-registry-token
  namespace: secret-rotation
 spec:
  provider: forgejo
  rotationInterval: 2160
  appId: rock/riotpiao.com
  secretRef:
    name: forgejo-registry-secret
    namespace: kube-system
    key: REGISTRY_TOKEN
  gitPath: k8s/argocd/secrets/forgejo-registry-secret.enc.yaml
 ---
 apiVersion: homelab.riotpiao.com/v1
 kind: SecretRotation
 metadata:
  name: minio-root-credentials
  namespace: secret-rotation
 spec:
  provider: minio
  rotationInterval: 4320  # 180 days in hours
  appId: root
  secretRef:
    name: minio-creds
    namespace: storage
  gitPath: k8s/argocd/secrets/minio-secrets.enc.yaml
@@ -0,0 +1,92 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: secret-rotation-controller
  namespace: secret-rotation
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: secret-rotation-controller
  template:
    metadata:
      labels:
        app: secret-rotation-controller
    spec:
      serviceAccountName: secret-rotation-controller
      containers:
      - name: controller
        image: secret-rotation-controller:latest
        imagePullPolicy: IfNotPresent
        env:
        # SOPS reads age key from this file
        - name: SOPS_AGE_KEY_FILE
          value: /etc/sops/age/private-key.txt
        # Vault auth (token in projected volume)
        - name: VAULT_ADDR
          value: http://vault.vault.svc.cluster.local:8200
        - name: VAULT_TOKEN_FILE
          value: /var/run/secrets/vault/token
        # Authentik
        - name: AUTHENTIK_URL
          value: http://authentik-server.iam.svc.cluster.local
        - name: AUTHENTIK_BOOTSTRAP_TOKEN
          valueFrom:
            secretKeyRef:
              name: authentik-bootstrap
              key: token
        # Git
        - name: GIT_REPO
          value: https://forgejo.riotpiao.com/rock/homelab.git
        - name: GIT_AUTHOR_EMAIL
          value: [email protected]
        - name: GIT_AUTHOR_NAME
          value: Secret Rotation Controller
        - name: FORGEJO_TOKEN
          valueFrom:
            secretKeyRef:
              name: forgejo-registry-secret
              key: REGISTRY_TOKEN
        volumeMounts:
        # Age key from ExternalSecret (synced from Vault)
        - name: age-key
          mountPath: /etc/sops/age
          readOnly: true
        # Vault auth token (projected)
        - name: vault-token
          mountPath: /var/run/secrets/vault
          readOnly: true
        # Temp working dir
        - name: tmp
          mountPath: /tmp
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi
      volumes:
      - name: age-key
        secret:
          secretName: sops-age-key
          defaultMode: 0400
      - name: vault-token
        projected:
          sources:
          - serviceAccountToken:
              path: token
              audience: vault
              expirationSeconds: 3600
      - name: tmp
        emptyDir: {}
@@ -0,0 +1,15 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: secret-rotation
 resources:
  - rbac.yaml
  - crd.yaml
  - external-secret.yaml
  - deployment.yaml
 commonLabels:
  app.kubernetes.io/name: secret-rotation-controller
  app.kubernetes.io/component: automation
  managed-by: argocd
@@ -0,0 +1,53 @@
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: secret-rotation-controller
  namespace: secret-rotation
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
  name: secret-rotation-controller
 rules:
 # Read SecretRotation CRDs
 - apiGroups: ["homelab.riotpiao.com"]
  resources: ["secretrotations"]
  verbs: ["get", "list", "watch"]
 # Update status
 - apiGroups: ["homelab.riotpiao.com"]
  resources: ["secretrotations/status"]
  verbs: ["get", "patch", "update"]
 # Read k8s secrets that will be rotated
 - apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]
 # For recording events
 - apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "patch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata:
  name: secret-rotation-controller
 roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: secret-rotation-controller
 subjects:
 - kind: ServiceAccount
  name: secret-rotation-controller
  namespace: secret-rotation
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: secret-rotation
  labels:
    kubernetes.io/metadata.name: secret-rotation
@@ -0,0 +1,115 @@
 # macOS VM (Docker-OSX) hosting the BlueBubbles server.
 #
 # ── Why a VM and not a container ────────────────────────────────────────────
 # Containers share the host kernel. macOS binaries are Mach-O and need XNU plus
 # Cocoa/IOKit, which a Linux kernel cannot provide, so no macOS container exists
 # or can exist. Docker-OSX is QEMU running a macOS guest, packaged in a
 # container — a VM in a box, not a macOS container.
 #
 # ── Why this works on worker-2 ──────────────────────────────────────────────
 # Verified on the existing hardware: amd64, `vmx` (Intel VT-x) present, and
 # /dev/kvm exists on Talos nodes (KVM is compiled into Talos' kernel, not a
 # module). Bare metal, so no nested virtualisation needed.
 #
 # ── Read this before relying on it ──────────────────────────────────────────
 # 1. Setup is INTERACTIVE. First boot runs the macOS installer: connect over
 #    VNC (:5999), erase the disk in Disk Utility, install, create a user, sign
 #    into iMessage, THEN install BlueBubbles inside the guest. This manifest
 #    only provides the machine; it does not provision macOS.
 # 2. iMessage activation on non-Apple hardware is a coin flip. BlueBubbles'
 #    own guidance: "test sending an iMessage to yourself. If it does not
 #    succeed, it's likely best to restart from the beginning."
 # 3. Apple's macOS licence permits virtualisation only on Apple hardware. This
 #    is a Hackintosh. Use a throwaway Apple ID, not a primary one.
 # 4. BlueBubbles labels this path "not for beginners", "no guarantees or
 #    warranty".
 #
 # Private API (reactions, typing indicators, edit/unsend) needs SIP disabled
 # inside the guest and is NOT required for plain send/receive. Skip it.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: macos-bluebubbles
  labels:
    app.kubernetes.io/name: macos-bluebubbles
    app.kubernetes.io/part-of: sms
 spec:
  replicas: 1
  # Recreate: the qcow2 disk is RWO and a second pod must never attach it
  # concurrently — two QEMU processes on one image corrupts it.
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: macos-bluebubbles
  template:
    metadata:
      labels:
        app.kubernetes.io/name: macos-bluebubbles
        app.kubernetes.io/part-of: sms
    spec:
      # Dedicated node. The taint keeps everything else off worker-2; this
      # toleration is what lets the VM on. Both halves are required.
      nodeSelector:
        workload: imessage
      tolerations:
        - key: workload
          operator: Equal
          value: imessage
          effect: NoSchedule
      containers:
        - name: macos
          image: sickcodes/docker-osx:latest@sha256:3a3c82c79bc4e73531f819ccdfa4053b3084efd7c1f645678dbf8b4b3a24369c
          # QEMU needs /dev/kvm; Talos enforces `baseline` cluster-wide, so this
          # only schedules because the sms namespace is labelled privileged.
          securityContext:
            privileged: true
          env:
            # Generates a unique serial / board-serial / UUID / MAC and persists
            # them to bootdisk.qcow2. This synthetic identity is what iMessage
            # activates against — it must stay stable across restarts, which is
            # why the PVC matters.
            - name: GENERATE_UNIQUE
              value: "true"
            # Identity is only plausible if it matches a real product line.
            - name: DEVICE_MODEL
              value: "iMacPro1,1"
            - name: RAM
              value: "12"
            - name: CORES
              value: "6"
            - name: EXTRA
              # Expose the BlueBubbles server port from the guest to the pod.
              # Guest :1234 (BlueBubbles default) -> pod :1234.
              value: "-device virtio-net-pci,netdev=net0 -netdev user,id=net0,hostfwd=tcp::1234-:1234"
          ports:
            - name: vnc
              containerPort: 5999
              protocol: TCP
            - name: bluebubbles
              containerPort: 1234
              protocol: TCP
          resources:
            requests:
              cpu: "6"
              memory: 14Gi
            limits:
              cpu: "12"
              memory: 20Gi
          volumeMounts:
            - name: macos-disk
              mountPath: /home/arch/OSX-KVM/disk
            - name: kvm
              mountPath: /dev/kvm
          # No readiness probe on purpose. The guest takes many minutes to boot,
          # and until macOS + BlueBubbles are installed BY HAND there is nothing
          # listening on 1234. A probe here would crash-loop the pod through the
          # entire interactive install.
      volumes:
        - name: macos-disk
          persistentVolumeClaim:
            claimName: macos-disk
        - name: kvm
          hostPath:
            path: /dev/kvm
            type: CharDevice
@@ -0,0 +1,10 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: sms
 resources:
  - namespace.yaml
  - storageclass.yaml
  - pvc-macos.yaml
  - deployment-macos.yaml
  - service.yaml
  - networkpolicy.yaml
@@ -0,0 +1,19 @@
 # iMessage delivery for the cluster.
 #
 # BlueBubbles' server is a macOS Electron app paired with an Objective-C helper
 # that hooks Messages.app private APIs — it cannot be containerised on Linux,
 # because containers share the host kernel and macOS needs XNU + Cocoa. The only
 # way to run it on Talos is a full macOS VM under QEMU/KVM (Docker-OSX), which
 # needs a privileged pod with /dev/kvm.
 #
 # Hence privileged PodSecurity: the cluster default from the Talos controlplane
 # is `enforce: baseline`, which forbids privileged containers and host devices.
 # Scope is limited to this namespace.
 apiVersion: v1
 kind: Namespace
 metadata:
  name: sms
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
@@ -0,0 +1,22 @@
 # Default-deny. This namespace runs a privileged QEMU VM signed into an Apple
 # ID and exposes an unauthenticated VNC console; nothing should reach it except
 # opted-in clients.
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
  name: sms-default-deny
 spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: sms
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              sms-client: "true"
      ports:
        - protocol: TCP
          port: 1234
@@ -0,0 +1,25 @@
 # Persistent macOS disk image + generated hardware identity (bootdisk.qcow2).
 #
 # This volume is NOT disposable: it holds the VM's serial number, board serial,
 # UUID and MAC, which together form the identity iMessage was activated against.
 # Losing it means re-running activation, which is the least reliable step of the
 # whole setup.
 #
 # Docker-OSX documents 128GB minimum for the guest image; 200Gi leaves room for
 # the installer, the base system, and qcow2 growth.
 #
 # ⚠️ Single replica (see storageclass.yaml — capacity and IO both rule out 3).
 # Losing worker-2's disk therefore means losing the activated identity and
 # redoing iMessage activation. Once the guest is installed and activated, take
 # a Longhorn snapshot/backup of this volume; that is the only redundancy here.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: macos-disk
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn-imessage-local
  resources:
    requests:
      storage: 200Gi
@@ -0,0 +1,31 @@
 # VNC is how you drive the interactive macOS install. Deliberately ClusterIP —
 # it is an unauthenticated console onto a machine holding a live Apple ID
 # session. Reach it with `kubectl port-forward`, never an Ingress.
 apiVersion: v1
 kind: Service
 metadata:
  name: macos-vnc
 spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: macos-bluebubbles
  ports:
    - name: vnc
      port: 5999
      targetPort: vnc
 ---
 # The BlueBubbles REST API, once installed inside the guest. This is the stable
 # name cluster services use, so callers never depend on the pod IP or on whether
 # the backend is this VM or a real Mac mini later.
 apiVersion: v1
 kind: Service
 metadata:
  name: bluebubbles
 spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: macos-bluebubbles
  ports:
    - name: http
      port: 1234
      targetPort: bluebubbles
@@ -0,0 +1,32 @@
 # Dedicated StorageClass for the macOS VM disk.
 #
 # The default `longhorn` class does not work here, for two independent reasons:
 #
 # 1. Replica count. Default is 3, and Longhorn schedules against
 #    storageMaximum - storageScheduled with over-provisioning at 100%. Free
 #    space is cp-1 146Gi / cp-2 8Gi / cp-3 146Gi / worker-1 292Gi, so a 200Gi
 #    volume has only one node that can hold even a single replica — a 3-replica
 #    volume fails outright with ReplicaSchedulingFailure.
 # 2. Binding mode. `Immediate` provisions the volume the moment the PVC is
 #    created, before any pod is scheduled. Combined with strict-local that
 #    pins the data to an arbitrary node, not the one the VM runs on.
 #
 # So: one replica, kept local to the VM, bound only once the pod has a node.
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
  name: longhorn-imessage-local
 provisioner: driver.longhorn.io
 allowVolumeExpansion: true
 reclaimPolicy: Delete
 # The pod is pinned to worker-2 by nodeSelector; wait for it to be scheduled so
 # the replica is placed on that node and not somewhere else.
 volumeBindingMode: WaitForFirstConsumer
 parameters:
  # A qcow2 backing a live VM is latency-sensitive and rewritten constantly.
  # Serving it over the network from another node's disk would be the single
  # worst thing for guest responsiveness, so force it local.
  numberOfReplicas: "1"
  dataLocality: "strict-local"
  staleReplicaTimeout: "30"
  fsType: "ext4"
@@ -1,18 +0,0 @@
 temporal:
    oidc_client_id: ENC[AES256_GCM,data:y6eSNJpCNCg=,iv:4n5fIeiVG2JZOCD4yZ4Asm/gWvRl0QNFiRGTR1bs8ws=,tag:liLSr9sadR82UPq6d3gqGA==,type:str]
    oidc_client_secret: ENC[AES256_GCM,data:ebk0FVtSfq7JbIT+84cigdLg8zfN+4HTfGxlgqLw6hs1H92SFRHQFXO1l3U=,iv:qxa8aqUVoN31PpzAhs6bNMDCM6yPvOT27urIrM1rfo0=,tag:Vo25X8Znw7eh5DNq1eIJEw==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArZ1pyMXZ0Y3RFQVZkcTg5
            aENyemRueXpIM0FaUVhMc2ZyVDRyWWlzK1JzCjA0OTZ0aGJDNVVSM29CY1RzQWJL
            eVNiSCtMSXdmbEtKeGt1L3NYMy9ibUEKLS0tIGJnU3A4MjVnZ1pCa1lPMnVoM0xo
            dWs1cVM5ZCszbXp6eFltRVhGbFc0ajQKyCc8lClnSqWUxhNOr1FDCwn5V7nvjxPN
            7kjQpldseaRbsy+TM5sFQ1w6Acmun9uYjzs8PtmTNaayc/AFfspufA==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1smu533f803gmd0jq60s2zaj9zlznajy0ca6rtewd4r37mr2hs3uqsrldfh
    lastmodified: "2026-07-15T23:22:55Z"
    mac: ENC[AES256_GCM,data:0YuRxQHfdeKbjJzxHI4iBc7I2KnalNJmybB6bVQxvdRutsxKXW/bu99yqHkD4y/nI66Pi6d7LhrPhhGat3x2c6Drs7QRYVWnZ75thil6yWsOcpH7H1esxEWTP3cpoMzGdS0aF6zwU9H+WDvfm/t1wpeDpJly3eczcPvfbK1joDU=,iv:TgwfpI+ZVLyd2c/PKWsJbCI9yKkADpPPafJ5No0fLks=,tag:nqwQHtZ263CI+fY9iTUAzQ==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -0,0 +1,18 @@
 apiVersion: kmsvc.io/v1
 kind: TemporalWorker
 metadata:
  name: worker-production
  namespace: temporal
 spec:
  namespace: production
  taskQueue: worker-production
  concurrency: 10
  workflowTypes:
    - HelloWorldWorkflow
    - GreeterWorkflow
    - ProcessOrderWorkflow
  activityTypes:
    - GreetActivity
    - ValidateOrderActivity
    - ProcessPaymentActivity
    - NotifyCustomerActivity
@@ -0,0 +1,25 @@
 # Wave -1 — AppProject definitions (must sync before any Application that references them).
 # Syncs k8s/argocd/projects/ which was previously applied by hand.
 # Enabled by Stage 1 (A2).
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: argocd-projects
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
 spec:
  project: homelab
  revisionHistoryLimit: 3
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/argocd/projects
  destination:
    server: https://kubernetes.default.svc
@@ -1,42 +0,0 @@
 # Wave 0 — Nginx Ingress Controller
 # Foundational infrastructure required for all ingress resources and ArgoCD UI access.
 # Must be wave 0 to ensure LoadBalancer IP is available before other apps deploy.
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: ingress-nginx
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
 spec:
  project: homelab
  revisionHistoryLimit: 3
  sources:
    - repoURL: https://kubernetes.github.io/ingress-nginx
      chart: ingress-nginx
      targetRevision: "4.15.1"
      helm:
        valueFiles:
          - $values/k8s/bootstrap/ingress/nginx-values.yaml
    - repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: ingress-nginx
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
    retry:
      limit: 3
      backoff:
        duration: 10s
        factor: 2
        maxDuration: 3m
@@ -17,10 +17,12 @@ spec:
    syncOptions:
      - CreateNamespace=true
  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
-    path: .
+    # ksops decrypts every *.enc.yaml here at kustomize-build time (repo-server
-    plugin:
+    # runs `kustomize build --enable-alpha-plugins --enable-exec`). Replaces the
-      name: sops-secrets-v1.0
+    # old sops-secrets-v1.0 CMP whose discover glob silently hijacked kustomize
    # rendering of any app whose path contained a *.enc.yaml.
    path: k8s/argocd/secrets
  destination:
    server: https://kubernetes.default.svc
@@ -21,7 +21,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/bootstrap/cert-manager/cert-manager-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -57,6 +57,13 @@ spec:
    helm:
      values: |
        reloader:
          # Watch every workload — no per-Deployment reloader annotation needed
          # (several charts, e.g. homarr, don't expose Deployment-level
          # annotations). reloadOnCreate rolls a workload when a Secret/ConfigMap
          # it references is first CREATED, not only updated — so ksops-delivered
          # secrets landing after a pod started auto-restart it.
          autoReloadAll: true
          reloadOnCreate: true
          deployment:
            tolerations:
              - key: node-role.kubernetes.io/control-plane
@@ -86,11 +93,14 @@ spec:
  project: homelab
  revisionHistoryLimit: 3
  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    # A real kustomization.yaml (resources: the 3 issuer/CA files) renders these
    # deterministically. The previous directory.include with bare filenames
    # rendered EMPTY — ArgoCD's include glob never matched — so this app silently
    # tracked 0 resources; its ConfigMaps/Issuers only existed from bootstrap
    # kubectl apply, and an automated prune wiped them.
    path: k8s/bootstrap/cert-manager
    directory:
      include: "letsencrypt-issuer.yaml"
  destination:
    server: https://kubernetes.default.svc
    namespace: cert-manager
@@ -98,6 +108,8 @@ spec:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
 ---
 # Consolidated: wildcard-cert + homelab-ingress → ingress-config
 # Manages both the wildcard TLS certificate and all Ingress rules.
@@ -115,7 +127,7 @@ spec:
  project: homelab
  revisionHistoryLimit: 3
  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/bootstrap/ingress
  destination:
@@ -126,3 +138,75 @@ spec:
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: cluster-maintenance
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/cluster-maintenance
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: kyverno
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
 spec:
  project: homelab
  source:
    repoURL: https://kyverno.github.io/kyverno/
    chart: kyverno
    targetRevision: "1.14.0"
    helm:
      valueFiles:
        - $values/k8s/bootstrap/kyverno/kyverno-values.yaml
  sources:
    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: kyverno
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: kyverno-policies
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/bootstrap/kyverno
  destination:
    server: https://kubernetes.default.svc
    namespace: kyverno
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
@@ -0,0 +1,33 @@
 # ArgoCD Image Updater - auto-updates Application images from registry
 # Watches forgejo.riotpiao.com for new image tags and updates Applications
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: argocd-image-updater
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "1"
 spec:
  project: homelab
  revisionHistoryLimit: 3
  sources:
    - repoURL: https://argoproj.github.io/argo-helm
      chart: argocd-image-updater
      targetRevision: "0.11.2"
      helm:
        valueFiles:
          - $values/k8s/infra/argocd-image-updater/values.yaml
    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=false
@@ -1,43 +1,8 @@
-# Wave 0 — networking policies layered on the Cilium CNI + CoreDNS that the
+# Wave 0 — networking substrate is Talos-owned (terraform inlineManifests), not
-# cluster bootstrap already installed (substrate). These are raw manifests only.
+# ArgoCD:
-apiVersion: argoproj.io/v1alpha1
+#   - CoreDNS Corefile + hostname rewrites  -> terraform/files/coredns/Corefile
-kind: Application
+#   - Cilium LB-IPAM pool + L2 announcement -> terraform/files/cilium/*.yaml
-metadata:
+# Both were previously ArgoCD apps here whose empty `resources: []`
-  name: cilium-policy
+# kustomizations never actually applied them (live objects came from manual
-  namespace: argocd
+# kubectl). Managing them from ArgoCD too would let two reconcilers fight. This
-  annotations:
+# file intentionally defines no Applications now.
    argocd.argoproj.io/sync-wave: "0"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
    targetRevision: main
    path: k8s/bootstrap/cilium
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: coredns-config
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
    targetRevision: main
    path: k8s/bootstrap/coredns
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
@@ -0,0 +1,32 @@
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: secret-rotation
  namespace: argocd
  labels:
    app.kubernetes.io/name: secret-rotation
 spec:
  project: homelab
  sources:
  - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    path: k8s/apps/secret-rotation-controller
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: secret-rotation
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
    - RespectIgnoreDifferences=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
@@ -17,7 +17,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/minio/minio-operator-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -41,7 +41,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/minio
  destination:
@@ -66,12 +66,20 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/longhorn
  destination:
    server: https://kubernetes.default.svc
    namespace: longhorn-system
  # Longhorn writes disk state back into its own Node CRs — the disk key it
  # generates, storageReserved, diskType, evictionRequested. Git declares only
  # allowScheduling; without this the controller's writes read as drift forever.
  ignoreDifferences:
    - group: longhorn.io
      kind: Node
      jsonPointers:
        - /spec/disks
  syncPolicy:
    automated:
      prune: true
@@ -94,7 +102,7 @@ spec:
        skipCrds: true
        valueFiles:
          - $values/k8s/infra/monitoring/prometheus-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -144,7 +152,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/monitoring/crds
  destination:
@@ -175,7 +183,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/monitoring
  destination:
@@ -205,7 +213,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/monitoring/blackbox-exporter-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -215,3 +223,29 @@ spec:
    automated:
      prune: true
      selfHeal: true
 ---
 # Distributed tracing: Tempo + OpenTelemetry Collector.
 # Receives traces from instrumented services, stores in local volume (72h retention).
 # Grafana datasource auto-configured, service graph + latency dashboards included.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: tracing
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/tracing
  destination:
    server: https://kubernetes.default.svc
    namespace: tracing
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
@@ -19,7 +19,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/logging/loki-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -53,7 +53,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/logging/grafana-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -87,7 +87,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/logging/promtail-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -17,7 +17,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/iam/vault-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -46,7 +46,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/iam/authentik-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -68,7 +68,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/iam
  destination:
@@ -79,18 +79,90 @@ spec:
      prune: true
      selfHeal: true
 ---
-# Forgejo runner (local chart). Forgejo itself is Phase 0 (bootstrap).
+# Forgejo itself. Was a bootstrap Helm release (phase 3) until it was brought
 # under Argo, because values changes there were inert — a proxy-body-size fix
 # sat committed while the live Ingress kept nginx's 1m default and rejected
 # every OCI push with 413.
 #
 # Wave 3: after databases (wave 2) — Forgejo needs CNPG and Redis up first.
 #
 # Retiring the Helm release: Argo adopts the existing objects on first sync.
 # Delete the release secrets afterwards so helm stops claiming ownership:
 #   kubectl -n cicd delete secret -l owner=helm,name=forgejo
 #
 # automated sync is deliberately absent. This chart owns the Forgejo PVC and
 # the git forge itself; the first sync is manual so its diff can be read before
 # anything is applied. Turn on automated+selfHeal once that diff is clean.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
-  name: forgejo-runner
+  name: forgejo
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "3"
 spec:
  project: homelab
  sources:
    - repoURL: https://dl.gitea.com/charts/
      chart: gitea
      targetRevision: 12.7.0
      helm:
        valueFiles:
          - $values/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml
    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: cicd
  # Reloader injects a STAKATER_* env var carrying a hash of the config Secret,
  # so the pod rolls when that Secret changes. The chart does not render it, so
  # Argo would strip it on every sync — and with selfHeal on, Argo and Reloader
  # would fight over the field and Recreate the forge each round.
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: forgejo-gitea
      jqPathExpressions:
        - '.spec.template.spec.containers[].env[] | select(.name | startswith("STAKATER_"))'
  syncPolicy:
    syncOptions:
      # Adopt the objects the bootstrap Helm release already created rather
      # than failing on "already exists".
      - ServerSideApply=true
 ---
 # Forgejo runners (local chart, one instance per language), replacing the
 # single generic "docker"-labeled runner. Each instance is a full standalone
 # Deployment with its own dind sidecar, own PVCs (registration + layer
 # cache) and own registered label -- there is no shared generic runner
 # anymore, so each instance also builds and pushes images for the repos it
 # serves (the chart's ConfigMap/NetworkPolicy fixes for that -- valid_volumes,
 # network: host, egress to ingress-nginx -- apply identically to all three).
 #
 # `values.yaml` is the chart's default and doubles as the golang instance's
 # config; node and rust layer a small values-<lang>.yaml override on top for
 # just runner.name/runner.labels. All three share one runner-token Secret
 # (Forgejo registration tokens are reusable across multiple runners, unlike
 # GitHub's one-time tokens) -- if that assumption is ever wrong, registration
 # will fail loudly in the register initContainer's logs, not silently.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: forgejo-runner-golang
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-golang
    argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
    argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
    argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
    argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/forgejo-runner
  destination:
@@ -100,3 +172,65 @@ spec:
    automated:
      prune: true
      selfHeal: true
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: forgejo-runner-node
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-node
    argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
    argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
    argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
    argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/forgejo-runner
    helm:
      valueFiles:
        - values-node.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: cicd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: forgejo-runner-rust
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-rust
    argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
    argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
    argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
    argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/forgejo-runner
    helm:
      valueFiles:
        - values-rust.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: cicd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
@@ -14,7 +14,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/databases
  destination:
@@ -0,0 +1,20 @@
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: memory-queues
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "7"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/messaging/memory-queues
  destination:
    server: https://kubernetes.default.svc
    namespace: sqs
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
@@ -1,6 +1,6 @@
-# Wave 5 — Kafka (Strimzi operator + cluster CR), Redis, and the SQS-like
+# Wave 5 — Kafka (Strimzi operator + cluster CR), Redis infrastructure.
-# queue services. Strimzi/Redis are public Helm charts; kafka-cluster/queue-crd/
+# Strimzi/Redis are public Helm charts; kafka-cluster is a local chart.
-# management-service are local charts (rendered from their own Chart.yaml).
+# queue-crd and management-service are managed by kmsvc-root (kmsvc-manage.git).
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
@@ -68,7 +68,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/messaging/kafka-cluster
  destination:
@@ -78,45 +78,5 @@ spec:
    automated:
      prune: true
      selfHeal: true
---
+# queue-crd and management-service moved to kmsvc-manage.git repo
-apiVersion: argoproj.io/v1alpha1
+# Managed by kmsvc-root Application
 kind: Application
 metadata:
  name: queue-crd
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "6"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
    targetRevision: main
    path: k8s/apps/messaging/queue-crd
  destination:
    server: https://kubernetes.default.svc
    namespace: sqs
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: management-service
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "7"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
    targetRevision: main
    path: k8s/apps/messaging/management-service
  destination:
    server: https://kubernetes.default.svc
    namespace: sqs
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
--- a/Show More
+++ b/Show More