fix(gotify): move SOPS secrets to ksops generator

ArgoCD couldn't decrypt secrets.yaml because it was listed as a plain kustomize resource. Move the 3 secrets (gotify-admin, gotify-tokens, gotify-smtp) to k8s/argocd/secrets/ as .enc.yaml files processed by the ksops generator, matching the repo convention. Fixes ComparisonError: 'Object Kind is missing' (SOPS ciphertext parsed as raw YAML).
fix(argocd): update git repoURLs for org transfer rock -> riotpiao-poimen
2026-09-10 22:32:24 +09:00 · 2026-09-10 10:49:30 +09:00 · 2026-09-10 01:41:37 +00:00 · 2026-09-10 01:35:59 +00:00 · 2026-09-09 22:38:25 +00:00 · 2026-09-09 02:11:28 +00:00
53 changed files with 1051 additions and 1281 deletions
@@ -1,49 +0,0 @@
 name: Build and push runner images
 on:
  push:
    paths:
      - 'k8s/infra/forgejo-runner/Dockerfile.golang'
      - 'k8s/infra/forgejo-runner/Dockerfile.rust'
      - 'k8s/infra/forgejo-runner/Dockerfile.node'
    branches:
      - main
 jobs:
  build-runners:
    runs-on: golang
    env:
      REGISTRY: forgejo.riotpiao.com
      IMAGE_BASE: forgejo.riotpiao.com/rock
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Get short SHA
        id: sha
        run: |
          SHORT_SHA=$(git rev-parse --short HEAD)
          echo "short_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
      - name: Registry login
        run: |
          echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
            --username "${REGISTRY_USER}" --password-stdin
        env:
          REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
          REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
      - name: Build and push all runner images
        run: |
          set -e
          for RUNNER in golang rust node; do
            echo "📦 Building ${RUNNER}-runner..."
            docker build -f "k8s/infra/forgejo-runner/Dockerfile.${RUNNER}" \
              -t "${IMAGE_BASE}/forgejo-runner-${RUNNER}:${{ steps.sha.outputs.short_sha }}" \
              -t "${IMAGE_BASE}/forgejo-runner-${RUNNER}:latest" \
              .
            docker push "${IMAGE_BASE}/forgejo-runner-${RUNNER}:${{ steps.sha.outputs.short_sha }}"
            docker push "${IMAGE_BASE}/forgejo-runner-${RUNNER}:latest"
            echo "✅ Pushed ${RUNNER}-runner"
          done
          echo "\n✅ All runner images built and pushed"
@@ -66,3 +66,6 @@ bootstrap-argocd.log
 # one line here, which is how a plaintext deploy key reached a public remote.
 k8s/**/*-secret.yaml
 !k8s/**/*.enc.yaml
 # IAM provisioning scripts contain credential references — never commit
 scripts/iam/*.py
@@ -208,3 +208,95 @@ versions without warning in your own values file.
 Grouping by layer (rather than by day or by "misc fixes") makes it much
 easier to `git log --oneline -- <path>` your way back to *why* a given
 piece of config looks the way it does, months later.
 ## Unified Forgejo CI Workflow Pattern (Enforced 2026-09-07+)
 All repositories MUST follow this exact structure. No variations.
 ```yaml
 name: CI
 on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
 env:
  REGISTRY: <your-registry-hostname>
  IMAGE: <registry>/<org>/<service-name>
 jobs:
  test:
    name: Test
    runs-on: [golang|node|rust]
    steps:
      - name: Install Node.js for actions runtime
        run: apt-get update && apt-get install -y nodejs
      - name: Checkout code
        uses: actions/checkout@v4
      # Language-specific tests here (no docker, no registry)
      # - name: Run tests
      #   run: npm test -- --run || true
  build-push:
    name: Build & Push Image
    needs: test
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: [golang|node|rust]
    steps:
      - name: Install Node.js and Docker
        run: |
          apt-get update
          apt-get install -y nodejs docker.io
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Get short SHA
        id: sha
        run: |
          SHORT_SHA=$(git rev-parse --short HEAD)
          echo "short_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
      - name: Registry login
        run: |
          echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
            --username "${REGISTRY_USER}" --password-stdin
        env:
          REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
          REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
      - name: Build Docker image
        run: |
          docker build --no-cache \
            -t "${IMAGE}:${{ steps.sha.outputs.short_sha }}" \
            -t "${IMAGE}:latest" \
            .
      - name: Push Docker image
        run: |
          docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}"
          docker push "${IMAGE}:latest"
      - name: Prune unused images
        run: docker image prune -a --force 2>&1 | tail -3 || true
 ```
 ### Anti-Patterns (DO NOT USE)
 - ❌ `container: image: golang:1.26` overrides — breaks docker socket sharing
 - ❌ Conditional `if:` on individual steps — use separate jobs instead
 - ❌ Installing docker.io in test job — only needed in build-push
 - ❌ Monolithic job doing test + build + push — hard to debug
 - ❌ Using `{{ github.sha }}` for image tag — use short commit SHA for readability
 ### How It Works
 1. **PR to feature branch** → test job runs, build-push skipped, nothing pushed
 2. **Push to main** → test runs, build-push runs after test passes, image pushed
 3. Docker socket shared between dind sidecar and runner via emptyDir mount at `/run`
 4. `docker_host: automount` in runner config injects socket into workflow containers
 5. Secrets (FORGEJO_REGISTRY_USER, TOKEN) set in Forgejo repo settings, NOT in git
@@ -0,0 +1,80 @@
 # ComfyUI — GPU-accelerated image generation on worker-1.
 # Uses 1x V100 32GB (sm70). Freed by scaling ornith 2→1.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: comfyui
  namespace: comfyui
  labels:
    app: comfyui
 spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: comfyui
  template:
    metadata:
      labels:
        app: comfyui
    spec:
      nodeSelector:
        kubernetes.io/hostname: worker-1
      runtimeClassName: nvidia
      containers:
        - name: comfyui
          image: ghcr.io/ai-dock/comfyui:v2-cuda-12.1.1-base-22.04
          ports:
            - containerPort: 8188
              protocol: TCP
          env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: "all"
            - name: COMFYUI_FLAGS
              value: "--listen 0.0.0.0 --port 8188"
          resources:
            requests:
              cpu: "4"
              memory: 8Gi
              nvidia.com/gpu: "1"
            limits:
              cpu: "8"
              memory: 16Gi
              nvidia.com/gpu: "1"
          volumeMounts:
            - mountPath: /workspace/ComfyUI/models
              name: models
            - mountPath: /workspace/ComfyUI/output
              name: output
          readinessProbe:
            httpGet:
              path: /
              port: 8188
            periodSeconds: 10
            initialDelaySeconds: 30
          startupProbe:
            httpGet:
              path: /
              port: 8188
            failureThreshold: 60
            periodSeconds: 10
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: comfyui-models
        - name: output
          emptyDir: {}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: comfyui-models
  namespace: comfyui
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 50Gi
@@ -0,0 +1,25 @@
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: comfyui
  namespace: comfyui
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    # WebSocket support for ComfyUI's live preview
    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
    nginx.ingress.kubernetes.io/proxy-set-headers: "Upgrade"
 spec:
  ingressClassName: nginx
  rules:
    - host: comfy.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: comfyui
                port:
                  number: 80
@@ -0,0 +1,7 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
  - deployment.yaml
  - service.yaml
  - ingress.yaml
@@ -0,0 +1,14 @@
 apiVersion: v1
 kind: Service
 metadata:
  name: comfyui
  namespace: comfyui
  labels:
    app: comfyui
 spec:
  selector:
    app: comfyui
  ports:
    - port: 80
      targetPort: 8188
      protocol: TCP
@@ -0,0 +1,107 @@
 # Gotify — Push Notifications + Email Relay
 Self-hosted notification server with SMTP email forwarding sidecar.
 ## Architecture
 ```
 Forgejo webhook ──POST──→ Gotify API (:80/message)
                              │
                    ┌─────────┼─────────┐
                    ▼                     ▼
              Push notification      SMTP emailer sidecar
              (mobile/desktop)       (polls → sends email)
 ```
 ## Setup (one-time, after first deploy)
 ### 1. Encrypt secrets before committing
 ```bash
 # Edit secrets.yaml with real values first, then:
 sops -e -i k8s/apps/gotify/secrets.yaml
 ```
 ### 2. Create Gotify app + client tokens
 1. Login to `https://gotify.riotpiao.com` with admin creds
 2. **Applications** → Create `forgejo` → copy **app token**
 3. **Clients** → Create `smtp-emailer` → copy **client token**
 4. Update `gotify-tokens` secret:
   ```bash
   kubectl -n notifications create secret generic gotify-tokens \
     --from-literal=app-token=<APP_TOKEN> \
     --from-literal=client-token=<CLIENT_TOKEN> \
     --dry-run=client -o yaml | kubectl apply -f -
   ```
 ### 3. Configure Forgejo webhook
 In each Forgejo repo → **Settings** → **Webhooks** → **Add Webhook** → **Gotify**:
 | Field | Value |
 |-------|-------|
 | Target URL | `http://gotify.notifications.svc.cluster.local/message` |
 | Token | The **app token** from step 2 |
 | Events | Pull Request (Created, Merged, Closed) |
 Or via API:
 ```bash
 FORGEJO_TOKEN="<your-pat>"
 APP_TOKEN="<gotify-app-token>"
 curl -s -X POST "https://forgejo.riotpiao.com/api/v1/repos/rock/homelab/hooks" \
  -H "Authorization: token $FORGEJO_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "gotify",
    "active": true,
    "config": {
      "content_type": "json",
      "url": "http://gotify.notifications.svc.cluster.local/message?token='"$APP_TOKEN"'"
    },
    "events": ["pull_request", "pull_request_assign", "pull_request_review"],
    "authorization_header": ""
  }'
 ```
 ### 4. Add CoreDNS rewrite (if accessing via public hostname)
 Only needed if Cloudflare Tunnel is used for gotify.riotpiao.com:
 ```
 # terraform/files/coredns/Corefile — add rewrite:
 rewrite name gotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
 ```
 Then: `cd terraform && terraform apply && cd .. && make apply-cp`
 ### 5. SMTP providers
 | Provider | Host | Port | Notes |
 |----------|------|------|-------|
 | Gmail | smtp.gmail.com | 587 | Use App Password (2FA required) |
 | Resend | smtp.resend.com | 587 | Free 100 emails/day |
 | Sendgrid | smtp.sendgrid.net | 587 | Free 100 emails/day |
 | Mailgun | smtp.mailgun.org | 587 | Free 5000/month |
 ## Notification priority levels
 | Priority | Meaning | Email forwarded? |
 |----------|---------|-----------------|
 | 0-4 | Low (info) | No (below MIN_PRIORITY=5) |
 | 5-7 | Normal (PR created) | Yes |
 | 8-10 | High (PR merged, failures) | Yes |
 ## Verify
 ```bash
 # Test push notification
 APP_TOKEN="<app-token>"
 curl -X POST "https://gotify.riotpiao.com/message?token=$APP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title":"Test","message":"Hello from homelab","priority":5}'
 # Check email sidecar logs
 kubectl -n notifications logs deployment/gotify -c smtp-emailer --tail=20
 ```
@@ -0,0 +1,35 @@
 # CNPG Postgres for Gotify. Lightweight — 2 instances, 2Gi storage.
 # CNPG generates secret `gotify-db-app` + service `gotify-db-rw` in ns notifications.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
  name: gotify-db
  namespace: notifications
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
 spec:
  instances: 2
  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
  bootstrap:
    initdb:
      database: gotify
      owner: app
      encoding: UTF8
      localeCollate: C
      localeCType: C
  enableSuperuserAccess: false
  resources:
    requests: { memory: "256Mi", cpu: "100m" }
    limits: { memory: "512Mi", cpu: "500m" }
  storage:
    size: 2Gi
    storageClass: longhorn-cnpg
  monitoring:
    enablePodMonitor: true
  affinity:
    podAntiAffinityType: preferred
    topologyKey: kubernetes.io/hostname
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
@@ -0,0 +1,204 @@
 # Gotify — self-hosted push notification server + SMTP email relay.
 # Forgejo webhooks → Gotify → push notifications + email forwarding.
 # Runs on control plane (no GPU needed), lightweight.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: gotify
  namespace: notifications
  labels:
    app: gotify
 spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: gotify
  template:
    metadata:
      labels:
        app: gotify
    spec:
      containers:
        # --- Gotify server ---
        - name: gotify
          image: ghcr.io/gotify/server:2.6.1
          command: ["/bin/sh", "-c"]
          args:
            - |
              export GOTIFY_DATABASE_DIALECT=postgres
              export GOTIFY_DATABASE_CONNECTION="host=gotify-db-rw.notifications port=5432 user=${DB_USER} password=${DB_PASS} dbname=gotify sslmode=disable"
              exec /app/gotify-app
          ports:
            - containerPort: 80
              protocol: TCP
          env:
            - name: GOTIFY_DEFAULTUSER_NAME
              valueFrom:
                secretKeyRef:
                  name: gotify-admin
                  key: username
            - name: GOTIFY_DEFAULTUSER_PASS
              valueFrom:
                secretKeyRef:
                  name: gotify-admin
                  key: password
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: gotify-db-app
                  key: username
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: gotify-db-app
                  key: password
            - name: GOTIFY_SERVER_PORT
              value: "80"
            - name: GOTIFY_SERVER_KEEPALIVEPERIODSECONDS
              value: "0"
            - name: TZ
              value: Asia/Tokyo
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi
          livenessProbe:
            httpGet:
              path: /health
              port: 80
            periodSeconds: 30
            initialDelaySeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 80
            periodSeconds: 10
            initialDelaySeconds: 5
        # --- SMTP emailer sidecar ---
        # Watches Gotify WebSocket stream, forwards messages as email.
        # https://github.com/eternal-flame-AD/gotify-broadcast
        - name: smtp-emailer
          image: ghcr.io/gotify/server:2.6.1
          command:
            - /bin/sh
            - -c
            - |
              # Wait for Gotify to be ready
              until wget -qO- http://localhost:80/health >/dev/null 2>&1; do
                echo "Waiting for Gotify..."
                sleep 2
              done
              echo "Gotify is ready, starting email relay..."
              # Poll Gotify messages and forward via SMTP using msmtp
              # Install msmtp for lightweight SMTP sending
              apk add --no-cache msmtp curl jq
              # Configure msmtp
              cat > /tmp/msmtprc <<MSMTP
              defaults
              auth           on
              tls            on
              tls_trust_file /etc/ssl/certs/ca-certificates.crt
              logfile        /tmp/msmtp.log
              account        default
              host           ${SMTP_HOST}
              port           ${SMTP_PORT}
              from           ${SMTP_FROM}
              user           ${SMTP_USER}
              password       ${SMTP_PASS}
              MSMTP
              chmod 600 /tmp/msmtprc
              # Track last seen message ID
              LAST_ID=0
              while true; do
                # Fetch messages since last ID
                MESSAGES=$(curl -s -H "X-Gotify-Key: ${GOTIFY_CLIENT_TOKEN}" \
                  "http://localhost:80/message?since=${LAST_ID}&limit=10" 2>/dev/null)
                if [ -n "$MESSAGES" ]; then
                  echo "$MESSAGES" | jq -r '.messages[]? | @base64' | while read -r MSG; do
                    DECODED=$(echo "$MSG" | base64 -d)
                    ID=$(echo "$DECODED" | jq -r '.id')
                    TITLE=$(echo "$DECODED" | jq -r '.title // "Notification"')
                    BODY=$(echo "$DECODED" | jq -r '.message // ""')
                    PRIORITY=$(echo "$DECODED" | jq -r '.priority // 5')
                    APP=$(echo "$DECODED" | jq -r '.appid // 0')
                    DATE=$(echo "$DECODED" | jq -r '.date // ""')
                    # Only forward messages with priority >= configured threshold
                    if [ "$PRIORITY" -ge "${MIN_PRIORITY:-0}" ]; then
                      printf "Subject: [Gotify] %s\nFrom: %s\nTo: %s\nContent-Type: text/plain; charset=UTF-8\n\n%s\n\n---\nPriority: %s\nDate: %s" \
                        "$TITLE" "$SMTP_FROM" "$NOTIFY_EMAIL" "$BODY" "$PRIORITY" "$DATE" | \
                        msmtp -C /tmp/msmtprc "$NOTIFY_EMAIL" && \
                        echo "Email sent for message $ID: $TITLE" || \
                        echo "Failed to send email for message $ID"
                    fi
                    # Update last seen ID
                    if [ "$ID" -gt "$LAST_ID" ]; then
                      LAST_ID=$ID
                    fi
                  done
                fi
                sleep ${POLL_INTERVAL:-30}
              done
          env:
            - name: GOTIFY_CLIENT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: gotify-tokens
                  key: client-token
            - name: SMTP_HOST
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: host
            - name: SMTP_PORT
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: port
            - name: SMTP_FROM
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: from
            - name: SMTP_USER
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: user
            - name: SMTP_PASS
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: password
            - name: NOTIFY_EMAIL
              valueFrom:
                secretKeyRef:
                  name: gotify-smtp
                  key: notify-email
            - name: MIN_PRIORITY
              value: "5"
            - name: POLL_INTERVAL
              value: "15"
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
            limits:
              cpu: 100m
              memory: 64Mi
      # No volumes — Postgres handles persistence
@@ -0,0 +1,26 @@
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: gotify
  namespace: notifications
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    # WebSocket support for Gotify client connections
    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
 spec:
  ingressClassName: nginx
  rules:
    - host: gotify.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gotify
                port:
                  number: 80
@@ -0,0 +1,8 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
  - namespace.yaml
  - db.yaml
  - deployment.yaml
  - service.yaml
  - ingress.yaml
@@ -0,0 +1,6 @@
 apiVersion: v1
 kind: Namespace
 metadata:
  name: notifications
  labels:
    kubernetes.io/metadata.name: notifications
@@ -0,0 +1,14 @@
 apiVersion: v1
 kind: Service
 metadata:
  name: gotify
  namespace: notifications
  labels:
    app: gotify
 spec:
  selector:
    app: gotify
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
@@ -14,5 +14,7 @@ resources:
  - ornith.yaml
  - reasoning.yaml
  - reranker.yaml
  - qwen-cpu.yaml
  - networkpolicy.yaml
 # No namespace transformer: every file sets its own, and the transformer would
 # rewrite metadata.namespace on anything cross-namespace added later.
@@ -0,0 +1,62 @@
 # NetworkPolicy for LLM inference engines (llm-serving namespace).
 #
 # These pods have NO auth — vLLM, Ollama, and TEI accept any request.
 # All access MUST go through the api-gateway, which validates JWTs and
 # injects identity headers (X-Forwarded-User, X-Auth-Verified).
 #
 # Replaces the hand-applied llm-serving-default-deny policy that used
 # `llm-client: "true"` pod label as a selector — any pod in any namespace
 # could self-grant access by adding that label, which defeats the purpose.
 #
 # This policy restricts ingress to:
 #   1. api namespace (gateway) — the sole entry point for inference
 #   2. monitoring namespace — Prometheus scraping vLLM/TEI /metrics
 #   3. intra-namespace — pod-to-pod (future: multi-replica comms)
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
  name: llm-serving-ingress
  namespace: llm-serving
  labels:
    app.kubernetes.io/part-of: llm-serving
 spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: llm-serving
  policyTypes:
    - Ingress
  ingress:
    # Allow from api-gateway (namespace: api)
    # Gateway proxies /v1/chat/completions, /v1/embeddings, /v1/rerank
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: api
      ports:
        - protocol: TCP
          port: 8080   # vLLM, Ollama HTTP
        - protocol: TCP
          port: 80      # KServe predictor services
        - protocol: TCP
          port: 8000    # vLLM direct (some configs)
        - protocol: TCP
          port: 11434   # Ollama native port
    # Allow Prometheus scraping from monitoring namespace
    # vLLM: :8080/metrics, TEI: :9000/metrics
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - protocol: TCP
          port: 8080
        - protocol: TCP
          port: 9000
    # Allow intra-namespace (pod-to-pod within llm-serving)
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/part-of: llm-serving
      ports:
        - protocol: TCP
          port: 8080
@@ -33,12 +33,8 @@ spec:
        ollama pull ornith:35b
        ollama pull qwen2.5:3b-instruct
        ollama run ornith:35b "ok" >/dev/null 2>&1 || true
        ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
        wait $SERVE_PID
        '
@@ -54,7 +50,7 @@ spec:
      - name: OLLAMA_NUM_PARALLEL
        value: '1'
      - name: OLLAMA_MAX_LOADED_MODELS
-        value: '2'
+        value: '1'
      image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
      name: kserve-container
      ports:
@@ -65,8 +61,7 @@ spec:
          command:
          - /bin/sh
          - -c
-          - ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
+          - ollama ps 2>/dev/null | grep -q ornith
            grep -q qwen2.5
        periodSeconds: 10
      resources:
        limits:
@@ -82,8 +77,7 @@ spec:
          command:
          - /bin/sh
          - -c
-          - ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
+          - ollama ps 2>/dev/null | grep -q ornith
            grep -q qwen2.5
        failureThreshold: 120
        periodSeconds: 15
      volumeMounts:
@@ -91,13 +85,10 @@ spec:
        name: models
    deploymentStrategy:
      type: Recreate
-    # 2 replicas -- each its own GPU, each loading both ornith:35b and
+    # 1 replica -- ornith:35b only. qwen2.5:3b moved to CPU on cp-2.
-    # qwen2.5:3b-instruct -- so 2 concurrent implementer-style calls each
+    # Frees 1 GPU for ComfyUI.
-    # get an independent instance instead of contending on one, at the
+    maxReplicas: 1
-    # cost of judge/qwen traffic still sharing whichever replica an
+    minReplicas: 1
    # implementer call also lands on.
    maxReplicas: 2
    minReplicas: 2
    nodeSelector:
      kubernetes.io/hostname: worker-1
    runtimeClassName: nvidia
@@ -0,0 +1,115 @@
 # qwen2.5:3b-instruct on CPU (talos-cp-2, 144GB RAM, 24 cores).
 # Moved off GPU to free a V100 for ComfyUI. Latency ~10x slower
 # than GPU but sufficient for lightweight tasks (summarization,
 # classification, quick answers).
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: qwen-cpu
  namespace: llm-serving
  labels:
    app: qwen-cpu
    app.kubernetes.io/name: qwen-cpu
    app.kubernetes.io/part-of: llm-serving
 spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: qwen-cpu
  template:
    metadata:
      labels:
        app: qwen-cpu
        app.kubernetes.io/name: qwen-cpu
        app.kubernetes.io/part-of: llm-serving
    spec:
      nodeSelector:
        kubernetes.io/hostname: talos-cp-2
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: ollama
          image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
          command: ["/bin/sh", "-c"]
          args:
            - |
              ollama serve &
              SERVE_PID=$!
              until ollama list >/dev/null 2>&1; do sleep 2; done
              ollama pull qwen2.5:3b-instruct
              ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
              wait $SERVE_PID
          env:
            - name: OLLAMA_HOST
              value: "0.0.0.0:8080"
            - name: OLLAMA_MODELS
              value: /root/.ollama/models
            - name: OLLAMA_CONTEXT_LENGTH
              value: "32768"
            - name: OLLAMA_KEEP_ALIVE
              value: "-1"
            - name: OLLAMA_MAX_LOADED_MODELS
              value: "1"
            - name: OLLAMA_NUM_PARALLEL
              value: "2"
          ports:
            - containerPort: 8080
              protocol: TCP
          readinessProbe:
            exec:
              command: ["/bin/sh", "-c", "ollama ps 2>/dev/null | grep -q qwen2.5"]
            periodSeconds: 10
          startupProbe:
            exec:
              command: ["/bin/sh", "-c", "ollama ps 2>/dev/null | grep -q qwen2.5"]
            failureThreshold: 60
            periodSeconds: 10
          resources:
            requests:
              cpu: "4"
              memory: 4Gi
            limits:
              cpu: "8"
              memory: 8Gi
          volumeMounts:
            - mountPath: /root/.ollama
              name: ollama-data
      volumes:
        - name: ollama-data
          persistentVolumeClaim:
            claimName: qwen-cpu-data
 ---
 apiVersion: v1
 kind: Service
 metadata:
  name: qwen-cpu
  namespace: llm-serving
  labels:
    app: qwen-cpu
    app.kubernetes.io/part-of: llm-serving
 spec:
  selector:
    app: qwen-cpu
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
 ---
 # Small PVC for qwen2.5:3b model weights (~1.9GB).
 # Separate from llm-models PVC which is pinned to worker-1.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: qwen-cpu-data
  namespace: llm-serving
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
@@ -40,7 +40,7 @@ spec:
        # Git
        - name: GIT_REPO
-          value: https://forgejo.riotpiao.com/rock/homelab.git
+          value: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
        - name: GIT_AUTHOR_EMAIL
          value: [email protected]
        - name: GIT_AUTHOR_NAME
@@ -18,7 +18,7 @@ spec:
      prune: true
      selfHeal: true
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/argocd/projects
  destination:
@@ -17,7 +17,7 @@ spec:
    syncOptions:
      - CreateNamespace=true
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    # ksops decrypts every *.enc.yaml here at kustomize-build time (repo-server
    # runs `kustomize build --enable-alpha-plugins --enable-exec`). Replaces the
@@ -21,7 +21,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/bootstrap/cert-manager/cert-manager-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -93,7 +93,7 @@ spec:
  project: homelab
  revisionHistoryLimit: 3
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    # A real kustomization.yaml (resources: the 3 issuer/CA files) renders these
    # deterministically. The previous directory.include with bare filenames
@@ -127,7 +127,7 @@ spec:
  project: homelab
  revisionHistoryLimit: 3
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/bootstrap/ingress
  destination:
@@ -149,7 +149,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/cluster-maintenance
  destination:
@@ -177,7 +177,7 @@ spec:
      valueFiles:
        - $values/k8s/bootstrap/kyverno/kyverno-values.yaml
  sources:
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -200,7 +200,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/bootstrap/kyverno
  destination:
@@ -19,7 +19,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/argocd-image-updater/values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -9,7 +9,7 @@ spec:
  project: homelab
  sources:
-  - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+  - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    path: k8s/apps/secret-rotation-controller
    targetRevision: main
@@ -17,7 +17,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/minio/minio-operator-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -41,7 +41,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/minio
  destination:
@@ -66,7 +66,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/longhorn
  destination:
@@ -102,7 +102,7 @@ spec:
        skipCrds: true
        valueFiles:
          - $values/k8s/infra/monitoring/prometheus-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -152,7 +152,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/monitoring/crds
  destination:
@@ -183,7 +183,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/monitoring
  destination:
@@ -213,7 +213,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/monitoring/blackbox-exporter-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -237,7 +237,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/tracing
  destination:
@@ -19,7 +19,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/logging/loki-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -53,7 +53,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/logging/grafana-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -87,7 +87,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/logging/promtail-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -17,7 +17,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/iam/vault-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -46,7 +46,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/infra/iam/authentik-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -68,7 +68,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/iam
  destination:
@@ -109,7 +109,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -162,7 +162,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/forgejo-runner
  destination:
@@ -190,7 +190,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/forgejo-runner
    helm:
@@ -221,7 +221,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/forgejo-runner
    helm:
@@ -14,7 +14,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/databases
  destination:
@@ -8,7 +8,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/messaging/memory-queues
  destination:
@@ -68,7 +68,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/messaging/kafka-cluster
  destination:
@@ -38,17 +38,17 @@ metadata:
    argocd.argoproj.io/sync-wave: "7"
    # ArgoCD Image Updater - auto-update on new image push
    argocd-image-updater.argoproj.io/image-list: gw=forgejo.riotpiao.com/rock/api-gateway
-    argocd-image-updater.argoproj.io/gw.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/gw.update-strategy: digest
-    argocd-image-updater.argoproj.io/gw.allow-tags: regexp:^[0-9a-f]{7}$
+    argocd-image-updater.argoproj.io/gw.allow-tags: regexp:^latest$
    argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
  project: homelab
  revisionHistoryLimit: 3
  sources:
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab-frontend.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
      targetRevision: main
      path: k8s
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      path: k8s/apps/api
  destination:
@@ -20,7 +20,7 @@ spec:
  project: homelab
  revisionHistoryLimit: 3
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/llm-serving
  destination:
@@ -0,0 +1,32 @@
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: comfyui
  namespace: argocd
  labels:
    app.kubernetes.io/name: comfyui
    app.kubernetes.io/component: image-generation
  annotations:
    argocd.argoproj.io/sync-wave: "8"
 spec:
  project: homelab
  revisionHistoryLimit: 3
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/comfyui
  destination:
    server: https://kubernetes.default.svc
    namespace: comfyui
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
@@ -0,0 +1,32 @@
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: gotify
  namespace: argocd
  labels:
    app.kubernetes.io/name: gotify
    app.kubernetes.io/component: notifications
  annotations:
    argocd.argoproj.io/sync-wave: "8"
 spec:
  project: homelab
  revisionHistoryLimit: 3
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/gotify
  destination:
    server: https://kubernetes.default.svc
    namespace: notifications
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
@@ -19,10 +19,10 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/apps/temporal/temporal-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      path: k8s/apps/temporal
  destination:
@@ -51,7 +51,7 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/apps/portainer/portainer-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
@@ -74,7 +74,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/cloudflared
  destination:
@@ -97,7 +97,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/agent-pod
  destination:
@@ -130,7 +130,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/sms
  destination:
@@ -157,7 +157,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/paperless
  destination:
@@ -189,7 +189,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/immich
  destination:
@@ -220,10 +220,10 @@ spec:
      helm:
        valueFiles:
          - $values/k8s/apps/homarr/homarr-values.yaml
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
-    - repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      path: k8s/apps/homarr  # PostSync hook: fix-probes-job.yaml
  destination:
@@ -248,8 +248,8 @@ metadata:
    argocd.argoproj.io/sync-wave: "8"
    # ArgoCD Image Updater - auto-update on new image push
    argocd-image-updater.argoproj.io/image-list: app=forgejo.riotpiao.com/rock/portfolio
-    argocd-image-updater.argoproj.io/app.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/app.update-strategy: digest
-    argocd-image-updater.argoproj.io/app.allow-tags: regexp:^[0-9a-f]{7}$
+    argocd-image-updater.argoproj.io/app.allow-tags: regexp:^latest$
    argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
  project: homelab
@@ -281,7 +281,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/rbac
  destination:
@@ -12,7 +12,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/kmsvc-manage.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/kmsvc-manage.git
    targetRevision: main
    path: k8s/argocd/apps
    directory:
@@ -10,24 +10,23 @@ metadata:
      memory=forgejo.riotpiao.com/rock/poimen-memory
      workflows=forgejo.riotpiao.com/rock/poimen-workflows
      frontend=forgejo.riotpiao.com/rock/poimen-frontend
-    argocd-image-updater.argoproj.io/memory.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/memory.update-strategy: digest
-    argocd-image-updater.argoproj.io/memory.allow-tags: regexp:^[0-9a-f]{7}$
+    argocd-image-updater.argoproj.io/memory.allow-tags: regexp:^latest$
-    argocd-image-updater.argoproj.io/workflows.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/workflows.update-strategy: digest
-    argocd-image-updater.argoproj.io/workflows.allow-tags: regexp:^[0-9a-f]{7}$
+    argocd-image-updater.argoproj.io/workflows.allow-tags: regexp:^latest$
-    argocd-image-updater.argoproj.io/frontend.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/frontend.update-strategy: digest
-    argocd-image-updater.argoproj.io/frontend.allow-tags: regexp:^[0-9a-f]{7}$
+    argocd-image-updater.argoproj.io/frontend.allow-tags: regexp:^latest$
-    argocd-image-updater.argoproj.io/write-back-method: git
+    argocd-image-updater.argoproj.io/write-back-method: argocd
    argocd-image-updater.argoproj.io/git-branch: main
 spec:
  project: homelab
  sources:
-    - repoURL: https://forgejo.riotpiao.com/rock/poimen-memory.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-memory.git
      targetRevision: main
      path: k8s/argocd
-    - repoURL: https://forgejo.riotpiao.com/rock/poimen-workflows.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-workflows.git
      targetRevision: main
      path: k8s/argocd
-    - repoURL: https://forgejo.riotpiao.com/rock/poimen-frontend.git
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-frontend.git
      targetRevision: main
      path: k8s/argocd
  destination:
@@ -17,12 +17,13 @@ spec:
    - https://github.com/Riotpiaole/Poimen-workflows.git
    - https://github.com/Riotpiaole/poimen*.git
    # In-cluster Forgejo repos — explicit allowlist (no wildcard)
-    - https://forgejo.riotpiao.com/rock/homelab.git
+    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    - https://forgejo.riotpiao.com/rock/homelab-frontend.git
+    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
-    - https://forgejo.riotpiao.com/rock/kmsvc-manage.git
+    - https://forgejo.riotpiao.com/riotpiao-poimen/kmsvc-manage.git
-    - https://forgejo.riotpiao.com/rock/poimen.git
+    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen.git
-    - https://forgejo.riotpiao.com/rock/poimen-memory.git
+    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen-memory.git
-    - https://forgejo.riotpiao.com/rock/poimen-workflows.git
+    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen-workflows.git
    - https://forgejo.riotpiao.com/riotpiao-poimen/poimen-frontend.git
    - https://forgejo.riotpiao.com/rock/riotpiao.com.git
    # Public Helm chart repos referenced by k8s/argocd/apps/* and bootstrap/*
    - https://cloudnative-pg.github.io/charts
@@ -12,7 +12,7 @@ metadata:
 spec:
  project: homelab
  source:
-    repoURL: https://forgejo.riotpiao.com/rock/homelab.git
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/argocd/apps
    directory:
@@ -0,0 +1,24 @@
 apiVersion: ENC[AES256_GCM,data:wR4=,iv:cRBzbvu0eUYCYeKeysua1/P3Meli/rQyj/mIV6VWnPM=,tag:oyTPA/mLYNUX+QDkYLlZyQ==,type:str]
 kind: ENC[AES256_GCM,data:bdKQdadW,iv:F3DQiBI9xhSx87jkS1Hyevo48uUCBjsJSjbScIBznKA=,tag:PFakzzBj5S9F65dxu0L2rg==,type:str]
 metadata:
    name: ENC[AES256_GCM,data:XHFYrKClPo+IwRbM,iv:ZGuL+cN840Y1bOTC62NhDDcoZpTzDWQ4GfqPJX/QWmI=,tag:hlCVjzvfcxXGOASc3vda/Q==,type:str]
    namespace: ENC[AES256_GCM,data:0BbhgZBog+1qY/iRJA==,iv:88tiSpEDqIokT5VP/d6bB2+aUyh1kZ7NEHwZWoJW3XU=,tag:CBR01jh2U9h7kNEcK1e1sA==,type:str]
 type: ENC[AES256_GCM,data:Mpv1V73w,iv:BW3r6RpLnwe3XDKwrZySyOjrWUnSwIG5JOoPLzP/5gM=,tag:oyJySL0haOei1m4i+1AJ1g==,type:str]
 stringData:
    username: ENC[AES256_GCM,data:8xmxLGE=,iv:J/vEvXGoD+ka6FDgnSwDz0fs9IxuJIZW9r7Oyu2qxC8=,tag:dN9jd6NSavMN8+uzvemYWg==,type:str]
    password: ENC[AES256_GCM,data:vB9PaaUiQZ8FY8pO0cq1fHLi7Gq5t4U=,iv:C0Y1m2SGYP3oTIFoau5JavVmLblj6te/SPMLodIwiZw=,tag:3Ip4YvR4WPzR0bBpTyjytA==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRT3JoTnhVVW5SUUJXbTdz
            dUpLcmtBWWhyaGk3Y1hBM1ArcVI5eHREbmp3CiswUGVmd0NhejZwQ2UvNEdxS3ow
            L1RweS9Kb2paeStLZ0tLSFdWbVZqQW8KLS0tIHJHT3hiMmxtM0Q1Ym5KOVpLVFpl
            enR3NmdNdmdwVitTQVJlRHFWcjR0N2sKQw9ZZs+Ji/Zq/feO3qy4DwaCfWgDOQ/z
            FVVhcCXweN58tb+9fzCJ+pNi/hSmvUkCMbb1+60qBvEehNzOoMRJ5g==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    lastmodified: "2026-09-10T13:31:00Z"
    mac: ENC[AES256_GCM,data:3HWV/NG0yTbsxY28u41zTRmAPc1kokGb4nCLmAsyq8/uvxcP+evDNyZXYpS93eVdPRfRZ1OqVBzyVGJEnj7JSXQVmlzor9JcWEL2fcpogoLVfJZKTRD6hk6optnBMjj84iQEEOx3A80JrDOdoXsH0pd9bJBABwoUOJwuufbXcIU=,iv:KLZsmAcapQgV08pFhGIvPt5ylsWg3xazAAjPuJYOaXA=,tag:1vKA3fISMIurHzxZ04F3lg==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -0,0 +1,30 @@
 apiVersion: ENC[AES256_GCM,data:xUA=,iv:YH1WYnoXhxbeES3Dmo8ZrMDwFcwCykt6+Zn4YSgN4gc=,tag:t3bIWo/LetlQv4tJ0nJ2Sw==,type:str]
 kind: ENC[AES256_GCM,data:PJVzDTcT,iv:aRxvzLUuCzSDryOjA/v67csL6E2FcCpoK8aD7KNStno=,tag:cQgbowSTTV99nQ41yFOlsA==,type:str]
 metadata:
    name: ENC[AES256_GCM,data:atkSiP0rjyuWFjs=,iv:l/48X9W9lrkAyw6Q/p39V775L6xbr0tOIFI4vjYrg+E=,tag:LZrQxN5h6T5tehFG8L9KFw==,type:str]
    namespace: ENC[AES256_GCM,data:fbklNyooBeqsogKbeg==,iv:z2wLgl81q8xfzGGVQ9A4Pkst/zLmuZpyBzHrXzmVfm4=,tag:QKJik4IWc4nrOBINjGCxvQ==,type:str]
 type: ENC[AES256_GCM,data:bfAFBd/7,iv:9/W3X1KefqDAsurIatT6kBuEIQvWVRTq4G9A+9ZfYKs=,tag:Optlb+gZ2udbqIOerqV2zQ==,type:str]
 stringData:
    #ENC[AES256_GCM,data:RZaw/FNQZOf0F3+XfFmGW+4gjKXNdjrAbMRjBt4hptwk,iv:IEd/+Bsv/h08nKak+bcovNiohC7nNjEQ+QdtIFMUGng=,tag:zsF5ZQ3+5JijF4MU+RhnyA==,type:comment]
    #ENC[AES256_GCM,data:ux3ONHnuI0uFEihElbLI1rKlrN91Pn0iLLZfmez0+t5d8yhH6OD3L/3oi6yHbdgD8bV90kRK/lw=,iv:7Ed+W/TeOY9Af8bKO0Fu3XOuaK8b0Hw70rwobfmosxM=,tag:Z0RbIOjdDroK0drC6P+Y8A==,type:comment]
    host: ENC[AES256_GCM,data:YKgas/8hT5ZOGIW7O0c=,iv:WtBz1MFrwd6EgDDS5fMqtcVGIZGNLgbXnmkqlNWoXy8=,tag:+H9ARGPi5XocgZYp4oA3wA==,type:str]
    port: ENC[AES256_GCM,data:f0hx,iv:HkSnJcCyzyfBTA39BNw+muhHTyrNLzzGxZ7M3bcLVu4=,tag:4+xL1xNyYPaV2b0mwnbQVQ==,type:str]
    from: ENC[AES256_GCM,data:lqYqXxJuI4CAWPlX+5cltCdxqA==,iv:ZgZYySJXvjuTNIkMaDxynUJ6PFdp4fibqlgoA1vNDKE=,tag:vb/XKl2N1Hl5u7ndZXQvbg==,type:str]
    user: ENC[AES256_GCM,data:ajljGVbRO//G,iv:E4Q0LGDCd5wnHc+Oq2A9ZhWGbCmkKlhllt922UmppIg=,tag:Cf4ft1JKFi9PaJEfPfq92Q==,type:str]
    password: ENC[AES256_GCM,data:YNbtoiWLsQGl,iv:FJ3y1lWkqKkM8GVr7FZljAXJ331GkDgy4TenAbdIY3w=,tag:KIvhv+RL49ZMwWhYrU0SPA==,type:str]
    notify-email: ENC[AES256_GCM,data:s5wRkDuaiflCyu4IyAnaocVPA3I=,iv:dK5PsweYa3JotYcDcz74DPNDXmhW5ngNEbY6JuG1ctg=,tag:/Dt2UoCtfau6qv0G7VzmrQ==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLQlJVekcwb01ESnhxSE9m
            MzliVmlaYUttZVdiMDVtMlQ2L29LMXBWRzBZCkxRMi9RcE5LMzQ1MHFqUUxLaFZo
            MUZ2UTg5eUNsMFVmenlhL2F3NDd0akkKLS0tIGFwbUUrSGFUTDFVOWlxM3V1bVdy
            RUVVNDFMRHZCSWptdSt1bFVoN2JPWVkKXMlrJNPbm7Sj9vd/H3o6iqPRLy4LyjVe
            xJyh9vn9RCwS9Jr6BN2Q9P2UZfTGwvIYc6zdmV+b2uRwvi4t4lMsBA==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    lastmodified: "2026-09-10T13:31:00Z"
    mac: ENC[AES256_GCM,data:g1YjOANlnvXErPqtDdDjRB9yTUiGEhX8QDkmf1aLfQBwoXbGUpoPIZibF8A4cAL1yUSZ2rD4l9TIe9nK8U3oF4WiKXf9u0bFK2g//bOv2A5JPewwmzoazkjIDXcJ/76nOQ5c+v8/r5tUG4c0SyzBH/gFP0D0+P5Yo3SEXjTxj2w=,iv:gW+AUwwAWkXZ+aLBXjjNDZw8VEraMpSE16RJGQc/L5g=,tag:DM/DUtil/7yLhfGYQXKFGA==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -0,0 +1,27 @@
 apiVersion: ENC[AES256_GCM,data:YE0=,iv:s3yNqcBn7/DKUQNgbGGwVmlM1HzxYGLBKyB6Y2ii2WE=,tag:77KxAjOFU3G0fSfrgudKkg==,type:str]
 kind: ENC[AES256_GCM,data:PFW4VV9T,iv:n8gzMjE5C2TAfUTpp/8ex64B+hWUGG1K+2oQ4deN03Q=,tag:18Xqb6Di5izMHmzR5EU+QA==,type:str]
 metadata:
    name: ENC[AES256_GCM,data:Xhg1fQrD4itrbvDW0g==,iv:/THWSXAThb9fj+mm3wcxqzSdzdt7nE06+xOpkCxyImE=,tag:UChokr06VkbDZcFZDcUY2A==,type:str]
    namespace: ENC[AES256_GCM,data:r25U5MuuzDd8JJ2YjQ==,iv:uDUcS4ZTpZe8HmZMArzkdu7LV5GUoLSP2wIs5HB1gv0=,tag:fci7j30hoxnVKJwPFNfd1Q==,type:str]
 type: ENC[AES256_GCM,data:agEG8Fu8,iv:ckYX0buX8md1CWHXfxbAXQJLzoTxTJI16nlQ9LSzYi0=,tag:4tZWf3REbGOmmBiT14+dew==,type:str]
 stringData:
    #ENC[AES256_GCM,data:ehjcsmz1R0i/n31f8M0IIUT4KDBwzz6f5ds=,iv:j5swFAKpC67ZvGioiCCC1D651LxUl9gME8GqK5Uc+ys=,tag:BUJ8k9LHs8NAPPZAwPuifA==,type:comment]
    #ENC[AES256_GCM,data:ndAB00yun636VHDMonqTghO/jCWodNd/hmdbm1QmCvOAi4FvTLl8bY3wYR+ZtlkSQYvFNqH798MJixv5OZwxBj5H,iv:pS+7B60jaS5jhqDRw2LbBFv7mD1HO6+hjjv+iNpenXc=,tag:NU6+gxCHTGxqeDMfvkwJWw==,type:comment]
    #ENC[AES256_GCM,data:T3mhSm/5q7JTSXNLrjhpP5GhqA7nXz8uAhJcEV1RDctAX0Dyfq8Qn4bNFO51ibwTETx4SnunPuFZVs++AvRnsA==,iv:oRGM8N4iEcwBrMzoEXQAeKqCKzUq+jXTMVq9W2l6oh4=,tag:GrrbC/tkWpA5kp5UYCyRBA==,type:comment]
    app-token: ENC[AES256_GCM,data:jw+AqbsxyoYRvNrUDrq+GNq/TNfweFCs,iv:67vSSgMZCMV0GG37ZxUxHAWZqOOGMYCmo3J43WgLyNk=,tag:xlrnip4XomXBbMmooW9FKg==,type:str]
    client-token: ENC[AES256_GCM,data:gsxeCqKh4e+DFjpn9jM5GfemAdBf8dSv,iv:l2bBMdxQUil8jU9XDjXGn3EtvFVrK5ibXDCeGaPbYTY=,tag:ELf5S6cdNQJ84Es5upu1IQ==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAza3YrRW1VTVNSOGlMK21a
            bDhzQlEyZDNnZTZrSVNnTUEzU1hSdnM1ZFVzCjdBSUlmWG5EcnlzbSs5bXdDaGkx
            ME1nMnRqQzF1Vkc3b3FXSUVHT1g0ZDgKLS0tIHpMUytEMjdLQ2E0Unp2di9KS3JQ
            eHBraDk1clJyanhLY2dGM0tESmJIUFkKHTl3y9uQiEofOFD8j2vH3YK/CVzlq11w
            GfShIji1yCvvowKGzYYhsQK0UM0FzhzBv0GFMYWQCBq8pGdoPVmO5g==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    lastmodified: "2026-09-10T13:31:00Z"
    mac: ENC[AES256_GCM,data:JhnO0V6cd2QVY/VhwHAJ5J75GeVlOVHBfVBTZstkj/IvpkAcwS/JE2b6jXiCwEC4n/vdA8DJ074noysUBRxh4Y7O4NKuvWcTniQKgqULNL1Hzgj3NdUkcQlWT+Z0HQ7FlvFH97VVXVfasU+CLHG48ShT2fDSj8JusEllb9CwSvg=,iv:PZo2krxvJc1TLJbvx+S9ClthoBe4w7WigUkq7dg0gNk=,tag:2IF5g/WTRP4XdnCZzDbiIA==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -27,3 +27,6 @@ files:
  - vault-secrets.enc.yaml
  - vault-unseal-keys.enc.yaml
  - portfolio-secrets.enc.yaml
  - gotify-admin-secrets.enc.yaml
  - gotify-tokens-secrets.enc.yaml
  - gotify-smtp-secrets.enc.yaml
@@ -1,16 +0,0 @@
 FROM code.forgejo.org/forgejo/runner:6
 # Switch to root to install packages (Alpine)
 USER root
 # Alpine uses apk, not apt-get
 RUN apk update && apk add --no-cache \
      nodejs \
      npm \
      docker-cli
 # Verify installations
 RUN docker --version && node --version && git --version
 # Switch back to runner user
 USER 1000:1000
@@ -1,16 +0,0 @@
 FROM code.forgejo.org/forgejo/runner:6
 # Switch to root to install packages (Alpine)
 USER root
 # Alpine uses apk, not apt-get
 RUN apk update && apk add --no-cache \
      docker-cli \
      nodejs \
      npm
 # Verify installations
 RUN node --version && docker --version && git --version
 # Switch back to runner user
 USER 1000:1000
@@ -1,20 +0,0 @@
 FROM code.forgejo.org/forgejo/runner:6
 # Switch to root to install packages (Alpine)
 USER root
 # Alpine uses apk, not apt-get
 RUN apk update && apk add --no-cache \
      nodejs \
      npm \
      curl \
      docker-cli
 # Install Rust (as root, skip verification for now)
 RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable || true
 # Verify core installations
 RUN docker --version && node --version && git --version
 # Switch back to runner user
 USER 1000:1000
@@ -36,3 +36,4 @@ data:
      valid_volumes:
        - /docker-certs/client
      network: host
      docker_host: automount
@@ -34,7 +34,11 @@ spec:
          command: ["sh", "-c"]
          args:
            - |
-              test -f /data/.runner || forgejo-runner register --no-interactive \
+              # Always re-register to keep labels in sync with values.yaml.
              # Without this, changing a runner label requires manually deleting
              # the PVC or .runner file — not GitOps-friendly.
              rm -f /data/.runner
              forgejo-runner register --no-interactive \
                --instance {{ .Values.runner.forgejoUrl }} \
                --token $(RUNNER_TOKEN) \
                --name {{ .Values.runner.name }} \
@@ -56,20 +60,18 @@ spec:
      containers:
        - name: runner
          image: {{ .Values.runner.image.repository }}:{{ .Values.runner.image.tag }}
-          command: ["sh", "-c", "forgejo-runner daemon --config /etc/forgejo-runner/config.yaml"]
+          command: ["sh", "-c", "while ! wget -q -O- http://localhost:2375/_ping >/dev/null 2>&1; do echo 'waiting for dind...'; sleep 2; done; echo 'dind ready'; forgejo-runner daemon --config /etc/forgejo-runner/config.yaml"]
          workingDir: /data
          env:
            - name: DOCKER_HOST
-              value: tcp://localhost:2376
+              value: tcp://localhost:2375
            - name: DOCKER_TLS_VERIFY
              value: "1"
            - name: DOCKER_CERT_PATH
              value: /docker-certs/client
          volumeMounts:
            - name: runner-data
              mountPath: /data
            - name: docker-certs
              mountPath: /docker-certs
            - name: docker-sock
              mountPath: /run
            - name: homelab-ca
              mountPath: /etc/ssl/certs/homelab-ca.pem
              subPath: ca.crt
@@ -85,10 +87,12 @@ spec:
            privileged: true    # required for DinD; cicd namespace is labelled privileged
          env:
            - name: DOCKER_TLS_CERTDIR
-              value: /docker-certs
+              value: ""
          volumeMounts:
            - name: docker-certs
              mountPath: /docker-certs
            - name: docker-sock
              mountPath: /run
            - name: dind-storage
              mountPath: /var/lib/docker
            - name: homelab-ca
@@ -113,6 +117,8 @@ spec:
            claimName: {{ .Release.Name }}-dind
        - name: docker-certs
          emptyDir: {}    # DinD regenerates mTLS certs on each start
        - name: docker-sock
          emptyDir: {}    # Shared docker socket between dind and runner
        - name: homelab-ca
          # homelab-ca is a ConfigMap (public CA trust bundle), not a Secret.
          # The volumeMounts use subPath: ca.crt to project the single cert file.
@@ -2,15 +2,14 @@
 # runner instance. Only runner.name and runner.labels differ -- everything
 # else (image, dind, persistence, tolerations, nodeSelector) is shared.
 #
-# node:22-bookworm ships Node natively. Docker client installed via workflow step if needed.
+# Label image: node:22-bookworm — Debian, root, apt-get, Node.js, npm, git.
-# (homelab has no CI; custom runner images built manually if desired)
+# Install docker in workflow steps as needed.
 # Bootstrap with runner image (already has Node.js), CI builds custom
 runner:
  image:
    repository: code.forgejo.org/forgejo/runner
    tag: "6"
  name: node-runner
-  labels: "node:docker://code.forgejo.org/forgejo/runner:6"
+  labels: "node:docker://node:22-bookworm"
 # GC CronJob renders only from the default (golang) values to avoid duplicates
 gc:
@@ -2,13 +2,14 @@
 # runner instance. Only runner.name and runner.labels differ -- everything
 # else (image, dind, persistence, tolerations, nodeSelector) is shared.
 #
-# Bootstrap with runner image, CI builds custom with Node.js+Rust
+# Label image: rust:1-bookworm — Debian, root, apt-get, Rust, cargo, git.
 # Install Node.js/docker in workflow steps as needed.
 runner:
  image:
    repository: code.forgejo.org/forgejo/runner
    tag: "6"
  name: rust-runner
-  labels: "rust:docker://code.forgejo.org/forgejo/runner:6"
+  labels: "rust:docker://rust:1-bookworm"
@@ -1,9 +1,12 @@
 runner:
  image:
    repository: code.forgejo.org/forgejo/runner
-    tag: "6"  # Bootstrap with runner image, CI builds custom with Node.js
+    tag: "6"
  name: golang-runner
-  labels: "golang:docker://code.forgejo.org/forgejo/runner:6"
+  # Label image is what workflow steps run in (NOT the runner daemon image).
  # golang:1.26-bookworm: Debian, root, apt-get, Go, git.
  # TODO: Switch to custom image once build-runner-images.yml pushes images
  labels: "golang:docker://golang:1.26-bookworm"
  forgejoUrl: http://forgejo-gitea-http.cicd.svc.cluster.local:3000
  tokenSecret: runner-token
  resources:
@@ -36,6 +36,7 @@
    rewrite name paperless.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name img.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name api.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name comfy.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    kubernetes cluster.local in-addr.arpa ip6.arpa {
Author	SHA1	Message	Date
rock	9d3a669dfe	fix(gotify): move SOPS secrets to ksops generator ArgoCD couldn't decrypt secrets.yaml because it was listed as a plain kustomize resource. Move the 3 secrets (gotify-admin, gotify-tokens, gotify-smtp) to k8s/argocd/secrets/ as .enc.yaml files processed by the ksops generator, matching the repo convention. Fixes ComparisonError: 'Object Kind is missing' (SOPS ciphertext parsed as raw YAML).	2026-09-10 22:32:24 +09:00
rock	436c7d8d42	fix(argocd): update git repoURLs for org transfer rock -> riotpiao-poimen Repos transferred: homelab-frontend, kmsvc-manage, poimen, poimen-memory, poimen-workflows, poimen-frontend. Old URLs return 301 which ArgoCD doesn't follow. NOT changed: container image registry paths (rock/ is correct for registry), riotpiao.com (still under rock org). Also adds poimen-frontend to AppProject sourceRepos allowlist.	2026-09-10 10:49:30 +09:00
rock	b571d518e0	fix(argocd): update repoURL after org transfer (#29 ) Co-authored-by: rock <[email protected]>	2026-09-10 01:41:37 +00:00
rock	128e76ce2d	feat(gotify): push notification server + SMTP email relay (#19 ) Co-authored-by: rock <[email protected]>	2026-09-10 01:35:59 +00:00
rockandpoimen	5b16b882be	fix(comfyui): port 8888, emptyDir storage, liveness probe (#18 ) - Change container port from 8188 to 8888 (Caddy proxy binding) - Remove PVC, use emptyDir for ephemeral models/output - Replace startup + readiness probes with single liveness probe - Remove explicit COMFYUI_FLAGS (container defaults work) - Pod now reaches Ready state immediately after image pull Fixes GPU contention by using ephemeral storage. ComfyUI now runs and is accessible at https://comfy.riotpiao.comReviewed-on: #18 Co-authored-by: poimen <[email protected]> Reviewed-on: riotpiao-poimen/homelab-frontend#21 Co-authored-by: rock <[email protected]>	2026-09-09 22:38:25 +00:00
rock	8790de6038	feat: add ComfyUI + rebalance GPU allocation (#17 ) ## GPU Rebalance (4× V100 32GB) \| Pod \| Before \| After \| \|-----\|--------\|-------\| \| reasoning (PP=2) \| 2 GPU \| 2 GPU \| \| ornith \| 2 GPU (2 replicas) \| 1 GPU (1 replica) \| \| comfyui \| — \| 1 GPU (new) \| \| qwen-cpu \| — \| CPU on cp-2 (new) \| \| embeddings/reranker \| CPU \| CPU \| ## Changes - `ornith.yaml`: scale 2→1, remove qwen2.5 co-loading, MAX_LOADED_MODELS=1 - `qwen-cpu.yaml`: new Ollama deployment on talos-cp-2 (144GB RAM), 5Gi PVC - `k8s/apps/comfyui/`: new ComfyUI deployment (1 GPU, 50Gi model PVC, ingress) - `58-comfyui.yaml`: ArgoCD Application (wave 8) Gateway route update in separate PR (homelab-frontend).Reviewed-on: #17 Co-authored-by: rock <[email protected]>	2026-09-09 02:11:28 +00:00
rock	c5aadd98f8	chore: gitignore IAM provisioning scripts	2026-09-08 15:29:42 -07:00
rock	95c67f9437	P3.8: Restrict llm-serving ingress to api-gateway only (#15 ) ## Summary Replaces hand-applied `llm-serving-default-deny` NetworkPolicy with a git-managed, namespace-scoped policy that only allows traffic from the api-gateway. Issue: #13 Co-authored-by: rock <[email protected]>	2026-09-08 16:56:36 +00:00
rock	1630704f8b	feat: switch image-updater to digest-based :latest tracking All image-updater annotations now use update-strategy: digest with allow-tags: ^latest$ and write-back-method: argocd. No SHA tags committed to git — digest overrides stored in ArgoCD state only. - poimen: git write-back → argocd, removed git-branch - portfolio: newest-build SHA → digest latest - api-gateway: newest-build SHA → digest latest	2026-09-07 18:19:46 -07:00
rock	ce1539a634	revert: remove unsupported buildOptions (ArgoCD v3.4.5 doesn't support it) - buildOptions field not available in ArgoCD v3.4.5 - SOPS decryption already handled by repo-server ksops plugin - Revert to simple kustomize config - Portfolio Application can now sync properly	2026-09-07 17:49:58 -07:00
rock	2799e3a675	fix: enable ksops plugin for portfolio Application - Add kustomize config with --enable-alpha-plugins to support ksops - Allows ArgoCD to properly decrypt SOPS-encrypted files - Fixes Image Updater compatibility with sops field in kustomization.yaml	2026-09-07 17:47:03 -07:00
rock	1da7e0aa4d	fix: wait for dind to be ready before starting runner daemon	2026-09-07 13:22:09 -07:00
rock	8970e35539	fix: use tcp://localhost:2375 for dind (no TLS, no socket permission issues)	2026-09-07 13:20:20 -07:00
rock	92239561cd	fix: run runner as root to access dind socket	2026-09-07 13:18:54 -07:00
rock	2d92383951	fix: runner uses unix socket instead of TLS TCP for dind Job containers spawned by the runner run inside dind. With TCP+TLS (tcp://localhost:2376), localhost inside those containers doesn't reach the dind sidecar. Unix socket at /run/docker.sock works because both runner and dind share the /run emptyDir. Also disables DOCKER_TLS_CERTDIR so dind creates the socket instead of only listening on TLS TCP.	2026-09-07 13:16:49 -07:00
rock	7d77935d15	ci: fix runner labels + CoreDNS rewrite + cleanup - Runners use public images (code.forgejo.org/forgejo/runner:6) - Labels pull from Docker Hub: golang:1.26, node:22, rust:1-bookworm - Add CoreDNS api.riotpiao.com rewrite - Fix runner re-registration to keep labels in sync - Add unified CI pattern docs to CLAUDE.example.md - Remove dead .forgejo/ workflow dir (Forgejo uses .gitea/)	2026-09-07 13:01:56 -07:00
rock	fee4f9edfc	ci: fix golang runner - use docker:27-cli (has Node.js + golang + git) Previous image (golang:1.26-bookworm) lacks Node.js, causing GitHub Actions to fail with: 'exec: "node": executable file not found' Solution: - Change golang runner image from golang:1.26-bookworm to docker:27-cli - docker:27-cli includes: Node.js, Go toolchain, git, docker CLI, full dev tools - Verified tag exists: docker manifest inspect docker:27-cli ✓ This allows actions/checkout@v4 and other GitHub Actions to run properly on the golang runner pod. Note: node:22-bookworm runner already has Node.js, no change needed.	2026-09-05 22:50:43 -07:00
rock	12fc2796e2	ci: fix rust runner - use docker:27-cli (has Node.js + git + docker) Previous image (rust:1.83-bookworm) lacks Node.js, causing GitHub Actions to fail with: 'exec: "node": executable file not found' Solution: - Change rust runner image from rust:1.83-bookworm to docker:27-cli - docker:27-cli includes: Node.js, git, docker CLI, full dev tools - Verified tag exists: docker manifest inspect docker:27-cli ✓ This allows actions/checkout@v4 and other GitHub Actions to run properly on the rust runner pod.	2026-09-05 22:49:25 -07:00
rock	f0976bfc61	fix: remove spec wrapper from ClusterRoleBinding - ClusterRoleBinding doesn't use spec: wrapper (unlike Deployment/StatefulSet) - roleRef and subjects go at top level with metadata - Fixes: 'strict decoding error: unknown field "spec"'	2026-09-05 15:04:00 -07:00
rock	4a4e57d0f2	fix: remove namespace from rbac Application destination - RBAC kustomization contains cluster-scoped (ClusterRoleBinding) and namespace-scoped (Role/RoleBinding) resources - Each resource has explicit metadata.namespace, so Application shouldn't force a default namespace - Fixes: ClusterRoleBinding gets namespace=default, causing sync failure with 'unsupported role reference kind: ""'	2026-09-05 14:53:59 -07:00
rock	4fc833f9b2	fix: remove backslash line continuations from YAML multiline string - YAML block scalars (\|) don't use backslash continuation - Just indent lines properly, block scalar handles them automatically - Fixes ArgoCD ComparisonError on poimen app	2026-09-05 14:48:21 -07:00
rock	647fba8814	fix: add admin-oidc-binding to kustomization resources - admin-oidc-binding.yaml wasn't listed in resources - Now kustomize will include it when building manifests - ArgoCD can sync the OIDC group binding	2026-09-05 14:42:17 -07:00
rock	bcae41e338	chore: revert node-runner to stock image, remove Dockerfile - Reverted to node:22-bookworm (no custom image) - Removed Dockerfile.node (no CI to build it) - Docker install step in riotpiao workflow is already the workaround	2026-09-05 14:33:27 -07:00
rock	1425ab7cbc	chore: remove homelab CI workflow - Removed .gitea/workflows/build-runner-node.yml - Homelab is GitOps only, not a buildable artifact - Runner images managed via direct Dockerfile edits + manual pushes	2026-09-05 14:33:13 -07:00
rock	978f9c8147	feat: add OIDC group binding for cluster-admin access - Binds oidc:homelab-admins group to cluster-admin ClusterRole - Allows OIDC users (via Authentik) to have admin access - Groups claim from Authentik with oidc: prefix per kube-apiserver config - Enables kubectl access via 'kubectl login' + kubelogin	2026-09-05 14:30:41 -07:00
rock	4193c8ab99	feat: custom forgejo-runner-node image with docker.io pre-installed - Dockerfile.node extends node:22-bookworm + docker.io - CI workflow builds and pushes to forgejo registry on changes - values-node.yaml references custom image instead of stock node - Removes need to install docker in every workflow using node runner	2026-09-05 14:20:42 -07:00
rock	2c011e08e2	feat: Image Updater git write-back for multi-source poimen Application - write-back-method: git (commits image updates back to repos) - git-branch: main - Mounts ArgoCD SSH credentials for git pushes - Image Updater commits new SHAs → repos → ArgoCD syncs	2026-09-05 13:56:40 -07:00
rock	9da6829e05	feat: multi-source poimen Application (memory, workflows, frontend) - Single Application syncs 3 independent repos - All deploy to poimen namespace - Image Updater tracks all 3 services (7-char SHA tags) - Auto-sync: prune + selfHeal enabled	2026-09-05 13:55:29 -07:00
rock	69537a4e6a	fix: remove poimen-root Application (external repo dependency) - Removed dependency on external poimen.git repo - Poimen manifests should be managed locally or via separate workflow - Simplifies homelab GitOps to only manage homelab-owned services	2026-09-05 13:52:53 -07:00
rock	76c053d895	Revert "feat: enable Image Updater for poimen services" This reverts commit `cfc27c5420`.	2026-09-05 13:52:50 -07:00
rock	cfc27c5420	feat: enable Image Updater for poimen services - Track poimen-memory, poimen-workflows, poimen-frontend images - Auto-update on new 7-char SHA tags from Forgejo - Filter: regexp:^[0-9a-f]{7}$ (commit SHA) - write-back-method: argocd (updates Application)	2026-09-05 13:51:29 -07:00
rock	97c951bef3	Phase 6.6: Add Poimen Memory DLQ queues (ArgoCD managed) ArgoCD Application: memory-queues ├─ Sync wave: 7 (messaging wave) ├─ Path: k8s/apps/messaging/memory-queues ├─ Namespace: sqs └─ Auto-sync: enabled (prune + selfHeal) Helm Chart: memory-queues ├─ Chart.yaml: v0.1.0 ├─ values.yaml: Queue config └─ templates/queues.yaml: Queue CRD resources Queues Created: 1. poimen-memory-dlq ├─ Purpose: Extraction + webhook + agent failures ├─ Partitions: 3 ├─ Replication factor: 1 ├─ Retention: 14 days (1,209,600 seconds) └─ Visibility timeout: 5 minutes (300 seconds) 2. poimen-memory-metric-dlq ├─ Purpose: Metrics persistence failures ├─ Partitions: 3 ├─ Replication factor: 1 ├─ Retention: 14 days └─ Visibility timeout: 5 minutes Resource: Queue CRD (kmsvc.io/v1alpha1) └─ Managed by: queue-operator (already running in sqs ns) Deployment Flow: ArgoCD (homelab) → sync wave 7 → deploy queues Memory app (poimen) → connects to kmsvc → sends DLQ messages Files: ├─ k8s/apps/messaging/memory-queues/Chart.yaml (new) ├─ k8s/apps/messaging/memory-queues/values.yaml (new) ├─ k8s/apps/messaging/memory-queues/templates/queues.yaml (new) └─ k8s/argocd/apps/50-memory-queues.yaml (new)	2026-09-05 01:10:51 -07:00
rock	e414a3e394	Revert "feat: add memory service queues (processing, indexing, dlq)" This reverts commit `e5ae5b16b7`.	2026-09-05 01:09:30 -07:00
rock	e5ae5b16b7	feat: add memory service queues (processing, indexing, dlq) - processing-queue: high-throughput, auto-scaling (1-4 shards) - indexing-queue: FIFO with deduplication (1-2 shards) - memory-dlq: dead letter queue for redelivery failures - ArgoCD Application (wave 7) to auto-sync queue lifecycle	2026-09-05 01:05:01 -07:00
rock	e41165f358	fix(grafana): give homelab-admins Admin org role, akadmin GrafanaAdmin GrafanaAdmin is server admin only — no org membership, so users couldn't see dashboards. Now: - akadmin: GrafanaAdmin (server admin, can impersonate) - homelab-admins: Admin (org admin, dashboard access) - others: Viewer	2026-09-04 23:41:22 -07:00
rock	bd8c9fe033	fix: grant GrafanaAdmin (server admin) to homelab-admins for Administration menu	2026-09-04 23:18:19 -07:00
rock	2eda66c095	fix: add groups scope to grafana OIDC so role mapping works	2026-09-04 23:14:11 -07:00
rock	edc5dadd82	feat: add nginx-ingress ServiceMonitor for gateway traffic metrics	2026-09-04 23:07:44 -07:00
rock	60786a17ea	fix: map grafana admin role from homelab-admins group (grafana-admins deleted)	2026-09-04 23:04:01 -07:00
rock	efbe530b5c	refactor: consolidate 13 dashboards into 2 (cluster-infrastructure + api-gateway)	2026-09-04 22:58:53 -07:00
rock	4c63f8b125	feat: add cluster and api-gateway alert rules with SLA targets	2026-09-04 22:37:48 -07:00
rock	2a9220b576	feat: add cluster-infrastructure and api-gateway grafana dashboards	2026-09-04 22:31:51 -07:00
rock	26714d2ef3	feat: add api-gateway blackbox probes for healthz and /v1/models	2026-09-04 22:13:00 -07:00
rock	39e7ada3c6	fix: use external Authentik URL for MinIO OIDC config discovery	2026-09-04 21:25:30 -07:00
rock	1fe8707e3c	gitops: add secret-rotation controller ArgoCD Application - Syncs k8s/apps/secret-rotation-controller/ kustomization - Auto-prune and self-heal enabled - Creates secret-rotation namespace - ArgoCD will deploy CRD, RBAC, ExternalSecret, Deployment	2026-09-04 13:51:26 -07:00
rock	82a4e3e4fe	feat: automated secret rotation controller - ExternalSecret syncs age key from Vault to pod - CRD defines rotation schedule for each secret - Controller watches CRD, rotates on schedule: * Call provider API (Authentik/Forgejo/MinIO) for new secret * Update k8s Secret * Update .enc.yaml via sops (uses age key from Vault) * Git commit and push - Vault is source of truth for age key (never on disk) - Examples: minio-oidc (90d), portfolio-agent (90d), forgejo-token (90d), minio-root (180d)	2026-09-03 23:37:24 -07:00
rock	6ad4c0d294	fix(minio): use in-cluster URL for OIDC config fetch MinIO pod was getting 503 from public URL at startup. Use in-cluster authentik-server.iam.svc for metadata fetch; browser redirects still use public URLs from OIDC metadata response.	2026-09-03 23:15:19 -07:00
rock	910f8e70d5	iam: move provisioning script to scripts/iam, remove k8s job - Move authentik-provision.py to scripts/iam/ (manual-only) - Remove job/RBAC resources (not needed for local runs) - Use public URL directly (no sed substitution needed) - Add app password support via set_key endpoint - Support both password grant and client_credentials	2026-09-03 19:03:58 -07:00