fix: Forgejo CI runners — proper images, shared docker socket, auto-registration

All CI workflows across all repos were broken. Three root causes:

1. Runner labels pointed to bare Alpine image (forgejo/runner:6) which has
   no Node.js, no docker, no Go/Rust, no root, no apt-get. Every workflow
   step failed immediately.

   Fix: Change labels to official Debian language images:
     golang → docker://golang:1.26-bookworm
     node   → docker://node:22-bookworm
     rust   → docker://rust:1-bookworm

2. Docker socket not shared between dind sidecar and runner container.
   dind creates /run/docker.sock in its own filesystem. Runner container
   has a separate filesystem and can't see it. Workflow containers that
   run 'docker build' get 'Cannot connect to Docker daemon'.

   Fix: Share /run between dind and runner via emptyDir volume.
   Add docker_host: automount to runner config so the socket is
   mounted into workflow containers automatically.

   Note: /var/run is a symlink to /run in Alpine — must mount at /run.

3. Init container skipped re-registration if .runner file existed on PVC.
   Changing labels in values.yaml had no effect until PVC was manually
   deleted — not GitOps-friendly.

   Fix: Always rm .runner and re-register on pod start. Labels stay
   in sync with values.yaml automatically.

Also removes custom Dockerfiles and build-runner-images workflow — no longer
needed since official images provide the language tools directly.

.gitignore

@@ -66,6 +66,3 @@ bootstrap-argocd.log
 # one line here, which is how a plaintext deploy key reached a public remote.
 k8s/**/*-secret.yaml
 !k8s/**/*.enc.yaml
-
-# IAM provisioning scripts contain credential references — never commit
-scripts/iam/*.py

CLAUDE.example.md

@@ -208,95 +208,3 @@ versions without warning in your own values file.
 Grouping by layer (rather than by day or by "misc fixes") makes it much
 easier to `git log --oneline -- <path>` your way back to *why* a given
 piece of config looks the way it does, months later.
-
-## Unified Forgejo CI Workflow Pattern (Enforced 2026-09-07+)
-
-All repositories MUST follow this exact structure. No variations.
-
-```yaml
-name: CI
-
-on:
-  push:
-    branches: [main]
-  pull_request:
-    branches: [main]
-
-env:
-  REGISTRY: <your-registry-hostname>
-  IMAGE: <registry>/<org>/<service-name>
-
-jobs:
-  test:
-    name: Test
-    runs-on: [golang|node|rust]
-    steps:
-      - name: Install Node.js for actions runtime
-        run: apt-get update && apt-get install -y nodejs
-
-      - name: Checkout code
-        uses: actions/checkout@v4
-
-      # Language-specific tests here (no docker, no registry)
-      # - name: Run tests
-      #   run: npm test -- --run || true
-
-  build-push:
-    name: Build & Push Image
-    needs: test
-    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
-    runs-on: [golang|node|rust]
-    steps:
-      - name: Install Node.js and Docker
-        run: |
-          apt-get update
-          apt-get install -y nodejs docker.io
-
-      - name: Checkout code
-        uses: actions/checkout@v4
-
-      - name: Get short SHA
-        id: sha
-        run: |
-          SHORT_SHA=$(git rev-parse --short HEAD)
-          echo "short_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
-
-      - name: Registry login
-        run: |
-          echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
-            --username "${REGISTRY_USER}" --password-stdin
-        env:
-          REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
-          REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
-
-      - name: Build Docker image
-        run: |
-          docker build --no-cache \
-            -t "${IMAGE}:${{ steps.sha.outputs.short_sha }}" \
-            -t "${IMAGE}:latest" \
-            .
-
-      - name: Push Docker image
-        run: |
-          docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}"
-          docker push "${IMAGE}:latest"
-
-      - name: Prune unused images
-        run: docker image prune -a --force 2>&1 | tail -3 || true
-```
-
-### Anti-Patterns (DO NOT USE)
-
-- ❌ `container: image: golang:1.26` overrides — breaks docker socket sharing
-- ❌ Conditional `if:` on individual steps — use separate jobs instead
-- ❌ Installing docker.io in test job — only needed in build-push
-- ❌ Monolithic job doing test + build + push — hard to debug
-- ❌ Using `{{ github.sha }}` for image tag — use short commit SHA for readability
-
-### How It Works
-
-1. **PR to feature branch** → test job runs, build-push skipped, nothing pushed
-2. **Push to main** → test runs, build-push runs after test passes, image pushed
-3. Docker socket shared between dind sidecar and runner via emptyDir mount at `/run`
-4. `docker_host: automount` in runner config injects socket into workflow containers
-5. Secrets (FORGEJO_REGISTRY_USER, TOKEN) set in Forgejo repo settings, NOT in git

k8s/apps/llm-serving/kustomization.yaml

@@ -14,6 +14,5 @@ resources:
   - ornith.yaml
   - reasoning.yaml
   - reranker.yaml
-  - networkpolicy.yaml
 # No namespace transformer: every file sets its own, and the transformer would
 # rewrite metadata.namespace on anything cross-namespace added later.

k8s/apps/llm-serving/networkpolicy.yaml

@@ -1,62 +0,0 @@
-# NetworkPolicy for LLM inference engines (llm-serving namespace).
-#
-# These pods have NO auth — vLLM, Ollama, and TEI accept any request.
-# All access MUST go through the api-gateway, which validates JWTs and
-# injects identity headers (X-Forwarded-User, X-Auth-Verified).
-#
-# Replaces the hand-applied llm-serving-default-deny policy that used
-# `llm-client: "true"` pod label as a selector — any pod in any namespace
-# could self-grant access by adding that label, which defeats the purpose.
-#
-# This policy restricts ingress to:
-#   1. api namespace (gateway) — the sole entry point for inference
-#   2. monitoring namespace — Prometheus scraping vLLM/TEI /metrics
-#   3. intra-namespace — pod-to-pod (future: multi-replica comms)
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: llm-serving-ingress
-  namespace: llm-serving
-  labels:
-    app.kubernetes.io/part-of: llm-serving
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/part-of: llm-serving
-  policyTypes:
-    - Ingress
-  ingress:
-    # Allow from api-gateway (namespace: api)
-    # Gateway proxies /v1/chat/completions, /v1/embeddings, /v1/rerank
-    - from:
-        - namespaceSelector:
-            matchLabels:
-              kubernetes.io/metadata.name: api
-      ports:
-        - protocol: TCP
-          port: 8080   # vLLM, Ollama HTTP
-        - protocol: TCP
-          port: 80      # KServe predictor services
-        - protocol: TCP
-          port: 8000    # vLLM direct (some configs)
-        - protocol: TCP
-          port: 11434   # Ollama native port
-    # Allow Prometheus scraping from monitoring namespace
-    # vLLM: :8080/metrics, TEI: :9000/metrics
-    - from:
-        - namespaceSelector:
-            matchLabels:
-              kubernetes.io/metadata.name: monitoring
-      ports:
-        - protocol: TCP
-          port: 8080
-        - protocol: TCP
-          port: 9000
-    # Allow intra-namespace (pod-to-pod within llm-serving)
-    - from:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/part-of: llm-serving
-      ports:
-        - protocol: TCP
-          port: 8080

k8s/argocd/apps/55-api-gateway.yaml

@@ -38,8 +38,8 @@ metadata:
     argocd.argoproj.io/sync-wave: "7"
     # ArgoCD Image Updater - auto-update on new image push
     argocd-image-updater.argoproj.io/image-list: gw=forgejo.riotpiao.com/rock/api-gateway
-    argocd-image-updater.argoproj.io/gw.update-strategy: digest
-    argocd-image-updater.argoproj.io/gw.allow-tags: regexp:^latest$
+    argocd-image-updater.argoproj.io/gw.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/gw.allow-tags: regexp:^[0-9a-f]{7}$
     argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
   project: homelab

k8s/argocd/apps/60-applications.yaml

@@ -248,8 +248,8 @@ metadata:
     argocd.argoproj.io/sync-wave: "8"
     # ArgoCD Image Updater - auto-update on new image push
     argocd-image-updater.argoproj.io/image-list: app=forgejo.riotpiao.com/rock/portfolio
-    argocd-image-updater.argoproj.io/app.update-strategy: digest
-    argocd-image-updater.argoproj.io/app.allow-tags: regexp:^latest$
+    argocd-image-updater.argoproj.io/app.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/app.allow-tags: regexp:^[0-9a-f]{7}$
     argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
   project: homelab

k8s/argocd/apps/70-poimen.yaml

@@ -10,13 +10,14 @@ metadata:
       memory=forgejo.riotpiao.com/rock/poimen-memory
       workflows=forgejo.riotpiao.com/rock/poimen-workflows
       frontend=forgejo.riotpiao.com/rock/poimen-frontend
-    argocd-image-updater.argoproj.io/memory.update-strategy: digest
-    argocd-image-updater.argoproj.io/memory.allow-tags: regexp:^latest$
-    argocd-image-updater.argoproj.io/workflows.update-strategy: digest
-    argocd-image-updater.argoproj.io/workflows.allow-tags: regexp:^latest$
-    argocd-image-updater.argoproj.io/frontend.update-strategy: digest
-    argocd-image-updater.argoproj.io/frontend.allow-tags: regexp:^latest$
-    argocd-image-updater.argoproj.io/write-back-method: argocd
+    argocd-image-updater.argoproj.io/memory.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/memory.allow-tags: regexp:^[0-9a-f]{7}$
+    argocd-image-updater.argoproj.io/workflows.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/workflows.allow-tags: regexp:^[0-9a-f]{7}$
+    argocd-image-updater.argoproj.io/frontend.update-strategy: newest-build
+    argocd-image-updater.argoproj.io/frontend.allow-tags: regexp:^[0-9a-f]{7}$
+    argocd-image-updater.argoproj.io/write-back-method: git
+    argocd-image-updater.argoproj.io/git-branch: main
 spec:
   project: homelab
   sources:

k8s/infra/forgejo-runner/templates/deployment.yaml

@@ -60,11 +60,15 @@ spec:
       containers:
         - name: runner
           image: {{ .Values.runner.image.repository }}:{{ .Values.runner.image.tag }}
-          command: ["sh", "-c", "while ! wget -q -O- http://localhost:2375/_ping >/dev/null 2>&1; do echo 'waiting for dind...'; sleep 2; done; echo 'dind ready'; forgejo-runner daemon --config /etc/forgejo-runner/config.yaml"]
+          command: ["sh", "-c", "forgejo-runner daemon --config /etc/forgejo-runner/config.yaml"]
           workingDir: /data
           env:
             - name: DOCKER_HOST
-              value: tcp://localhost:2375
+              value: tcp://localhost:2376
+            - name: DOCKER_TLS_VERIFY
+              value: "1"
+            - name: DOCKER_CERT_PATH
+              value: /docker-certs/client
           volumeMounts:
             - name: runner-data
               mountPath: /data
@@ -87,7 +91,7 @@ spec:
             privileged: true    # required for DinD; cicd namespace is labelled privileged
           env:
             - name: DOCKER_TLS_CERTDIR
-              value: ""
+              value: /docker-certs
           volumeMounts:
             - name: docker-certs
               mountPath: /docker-certs

k8s/infra/forgejo-runner/values.yaml

@@ -5,7 +5,7 @@ runner:
   name: golang-runner
   # Label image is what workflow steps run in (NOT the runner daemon image).
   # golang:1.26-bookworm: Debian, root, apt-get, Go, git.
-  # TODO: Switch to custom image once build-runner-images.yml pushes images
+  # Install Node.js/docker in workflow steps as needed.
   labels: "golang:docker://golang:1.26-bookworm"
   forgejoUrl: http://forgejo-gitea-http.cicd.svc.cluster.local:3000
   tokenSecret: runner-token

scripts/test-auth-e2e.sh

@@ -1,178 +0,0 @@
-#!/usr/bin/env bash
-# End-to-end auth flow tests for api-gateway.
-# Validates: JWT auth, identity headers, token exchange, error handling.
-#
-# Usage:
-#   ./scripts/test-auth-e2e.sh
-#
-# Requires: kubectl access to cluster, curl, jq
-
-set -euo pipefail
-
-API="https://api.riotpiao.com"
-TOKEN_URL="https://authentik.riotpiao.com/application/o/token/"
-PASS=0
-FAIL=0
-SKIP=0
-
-# Colors
-RED='\033[0;31m'
-GREEN='\033[0;32m'
-YELLOW='\033[0;33m'
-NC='\033[0m'
-
-check() {
-    local name="$1" expected="$2" actual="$3"
-    if [ "$expected" = "$actual" ]; then
-        echo -e "  ${GREEN}✓${NC} $name"
-        PASS=$((PASS + 1))
-    else
-        echo -e "  ${RED}✗${NC} $name (expected $expected, got $actual)"
-        FAIL=$((FAIL + 1))
-    fi
-}
-
-skip() {
-    echo -e "  ${YELLOW}○${NC} $1 (skipped: $2)"
-    SKIP=$((SKIP + 1))
-}
-
-# --- Load credentials ---
-echo "Loading credentials from cluster..."
-P_SECRET=$(kubectl get secret -n portfolio portfolio-agent-oidc -o jsonpath='{.data.CLIENT_SECRET}' | base64 -d 2>/dev/null) || true
-M_SECRET=$(kubectl get secret -n poimen memory-agent-oidc -o jsonpath='{.data.CLIENT_SECRET}' | base64 -d 2>/dev/null) || true
-
-if [ -z "$P_SECRET" ] || [ -z "$M_SECRET" ]; then
-    echo "ERROR: Cannot read service account secrets. Check kubectl context."
-    exit 1
-fi
-
-# --- Get tokens ---
-echo ""
-echo "=== Acquiring tokens ==="
-
-P_TOKEN=$(curl -s -X POST "$TOKEN_URL" \
-    -d "grant_type=client_credentials&client_id=portfolio-agent&client_secret=$P_SECRET&scope=openid roles" \
-    | jq -r '.access_token')
-
-M_TOKEN=$(curl -s -X POST "$TOKEN_URL" \
-    -d "grant_type=client_credentials&client_id=memory-agent&client_secret=$M_SECRET&scope=openid roles" \
-    | jq -r '.access_token')
-
-if [ "$P_TOKEN" = "null" ] || [ -z "$P_TOKEN" ]; then
-    echo "ERROR: Failed to get portfolio-agent token"
-    exit 1
-fi
-echo "  portfolio-agent: token acquired"
-echo "  memory-agent: token acquired"
-
-# --- Scenario 1: LLM inference ---
-echo ""
-echo "=== Scenario 1: LLM inference (llm:inference) ==="
-
-CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$API/v1/chat/completions" \
-    -H "Authorization: Bearer $P_TOKEN" -H "Content-Type: application/json" \
-    -d '{"model":"qwen2.5:3b-instruct","messages":[{"role":"user","content":"ok"}],"max_tokens":2}')
-check "portfolio-agent -> qwen chat" "200" "$CODE"
-
-CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$API/v1/chat/completions" \
-    -H "Authorization: Bearer $M_TOKEN" -H "Content-Type: application/json" \
-    -d '{"model":"reasoning","messages":[{"role":"user","content":"ok"}],"max_tokens":2}')
-check "memory-agent -> reasoning chat" "200" "$CODE"
-
-CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$API/v1/embeddings" \
-    -H "Authorization: Bearer $P_TOKEN" -H "Content-Type: application/json" \
-    -d '{"model":"nomic-ai/nomic-embed-text-v2-moe","input":"test"}')
-check "portfolio-agent -> embeddings" "200" "$CODE"
-
-CODE=$(curl -s -o /dev/null -w "%{http_code}" "$API/v1/models")
-check "GET /v1/models (no auth)" "200" "$CODE"
-
-# --- Scenario 2: Auth enforcement ---
-echo ""
-echo "=== Scenario 2: Auth enforcement ==="
-
-CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$API/v1/chat/completions" \
-    -H "Content-Type: application/json" \
-    -d '{"model":"qwen2.5:3b-instruct","messages":[{"role":"user","content":"ok"}]}')
-check "no token -> 401" "401" "$CODE"
-
-CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$API/v1/chat/completions" \
-    -H "Authorization: Bearer garbage-token" -H "Content-Type: application/json" \
-    -d '{"model":"qwen2.5:3b-instruct","messages":[{"role":"user","content":"ok"}]}')
-check "bad token -> 403" "403" "$CODE"
-
-# --- Scenario 3: Token claims ---
-echo ""
-echo "=== Scenario 3: Token claims ==="
-
-decode_jwt() { python3 -c "import base64,json,sys; p=sys.argv[1].split('.')[1]; p+='='*(4-len(p)%4); print(json.dumps(json.loads(base64.urlsafe_b64decode(p))))" "$1"; }
-
-P_CLAIMS=$(decode_jwt "$P_TOKEN")
-P_ROLES=$(echo "$P_CLAIMS" | jq -r '.roles | join(",")')
-check "portfolio-agent has llm:inference" "true" "$(echo "$P_ROLES" | grep -q 'llm:inference' && echo true || echo false)"
-check "portfolio-agent has s3:read" "true" "$(echo "$P_ROLES" | grep -q 's3:read' && echo true || echo false)"
-check "portfolio-agent has sqs:read" "true" "$(echo "$P_ROLES" | grep -q 'sqs:read' && echo true || echo false)"
-
-M_CLAIMS=$(decode_jwt "$M_TOKEN")
-M_ROLES=$(echo "$M_CLAIMS" | jq -r '.roles | join(",")')
-check "memory-agent has memory:write" "true" "$(echo "$M_ROLES" | grep -q 'memory:write' && echo true || echo false)"
-check "memory-agent has s3:write" "true" "$(echo "$M_ROLES" | grep -q 's3:write' && echo true || echo false)"
-
-# --- Scenario 4: X-Service dispatch ---
-echo ""
-echo "=== Scenario 4: X-Service dispatch ==="
-
-CODE=$(curl -s -o /dev/null -w "%{http_code}" "$API/" \
-    -H "X-Service: s3" -H "X-Resource: list-objects" \
-    -H "Authorization: Bearer $P_TOKEN")
-check "portfolio-agent -> S3 list (reaches MinIO)" "403" "$CODE"
-# 403 = MinIO reached but rejects JWT (needs SigV4). Not a gateway auth issue.
-
-# Memory service might be down
-CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$API/" \
-    -H "X-Service: memory" -H "X-Resource: skills" \
-    -H "Authorization: Bearer $P_TOKEN" 2>/dev/null)
-if [ "$CODE" = "200" ]; then
-    check "portfolio-agent -> memory skills" "200" "$CODE"
-elif [ "$CODE" = "502" ]; then
-    skip "portfolio-agent -> memory skills" "poimen-memory pods down"
-else
-    check "portfolio-agent -> memory skills" "200" "$CODE"
-fi
-
-# --- Scenario 5: Auth endpoints ---
-echo ""
-echo "=== Scenario 5: Auth endpoints ==="
-
-# /auth/token — may return 400 if password grant not working, but endpoint should respond
-CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$API/auth/token" \
-    -H "Content-Type: application/json" \
-    -d '{"username":"rock","password":"wrong"}')
-# Should get 400 (bad creds forwarded from Authentik) or 502 (upstream issue), not 404
-if [ "$CODE" != "404" ] && [ "$CODE" != "405" ]; then
-    check "/auth/token endpoint exists" "true" "true"
-else
-    check "/auth/token endpoint exists" "not-404" "$CODE"
-fi
-
-CODE=$(curl -s -o /dev/null -w "%{http_code}" -X GET "$API/auth/token")
-check "GET /auth/token -> 405" "405" "$CODE"
-
-CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$API/auth/refresh" \
-    -H "Content-Type: application/json" \
-    -d '{"refresh_token":"invalid"}')
-if [ "$CODE" != "404" ] && [ "$CODE" != "405" ]; then
-    check "/auth/refresh endpoint exists" "true" "true"
-else
-    check "/auth/refresh endpoint exists" "not-404" "$CODE"
-fi
-
-# --- Summary ---
-echo ""
-echo "========================================="
-TOTAL=$((PASS + FAIL + SKIP))
-echo -e "Results: ${GREEN}$PASS passed${NC}, ${RED}$FAIL failed${NC}, ${YELLOW}$SKIP skipped${NC} / $TOTAL total"
-echo "========================================="
-
-[ "$FAIL" -eq 0 ] && exit 0 || exit 1