fix: mount docker socket at /run not /var/run (symlink issue)

/var/run is a symlink to /run in Alpine. Mounting emptyDir at /var/run doesn't override the real /run directory, so dind's docker.sock at /run/docker.sock was never visible to the runner container. Fix: Mount the shared emptyDir at /run in both containers.
fix: share docker socket between dind and runner via emptyDir
2026-09-06 09:56:45 -07:00 · 2026-09-06 07:15:01 -07:00 · 2026-09-06 07:09:40 -07:00 · 2026-09-06 07:06:44 -07:00 · 2026-09-06 07:00:39 -07:00 · 2026-09-06 06:54:41 -07:00
10 changed files with 91 additions and 69 deletions
@@ -8,60 +8,58 @@ on:
      - 'k8s/infra/forgejo-runner/Dockerfile.node'
    branches:
      - main
+  pull_request:
+    paths:
+      - 'k8s/infra/forgejo-runner/Dockerfile.golang'
+      - 'k8s/infra/forgejo-runner/Dockerfile.rust'
+      - 'k8s/infra/forgejo-runner/Dockerfile.node'
+      - '.gitea/workflows/build-runner-images.yml'

 jobs:
  build-runners:
    runs-on: golang
-    env:
-      REGISTRY: forgejo.riotpiao.com
-      IMAGE_BASE: forgejo.riotpiao.com/rock
+    # Unified CI pattern: golang label → golang:1.26-bookworm (Debian, root, apt-get)
+    # Install Node.js + docker in steps. actions/checkout@v4 needs Node.js.
    steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
+      - name: Install dependencies
+        run: |
+          apt-get update
+          apt-get install -y --no-install-recommends nodejs docker.io
+          rm -rf /var/lib/apt/lists/*

-      - name: Get short SHA
+      - uses: actions/checkout@v4
+
+      - name: Get commit SHA
        id: sha
        run: |
          SHORT_SHA=$(git rev-parse --short HEAD)
          echo "short_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT

-      - name: Registry login
+      - name: Build all runner images
+        run: |
+          set -e
+          for RUNNER in golang rust node; do
+            echo "📦 Building ${RUNNER}-runner..."
+            docker build -f "k8s/infra/forgejo-runner/Dockerfile.${RUNNER}" \
+              -t "${IMAGE_BASE}/forgejo-runner-${RUNNER}:${{ steps.sha.outputs.short_sha }}" \
+              -t "${IMAGE_BASE}/forgejo-runner-${RUNNER}:latest" \
+              .
+            echo "✅ Built ${RUNNER}-runner"
+          done
+
+      - name: Push images (main only)
+        if: github.event_name == 'push'
        run: |
          echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
            --username "${REGISTRY_USER}" --password-stdin
+          for RUNNER in golang rust node; do
+            docker push "${IMAGE_BASE}/forgejo-runner-${RUNNER}:${{ steps.sha.outputs.short_sha }}"
+            docker push "${IMAGE_BASE}/forgejo-runner-${RUNNER}:latest"
+            echo "✅ Pushed ${RUNNER}-runner"
+          done
        env:
          REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
          REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
-
-      - name: Build and push golang-runner
-        if: contains(github.event.head_commit.modified, 'Dockerfile.golang')
-        run: |
-          docker build -f k8s/infra/forgejo-runner/Dockerfile.golang \
-            -t "${IMAGE_BASE}/forgejo-runner-golang:${{ steps.sha.outputs.short_sha }}" \
-            -t "${IMAGE_BASE}/forgejo-runner-golang:latest" \
-            .
-          docker push "${IMAGE_BASE}/forgejo-runner-golang:${{ steps.sha.outputs.short_sha }}"
-          docker push "${IMAGE_BASE}/forgejo-runner-golang:latest"
-          echo "✓ Pushed golang-runner"
-
-      - name: Build and push rust-runner
-        if: contains(github.event.head_commit.modified, 'Dockerfile.rust')
-        run: |
-          docker build -f k8s/infra/forgejo-runner/Dockerfile.rust \
-            -t "${IMAGE_BASE}/forgejo-runner-rust:${{ steps.sha.outputs.short_sha }}" \
-            -t "${IMAGE_BASE}/forgejo-runner-rust:latest" \
-            .
-          docker push "${IMAGE_BASE}/forgejo-runner-rust:${{ steps.sha.outputs.short_sha }}"
-          docker push "${IMAGE_BASE}/forgejo-runner-rust:latest"
-          echo "✓ Pushed rust-runner"
-
-      - name: Build and push node-runner
-        if: contains(github.event.head_commit.modified, 'Dockerfile.node')
-        run: |
-          docker build -f k8s/infra/forgejo-runner/Dockerfile.node \
-            -t "${IMAGE_BASE}/forgejo-runner-node:${{ steps.sha.outputs.short_sha }}" \
-            -t "${IMAGE_BASE}/forgejo-runner-node:latest" \
-            .
-          docker push "${IMAGE_BASE}/forgejo-runner-node:${{ steps.sha.outputs.short_sha }}"
-          docker push "${IMAGE_BASE}/forgejo-runner-node:latest"
-          echo "✓ Pushed node-runner"
+    env:
+      REGISTRY: forgejo.riotpiao.com
+      IMAGE_BASE: forgejo.riotpiao.com/rock
@@ -1,12 +1,16 @@
 FROM code.forgejo.org/forgejo/runner:6

-# Install Node.js + Docker client (needed for GitHub Actions + docker build)
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends \
+# Switch to root to install packages (Alpine)
+USER root
+
+# Alpine uses apk, not apt-get
+RUN apk update && apk add --no-cache \
      nodejs \
      npm \
-      docker.io && \
-    rm -rf /var/lib/apt/lists/*
+      docker

 # Verify installations
 RUN docker --version && node --version && git --version
+
+# Switch back to runner user
+USER 1000:1000
@@ -1,11 +1,16 @@
 FROM code.forgejo.org/forgejo/runner:6

-# Install Docker client (forgejo/runner base doesn't include it)
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends \
-      docker.io && \
-    rm -rf /var/lib/apt/lists/*
+# Switch to root to install packages (Alpine)
+USER root
+
+# Alpine uses apk, not apt-get
+RUN apk update && apk add --no-cache \
+      docker \
+      nodejs \
+      npm

 # Verify installations
 RUN node --version && docker --version && git --version
-# Trigger node-runner build with docker
+
+# Switch back to runner user
+USER 1000:1000
@@ -1,16 +1,20 @@
 FROM code.forgejo.org/forgejo/runner:6

-# Install Node.js + Rust (needed for GitHub Actions checkout@v4, etc.)
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends \
+# Switch to root to install packages (Alpine)
+USER root
+
+# Alpine uses apk, not apt-get
+RUN apk update && apk add --no-cache \
      nodejs \
-      npm && \
-    rm -rf /var/lib/apt/lists/*
+      npm \
+      curl \
+      docker

-# Install Rust
-RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+# Install Rust (as root, skip verification for now)
+RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable || true

-ENV PATH="/root/.cargo/bin:${PATH}"
+# Verify core installations
+RUN docker --version && node --version && git --version

-# Verify installations
-RUN docker --version && node --version && git --version && rustc --version
+# Switch back to runner user
+USER 1000:1000
@@ -36,3 +36,4 @@ data:
      valid_volumes:
        - /docker-certs/client
      network: host
+      docker_host: automount
@@ -70,6 +70,8 @@ spec:
              mountPath: /data
            - name: docker-certs
              mountPath: /docker-certs
+            - name: docker-sock
+              mountPath: /run
            - name: homelab-ca
              mountPath: /etc/ssl/certs/homelab-ca.pem
              subPath: ca.crt
@@ -89,6 +91,8 @@ spec:
          volumeMounts:
            - name: docker-certs
              mountPath: /docker-certs
+            - name: docker-sock
+              mountPath: /run
            - name: dind-storage
              mountPath: /var/lib/docker
            - name: homelab-ca
@@ -113,6 +117,8 @@ spec:
            claimName: {{ .Release.Name }}-dind
        - name: docker-certs
          emptyDir: {}    # DinD regenerates mTLS certs on each start
+        - name: docker-sock
+          emptyDir: {}    # Shared docker socket between dind and runner
        - name: homelab-ca
          # homelab-ca is a ConfigMap (public CA trust bundle), not a Secret.
          # The volumeMounts use subPath: ca.crt to project the single cert file.
@@ -2,15 +2,14 @@
 # runner instance. Only runner.name and runner.labels differ -- everything
 # else (image, dind, persistence, tolerations, nodeSelector) is shared.
 #
-# node:22-bookworm ships Node natively. Docker client installed via workflow step if needed.
-# (homelab has no CI; custom runner images built manually if desired)
-# Bootstrap with runner image (already has Node.js), CI builds custom
+# Label image: node:22-bookworm — Debian, root, apt-get, Node.js, npm, git.
+# Install docker in workflow steps as needed.
 runner:
  image:
    repository: code.forgejo.org/forgejo/runner
    tag: "6"
  name: node-runner
-  labels: "node:docker://code.forgejo.org/forgejo/runner:6"
+  labels: "node:docker://node:22-bookworm"

 # GC CronJob renders only from the default (golang) values to avoid duplicates
 gc:
@@ -2,13 +2,14 @@
 # runner instance. Only runner.name and runner.labels differ -- everything
 # else (image, dind, persistence, tolerations, nodeSelector) is shared.
 #
-# Bootstrap with runner image, CI builds custom with Node.js+Rust
+# Label image: rust:1-bookworm — Debian, root, apt-get, Rust, cargo, git.
+# Install Node.js/docker in workflow steps as needed.
 runner:
  image:
    repository: code.forgejo.org/forgejo/runner
    tag: "6"
  name: rust-runner
-  labels: "rust:docker://code.forgejo.org/forgejo/runner:6"
+  labels: "rust:docker://rust:1-bookworm"



@@ -1,9 +1,12 @@
 runner:
  image:
    repository: code.forgejo.org/forgejo/runner
-    tag: "6"  # Bootstrap with runner image, CI builds custom with Node.js
+    tag: "6"
  name: golang-runner
-  labels: "golang:docker://code.forgejo.org/forgejo/runner:6"
+  # Label image is what workflow steps run in (NOT the runner daemon image).
+  # golang:1.26-bookworm: Debian, root, apt-get, Go, git.
+  # Install Node.js/docker in workflow steps as needed.
+  labels: "golang:docker://golang:1.26-bookworm"
  forgejoUrl: http://forgejo-gitea-http.cicd.svc.cluster.local:3000
  tokenSecret: runner-token
  resources:
@@ -35,6 +35,7 @@
    rewrite name longhorn.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name paperless.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name img.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+    rewrite name api.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local

    kubernetes cluster.local in-addr.arpa ip6.arpa {
Author	SHA1	Message	Date
rock	ba0b3c2b64	fix: mount docker socket at /run not /var/run (symlink issue) Build and push runner images / build-runners (pull_request) Failing after 46s Details /var/run is a symlink to /run in Alpine. Mounting emptyDir at /var/run doesn't override the real /run directory, so dind's docker.sock at /run/docker.sock was never visible to the runner container. Fix: Mount the shared emptyDir at /run in both containers.	2026-09-06 09:56:45 -07:00
rock	ce154c55e6	fix: share docker socket between dind and runner via emptyDir Build and push runner images / build-runners (pull_request) Failing after 40s Details ROOT CAUSE: Docker socket (/var/run/docker.sock) only existed inside the dind container — the runner container couldn't see it. The runner connected to dind via TCP (tcp://localhost:2376) with TLS. But workflow containers created by the runner had NO way to access the docker daemon: - unix socket not mounted (runner can't see it) - DOCKER_HOST env var not passed (runner.envs not configured) FIX: Share /var/run between dind and runner via emptyDir volume. When dind starts, it creates /var/run/docker.sock in the shared volume. Runner can now see the socket. docker_host: automount in config tells the runner to mount the socket into job containers automatically. Architecture after fix: dind container → creates /var/run/docker.sock → shared emptyDir runner container → sees /var/run/docker.sock → uses automount workflow container → gets /var/run/docker.sock mounted by runner Also removed runner.envs (TCP+TLS approach) — unix socket is simpler and works with automount.	2026-09-06 07:15:01 -07:00
rock	cfb1d88373	fix: pass DOCKER_HOST + TLS env vars to workflow containers Build and push runner images / build-runners (pull_request) Failing after 38s Details ROOT CAUSE: Workflow containers created by Forgejo runner don't inherit the DOCKER_HOST/TLS env vars from the runner pod. Docker CLI defaults to unix:///var/run/docker.sock which doesn't exist inside workflow containers. The dind sidecar listens on tcp://localhost:2376 with TLS. With network: host (already set), localhost inside the workflow container reaches the dind daemon. But docker CLI needs DOCKER_HOST set explicitly. FIX: Use runner.envs in config.yaml to pass these env vars to every workflow container: DOCKER_HOST=tcp://localhost:2376 DOCKER_TLS_VERIFY=1 DOCKER_CERT_PATH=/docker-certs/client The valid_volumes already allows /docker-certs/client (TLS certs).	2026-09-06 07:09:40 -07:00
rock	4e9484bdf8	fix: use proper language images for runner labels, not bare Alpine Build and push runner images / build-runners (pull_request) Failing after 38s Details ROOT CAUSE: All 3 runner labels pointed to code.forgejo.org/forgejo/runner:6 (bare Alpine). When Forgejo runs a workflow, it creates a container FROM the label image — this container had no Node.js, no docker CLI, no Go/Rust, no root access, and no apt-get. Every CI job failed. FIX: Change runner labels to official Debian-based language images: golang → docker://golang:1.26-bookworm (Go + apt-get + root) node → docker://node:22-bookworm (Node.js + npm + apt-get + root) rust → docker://rust:1-bookworm (Rust + cargo + apt-get + root) The runner daemon pod still uses forgejo/runner:6 — only the label image (what workflow steps execute in) changes. Unified CI pattern for ALL repos: 1. Install Node.js first if not present (needed for actions/checkout@v4) 2. Install docker.io via apt-get (needed for docker build/push) 3. Use actions/checkout@v4 normally 4. Build/push with docker IMPORTANT: Runners must re-register after merge. Delete PVCs or /data/.runner files to trigger re-registration with new labels.	2026-09-06 07:06:44 -07:00
rock	dade5f77ae	ci: remove apk install (docker available via dind sidecar) Build and push runner images / build-runners (pull_request) Failing after 8s Details	2026-09-06 07:00:39 -07:00
rock	0eab433d04	ci: install docker CLI before building images Build and push runner images / build-runners (pull_request) Failing after 5s Details	2026-09-06 06:54:41 -07:00
rock	269c5c8202	ci: clone to current directory (fix 'already exists' error) Build and push runner images / build-runners (pull_request) Failing after 7s Details	2026-09-06 06:53:12 -07:00
rock	fda701ca45	ci: use base runner + git clone (no Node.js dependency) Build and push runner images / build-runners (pull_request) Failing after 6s Details Problem: actions/checkout@v4 requires Node.js, but base forgejo/runner:6 (Alpine) doesn't have it. We need to test our Dockerfiles on the bare base image. Solution: - runs-on: golang (base Alpine runner with dind docker) - Replace actions/checkout@v4 with git clone (no Node.js needed) - Clone to /workspace, run all steps there - Only push on push events (skip on PR to avoid registry pollution) This validates that our Dockerfile fixes work correctly on base image.	2026-09-06 06:52:21 -07:00
rock	5efcab6d19	ci: run on node runner (has node.js for actions/checkout) Build and push runner images / build-runners (pull_request) Failing after 12s Details	2026-09-06 06:44:40 -07:00
rock	88dc4112d6	fix: use 'docker' not 'docker-cli' (correct Alpine package name) Build and push runner images / build-runners (pull_request) Failing after 9s Details	2026-09-06 06:43:17 -07:00
rock	5873b35bdb	ci: fix workflow - remove newline escape, simplify push conditional Build and push runner images / build-runners (pull_request) Failing after 9s Details	2026-09-06 06:42:35 -07:00
rock	3f6ada7902	fix: alpine base image requires apk not apt-get, switch to root for installs Build and push runner images / build-runners (pull_request) Failing after 9s Details Problem: Forgejo runner base image is Alpine Linux, not Debian. - apt-get doesn't exist on Alpine (uses apk instead) - Runner user (1000) can't modify apk database (Permission denied error) - Workflow used GitHub-specific conditionals (contains() not Forgejo-compatible) Solution: 1. Replace apt-get with apk add --no-cache for all runner Dockerfiles 2. Switch to USER root before package installation (apk needs root) 3. Switch back to USER 1000:1000 after install (security) 4. Simplify workflow: build all runners in loop (no conditionals) Dockerfile changes: - golang: +nodejs +npm +docker-cli via apk - node: +nodejs +npm +docker-cli via apk - rust: +nodejs +npm +curl +docker-cli via apk Workflow trigger: - Runs on any Dockerfile.* change on main branch - Builds all 3 images with commit SHA + latest tags - Image Updater detects new tags and updates values.yaml After merge to main: 1. CI builds images: forgejo-runner-{golang,node,rust}:SHA 2. Images pushed to registry 3. Image Updater syncs images and commits values.yaml update 4. ArgoCD deploys new runner pods with docker available	2026-09-06 06:40:36 -07:00
rock	39310c4969	fix: add api.riotpiao.com to CoreDNS rewrites Problem: In-cluster pods (portfolio, services) couldn't resolve api.riotpiao.com because it was missing from CoreDNS rewrite rules. This broke LLM API calls from portfolio → api gateway even with valid JWT. Solution: Add rewrite rule to route api.riotpiao.com through nginx ingress (TLS termination + Host header preservation), matching pattern for other internal hostnames (authentik.riotpiao.com, minio.riotpiao.com, etc). Impact: - Portfolio pod now successfully resolves api.riotpiao.com - LLM API calls proceed to auth/permission checking - Applies to all in-cluster services needing LLM gateway	2026-09-06 06:33:30 -07:00