No longer needed — runner labels now point to official language images
(golang:1.26-bookworm, node:22-bookworm, rust:1-bookworm) which already
have the language tools. Docker CLI installed via apt-get in workflow steps.
/var/run is a symlink to /run in Alpine. Mounting emptyDir at /var/run
doesn't override the real /run directory, so dind's docker.sock at
/run/docker.sock was never visible to the runner container.
Fix: Mount the shared emptyDir at /run in both containers.
ROOT CAUSE: Docker socket (/var/run/docker.sock) only existed inside the
dind container — the runner container couldn't see it. The runner connected
to dind via TCP (tcp://localhost:2376) with TLS. But workflow containers
created by the runner had NO way to access the docker daemon:
- unix socket not mounted (runner can't see it)
- DOCKER_HOST env var not passed (runner.envs not configured)
FIX: Share /var/run between dind and runner via emptyDir volume.
When dind starts, it creates /var/run/docker.sock in the shared volume.
Runner can now see the socket. docker_host: automount in config tells
the runner to mount the socket into job containers automatically.
Architecture after fix:
dind container → creates /var/run/docker.sock → shared emptyDir
runner container → sees /var/run/docker.sock → uses automount
workflow container → gets /var/run/docker.sock mounted by runner
Also removed runner.envs (TCP+TLS approach) — unix socket is simpler
and works with automount.
ROOT CAUSE: Workflow containers created by Forgejo runner don't inherit
the DOCKER_HOST/TLS env vars from the runner pod. Docker CLI defaults to
unix:///var/run/docker.sock which doesn't exist inside workflow containers.
The dind sidecar listens on tcp://localhost:2376 with TLS. With
network: host (already set), localhost inside the workflow container
reaches the dind daemon. But docker CLI needs DOCKER_HOST set explicitly.
FIX: Use runner.envs in config.yaml to pass these env vars to every
workflow container:
DOCKER_HOST=tcp://localhost:2376
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/docker-certs/client
The valid_volumes already allows /docker-certs/client (TLS certs).
ROOT CAUSE: All 3 runner labels pointed to code.forgejo.org/forgejo/runner:6
(bare Alpine). When Forgejo runs a workflow, it creates a container FROM the
label image — this container had no Node.js, no docker CLI, no Go/Rust,
no root access, and no apt-get. Every CI job failed.
FIX: Change runner labels to official Debian-based language images:
golang → docker://golang:1.26-bookworm (Go + apt-get + root)
node → docker://node:22-bookworm (Node.js + npm + apt-get + root)
rust → docker://rust:1-bookworm (Rust + cargo + apt-get + root)
The runner daemon pod still uses forgejo/runner:6 — only the label image
(what workflow steps execute in) changes.
Unified CI pattern for ALL repos:
1. Install Node.js first if not present (needed for actions/checkout@v4)
2. Install docker.io via apt-get (needed for docker build/push)
3. Use actions/checkout@v4 normally
4. Build/push with docker
IMPORTANT: Runners must re-register after merge. Delete PVCs or
/data/.runner files to trigger re-registration with new labels.
Problem: actions/checkout@v4 requires Node.js, but base forgejo/runner:6
(Alpine) doesn't have it. We need to test our Dockerfiles on the bare base image.
Solution:
- runs-on: golang (base Alpine runner with dind docker)
- Replace actions/checkout@v4 with git clone (no Node.js needed)
- Clone to /workspace, run all steps there
- Only push on push events (skip on PR to avoid registry pollution)
This validates that our Dockerfile fixes work correctly on base image.
Problem: Forgejo runner base image is Alpine Linux, not Debian.
- apt-get doesn't exist on Alpine (uses apk instead)
- Runner user (1000) can't modify apk database (Permission denied error)
- Workflow used GitHub-specific conditionals (contains() not Forgejo-compatible)
Solution:
1. Replace apt-get with apk add --no-cache for all runner Dockerfiles
2. Switch to USER root before package installation (apk needs root)
3. Switch back to USER 1000:1000 after install (security)
4. Simplify workflow: build all runners in loop (no conditionals)
Dockerfile changes:
- golang: +nodejs +npm +docker-cli via apk
- node: +nodejs +npm +docker-cli via apk
- rust: +nodejs +npm +curl +docker-cli via apk
Workflow trigger:
- Runs on any Dockerfile.* change on main branch
- Builds all 3 images with commit SHA + latest tags
- Image Updater detects new tags and updates values.yaml
After merge to main:
1. CI builds images: forgejo-runner-{golang,node,rust}:SHA
2. Images pushed to registry
3. Image Updater syncs images and commits values.yaml update
4. ArgoCD deploys new runner pods with docker available
Problem: In-cluster pods (portfolio, services) couldn't resolve
api.riotpiao.com because it was missing from CoreDNS rewrite rules.
This broke LLM API calls from portfolio → api gateway even with valid JWT.
Solution: Add rewrite rule to route api.riotpiao.com through nginx ingress
(TLS termination + Host header preservation), matching pattern for other
internal hostnames (authentik.riotpiao.com, minio.riotpiao.com, etc).
Impact:
- Portfolio pod now successfully resolves api.riotpiao.com
- LLM API calls proceed to auth/permission checking
- Applies to all in-cluster services needing LLM gateway
rock