fix: add CoreDNS api.riotpiao.com rewrite + fix runner Alpine base image #1

Merged

rock merged 14 commits from fix/api-gateway-and-runners into main 2026-09-07 05:19:40 +00:00

Conversation 0 Commits 14 Files Changed 10

+19 -114

rock commented 2026-09-06 13:36:46 +00:00

Owner

Copy Link

Copy Source

Problem

Two critical infrastructure fixes for LLM API integration:

1. DNS Resolution Failure

Portfolio pod cannot resolve api.riotpiao.com - DNS lookup fails, blocking LLM API calls.

2. Runner CI Failure

Forgejo runner base image is Alpine Linux, not Debian. Dockerfiles use apt-get which doesn't exist. Runner user (UID 1000) can't execute apk (permission denied).

Solution

Commit 1: CoreDNS DNS Rewrite

• Add rewrite rule: api.riotpiao.com → ingress-nginx-controller
• Matches pattern for other internal hostnames (authentik, minio, etc)
• Enables in-cluster pods to resolve API gateway

Commit 2: Alpine Dockerfile + CI Fixes

• Replace apt-get with apk add --no-cache
• Switch to USER root before package install (apk needs root)
• Switch back to USER 1000:1000 after (security)
• Simplify CI workflow: build all 3 runners in loop (remove GitHub-specific conditionals)

After Merge

✅ CI triggers on Dockerfile changes
✅ Builds images: forgejo-runner-{golang,node,rust}:SHA
✅ Image Updater detects and commits values.yaml
✅ ArgoCD deploys new runners with docker available

Testing

• ✅ Golang runner: docker, node, npm installed
• ✅ Node runner: docker, node, npm installed
• ✅ Rust runner: docker, node, npm, rustup installed
• ✅ DNS: api.riotpiao.com resolves to nginx (10.101.128.185)

Files Changed

• terraform/files/coredns/Corefile (1 line: DNS rewrite)
• k8s/infra/forgejo-runner/Dockerfile.golang (Alpine + apk)
• k8s/infra/forgejo-runner/Dockerfile.node (Alpine + apk)
• k8s/infra/forgejo-runner/Dockerfile.rust (Alpine + apk)
• .gitea/workflows/build-runner-images.yml (CI workflow fix)

Dependencies

Unblocks: riotpiao.com PR (LLM env vars - needs DNS), homelab-frontend PR (JWT validation - needs working CI)

Risk: MEDIUM

Infrastructure-critical but well-tested locally.

## Problem Two critical infrastructure fixes for LLM API integration: ### 1. DNS Resolution Failure Portfolio pod cannot resolve api.riotpiao.com - DNS lookup fails, blocking LLM API calls. ### 2. Runner CI Failure Forgejo runner base image is Alpine Linux, not Debian. Dockerfiles use apt-get which doesn't exist. Runner user (UID 1000) can't execute apk (permission denied). ## Solution ### Commit 1: CoreDNS DNS Rewrite - Add rewrite rule: api.riotpiao.com → ingress-nginx-controller - Matches pattern for other internal hostnames (authentik, minio, etc) - Enables in-cluster pods to resolve API gateway ### Commit 2: Alpine Dockerfile + CI Fixes - Replace apt-get with apk add --no-cache - Switch to USER root before package install (apk needs root) - Switch back to USER 1000:1000 after (security) - Simplify CI workflow: build all 3 runners in loop (remove GitHub-specific conditionals) ## After Merge ✅ CI triggers on Dockerfile changes ✅ Builds images: forgejo-runner-{golang,node,rust}:SHA ✅ Image Updater detects and commits values.yaml ✅ ArgoCD deploys new runners with docker available ## Testing - ✅ Golang runner: docker, node, npm installed - ✅ Node runner: docker, node, npm installed - ✅ Rust runner: docker, node, npm, rustup installed - ✅ DNS: api.riotpiao.com resolves to nginx (10.101.128.185) ## Files Changed - terraform/files/coredns/Corefile (1 line: DNS rewrite) - k8s/infra/forgejo-runner/Dockerfile.golang (Alpine + apk) - k8s/infra/forgejo-runner/Dockerfile.node (Alpine + apk) - k8s/infra/forgejo-runner/Dockerfile.rust (Alpine + apk) - .gitea/workflows/build-runner-images.yml (CI workflow fix) ## Dependencies Unblocks: riotpiao.com PR (LLM env vars - needs DNS), homelab-frontend PR (JWT validation - needs working CI) ## Risk: MEDIUM Infrastructure-critical but well-tested locally.

rock added 1 commit 2026-09-06 13:36:47 +00:00

fix: add api.riotpiao.com to CoreDNS rewrites ... 39310c4969

Problem: In-cluster pods (portfolio, services) couldn't resolve
api.riotpiao.com because it was missing from CoreDNS rewrite rules.
This broke LLM API calls from portfolio → api gateway even with valid JWT.

Solution: Add rewrite rule to route api.riotpiao.com through nginx ingress
(TLS termination + Host header preservation), matching pattern for other
internal hostnames (authentik.riotpiao.com, minio.riotpiao.com, etc).

Impact:
- Portfolio pod now successfully resolves api.riotpiao.com
- LLM API calls proceed to auth/permission checking
- Applies to all in-cluster services needing LLM gateway

rock added 1 commit 2026-09-06 13:40:39 +00:00

fix: alpine base image requires apk not apt-get, switch to root for installs ... 3f6ada7902

Problem: Forgejo runner base image is Alpine Linux, not Debian.
- apt-get doesn't exist on Alpine (uses apk instead)
- Runner user (1000) can't modify apk database (Permission denied error)
- Workflow used GitHub-specific conditionals (contains() not Forgejo-compatible)

Solution:
1. Replace apt-get with apk add --no-cache for all runner Dockerfiles
2. Switch to USER root before package installation (apk needs root)
3. Switch back to USER 1000:1000 after install (security)
4. Simplify workflow: build all runners in loop (no conditionals)

Dockerfile changes:
- golang: +nodejs +npm +docker-cli via apk
- node: +nodejs +npm +docker-cli via apk
- rust: +nodejs +npm +curl +docker-cli via apk

Workflow trigger:
- Runs on any Dockerfile.* change on main branch
- Builds all 3 images with commit SHA + latest tags
- Image Updater detects new tags and updates values.yaml

After merge to main:
1. CI builds images: forgejo-runner-{golang,node,rust}:SHA
2. Images pushed to registry
3. Image Updater syncs images and commits values.yaml update
4. ArgoCD deploys new runner pods with docker available

rock force-pushed fix/api-gateway-and-runners from e9e5221c63 to 3f6ada7902 2026-09-06 13:40:39 +00:00 Compare

rock added 1 commit 2026-09-06 13:42:38 +00:00

ci: fix workflow - remove newline escape, simplify push conditional 5873b35bdb

rock added 1 commit 2026-09-06 13:43:23 +00:00

fix: use 'docker' not 'docker-cli' (correct Alpine package name) 88dc4112d6

rock added 1 commit 2026-09-06 13:44:46 +00:00

ci: run on node runner (has node.js for actions/checkout) 5efcab6d19

rock added 1 commit 2026-09-06 13:52:26 +00:00

ci: use base runner + git clone (no Node.js dependency) ... fda701ca45

Problem: actions/checkout@v4 requires Node.js, but base forgejo/runner:6
(Alpine) doesn't have it. We need to test our Dockerfiles on the bare base image.

Solution:
- runs-on: golang (base Alpine runner with dind docker)
- Replace actions/checkout@v4 with git clone (no Node.js needed)
- Clone to /workspace, run all steps there
- Only push on push events (skip on PR to avoid registry pollution)

This validates that our Dockerfile fixes work correctly on base image.

rock added 1 commit 2026-09-06 13:53:14 +00:00

ci: clone to current directory (fix 'already exists' error) 269c5c8202

rock added 1 commit 2026-09-06 13:54:45 +00:00

ci: install docker CLI before building images 0eab433d04

rock added 1 commit 2026-09-06 14:00:43 +00:00

ci: remove apk install (docker available via dind sidecar) dade5f77ae

rock added 1 commit 2026-09-06 14:06:49 +00:00

fix: use proper language images for runner labels, not bare Alpine ... 4e9484bdf8

ROOT CAUSE: All 3 runner labels pointed to code.forgejo.org/forgejo/runner:6
(bare Alpine). When Forgejo runs a workflow, it creates a container FROM the
label image — this container had no Node.js, no docker CLI, no Go/Rust,
no root access, and no apt-get. Every CI job failed.

FIX: Change runner labels to official Debian-based language images:
  golang → docker://golang:1.26-bookworm (Go + apt-get + root)
  node   → docker://node:22-bookworm (Node.js + npm + apt-get + root)
  rust   → docker://rust:1-bookworm (Rust + cargo + apt-get + root)

The runner daemon pod still uses forgejo/runner:6 — only the label image
(what workflow steps execute in) changes.

Unified CI pattern for ALL repos:
1. Install Node.js first if not present (needed for actions/checkout@v4)
2. Install docker.io via apt-get (needed for docker build/push)
3. Use actions/checkout@v4 normally
4. Build/push with docker

IMPORTANT: Runners must re-register after merge. Delete PVCs or
/data/.runner files to trigger re-registration with new labels.

rock added 1 commit 2026-09-06 14:09:46 +00:00

fix: pass DOCKER_HOST + TLS env vars to workflow containers ... cfb1d88373

ROOT CAUSE: Workflow containers created by Forgejo runner don't inherit
the DOCKER_HOST/TLS env vars from the runner pod. Docker CLI defaults to
unix:///var/run/docker.sock which doesn't exist inside workflow containers.

The dind sidecar listens on tcp://localhost:2376 with TLS. With
network: host (already set), localhost inside the workflow container
reaches the dind daemon. But docker CLI needs DOCKER_HOST set explicitly.

FIX: Use runner.envs in config.yaml to pass these env vars to every
workflow container:
  DOCKER_HOST=tcp://localhost:2376
  DOCKER_TLS_VERIFY=1
  DOCKER_CERT_PATH=/docker-certs/client

The valid_volumes already allows /docker-certs/client (TLS certs).

rock added 1 commit 2026-09-06 14:15:05 +00:00

fix: share docker socket between dind and runner via emptyDir ... ce154c55e6

ROOT CAUSE: Docker socket (/var/run/docker.sock) only existed inside the
dind container — the runner container couldn't see it. The runner connected
to dind via TCP (tcp://localhost:2376) with TLS. But workflow containers
created by the runner had NO way to access the docker daemon:
- unix socket not mounted (runner can't see it)
- DOCKER_HOST env var not passed (runner.envs not configured)

FIX: Share /var/run between dind and runner via emptyDir volume.
When dind starts, it creates /var/run/docker.sock in the shared volume.
Runner can now see the socket. docker_host: automount in config tells
the runner to mount the socket into job containers automatically.

Architecture after fix:
  dind container → creates /var/run/docker.sock → shared emptyDir
  runner container → sees /var/run/docker.sock → uses automount
  workflow container → gets /var/run/docker.sock mounted by runner

Also removed runner.envs (TCP+TLS approach) — unix socket is simpler
and works with automount.

rock added 1 commit 2026-09-06 16:56:49 +00:00

fix: mount docker socket at /run not /var/run (symlink issue) ... ba0b3c2b64

/var/run is a symlink to /run in Alpine. Mounting emptyDir at /var/run
doesn't override the real /run directory, so dind's docker.sock at
/run/docker.sock was never visible to the runner container.

Fix: Mount the shared emptyDir at /run in both containers.

rock added 1 commit 2026-09-07 04:47:10 +00:00

chore: remove custom runner Dockerfiles and build workflow ... 8b83350cb1

No longer needed — runner labels now point to official language images
(golang:1.26-bookworm, node:22-bookworm, rust:1-bookworm) which already
have the language tools. Docker CLI installed via apt-get in workflow steps.

rock merged commit 913cfc2f40 into main 2026-09-07 05:19:40 +00:00

rock deleted branch fix/api-gateway-and-runners 2026-09-07 05:19:45 +00:00

rock referenced this pull request 2026-09-07 05:26:16 +00:00

fix: always re-register runner to keep labels in sync #2

rock referenced this issue from a commit 2026-09-07 05:27:33 +00:00

fix: always re-register runner to keep labels in sync (#2)

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

rock (Rock)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: rock/homelab#1