ROOT CAUSE: Workflow containers created by Forgejo runner don't inherit
the DOCKER_HOST/TLS env vars from the runner pod. Docker CLI defaults to
unix:///var/run/docker.sock which doesn't exist inside workflow containers.
The dind sidecar listens on tcp://localhost:2376 with TLS. With
network: host (already set), localhost inside the workflow container
reaches the dind daemon. But docker CLI needs DOCKER_HOST set explicitly.
FIX: Use runner.envs in config.yaml to pass these env vars to every
workflow container:
DOCKER_HOST=tcp://localhost:2376
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/docker-certs/client
The valid_volumes already allows /docker-certs/client (TLS certs).
ROOT CAUSE: All 3 runner labels pointed to code.forgejo.org/forgejo/runner:6
(bare Alpine). When Forgejo runs a workflow, it creates a container FROM the
label image — this container had no Node.js, no docker CLI, no Go/Rust,
no root access, and no apt-get. Every CI job failed.
FIX: Change runner labels to official Debian-based language images:
golang → docker://golang:1.26-bookworm (Go + apt-get + root)
node → docker://node:22-bookworm (Node.js + npm + apt-get + root)
rust → docker://rust:1-bookworm (Rust + cargo + apt-get + root)
The runner daemon pod still uses forgejo/runner:6 — only the label image
(what workflow steps execute in) changes.
Unified CI pattern for ALL repos:
1. Install Node.js first if not present (needed for actions/checkout@v4)
2. Install docker.io via apt-get (needed for docker build/push)
3. Use actions/checkout@v4 normally
4. Build/push with docker
IMPORTANT: Runners must re-register after merge. Delete PVCs or
/data/.runner files to trigger re-registration with new labels.
Problem: actions/checkout@v4 requires Node.js, but base forgejo/runner:6
(Alpine) doesn't have it. We need to test our Dockerfiles on the bare base image.
Solution:
- runs-on: golang (base Alpine runner with dind docker)
- Replace actions/checkout@v4 with git clone (no Node.js needed)
- Clone to /workspace, run all steps there
- Only push on push events (skip on PR to avoid registry pollution)
This validates that our Dockerfile fixes work correctly on base image.
Problem: Forgejo runner base image is Alpine Linux, not Debian.
- apt-get doesn't exist on Alpine (uses apk instead)
- Runner user (1000) can't modify apk database (Permission denied error)
- Workflow used GitHub-specific conditionals (contains() not Forgejo-compatible)
Solution:
1. Replace apt-get with apk add --no-cache for all runner Dockerfiles
2. Switch to USER root before package installation (apk needs root)
3. Switch back to USER 1000:1000 after install (security)
4. Simplify workflow: build all runners in loop (no conditionals)
Dockerfile changes:
- golang: +nodejs +npm +docker-cli via apk
- node: +nodejs +npm +docker-cli via apk
- rust: +nodejs +npm +curl +docker-cli via apk
Workflow trigger:
- Runs on any Dockerfile.* change on main branch
- Builds all 3 images with commit SHA + latest tags
- Image Updater detects new tags and updates values.yaml
After merge to main:
1. CI builds images: forgejo-runner-{golang,node,rust}:SHA
2. Images pushed to registry
3. Image Updater syncs images and commits values.yaml update
4. ArgoCD deploys new runner pods with docker available
Problem: In-cluster pods (portfolio, services) couldn't resolve
api.riotpiao.com because it was missing from CoreDNS rewrite rules.
This broke LLM API calls from portfolio → api gateway even with valid JWT.
Solution: Add rewrite rule to route api.riotpiao.com through nginx ingress
(TLS termination + Host header preservation), matching pattern for other
internal hostnames (authentik.riotpiao.com, minio.riotpiao.com, etc).
Impact:
- Portfolio pod now successfully resolves api.riotpiao.com
- LLM API calls proceed to auth/permission checking
- Applies to all in-cluster services needing LLM gateway
- All runners now based on code.forgejo.org/forgejo/runner:6 (has runner binary)
- Dockerfile adds Node.js + specialized tools (Go, Rust, docker)
- CI workflow will build custom images and push to registry
- Image Updater will auto-detect and update values.yaml
- ArgoCD will sync new custom images when available
- Dockerfile.node extends node:22-bookworm with docker.io
- No need for install step in every workflow
- Values-node.yaml references custom image
- Build and push manually (no CI needed)
Previous image (golang:1.26-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'
Solution:
- Change golang runner image from golang:1.26-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, Go toolchain, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓
This allows actions/checkout@v4 and other GitHub Actions to run properly
on the golang runner pod.
Note: node:22-bookworm runner already has Node.js, no change needed.
Previous image (rust:1.83-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'
Solution:
- Change rust runner image from rust:1.83-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓
This allows actions/checkout@v4 and other GitHub Actions to run properly
on the rust runner pod.
- Kubernetes provider now uses default-provider-authorization-explicit-consent
instead of implicit-consent flow
- Better handling for public clients like kubelogin (authcode grant)
- Prevents fallback to password grant prompts
- ClusterRoleBinding doesn't use spec: wrapper (unlike Deployment/StatefulSet)
- roleRef and subjects go at top level with metadata
- Fixes: 'strict decoding error: unknown field "spec"'
- RBAC kustomization contains cluster-scoped (ClusterRoleBinding) and
namespace-scoped (Role/RoleBinding) resources
- Each resource has explicit metadata.namespace, so Application shouldn't
force a default namespace
- Fixes: ClusterRoleBinding gets namespace=default, causing sync failure
with 'unsupported role reference kind: ""'
- Reverted to node:22-bookworm (no custom image)
- Removed Dockerfile.node (no CI to build it)
- Docker install step in riotpiao workflow is already the workaround
- Removed .gitea/workflows/build-runner-node.yml
- Homelab is GitOps only, not a buildable artifact
- Runner images managed via direct Dockerfile edits + manual pushes
- Binds oidc:homelab-admins group to cluster-admin ClusterRole
- Allows OIDC users (via Authentik) to have admin access
- Groups claim from Authentik with oidc: prefix per kube-apiserver config
- Enables kubectl access via 'kubectl login' + kubelogin
- Dockerfile.node extends node:22-bookworm + docker.io
- CI workflow builds and pushes to forgejo registry on changes
- values-node.yaml references custom image instead of stock node
- Removes need to install docker in every workflow using node runner
- Removed dependency on external poimen.git repo
- Poimen manifests should be managed locally or via separate workflow
- Simplifies homelab GitOps to only manage homelab-owned services
GrafanaAdmin is server admin only — no org membership, so users couldn't
see dashboards. Now:
- akadmin: GrafanaAdmin (server admin, can impersonate)
- homelab-admins: Admin (org admin, dashboard access)
- others: Viewer
rock