Init container skipped registration if .runner file existed on PVC.
Changing labels in values.yaml had no effect until PVC was manually
deleted — not GitOps-friendly.
Fix: rm -f .runner and re-register on every pod start.
1. Forgejo CI is broken across all repos
Every workflow fails because runner labels point to a bare Alpine image with nothing in it.
┌────────────────────────────────────────┬──────────────────────────────────────┐
│ Before │ After │
├────────────────────────────────────────┼──────────────────────────────────────┤
│ golang:docker://forgejo/runner:6 │ golang:docker://golang:1.26-bookworm │
├────────────────────────────────────────┼──────────────────────────────────────┤
│ No Go, no Node.js, no apt-get, no root │ Go, git, apt-get, root │
└────────────────────────────────────────┴──────────────────────────────────────┘
Plus the docker socket isn't shared between dind sidecar and runner, so even if docker CLI existed, it can't reach the daemon.
┌───────────────────────────────────────────┬─────────────────────────────────────────────────────────────┐
│ Before │ After │
├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ dind creates socket in its own filesystem │ Shared /run emptyDir volume │
├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ Runner can't see it │ Both containers see /run/docker.sock │
├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ No docker_host config │ docker_host: automount passes socket to workflow containers │
Co-authored-by: rock <[email protected]>
- All runners now based on code.forgejo.org/forgejo/runner:6 (has runner binary)
- Dockerfile adds Node.js + specialized tools (Go, Rust, docker)
- CI workflow will build custom images and push to registry
- Image Updater will auto-detect and update values.yaml
- ArgoCD will sync new custom images when available
- Dockerfile.node extends node:22-bookworm with docker.io
- No need for install step in every workflow
- Values-node.yaml references custom image
- Build and push manually (no CI needed)
Previous image (golang:1.26-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'
Solution:
- Change golang runner image from golang:1.26-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, Go toolchain, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓
This allows actions/checkout@v4 and other GitHub Actions to run properly
on the golang runner pod.
Note: node:22-bookworm runner already has Node.js, no change needed.
Previous image (rust:1.83-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'
Solution:
- Change rust runner image from rust:1.83-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓
This allows actions/checkout@v4 and other GitHub Actions to run properly
on the rust runner pod.
- Kubernetes provider now uses default-provider-authorization-explicit-consent
instead of implicit-consent flow
- Better handling for public clients like kubelogin (authcode grant)
- Prevents fallback to password grant prompts
- ClusterRoleBinding doesn't use spec: wrapper (unlike Deployment/StatefulSet)
- roleRef and subjects go at top level with metadata
- Fixes: 'strict decoding error: unknown field "spec"'
- RBAC kustomization contains cluster-scoped (ClusterRoleBinding) and
namespace-scoped (Role/RoleBinding) resources
- Each resource has explicit metadata.namespace, so Application shouldn't
force a default namespace
- Fixes: ClusterRoleBinding gets namespace=default, causing sync failure
with 'unsupported role reference kind: ""'
- Reverted to node:22-bookworm (no custom image)
- Removed Dockerfile.node (no CI to build it)
- Docker install step in riotpiao workflow is already the workaround
- Removed .gitea/workflows/build-runner-node.yml
- Homelab is GitOps only, not a buildable artifact
- Runner images managed via direct Dockerfile edits + manual pushes
- Binds oidc:homelab-admins group to cluster-admin ClusterRole
- Allows OIDC users (via Authentik) to have admin access
- Groups claim from Authentik with oidc: prefix per kube-apiserver config
- Enables kubectl access via 'kubectl login' + kubelogin
- Dockerfile.node extends node:22-bookworm + docker.io
- CI workflow builds and pushes to forgejo registry on changes
- values-node.yaml references custom image instead of stock node
- Removes need to install docker in every workflow using node runner
- Removed dependency on external poimen.git repo
- Poimen manifests should be managed locally or via separate workflow
- Simplifies homelab GitOps to only manage homelab-owned services
GrafanaAdmin is server admin only — no org membership, so users couldn't
see dashboards. Now:
- akadmin: GrafanaAdmin (server admin, can impersonate)
- homelab-admins: Admin (org admin, dashboard access)
- others: Viewer
- ExternalSecret syncs age key from Vault to pod
- CRD defines rotation schedule for each secret
- Controller watches CRD, rotates on schedule:
* Call provider API (Authentik/Forgejo/MinIO) for new secret
* Update k8s Secret
* Update .enc.yaml via sops (uses age key from Vault)
* Git commit and push
- Vault is source of truth for age key (never on disk)
- Examples: minio-oidc (90d), portfolio-agent (90d), forgejo-token (90d), minio-root (180d)
MinIO pod was getting 503 from public URL at startup. Use in-cluster
authentik-server.iam.svc for metadata fetch; browser redirects still
use public URLs from OIDC metadata response.
- Roles stored in user attributes, not groups
- Property mapping looks up roles by client_id for client_credentials
- Service account apps have no policy bindings (client_secret = access control)
- Cleanup stale bindings on re-provision
- JWT claims: azp (service identity) + roles (capabilities)
- Move authentik-provision.py to scripts/iam/ (manual-only)
- Remove job/RBAC resources (not needed for local runs)
- Use public URL directly (no sed substitution needed)
- Add app password support via set_key endpoint
- Support both password grant and client_credentials
- Applications now managed by kmsvc-root from kmsvc-manage.git
- Added ServerSideApply to homelab-root for proper annotation sync
- Avoids duplicate Application conflicts with Image Updater
- Fix template to use .Values.gc.schedule instead of hardcoded cron
- Change schedule from daily 03:00 UTC to every 30 minutes
- Prevents DinD PVC fill-up (was at 93% before manual prune)
- portfolio-secrets.enc.env: FORGEJO_TOKEN + LLM_API_TOKEN for api.riotpiao.com
- kustomization: secretGenerator for ksops handling at deploy time
- Will be SOPS encrypted with homelab age key before merge
- Add local-llm OAuth2 provider and application to Authentik provisioning
- Configure JWT-compatible OAuth2 provider (client_id: local-llm)
- Generate client secret on first run, stored in llm-serving/local-llm-jwt Secret
- Bind llm-admins group to local-llm application for admin access
- Add RBAC for provisioning job to create secrets in llm-serving namespace
- Output JWT issuer URL and certificate for local-llm token validation
Schedule runners on az-b (talos-cp-2) which has more Longhorn storage
and breathing room (367Gi available vs worker-1's 369Gi but over-provisioned).
PVCs deleted and will recreate on new zone. Runners will re-register.
Daily 04:00 UTC sweeper in kube-system:
- Delete failed Jobs older than 24h (any namespace)
- Delete completed standalone Jobs older than 72h (no CronJob owner)
- Delete orphan Error/Evicted pods older than 1h
- Self-cleans via ttlSecondsAfterFinished
- Add gc-cronjob.yaml: daily prune of DinD Docker images/volumes/build-cache
and actcache across all forgejo-runner pods. Keeps :latest tagged images,
deletes non-latest older than 72h.
- Expand runner reg PVCs from 1Gi to 20Gi (all three runners) to prevent
action tool cache from filling disk.
- Rust runner gets explicit 20Gi persistence override.
- GC only renders from golang (default) values to avoid duplicate resources.
kmsvc-manage, kmsvc-cli require Go 1.26.0 in go.mod but runner was
using golang:1.25-bookworm container. Update to golang:1.26-bookworm
to match project requirements.
rock