Reverts to standard OpenID 'preferred_username' claim which is guaranteed to be present in all userinfo responses. The 'email' claim requires explicit scope mapping in Authentik that may not be consistently returned.
Authentik user 'rock' has preferred_username='rock' which matches the existing Grafana user login.
- Change login_attribute_path from preferred_username to email for stable user matching
- Enable allow_sign_up to permit OAuth user sync with existing local users
- Root cause: Authentik's 'rock' user matches existing Grafana 'rock' by email, not by preferred_username claim
- stale-job-cleanup: run every 2m, delete completed/failed jobs+pods after 2m
- paperless-ai: add PAPERLESS_USERNAME=admin (required to find own user ID)
Standard K8s NetworkPolicy ipBlock rules don't work under Cilium for API server access. Use Cilium-native kube-apiserver entity instead.
Tested: runner pod can now reach K8s API server.
Required by: homelab-frontend PR #25 (Tekton CI).
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #49
Co-authored-by: poimen <[email protected]>
CI runner needs kubectl access to create Tekton PipelineRuns for integration testing.
## Root Cause
The runner egress NetworkPolicy blocks `192.168.1.0/24` (LAN). The K8s API server runs on control-plane nodes in that subnet (`192.168.1.166:6443`). kubectl from inside the DinD container times out.
## Fix
Allow TCP port 6443 to `192.168.1.0/24` — scoped to control-plane API server only.
## Required By
homelab-frontend PR #25 (Tekton integration testing) — CI creates PipelineRuns via kubectl.
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #48
Co-authored-by: poimen <[email protected]>
ROOT CAUSE:
- tektoncd/pipeline config/ dir uses ko:// image refs (build-time placeholders)
- ArgoCD synced the raw dev manifests → InvalidImageName on all pods
- tektoncd/operator requires its own CRDs and controller (too heavy)
- Tekton has no official Helm chart
FIX:
- Vendor the pre-built release.yaml (v0.68.0) into k8s/infra/tekton/
- Point ArgoCD Application at our own repo (forgejo)
- Release contains real container images (ghcr.io/tektoncd/pipeline/*)
- Remove external tektoncd repo from AppProject (not needed)
TO UPGRADE TEKTON:
1. Download new release from github.com/tektoncd/pipeline/releases
2. Replace k8s/infra/tekton/release.yaml
3. Commit — ArgoCD syncs automatically
Liveness probe was firing after 60s with failureThreshold:6, killing
the server container before it finished applying 200+ DB migrations.
The startup probe (20min timeout) never got a chance to complete.
Root cause: Authentik health checks fail during long DB bootstrap.
Both liveness and startup probes run in parallel. Liveness killed the
pod at 60s; migrations need 2-3min minimum.
Solution: Add initialDelaySeconds:300 to liveness/readiness probes so
they don't fire until 5min have passed (migrations definitely complete).
Worker gets same treatment since it depends on server's DB bootstrap.
uid/gid are FAT/NTFS/FUSE mount options, not ext4. Caused mount exit 32
on every newly provisioned CNPG PVC, blocking authentik-db initdb.
CNPG handles postgres ownership via its own init container.
Add ServiceMonitor for llm-serving namespace. Wire prometheus.io annotations to all vLLM pods (reasoning, ornith, embeddings, reranker). Scrape /metrics@8080 every 30s with proper relabeling.
Job containers spawned by the runner run inside dind. With TCP+TLS
(tcp://localhost:2376), localhost inside those containers doesn't
reach the dind sidecar. Unix socket at /run/docker.sock works because
both runner and dind share the /run emptyDir.
Also disables DOCKER_TLS_CERTDIR so dind creates the socket instead
of only listening on TLS TCP.
Previous image (golang:1.26-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'
Solution:
- Change golang runner image from golang:1.26-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, Go toolchain, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓
This allows actions/checkout@v4 and other GitHub Actions to run properly
on the golang runner pod.
Note: node:22-bookworm runner already has Node.js, no change needed.
Previous image (rust:1.83-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'
Solution:
- Change rust runner image from rust:1.83-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓
This allows actions/checkout@v4 and other GitHub Actions to run properly
on the rust runner pod.
- ClusterRoleBinding doesn't use spec: wrapper (unlike Deployment/StatefulSet)
- roleRef and subjects go at top level with metadata
- Fixes: 'strict decoding error: unknown field "spec"'
- Reverted to node:22-bookworm (no custom image)
- Removed Dockerfile.node (no CI to build it)
- Docker install step in riotpiao workflow is already the workaround
- Removed .gitea/workflows/build-runner-node.yml
- Homelab is GitOps only, not a buildable artifact
- Runner images managed via direct Dockerfile edits + manual pushes
- Binds oidc:homelab-admins group to cluster-admin ClusterRole
- Allows OIDC users (via Authentik) to have admin access
- Groups claim from Authentik with oidc: prefix per kube-apiserver config
- Enables kubectl access via 'kubectl login' + kubelogin
- Dockerfile.node extends node:22-bookworm + docker.io
- CI workflow builds and pushes to forgejo registry on changes
- values-node.yaml references custom image instead of stock node
- Removes need to install docker in every workflow using node runner
GrafanaAdmin is server admin only — no org membership, so users couldn't
see dashboards. Now:
- akadmin: GrafanaAdmin (server admin, can impersonate)
- homelab-admins: Admin (org admin, dashboard access)
- others: Viewer
MinIO pod was getting 503 from public URL at startup. Use in-cluster
authentik-server.iam.svc for metadata fetch; browser redirects still
use public URLs from OIDC metadata response.
- Move authentik-provision.py to scripts/iam/ (manual-only)
- Remove job/RBAC resources (not needed for local runs)
- Use public URL directly (no sed substitution needed)
- Add app password support via set_key endpoint
- Support both password grant and client_credentials
rock
poimenandrock