Standard K8s NetworkPolicy ipBlock rules don't work under Cilium for
the API server — the except clause on 192.168.1.0/24 blocks the
post-DNAT destination even when a separate rule re-allows the subnet.
Cilium's native kube-apiserver entity tracks API server endpoints
regardless of ClusterIP vs node-IP routing.
Replaces: ipBlock 192.168.1.0/24:6443 (broken under Cilium)
With: CiliumNetworkPolicy toEntities: kube-apiserver (works)
CI runner needs kubectl access to create Tekton PipelineRuns for integration testing.
## Root Cause
The runner egress NetworkPolicy blocks `192.168.1.0/24` (LAN). The K8s API server runs on control-plane nodes in that subnet (`192.168.1.166:6443`). kubectl from inside the DinD container times out.
## Fix
Allow TCP port 6443 to `192.168.1.0/24` — scoped to control-plane API server only.
## Required By
homelab-frontend PR #25 (Tekton integration testing) — CI creates PipelineRuns via kubectl.
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #48
Co-authored-by: poimen <[email protected]>
ROOT CAUSE:
- tektoncd/pipeline config/ dir uses ko:// image refs (build-time placeholders)
- ArgoCD synced the raw dev manifests → InvalidImageName on all pods
- tektoncd/operator requires its own CRDs and controller (too heavy)
- Tekton has no official Helm chart
FIX:
- Vendor the pre-built release.yaml (v0.68.0) into k8s/infra/tekton/
- Point ArgoCD Application at our own repo (forgejo)
- Release contains real container images (ghcr.io/tektoncd/pipeline/*)
- Remove external tektoncd repo from AppProject (not needed)
TO UPGRADE TEKTON:
1. Download new release from github.com/tektoncd/pipeline/releases
2. Replace k8s/infra/tekton/release.yaml
3. Commit — ArgoCD syncs automatically
Required by paperless-ai to find its own user ID for scanning.
Without this, scanning aborts with "Failed to get own user ID".Reviewed-on: #45
Co-authored-by: rock <[email protected]>
Root cause: k8s Service named 'comfyui' auto-injects COMFYUI_PORT=tcp://...
into pod env, clobbering ai-dock's own COMFYUI_PORT variable which expects
a port number. This broke caddy's proxy config, leaving port 8188 dead.
Fix: enableServiceLinks: false, revert to port 8188 (ai-dock default).
Also bump startup probe failureThreshold 60->120 (20min budget for model loading).
- paperless-ai deployment using clusterzx/paperless-ai
- LLM via local api-gateway (reasoning model, no auth - phase 3 pending)
- Paperless API token SOPS-encrypted
- Auto-tags new documents, 5min scan interval
- Adds 'ai-processed' tag to classified documents
Liveness probe was firing after 60s with failureThreshold:6, killing
the server container before it finished applying 200+ DB migrations.
The startup probe (20min timeout) never got a chance to complete.
Root cause: Authentik health checks fail during long DB bootstrap.
Both liveness and startup probes run in parallel. Liveness killed the
pod at 60s; migrations need 2-3min minimum.
Solution: Add initialDelaySeconds:300 to liveness/readiness probes so
they don't fire until 5min have passed (migrations definitely complete).
Worker gets same treatment since it depends on server's DB bootstrap.
- Added queue OAuth2 provider (pk=13, client_id=queue-sqs)
- Added client_credentials grant type to all OAuth2 providers
- Updated temporal-worker-agent service account:
- Added queue:send role
- Added sqs_queues=* claim
- Provisioning script now creates 6 OAuth2 providers (all complete)
- All groups, scopes, and service accounts ready for Phase 2
Updated 2026-09-12 in provision-rbac.py
uid/gid are FAT/NTFS/FUSE mount options, not ext4. Caused mount exit 32
on every newly provisioned CNPG PVC, blocking authentik-db initdb.
CNPG handles postgres ownership via its own init container.
Add ServiceMonitor for llm-serving namespace. Wire prometheus.io annotations to all vLLM pods (reasoning, ornith, embeddings, reranker). Scrape /metrics@8080 every 30s with proper relabeling.
ArgoCD couldn't decrypt secrets.yaml because it was listed as a plain
kustomize resource. Move the 3 secrets (gotify-admin, gotify-tokens,
gotify-smtp) to k8s/argocd/secrets/ as .enc.yaml files processed by
the ksops generator, matching the repo convention.
Fixes ComparisonError: 'Object Kind is missing' (SOPS ciphertext
parsed as raw YAML).
Repos transferred: homelab-frontend, kmsvc-manage, poimen, poimen-memory,
poimen-workflows, poimen-frontend. Old URLs return 301 which ArgoCD
doesn't follow.
NOT changed: container image registry paths (rock/ is correct for registry),
riotpiao.com (still under rock org).
Also adds poimen-frontend to AppProject sourceRepos allowlist.
- Change container port from 8188 to 8888 (Caddy proxy binding)
- Remove PVC, use emptyDir for ephemeral models/output
- Replace startup + readiness probes with single liveness probe
- Remove explicit COMFYUI_FLAGS (container defaults work)
- Pod now reaches Ready state immediately after image pull
Fixes GPU contention by using ephemeral storage. ComfyUI now runs and is accessible at https://comfy.riotpiao.comReviewed-on: #18
Co-authored-by: poimen <[email protected]>
Reviewed-on: riotpiao-poimen/homelab-frontend#21
Co-authored-by: rock <[email protected]>
## Summary
Replaces hand-applied `llm-serving-default-deny` NetworkPolicy with a git-managed, namespace-scoped policy that only allows traffic from the api-gateway.
**Issue:** #13
Co-authored-by: rock <[email protected]>
- buildOptions field not available in ArgoCD v3.4.5
- SOPS decryption already handled by repo-server ksops plugin
- Revert to simple kustomize config
- Portfolio Application can now sync properly
- Add kustomize config with --enable-alpha-plugins to support ksops
- Allows ArgoCD to properly decrypt SOPS-encrypted files
- Fixes Image Updater compatibility with sops field in kustomization.yaml
Job containers spawned by the runner run inside dind. With TCP+TLS
(tcp://localhost:2376), localhost inside those containers doesn't
reach the dind sidecar. Unix socket at /run/docker.sock works because
both runner and dind share the /run emptyDir.
Also disables DOCKER_TLS_CERTDIR so dind creates the socket instead
of only listening on TLS TCP.
rock
poimenandrock