- Add SOCIALACCOUNT_AUTO_SIGNUP for automatic social account creation
- Enable ACCOUNT_ALLOW_SIGNUPS for OIDC users
- Add SOCIALACCOUNT_EMAIL_AUTHENTICATION_AUTO_CONNECT to link existing users
- Fixes 'An account already exists with this email' error
- Allows Authentik login to auto-link to existing Paperless users
Re-enable skip_org_role_sync=false to sync Admin role from grafana-admins group.
User creation worked with skip_org_role_sync=true, now restore role sync.
Set skip_org_role_sync=true to allow user creation on first OAuth signin.
When false, Grafana tries to sync org roles before user exists, causing creation to fail.
Reverts to standard OpenID 'preferred_username' claim which is guaranteed to be present in all userinfo responses. The 'email' claim requires explicit scope mapping in Authentik that may not be consistently returned.
Authentik user 'rock' has preferred_username='rock' which matches the existing Grafana user login.
- Commit paperless-ai-config.yaml with production environment variables
- Configure Paperless API endpoint and token
- Enable AI processing with qwen2.5:3b model
- Set up auto-tagging, correspondent/document type extraction
- Configure 60s scan interval for document processing
- Add ConfigMap to kustomization.yaml for GitOps deployment
- Add ConfigMap with production paperless-ai config
- Mount .env config file for app startup
- Enable auto-tagging, correspondent extraction, document type detection
- Set LLM API endpoint and token file location
- Configure 60s scan interval for document processing
- Change login_attribute_path from preferred_username to email for stable user matching
- Enable allow_sign_up to permit OAuth user sync with existing local users
- Root cause: Authentik's 'rock' user matches existing Grafana 'rock' by email, not by preferred_username claim
- Remove hardcoded client secret from init container
- Reference secret from paperless-ai-config (SOPS-encrypted)
- Init container reads LLM_AUTH_CLIENT_SECRET env var
- Secret key must be added to k8s/argocd/secrets/paperless-ai-secrets.enc.yaml
by someone with SOPS/age key access
This removes the plaintext secret from commit history and future
deployments will source it securely from the encrypted Secret.
- Add paperless namespace to llm-serving NetworkPolicy ingress
- Point paperless-ai CUSTOM_BASE_URL to reasoning-predictor directly
- Gateway JWT auth not yet available for service accounts
- paperless-ai-agent has llm:inference role in Authentik
- stale-job-cleanup: run every 2m, delete completed/failed jobs+pods after 2m
- paperless-ai: add PAPERLESS_USERNAME=admin (required to find own user ID)
ROOT CAUSE:
- tektoncd/pipeline config/ dir uses ko:// image refs (build-time placeholders)
- ArgoCD synced the raw dev manifests → InvalidImageName on all pods
- tektoncd/operator requires its own CRDs and controller (too heavy)
- Tekton has no official Helm chart
FIX:
- Vendor the pre-built release.yaml (v0.68.0) into k8s/infra/tekton/
- Point ArgoCD Application at our own repo (forgejo)
- Release contains real container images (ghcr.io/tektoncd/pipeline/*)
- Remove external tektoncd repo from AppProject (not needed)
TO UPGRADE TEKTON:
1. Download new release from github.com/tektoncd/pipeline/releases
2. Replace k8s/infra/tekton/release.yaml
3. Commit — ArgoCD syncs automatically
Required by paperless-ai to find its own user ID for scanning.
Without this, scanning aborts with "Failed to get own user ID".Reviewed-on: #45
Co-authored-by: rock <[email protected]>
Root cause: k8s Service named 'comfyui' auto-injects COMFYUI_PORT=tcp://...
into pod env, clobbering ai-dock's own COMFYUI_PORT variable which expects
a port number. This broke caddy's proxy config, leaving port 8188 dead.
Fix: enableServiceLinks: false, revert to port 8188 (ai-dock default).
Also bump startup probe failureThreshold 60->120 (20min budget for model loading).
- paperless-ai deployment using clusterzx/paperless-ai
- LLM via local api-gateway (reasoning model, no auth - phase 3 pending)
- Paperless API token SOPS-encrypted
- Auto-tags new documents, 5min scan interval
- Adds 'ai-processed' tag to classified documents
Liveness probe was firing after 60s with failureThreshold:6, killing
the server container before it finished applying 200+ DB migrations.
The startup probe (20min timeout) never got a chance to complete.
Root cause: Authentik health checks fail during long DB bootstrap.
Both liveness and startup probes run in parallel. Liveness killed the
pod at 60s; migrations need 2-3min minimum.
Solution: Add initialDelaySeconds:300 to liveness/readiness probes so
they don't fire until 5min have passed (migrations definitely complete).
Worker gets same treatment since it depends on server's DB bootstrap.
- Added queue OAuth2 provider (pk=13, client_id=queue-sqs)
- Added client_credentials grant type to all OAuth2 providers
- Updated temporal-worker-agent service account:
- Added queue:send role
- Added sqs_queues=* claim
- Provisioning script now creates 6 OAuth2 providers (all complete)
- All groups, scopes, and service accounts ready for Phase 2
Updated 2026-09-12 in provision-rbac.py
uid/gid are FAT/NTFS/FUSE mount options, not ext4. Caused mount exit 32
on every newly provisioned CNPG PVC, blocking authentik-db initdb.
CNPG handles postgres ownership via its own init container.
Add ServiceMonitor for llm-serving namespace. Wire prometheus.io annotations to all vLLM pods (reasoning, ornith, embeddings, reranker). Scrape /metrics@8080 every 30s with proper relabeling.
ArgoCD couldn't decrypt secrets.yaml because it was listed as a plain
kustomize resource. Move the 3 secrets (gotify-admin, gotify-tokens,
gotify-smtp) to k8s/argocd/secrets/ as .enc.yaml files processed by
the ksops generator, matching the repo convention.
Fixes ComparisonError: 'Object Kind is missing' (SOPS ciphertext
parsed as raw YAML).
Repos transferred: homelab-frontend, kmsvc-manage, poimen, poimen-memory,
poimen-workflows, poimen-frontend. Old URLs return 301 which ArgoCD
doesn't follow.
NOT changed: container image registry paths (rock/ is correct for registry),
riotpiao.com (still under rock org).
Also adds poimen-frontend to AppProject sourceRepos allowlist.
- Change container port from 8188 to 8888 (Caddy proxy binding)
- Remove PVC, use emptyDir for ephemeral models/output
- Replace startup + readiness probes with single liveness probe
- Remove explicit COMFYUI_FLAGS (container defaults work)
- Pod now reaches Ready state immediately after image pull
Fixes GPU contention by using ephemeral storage. ComfyUI now runs and is accessible at https://comfy.riotpiao.comReviewed-on: #18
Co-authored-by: poimen <[email protected]>
Reviewed-on: riotpiao-poimen/homelab-frontend#21
Co-authored-by: rock <[email protected]>
## Summary
Replaces hand-applied `llm-serving-default-deny` NetworkPolicy with a git-managed, namespace-scoped policy that only allows traffic from the api-gateway.
**Issue:** #13
Co-authored-by: rock <[email protected]>
- buildOptions field not available in ArgoCD v3.4.5
- SOPS decryption already handled by repo-server ksops plugin
- Revert to simple kustomize config
- Portfolio Application can now sync properly
- Add kustomize config with --enable-alpha-plugins to support ksops
- Allows ArgoCD to properly decrypt SOPS-encrypted files
- Fixes Image Updater compatibility with sops field in kustomization.yaml
Job containers spawned by the runner run inside dind. With TCP+TLS
(tcp://localhost:2376), localhost inside those containers doesn't
reach the dind sidecar. Unix socket at /run/docker.sock works because
both runner and dind share the /run emptyDir.
Also disables DOCKER_TLS_CERTDIR so dind creates the socket instead
of only listening on TLS TCP.
Previous image (golang:1.26-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'
Solution:
- Change golang runner image from golang:1.26-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, Go toolchain, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓
This allows actions/checkout@v4 and other GitHub Actions to run properly
on the golang runner pod.
Note: node:22-bookworm runner already has Node.js, no change needed.
Previous image (rust:1.83-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'
Solution:
- Change rust runner image from rust:1.83-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓
This allows actions/checkout@v4 and other GitHub Actions to run properly
on the rust runner pod.
- ClusterRoleBinding doesn't use spec: wrapper (unlike Deployment/StatefulSet)
- roleRef and subjects go at top level with metadata
- Fixes: 'strict decoding error: unknown field "spec"'
- RBAC kustomization contains cluster-scoped (ClusterRoleBinding) and
namespace-scoped (Role/RoleBinding) resources
- Each resource has explicit metadata.namespace, so Application shouldn't
force a default namespace
- Fixes: ClusterRoleBinding gets namespace=default, causing sync failure
with 'unsupported role reference kind: ""'
- Reverted to node:22-bookworm (no custom image)
- Removed Dockerfile.node (no CI to build it)
- Docker install step in riotpiao workflow is already the workaround
- Removed .gitea/workflows/build-runner-node.yml
- Homelab is GitOps only, not a buildable artifact
- Runner images managed via direct Dockerfile edits + manual pushes
- Binds oidc:homelab-admins group to cluster-admin ClusterRole
- Allows OIDC users (via Authentik) to have admin access
- Groups claim from Authentik with oidc: prefix per kube-apiserver config
- Enables kubectl access via 'kubectl login' + kubelogin
- Dockerfile.node extends node:22-bookworm + docker.io
- CI workflow builds and pushes to forgejo registry on changes
- values-node.yaml references custom image instead of stock node
- Removes need to install docker in every workflow using node runner
- Removed dependency on external poimen.git repo
- Poimen manifests should be managed locally or via separate workflow
- Simplifies homelab GitOps to only manage homelab-owned services
GrafanaAdmin is server admin only — no org membership, so users couldn't
see dashboards. Now:
- akadmin: GrafanaAdmin (server admin, can impersonate)
- homelab-admins: Admin (org admin, dashboard access)
- others: Viewer
- ExternalSecret syncs age key from Vault to pod
- CRD defines rotation schedule for each secret
- Controller watches CRD, rotates on schedule:
* Call provider API (Authentik/Forgejo/MinIO) for new secret
* Update k8s Secret
* Update .enc.yaml via sops (uses age key from Vault)
* Git commit and push
- Vault is source of truth for age key (never on disk)
- Examples: minio-oidc (90d), portfolio-agent (90d), forgejo-token (90d), minio-root (180d)
MinIO pod was getting 503 from public URL at startup. Use in-cluster
authentik-server.iam.svc for metadata fetch; browser redirects still
use public URLs from OIDC metadata response.
- Move authentik-provision.py to scripts/iam/ (manual-only)
- Remove job/RBAC resources (not needed for local runs)
- Use public URL directly (no sed substitution needed)
- Add app password support via set_key endpoint
- Support both password grant and client_credentials
- Applications now managed by kmsvc-root from kmsvc-manage.git
- Added ServerSideApply to homelab-root for proper annotation sync
- Avoids duplicate Application conflicts with Image Updater
rock
