Re-enable skip_org_role_sync=false to sync Admin role from grafana-admins group.
User creation worked with skip_org_role_sync=true, now restore role sync.
Set skip_org_role_sync=true to allow user creation on first OAuth signin.
When false, Grafana tries to sync org roles before user exists, causing creation to fail.
Reverts to standard OpenID 'preferred_username' claim which is guaranteed to be present in all userinfo responses. The 'email' claim requires explicit scope mapping in Authentik that may not be consistently returned.
Authentik user 'rock' has preferred_username='rock' which matches the existing Grafana user login.
- Commit paperless-ai-config.yaml with production environment variables
- Configure Paperless API endpoint and token
- Enable AI processing with qwen2.5:3b model
- Set up auto-tagging, correspondent/document type extraction
- Configure 60s scan interval for document processing
- Add ConfigMap to kustomization.yaml for GitOps deployment
- Add ConfigMap with production paperless-ai config
- Mount .env config file for app startup
- Enable auto-tagging, correspondent extraction, document type detection
- Set LLM API endpoint and token file location
- Configure 60s scan interval for document processing
- Change login_attribute_path from preferred_username to email for stable user matching
- Enable allow_sign_up to permit OAuth user sync with existing local users
- Root cause: Authentik's 'rock' user matches existing Grafana 'rock' by email, not by preferred_username claim
- Remove hardcoded client secret from init container
- Reference secret from paperless-ai-config (SOPS-encrypted)
- Init container reads LLM_AUTH_CLIENT_SECRET env var
- Secret key must be added to k8s/argocd/secrets/paperless-ai-secrets.enc.yaml
by someone with SOPS/age key access
This removes the plaintext secret from commit history and future
deployments will source it securely from the encrypted Secret.
- Add paperless namespace to llm-serving NetworkPolicy ingress
- Point paperless-ai CUSTOM_BASE_URL to reasoning-predictor directly
- Gateway JWT auth not yet available for service accounts
- paperless-ai-agent has llm:inference role in Authentik
- stale-job-cleanup: run every 2m, delete completed/failed jobs+pods after 2m
- paperless-ai: add PAPERLESS_USERNAME=admin (required to find own user ID)
Standard K8s NetworkPolicy ipBlock rules don't work under Cilium for API server access. Use Cilium-native kube-apiserver entity instead.
Tested: runner pod can now reach K8s API server.
Required by: homelab-frontend PR #25 (Tekton CI).
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #49
Co-authored-by: poimen <[email protected]>
CI runner needs kubectl access to create Tekton PipelineRuns for integration testing.
## Root Cause
The runner egress NetworkPolicy blocks `192.168.1.0/24` (LAN). The K8s API server runs on control-plane nodes in that subnet (`192.168.1.166:6443`). kubectl from inside the DinD container times out.
## Fix
Allow TCP port 6443 to `192.168.1.0/24` — scoped to control-plane API server only.
## Required By
homelab-frontend PR #25 (Tekton integration testing) — CI creates PipelineRuns via kubectl.
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #48
Co-authored-by: poimen <[email protected]>
ROOT CAUSE:
- tektoncd/pipeline config/ dir uses ko:// image refs (build-time placeholders)
- ArgoCD synced the raw dev manifests → InvalidImageName on all pods
- tektoncd/operator requires its own CRDs and controller (too heavy)
- Tekton has no official Helm chart
FIX:
- Vendor the pre-built release.yaml (v0.68.0) into k8s/infra/tekton/
- Point ArgoCD Application at our own repo (forgejo)
- Release contains real container images (ghcr.io/tektoncd/pipeline/*)
- Remove external tektoncd repo from AppProject (not needed)
TO UPGRADE TEKTON:
1. Download new release from github.com/tektoncd/pipeline/releases
2. Replace k8s/infra/tekton/release.yaml
3. Commit — ArgoCD syncs automatically
Required by paperless-ai to find its own user ID for scanning.
Without this, scanning aborts with "Failed to get own user ID".Reviewed-on: #45
Co-authored-by: rock <[email protected]>
Root cause: k8s Service named 'comfyui' auto-injects COMFYUI_PORT=tcp://...
into pod env, clobbering ai-dock's own COMFYUI_PORT variable which expects
a port number. This broke caddy's proxy config, leaving port 8188 dead.
Fix: enableServiceLinks: false, revert to port 8188 (ai-dock default).
Also bump startup probe failureThreshold 60->120 (20min budget for model loading).
- paperless-ai deployment using clusterzx/paperless-ai
- LLM via local api-gateway (reasoning model, no auth - phase 3 pending)
- Paperless API token SOPS-encrypted
- Auto-tags new documents, 5min scan interval
- Adds 'ai-processed' tag to classified documents
Liveness probe was firing after 60s with failureThreshold:6, killing
the server container before it finished applying 200+ DB migrations.
The startup probe (20min timeout) never got a chance to complete.
Root cause: Authentik health checks fail during long DB bootstrap.
Both liveness and startup probes run in parallel. Liveness killed the
pod at 60s; migrations need 2-3min minimum.
Solution: Add initialDelaySeconds:300 to liveness/readiness probes so
they don't fire until 5min have passed (migrations definitely complete).
Worker gets same treatment since it depends on server's DB bootstrap.
- Added queue OAuth2 provider (pk=13, client_id=queue-sqs)
- Added client_credentials grant type to all OAuth2 providers
- Updated temporal-worker-agent service account:
- Added queue:send role
- Added sqs_queues=* claim
- Provisioning script now creates 6 OAuth2 providers (all complete)
- All groups, scopes, and service accounts ready for Phase 2
Updated 2026-09-12 in provision-rbac.py
uid/gid are FAT/NTFS/FUSE mount options, not ext4. Caused mount exit 32
on every newly provisioned CNPG PVC, blocking authentik-db initdb.
CNPG handles postgres ownership via its own init container.
Add ServiceMonitor for llm-serving namespace. Wire prometheus.io annotations to all vLLM pods (reasoning, ornith, embeddings, reranker). Scrape /metrics@8080 every 30s with proper relabeling.
rock
poimenandrock