- ExternalSecret syncs age key from Vault to pod
- CRD defines rotation schedule for each secret
- Controller watches CRD, rotates on schedule:
* Call provider API (Authentik/Forgejo/MinIO) for new secret
* Update k8s Secret
* Update .enc.yaml via sops (uses age key from Vault)
* Git commit and push
- Vault is source of truth for age key (never on disk)
- Examples: minio-oidc (90d), portfolio-agent (90d), forgejo-token (90d), minio-root (180d)
MinIO pod was getting 503 from public URL at startup. Use in-cluster
authentik-server.iam.svc for metadata fetch; browser redirects still
use public URLs from OIDC metadata response.
- Roles stored in user attributes, not groups
- Property mapping looks up roles by client_id for client_credentials
- Service account apps have no policy bindings (client_secret = access control)
- Cleanup stale bindings on re-provision
- JWT claims: azp (service identity) + roles (capabilities)
- Move authentik-provision.py to scripts/iam/ (manual-only)
- Remove job/RBAC resources (not needed for local runs)
- Use public URL directly (no sed substitution needed)
- Add app password support via set_key endpoint
- Support both password grant and client_credentials
- Applications now managed by kmsvc-root from kmsvc-manage.git
- Added ServerSideApply to homelab-root for proper annotation sync
- Avoids duplicate Application conflicts with Image Updater
- Fix template to use .Values.gc.schedule instead of hardcoded cron
- Change schedule from daily 03:00 UTC to every 30 minutes
- Prevents DinD PVC fill-up (was at 93% before manual prune)
- portfolio-secrets.enc.env: FORGEJO_TOKEN + LLM_API_TOKEN for api.riotpiao.com
- kustomization: secretGenerator for ksops handling at deploy time
- Will be SOPS encrypted with homelab age key before merge
- Add local-llm OAuth2 provider and application to Authentik provisioning
- Configure JWT-compatible OAuth2 provider (client_id: local-llm)
- Generate client secret on first run, stored in llm-serving/local-llm-jwt Secret
- Bind llm-admins group to local-llm application for admin access
- Add RBAC for provisioning job to create secrets in llm-serving namespace
- Output JWT issuer URL and certificate for local-llm token validation
Schedule runners on az-b (talos-cp-2) which has more Longhorn storage
and breathing room (367Gi available vs worker-1's 369Gi but over-provisioned).
PVCs deleted and will recreate on new zone. Runners will re-register.
Daily 04:00 UTC sweeper in kube-system:
- Delete failed Jobs older than 24h (any namespace)
- Delete completed standalone Jobs older than 72h (no CronJob owner)
- Delete orphan Error/Evicted pods older than 1h
- Self-cleans via ttlSecondsAfterFinished
- Add gc-cronjob.yaml: daily prune of DinD Docker images/volumes/build-cache
and actcache across all forgejo-runner pods. Keeps :latest tagged images,
deletes non-latest older than 72h.
- Expand runner reg PVCs from 1Gi to 20Gi (all three runners) to prevent
action tool cache from filling disk.
- Rust runner gets explicit 20Gi persistence override.
- GC only renders from golang (default) values to avoid duplicate resources.
kmsvc-manage, kmsvc-cli require Go 1.26.0 in go.mod but runner was
using golang:1.25-bookworm container. Update to golang:1.26-bookworm
to match project requirements.
Client credentials + device code grant, no browser redirect (empty
redirect_uris) - unlike every other SERVICES entry which is
authorization_code web SSO. First real step toward replacing
poimen-memory's static API key with a proper JWT flow.
Bind each service's operator Role to a ServiceAccount alongside the
existing oidc:*-admins Group, and wire serviceAccountName into the
pods we control (immich-server, immich-machine-learning,
management-service). poimen-memory's Deployment lives in its own
repo/ArgoCD app, so its SA is created here but not yet wired to a pod.
Confidential client for 'vault login -method=oidc' and the Vault UI's
oidc auth method. homelab-admins gets bound automatically like every
other app in SERVICES. Client secret generated on first provision run
into iam/vault-oidc.
Old root token and unseal keys were dead (lost access to the previous
Vault store). Wiped the S3 backend and vault-0, re-initialized fresh,
rotated these to match the new live unseal keys so ArgoCD's next sync
doesn't clobber them back to the dead ones.
Follows the portainer/kmsvc/temporal pattern - group + "permissions"
claim entry only, no Authentik Application/OAuth provider, since
poimen-memory is an internal API-key service, not browser OIDC login.
rock gets it automatically (already in every service_admin_group).
Cloudflare Tunnel's origin service for img.riotpiao.com self-references
the same public hostname (same pattern as paperless/forgejo/authentik) -
without this rewrite, cloudflared's in-cluster DNS resolution has
nowhere to loop back to, and TLS to the raw ingress-nginx service name
fails cert validation (cert is only valid for *.riotpiao.com).
immich-server crash-looped on "permission denied to create extension
vector" - pgvector's control file isn't marked trusted, and CNPG's
app owner role isn't superuser (enableSuperuserAccess: false).
Documents the fix for future cluster rebuilds; the live cluster
already had CREATE EXTENSION run manually via the postgres pod's
local socket.
2000Gi didn't schedule - "insufficient storage; tags not fulfilled".
The cp-3 HDD's real usable capacity (~3724GiB) minus paperless-media's
2000Gi and ~231GiB of other apps' default-class replicas that Longhorn
placed here anyway (tags only pull matching volumes in, don't exclude
others when the untagged pool elsewhere is full) only leaves ~1493Gi
of real scheduling headroom. 1400Gi fits with margin.
Self-hosted photo backup (Google Photos replacement) - raw manifests,
no Helm chart, self-contained under k8s/apps/immich including its own
CNPG Postgres. Media PVC shares the cp-3 HDD 2TB/2TB with
paperless-media.
Postgres is pg18, not this repo's usual 16.2: CNPG's official pgvector
extension image (ghcr.io/cloudnative-pg/pgvector) is only published
for pg18, loaded via CNPG's ImageVolume extension mechanism (operator
1.30.0 / k8s 1.36.1 both support it). Immich auto-manages CREATE
EXTENSION itself at startup.
OIDC via a new "immich_role" Authentik scope mapping (homelab-admins/
immich-admins -> "admin" claim, else "user"), consumed by Immich's
OAuth roleClaim setting which re-syncs isAdmin on every login - more
reliable than Immich's racy first-user-is-admin fallback. Config
composed into an immich-oidc Secret and mounted as IMMICH_CONFIG_FILE,
matching the paperless-oidc pattern. k8s RBAC (immich-operator Role +
oidc:immich-admins binding) mirrors paperless/rbac.yaml.
immich namespace pre-created in k8s/infra/databases/namespaces.yaml
(not just immich's own CreateNamespace=true) since the iam PostSync
job's RoleBinding needs it to exist before wave 8.
4TB disk on cp-3 was single-tenant for paperless (3500Gi). Splitting
2TB/2TB with the new Immich media PVC on the same disk/tag. Live PVC
and Longhorn volume already deleted+recreated manually (data was
outdated test uploads only, nightly MinIO backup covers it).
paperless-ngx v3 refuses to migrate from anything before v2.20.15
(paperless.E002). 2.20.15 already covers the API version range the
iOS app needs, so it fixes the phone upload issue without the v3
breaking changes.
rock
Story Crater Bot