- portfolio-secrets.enc.env: FORGEJO_TOKEN + LLM_API_TOKEN for api.riotpiao.com
- kustomization: secretGenerator for ksops handling at deploy time
- Will be SOPS encrypted with homelab age key before merge
- Add local-llm OAuth2 provider and application to Authentik provisioning
- Configure JWT-compatible OAuth2 provider (client_id: local-llm)
- Generate client secret on first run, stored in llm-serving/local-llm-jwt Secret
- Bind llm-admins group to local-llm application for admin access
- Add RBAC for provisioning job to create secrets in llm-serving namespace
- Output JWT issuer URL and certificate for local-llm token validation
Schedule runners on az-b (talos-cp-2) which has more Longhorn storage
and breathing room (367Gi available vs worker-1's 369Gi but over-provisioned).
PVCs deleted and will recreate on new zone. Runners will re-register.
Daily 04:00 UTC sweeper in kube-system:
- Delete failed Jobs older than 24h (any namespace)
- Delete completed standalone Jobs older than 72h (no CronJob owner)
- Delete orphan Error/Evicted pods older than 1h
- Self-cleans via ttlSecondsAfterFinished
- Add gc-cronjob.yaml: daily prune of DinD Docker images/volumes/build-cache
and actcache across all forgejo-runner pods. Keeps :latest tagged images,
deletes non-latest older than 72h.
- Expand runner reg PVCs from 1Gi to 20Gi (all three runners) to prevent
action tool cache from filling disk.
- Rust runner gets explicit 20Gi persistence override.
- GC only renders from golang (default) values to avoid duplicate resources.
kmsvc-manage, kmsvc-cli require Go 1.26.0 in go.mod but runner was
using golang:1.25-bookworm container. Update to golang:1.26-bookworm
to match project requirements.
Client credentials + device code grant, no browser redirect (empty
redirect_uris) - unlike every other SERVICES entry which is
authorization_code web SSO. First real step toward replacing
poimen-memory's static API key with a proper JWT flow.
Bind each service's operator Role to a ServiceAccount alongside the
existing oidc:*-admins Group, and wire serviceAccountName into the
pods we control (immich-server, immich-machine-learning,
management-service). poimen-memory's Deployment lives in its own
repo/ArgoCD app, so its SA is created here but not yet wired to a pod.
Confidential client for 'vault login -method=oidc' and the Vault UI's
oidc auth method. homelab-admins gets bound automatically like every
other app in SERVICES. Client secret generated on first provision run
into iam/vault-oidc.
Old root token and unseal keys were dead (lost access to the previous
Vault store). Wiped the S3 backend and vault-0, re-initialized fresh,
rotated these to match the new live unseal keys so ArgoCD's next sync
doesn't clobber them back to the dead ones.
Follows the portainer/kmsvc/temporal pattern - group + "permissions"
claim entry only, no Authentik Application/OAuth provider, since
poimen-memory is an internal API-key service, not browser OIDC login.
rock gets it automatically (already in every service_admin_group).
immich-server crash-looped on "permission denied to create extension
vector" - pgvector's control file isn't marked trusted, and CNPG's
app owner role isn't superuser (enableSuperuserAccess: false).
Documents the fix for future cluster rebuilds; the live cluster
already had CREATE EXTENSION run manually via the postgres pod's
local socket.
2000Gi didn't schedule - "insufficient storage; tags not fulfilled".
The cp-3 HDD's real usable capacity (~3724GiB) minus paperless-media's
2000Gi and ~231GiB of other apps' default-class replicas that Longhorn
placed here anyway (tags only pull matching volumes in, don't exclude
others when the untagged pool elsewhere is full) only leaves ~1493Gi
of real scheduling headroom. 1400Gi fits with margin.
Self-hosted photo backup (Google Photos replacement) - raw manifests,
no Helm chart, self-contained under k8s/apps/immich including its own
CNPG Postgres. Media PVC shares the cp-3 HDD 2TB/2TB with
paperless-media.
Postgres is pg18, not this repo's usual 16.2: CNPG's official pgvector
extension image (ghcr.io/cloudnative-pg/pgvector) is only published
for pg18, loaded via CNPG's ImageVolume extension mechanism (operator
1.30.0 / k8s 1.36.1 both support it). Immich auto-manages CREATE
EXTENSION itself at startup.
OIDC via a new "immich_role" Authentik scope mapping (homelab-admins/
immich-admins -> "admin" claim, else "user"), consumed by Immich's
OAuth roleClaim setting which re-syncs isAdmin on every login - more
reliable than Immich's racy first-user-is-admin fallback. Config
composed into an immich-oidc Secret and mounted as IMMICH_CONFIG_FILE,
matching the paperless-oidc pattern. k8s RBAC (immich-operator Role +
oidc:immich-admins binding) mirrors paperless/rbac.yaml.
immich namespace pre-created in k8s/infra/databases/namespaces.yaml
(not just immich's own CreateNamespace=true) since the iam PostSync
job's RoleBinding needs it to exist before wave 8.
4TB disk on cp-3 was single-tenant for paperless (3500Gi). Splitting
2TB/2TB with the new Immich media PVC on the same disk/tag. Live PVC
and Longhorn volume already deleted+recreated manually (data was
outdated test uploads only, nightly MinIO backup covers it).
paperless-ngx v3 refuses to migrate from anything before v2.20.15
(paperless.E002). 2.20.15 already covers the API version range the
iOS app needs, so it fixes the phone upload issue without the v3
breaking changes.
Swift Paperless needs REST API v3-9; server was on 2.13 (v1-5 only),
causing 406 on /api/token/ for all phone uploads. v3 requires
PAPERLESS_DBENGINE explicit instead of inferred from PAPERLESS_DBHOST.
Adds permissions claim + per-service admin groups in Authentik, scoped
Role/RoleBinding per service, public PKCE kubernetes OAuth2 client, and
kube-apiserver OIDC extraArgs. Also fixes paperless OIDC signup permissions
via adapter override and adds CoreDNS rewrite for authentik.riotpiao.com.
allauth defaulted to ACCOUNT_EMAIL_VERIFICATION=mandatory, and building the confirmation link 500'd with NoReverseMatch on account_confirm_email (paperless-ngx doesn't wire up that view, no SMTP configured either). Authentik already verifies identity via OIDC, so this step is redundant.
Service "paperless" made k8s inject PAPERLESS_PORT=tcp://<ip>:8000 as a legacy Docker-links env var, which paperless-ngx's own entrypoint also reads for gunicorn's bind port - collided, gunicorn crash-looped ("not a valid port number"), and the pod was 1/1 Running with nothing actually listening (nginx saw 502).
parameters is immutable on StorageClass, so ArgoCD's default patch sync kept failing after the nodeSelector removal ("field is immutable"). Same fix already used by longhorn-cnpg-storageclass.yaml.
Longhorn's StorageClass nodeSelector matches node tags (nodes.longhorn.io spec.tags), not k8s hostnames - "talos-cp-3" was never set as a node tag, so every PVC provision attempt failed with "specified node tag talos-cp-3 does not exist". diskSelector: paperless-media already pins placement correctly on its own.
homelab-root and every child Application still tracked github.com/Riotpiaole/riotpiao.homelab.com, which had diverged from origin (Forgejo) for a while - pushes to Forgejo were never picked up by ArgoCD. Repointed to forgejo.riotpiao.com/rock/homelab.git, already covered by the AppProject's rock/* wildcard.
Fixes controlplane.tftpl's install.wipe:true (should be false, live CPs already run false) and syncs coredns Corefile back to what's actually deployed (drops an unrolled-out, stale Kong-era rewrite).
rock
Story Crater Bot