- Fix template to use .Values.gc.schedule instead of hardcoded cron
- Change schedule from daily 03:00 UTC to every 30 minutes
- Prevents DinD PVC fill-up (was at 93% before manual prune)
- Add local-llm OAuth2 provider and application to Authentik provisioning
- Configure JWT-compatible OAuth2 provider (client_id: local-llm)
- Generate client secret on first run, stored in llm-serving/local-llm-jwt Secret
- Bind llm-admins group to local-llm application for admin access
- Add RBAC for provisioning job to create secrets in llm-serving namespace
- Output JWT issuer URL and certificate for local-llm token validation
Schedule runners on az-b (talos-cp-2) which has more Longhorn storage
and breathing room (367Gi available vs worker-1's 369Gi but over-provisioned).
PVCs deleted and will recreate on new zone. Runners will re-register.
Daily 04:00 UTC sweeper in kube-system:
- Delete failed Jobs older than 24h (any namespace)
- Delete completed standalone Jobs older than 72h (no CronJob owner)
- Delete orphan Error/Evicted pods older than 1h
- Self-cleans via ttlSecondsAfterFinished
- Add gc-cronjob.yaml: daily prune of DinD Docker images/volumes/build-cache
and actcache across all forgejo-runner pods. Keeps :latest tagged images,
deletes non-latest older than 72h.
- Expand runner reg PVCs from 1Gi to 20Gi (all three runners) to prevent
action tool cache from filling disk.
- Rust runner gets explicit 20Gi persistence override.
- GC only renders from golang (default) values to avoid duplicate resources.
kmsvc-manage, kmsvc-cli require Go 1.26.0 in go.mod but runner was
using golang:1.25-bookworm container. Update to golang:1.26-bookworm
to match project requirements.
Client credentials + device code grant, no browser redirect (empty
redirect_uris) - unlike every other SERVICES entry which is
authorization_code web SSO. First real step toward replacing
poimen-memory's static API key with a proper JWT flow.
Bind each service's operator Role to a ServiceAccount alongside the
existing oidc:*-admins Group, and wire serviceAccountName into the
pods we control (immich-server, immich-machine-learning,
management-service). poimen-memory's Deployment lives in its own
repo/ArgoCD app, so its SA is created here but not yet wired to a pod.
Confidential client for 'vault login -method=oidc' and the Vault UI's
oidc auth method. homelab-admins gets bound automatically like every
other app in SERVICES. Client secret generated on first provision run
into iam/vault-oidc.
Follows the portainer/kmsvc/temporal pattern - group + "permissions"
claim entry only, no Authentik Application/OAuth provider, since
poimen-memory is an internal API-key service, not browser OIDC login.
rock gets it automatically (already in every service_admin_group).
Self-hosted photo backup (Google Photos replacement) - raw manifests,
no Helm chart, self-contained under k8s/apps/immich including its own
CNPG Postgres. Media PVC shares the cp-3 HDD 2TB/2TB with
paperless-media.
Postgres is pg18, not this repo's usual 16.2: CNPG's official pgvector
extension image (ghcr.io/cloudnative-pg/pgvector) is only published
for pg18, loaded via CNPG's ImageVolume extension mechanism (operator
1.30.0 / k8s 1.36.1 both support it). Immich auto-manages CREATE
EXTENSION itself at startup.
OIDC via a new "immich_role" Authentik scope mapping (homelab-admins/
immich-admins -> "admin" claim, else "user"), consumed by Immich's
OAuth roleClaim setting which re-syncs isAdmin on every login - more
reliable than Immich's racy first-user-is-admin fallback. Config
composed into an immich-oidc Secret and mounted as IMMICH_CONFIG_FILE,
matching the paperless-oidc pattern. k8s RBAC (immich-operator Role +
oidc:immich-admins binding) mirrors paperless/rbac.yaml.
immich namespace pre-created in k8s/infra/databases/namespaces.yaml
(not just immich's own CreateNamespace=true) since the iam PostSync
job's RoleBinding needs it to exist before wave 8.
Adds permissions claim + per-service admin groups in Authentik, scoped
Role/RoleBinding per service, public PKCE kubernetes OAuth2 client, and
kube-apiserver OIDC extraArgs. Also fixes paperless OIDC signup permissions
via adapter override and adds CoreDNS rewrite for authentik.riotpiao.com.
parameters is immutable on StorageClass, so ArgoCD's default patch sync kept failing after the nodeSelector removal ("field is immutable"). Same fix already used by longhorn-cnpg-storageclass.yaml.
Longhorn's StorageClass nodeSelector matches node tags (nodes.longhorn.io spec.tags), not k8s hostnames - "talos-cp-3" was never set as a node tag, so every PVC provision attempt failed with "specified node tag talos-cp-3 does not exist". diskSelector: paperless-media already pins placement correctly on its own.
Fixes controlplane.tftpl's install.wipe:true (should be false, live CPs already run false) and syncs coredns Corefile back to what's actually deployed (drops an unrolled-out, stale Kong-era rewrite).
Ingress api/api now backs onto api-gateway:8080; the kong Application, its
Helm values, plugins and llm-routes are removed. Gateway image v0.0.0 is in
the Forgejo registry and the pull secret is in the api namespace.
- Runner egress: allow 192.168.1.160/32:443. forgejo.riotpiao.com resolves to
the ingress LB, inside the 192.168.1.0/24 block the NetworkPolicy denies, so
docker push hung until timeout.
- dind CA: also mount homelab-ca at /etc/docker/certs.d/forgejo.riotpiao.com/,
the path dockerd actually reads for per-registry trust.
- Pull secret: dockerconfigjson for the api namespace; /v2/ answers 401.
- AppProject: allow the Forgejo repo as a source for api-gw.
homelab-ca was referenced by 6 manifests (authentik, forgejo-runner,
blackbox-exporter, management-service) as a CA trust ConfigMap but never
existed anywhere - not in git, not live in cluster. Generated a new
10-year self-signed root CA, wired it as a ClusterIssuer (cert-manager
namespace) and distributed the public cert as a ConfigMap to every
consuming namespace (iam, cicd, monitoring, sqs). Private key lives only
in the encrypted Secret. Widened cert-manager-issuers' directory include
glob rather than creating a new Application - destination.namespace is
just a fallback default on a plain directory source, not a transformer,
so it doesn't fight with each ConfigMap's own explicit namespace.
Also adds grafana-oidc secret (GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET),
same pre-existing gap as grafana-admin - was meant to come from a deleted
manual script, value already available in .env.
Vault's S3 storage backend needs AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY
from vault-minio-creds, previously generated by a helmfile presync hook
that no longer exists post-Terraform/helmfile removal. Sourced from the
same MINIO_ROOT_USER/PASSWORD already in .env. vault-unseal-keys still
missing separately — needs a live 'vault operator init' run, deferred.
rock
Story Crater Bot