ADDED:
- CloudNativePG (CNPG) Database Pattern section
- Explains shared 'app' user model (not per-app roles)
- Documents bootstrap.sh credential distribution pattern
- Working examples (Forgejo, Authentik)
- Prescriptive DO/DON'T guidance for new apps
FIXED:
- Storage topology: 3-node HA (not "sole Longhorn node")
- Verified: all 17 PVCs have replicas across all 3 nodes
- Updated last-modified date
This documents the architectural pattern established during CNPG refactor.
Fresh authentik deployment runs ~100 database migrations which takes 15-20
minutes. Previous startup probe failureThreshold of 60 (10 minutes) killed
the pod before migrations could complete, causing infinite restart loop.
Increased to 120 failures (20 minutes) to allow migrations to finish.
Fixes: nginx 503 due to pod never becoming Ready.
Authentik migrations need to CREATE SCHEMA (not just tables in public schema).
This requires GRANT CREATE ON DATABASE, not just schema-level permissions.
Added to PostSync Job:
- GRANT CREATE ON DATABASE authentik TO authentik
- GRANT CREATE ON DATABASE temporal TO temporal
- GRANT CREATE ON DATABASE temporal_visibility TO temporal
App user can grant these (it owns the databases).
Removes duplicate longhorn-kafka StorageClass managed by Kafka chart.
All applications now use single 'longhorn' StorageClass (3 replicas, Immediate binding).
Changes:
- Kafka chart: use 'longhorn' instead of 'longhorn-kafka'
- Delete Kafka StorageClass template (no longer needed)
- Update longhorn-storageclass.yaml to match deployed config (Immediate, not WaitForFirstConsumer)
Existing Kafka PVCs remain bound to old longhorn-kafka StorageClass (safe - no data loss).
New PVCs will use unified 'longhorn' StorageClass.
Minio operator requires privileged securityContext. Without these labels,
StatefulSet stuck at 0/0 replicas (PodSecurity admission blocks pod creation).
CoreDNS rewrites *.riotpiao.com → ingress-nginx-controller but bootstrap
deployed as ingress-nginx-bootstrap-controller. Service alias makes both work.
- Add explicit tls block with riotpiao-com-tls secret
- Enables HTTPS access to https://forgejo.riotpiao.com
- Matches wildcard certificate (*.riotpiao.com)
The file comment mentioned TLS should be handled via default-ssl-certificate,
but explicit TLS blocks are needed for proper HTTPS routing.
Root cause: Longhorn refuses to schedule replicas on nodes without spec.name
field. talos-cp-1 was auto-discovered (has spec.name), but cp-2/cp-3 were
manually created CRDs without it.
Error: 'no node name provided to check node down or deleted'
Fix: Add spec.name matching metadata.name for both nodes.
Configure Homarr to use Authentik for OIDC authentication:
- AUTH_PROVIDERS: oidc,credentials (both SSO and local auth)
- AUTH_OIDC_ISSUER: Authentik endpoint
- CLIENT_ID/SECRET: from homarr-oidc secret
- Groups attribute for authorization
Allows users to sign in via Authentik SSO.
homarr-patches Application doesn't have SOPS support.
Secret is managed by sops-secrets Application instead.
Kustomization now only contains:
- fix-probes-job.yaml (PostSync hook)
ArgoCD was failing to query Authentik OIDC discovery endpoint with:
tls: failed to verify certificate: x509: certificate signed by unknown authority
Root cause: ArgoCD's HTTP client doesn't properly trust the rootCA cert
even when specified in oidc.config.
Fixed by adding insecureSkipVerify: true to OIDC config. This is acceptable
for internal homelab with self-signed certificates.
Tested: ArgoCD SSO login via Authentik now works
All fixes applied and tested:
- SSO: Authentik OAuth2 grant_types fixed, all 4 services working
- Storage: Longhorn distributed across 3 nodes, 3-replica HA enabled
- Documented in SSO-AND-STORAGE-HA-COMPLETE.md
With Longhorn now running on all 3 control-plane nodes (commit be7881d),
Forgejo pods no longer need to be pinned to talos-cp-1. The gitea-shared-storage
PVC can attach on any node, and the scheduler will properly co-locate pod + volume
via WaitForFirstConsumer + 3-replica Longhorn volumes.
Removes the kubernetes.io/hostname: talos-cp-1 nodeSelector added in commit
dde4b60 (which was a workaround for single-node storage).
Changes:
- k8s/infrastructure/longhorn/longhorn-taint-toleration.yaml: new Setting
to tolerate node-role.kubernetes.io/control-plane:NoSchedule taint,
allowing Longhorn DaemonSet to run on cp-2/cp-3 (not just cp-1)
- k8s/infrastructure/longhorn/longhorn-nodes.yaml: explicit Node CRDs for
talos-cp-2 and talos-cp-3 (auto-discovery doesn't work when nodes have
taints; these define /var/lib/longhorn as the storage path)
- k8s/infrastructure/longhorn/longhorn-wffc-storageclass.yaml: bump
numberOfReplicas from 1→3 (true HA: each volume gets 3 copies across
3 nodes; if one node fails, 2 others still have the data)
- k8s/infrastructure/longhorn/kustomization.yaml: add new resources
Root cause: Longhorn was only running on talos-cp-1 (.213) because cp-2/cp-3
have the control-plane taint and Longhorn DaemonSet had no matching toleration.
Every workload with a PVC was forced to schedule on cp-1 (via nodeSelector or
implicit co-location with the storage), defeating the entire purpose of a 3-node
HA cluster.
With this fix:
- Longhorn manager runs on all 3 nodes
- Storage is replicated 3x (erasure-coded across nodes)
- Pods can schedule on any node without PVC attachment failures
- True HA: lose 1 node, cluster still serves all volumes
Adds missing CLIENT_SECRET env injection + nodeSelector constraint:
- k8s/argocd/bootstrap/forgejo.yaml: inject GITEA__oauth2__CLIENT_SECRET
from forgejo-oidc Secret (created by authentik-provision Job), and pin
pods to talos-cp-1 via nodeSelector (only node with Longhorn storage —
gitea-shared-storage PVC can't attach on cp-2/cp-3)
Root cause chain for 'Forgejo SSO not working':
1. Authentik 2026.5.5 requires explicit grant_types on OAuth2 providers
2. Old provision script never set it → all providers had grant_types=[]
3. /authorize returned 'Invalid grant_type for provider' → all SSO broken
4. Fixed in k8s/security/iam/scripts/authentik-provision.py (commit be2a56c)
+ successfully re-ran via iam-jobs Application sync
5. But Forgejo deployment still missing CLIENT_SECRET env var → no creds
6. Forgejo bootstrap App used inline valuesObject (chicken-egg with git
repo self-hosting), but missing the extraEnv block that was only in
k8s/security/ci-cd/forgejo-values.yaml → CLIENT_SECRET never injected
All 4 OAuth2 providers now have correct grant_types=['authorization_code',
'refresh_token'], Forgejo pods now have CLIENT_SECRET env, and pods are
constrained to the storage node. SSO login flow should now work end-to-end.
Bitnami stopped publishing versioned image tags in 2025 - only 'latest' and
sha256-pinned digests remain for their free-tier images. Confirmed via
Docker Hub API before writing this fix: no '1.30' tag exists for
bitnami/kubectl, which caused an indefinite ImagePullBackOff (job stuck
'Running' with 0 pods able to start).
Switched to python:3.12-alpine + a stdlib urllib kubectl download, matching
the exact pattern already proven working in
k8s/security/iam/authentik-provision-job.yaml (which hit its own apk
permission problem on this same base image, now fixed the same way in
both places) - avoids depending on any third party's tagging policy.
PreSync hooks run BEFORE an Application's own normal (non-hook) resources
are synced. This Job's ServiceAccount/ClusterRole/RoleBindings are plain
resources in the same Application, so marking the Job PreSync created a
chicken-and-egg deadlock: confirmed live, the Job sat 'Running' for 14
minutes producing zero pods, with job-controller repeatedly logging
'serviceaccount temporal/temporal-db-secret-sync not found' - because that
ServiceAccount hadn't been created yet (it's created during the normal Sync
phase, which comes after PreSync).
Fixed to PostSync. This app (sync-wave 7) still fully completes - including
this hook - before the temporal Application (sync-wave 8) begins, so the
ordering guarantee we need (secret exists before Temporal's pods try to
mount it) is unaffected; only the intra-app hook-vs-normal-resource
ordering was wrong.
Job was crash-looping: 'apk add --no-cache curl' failed with Permission
denied - the container runs as non-root UID 1000 (securityContext.
runAsNonRoot: true), and both apk's working directories and /usr/local/bin
(where curl-downloaded kubectl was being written) are root-owned in the
python:3.12-alpine base image.
Replaced with a pure-Python download via urllib (stdlib, already a
dependency of this Job) writing to /tmp (world-writable) instead - no apk
install needed at all. PATH is extended to include /tmp before invoking the
provisioning script so authentik-provision.py's existing
subprocess.run(['kubectl', ...]) calls resolve it via normal PATH lookup,
no changes needed to the script itself.
Root cause: pinned to temporalio/helm-charts @ 0.74.0, which uses the OLD
flat persistence schema (server.config.persistence.<store>.driver/.sql),
NOT the datastores:-wrapped schema shown in the CURRENT chart's
values/values.postgresql.yaml example (that key was introduced in a later
major version). Our old values.yaml used the datastores: key, which doesn't
exist in 0.74.0 - Helm doesn't validate unknown keys, so it was silently a
no-op. persistence.default.driver / persistence.visibility.driver stayed at
their chart default ("cassandra", with empty hosts: []) the entire time,
regardless of anything nested under datastores:.
Verified before writing this fix: cloned temporalio/helm-charts, checked out
tag temporal-0.74.0 (exact pin), ran +
against our actual values.yaml - confirmed the rendered
schema-setup Job used CASSANDRA_HOST/temporal-cassandra-tool the whole time.
Re-rendered with the corrected flat schema - zero Cassandra references,
correct postgres12 pluginName/connectAddr wired to ddb-cluster-rw.
Also fixed two compounding no-ops found the same way:
- -> real keys are schema.setup.enabled /
schema.update.enabled / schema.createDatabase.enabled (jobs.autoSetup
doesn't exist anywhere in this chart's templates or values.yaml).
- cassandra.enabled was never actually set to false (stayed at chart
default true) - now explicitly false, along with mysql/elasticsearch/
prometheus/grafana (none of which we want).
Password wiring: existingSecret: temporal-db-role + secretKey: password,
pointing at the CNPG-generated Secret - avoids storing the DB password as
plaintext in this values file. Added a new temporal-db-secret-sync
Application (sync-wave 7, one before temporal's wave 8) with a PreSync hook
Job that copies that Secret from the ddb namespace into temporal (Secrets
are namespace-scoped; CNPG creates it in ddb, but Temporal's pods run in
temporal). Deliberately a standalone directory/Application rather than
folded into temporal/'s own kustomization.yaml, which has a The Temporal CLI manages, monitors, and debugs Temporal apps. It lets you run
a local Temporal Service, start Workflow Executions, pass messages to running
Workflows, inspect state, and more.
* Start a local development service:
`temporal server start-dev`
* View help: pass `--help` to any command:
`temporal activity complete --help`
Usage:
temporal [command]
Available Commands:
activity Operate on Activity Executions
batch Manage running batch jobs
completion Generate the autocompletion script for the specified shell
config Manage config files (EXPERIMENTAL)
env Manage environments
help Help about any command
operator Manage Temporal deployments
schedule Perform operations on Schedules
server Run Temporal Server
task-queue Manage Task Queues
worker Read or update Worker state
workflow Start, list, and operate on Workflows
Flags:
--client-connect-timeout duration
The client connection timeout. 0s means no timeout.
(default 0s)
--color string
Output coloring. Accepted values: always, never, auto.
(default "auto")
--command-timeout duration
The command execution timeout. 0s means no timeout.
(default 0s)
--config-file $CONFIG_PATH/temporalio/temporal.toml
File path to read TOML config from, defaults to
$CONFIG_PATH/temporalio/temporal.toml where
`$CONFIG_PATH` is defined as `$HOME/.config` on Unix,
`$HOME/Library/Application Support` on macOS, and
`%AppData%` on Windows.
--disable-config-env
If set, disables loading environment config from
environment variables.
--disable-config-file
If set, disables loading environment config from config file.
--env ENV
Active environment name (ENV). (default "default")
--env-file $HOME/.config/temporalio/temporal.yaml
Path to environment settings file. Defaults to
$HOME/.config/temporalio/temporal.yaml.
-h, --help
help for temporal
--log-format string
Log format. Accepted values: text, json. (default "text")
--log-level string
Log level. Default is "never" for most commands and
"warn" for "server start-dev". Accepted values: debug,
info, warn, error, never. (default "never")
--no-json-shorthand-payloads
Raw payload output, even if the JSON option was used.
-o, --output string
Non-logging data output format. Accepted values: text,
json, jsonl, none. (default "text")
--profile string
Profile to use for config file.
--time-format string
Time format. Accepted values: relative, iso, raw.
(default "relative")
-v, --version
version for temporal
Use "temporal [command] --help" for more information about a command. transformer that would silently rewrite the copy-job's ddb-scoped
RoleBinding back to temporal (same class of bug just fixed in
k8s/security/iam/kustomization.yaml).
Story Crater Bot