Bitnami stopped publishing versioned image tags in 2025 - only 'latest' and
sha256-pinned digests remain for their free-tier images. Confirmed via
Docker Hub API before writing this fix: no '1.30' tag exists for
bitnami/kubectl, which caused an indefinite ImagePullBackOff (job stuck
'Running' with 0 pods able to start).
Switched to python:3.12-alpine + a stdlib urllib kubectl download, matching
the exact pattern already proven working in
k8s/security/iam/authentik-provision-job.yaml (which hit its own apk
permission problem on this same base image, now fixed the same way in
both places) - avoids depending on any third party's tagging policy.
PreSync hooks run BEFORE an Application's own normal (non-hook) resources
are synced. This Job's ServiceAccount/ClusterRole/RoleBindings are plain
resources in the same Application, so marking the Job PreSync created a
chicken-and-egg deadlock: confirmed live, the Job sat 'Running' for 14
minutes producing zero pods, with job-controller repeatedly logging
'serviceaccount temporal/temporal-db-secret-sync not found' - because that
ServiceAccount hadn't been created yet (it's created during the normal Sync
phase, which comes after PreSync).
Fixed to PostSync. This app (sync-wave 7) still fully completes - including
this hook - before the temporal Application (sync-wave 8) begins, so the
ordering guarantee we need (secret exists before Temporal's pods try to
mount it) is unaffected; only the intra-app hook-vs-normal-resource
ordering was wrong.
Job was crash-looping: 'apk add --no-cache curl' failed with Permission
denied - the container runs as non-root UID 1000 (securityContext.
runAsNonRoot: true), and both apk's working directories and /usr/local/bin
(where curl-downloaded kubectl was being written) are root-owned in the
python:3.12-alpine base image.
Replaced with a pure-Python download via urllib (stdlib, already a
dependency of this Job) writing to /tmp (world-writable) instead - no apk
install needed at all. PATH is extended to include /tmp before invoking the
provisioning script so authentik-provision.py's existing
subprocess.run(['kubectl', ...]) calls resolve it via normal PATH lookup,
no changes needed to the script itself.
Root cause: pinned to temporalio/helm-charts @ 0.74.0, which uses the OLD
flat persistence schema (server.config.persistence.<store>.driver/.sql),
NOT the datastores:-wrapped schema shown in the CURRENT chart's
values/values.postgresql.yaml example (that key was introduced in a later
major version). Our old values.yaml used the datastores: key, which doesn't
exist in 0.74.0 - Helm doesn't validate unknown keys, so it was silently a
no-op. persistence.default.driver / persistence.visibility.driver stayed at
their chart default ("cassandra", with empty hosts: []) the entire time,
regardless of anything nested under datastores:.
Verified before writing this fix: cloned temporalio/helm-charts, checked out
tag temporal-0.74.0 (exact pin), ran +
against our actual values.yaml - confirmed the rendered
schema-setup Job used CASSANDRA_HOST/temporal-cassandra-tool the whole time.
Re-rendered with the corrected flat schema - zero Cassandra references,
correct postgres12 pluginName/connectAddr wired to ddb-cluster-rw.
Also fixed two compounding no-ops found the same way:
- -> real keys are schema.setup.enabled /
schema.update.enabled / schema.createDatabase.enabled (jobs.autoSetup
doesn't exist anywhere in this chart's templates or values.yaml).
- cassandra.enabled was never actually set to false (stayed at chart
default true) - now explicitly false, along with mysql/elasticsearch/
prometheus/grafana (none of which we want).
Password wiring: existingSecret: temporal-db-role + secretKey: password,
pointing at the CNPG-generated Secret - avoids storing the DB password as
plaintext in this values file. Added a new temporal-db-secret-sync
Application (sync-wave 7, one before temporal's wave 8) with a PreSync hook
Job that copies that Secret from the ddb namespace into temporal (Secrets
are namespace-scoped; CNPG creates it in ddb, but Temporal's pods run in
temporal). Deliberately a standalone directory/Application rather than
folded into temporal/'s own kustomization.yaml, which has a The Temporal CLI manages, monitors, and debugs Temporal apps. It lets you run
a local Temporal Service, start Workflow Executions, pass messages to running
Workflows, inspect state, and more.
* Start a local development service:
`temporal server start-dev`
* View help: pass `--help` to any command:
`temporal activity complete --help`
Usage:
temporal [command]
Available Commands:
activity Operate on Activity Executions
batch Manage running batch jobs
completion Generate the autocompletion script for the specified shell
config Manage config files (EXPERIMENTAL)
env Manage environments
help Help about any command
operator Manage Temporal deployments
schedule Perform operations on Schedules
server Run Temporal Server
task-queue Manage Task Queues
worker Read or update Worker state
workflow Start, list, and operate on Workflows
Flags:
--client-connect-timeout duration
The client connection timeout. 0s means no timeout.
(default 0s)
--color string
Output coloring. Accepted values: always, never, auto.
(default "auto")
--command-timeout duration
The command execution timeout. 0s means no timeout.
(default 0s)
--config-file $CONFIG_PATH/temporalio/temporal.toml
File path to read TOML config from, defaults to
$CONFIG_PATH/temporalio/temporal.toml where
`$CONFIG_PATH` is defined as `$HOME/.config` on Unix,
`$HOME/Library/Application Support` on macOS, and
`%AppData%` on Windows.
--disable-config-env
If set, disables loading environment config from
environment variables.
--disable-config-file
If set, disables loading environment config from config file.
--env ENV
Active environment name (ENV). (default "default")
--env-file $HOME/.config/temporalio/temporal.yaml
Path to environment settings file. Defaults to
$HOME/.config/temporalio/temporal.yaml.
-h, --help
help for temporal
--log-format string
Log format. Accepted values: text, json. (default "text")
--log-level string
Log level. Default is "never" for most commands and
"warn" for "server start-dev". Accepted values: debug,
info, warn, error, never. (default "never")
--no-json-shorthand-payloads
Raw payload output, even if the JSON option was used.
-o, --output string
Non-logging data output format. Accepted values: text,
json, jsonl, none. (default "text")
--profile string
Profile to use for config file.
--time-format string
Time format. Accepted values: relative, iso, raw.
(default "relative")
-v, --version
version for temporal
Use "temporal [command] --help" for more information about a command. transformer that would silently rewrite the copy-job's ddb-scoped
RoleBinding back to temporal (same class of bug just fixed in
k8s/security/iam/kustomization.yaml).
1. prometheus CRD sync failure (OutOfSync, permanently failing):
- helm.skipCrds: true on the prometheus Application - stop ArgoCD from
managing these CRDs through client-side apply (kube-prometheus-stack's
CRDs are large enough that the kubectl.kubernetes.io/last-applied-
configuration annotation exceeds etcd's 262144-byte limit on every sync).
- New prometheus-crds Application: plain git-sourced YAML (extracted via
helm show crds, committed under k8s/platform/monitoring/crds/), synced
with ServerSideApply=true. Chosen over a Helm-sourced 'CRDs only' app
because there's no clean way to ask ArgoCD's Helm source for 'render only
the crds/ directory' - a committed plain-YAML source is unambiguous.
- ServerSideApply=true can't go on the main prometheus Application: it
conflicts with managedNamespaceMetadata's forced namespace apply
('--force cannot be used with --server-side'), hence the split.
2. minio-tenant stuck OutOfSync (blocked 97+ minutes):
- minio-policy-setup PostSync hook Job was NAME:
mc alias set - set a new alias to configuration file
USAGE:
mc alias set ALIAS URL ACCESSKEY SECRETKEY
FLAGS:
--path value bucket path lookup supported by the server. Valid options are '[auto, on, off]' (default: "auto")
--api value API signature. Valid options are '[S3v4, S3v2]'
--config-dir value, -C value path to configuration folder (default: "/Users/rockliang/.mc") [$MC_CONFIG_DIR]
--quiet, -q disable progress bar display [$MC_QUIET]
--disable-pager, --dp disable mc internal pager and print to raw stdout [$MC_DISABLE_PAGER]
--no-color disable color theme [$MC_NO_COLOR]
--json enable JSON lines formatted output [$MC_JSON]
--debug enable debug output [$MC_DEBUG]
--resolve value resolves HOST[:PORT] to an IP address. Example: minio.local:9000=10.10.75.1 [$MC_RESOLVE]
--insecure disable SSL certificate verification [$MC_INSECURE]
--limit-upload value limits uploads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_UPLOAD]
--limit-download value limits downloads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_DOWNLOAD]
--custom-header value, -H value add custom HTTP header to the request. 'key:value' format.
--help, -h show help
EXAMPLES:
1. Add MinIO service under "myminio" alias. For security reasons turn off bash history momentarily.
$ set +o history
$ mc alias set myminio http://localhost:9000 minio minio123
$ set -o history
2. Add MinIO service under "myminio" alias, to use dns style bucket lookup. For security reasons
turn off bash history momentarily.
$ set +o history
$ mc alias set myminio http://localhost:9000 minio minio123 --api "s3v4" --path "off"
$ set -o history
3. Add Amazon S3 storage service under "mys3" alias. For security reasons turn off bash history momentarily.
$ set +o history
$ mc alias set mys3 https://s3.amazonaws.com \
BKIKJAA5BMMU2RHO6IBB V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
$ set -o history
4. Add Amazon S3 storage service under "mys3" alias, prompting for keys.
$ mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
Enter Access Key: BKIKJAA5BMMU2RHO6IBB
Enter Secret Key: V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
5. Add Amazon S3 storage service under "mys3" alias using piped keys.
$ set +o history
$ echo -e "BKIKJAA5BMMU2RHO6IBB\nV8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12" | \
mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
$ set -o history against
http://minio.storage.svc.cluster.local:9000 - stale port. The minio
Service's port now tracks requestAutoCert on the Tenant (443 when
auto-TLS is on, 80 when off - we set it to false earlier), so 9000
doesn't exist on that Service anymore and the job hung in its 'waiting
for minio...' retry loop indefinitely, blocking ArgoCD's sync operation
(PostSync hooks block the sync from completing until they succeed).
- Fixed to use minio-cluster-hl.storage.svc.cluster.local:9000 - the
headless per-pod Service, which always listens on 9000 regardless of
the Tenant's TLS mode, so this can't silently break again the same way.
Our Ingress objects intentionally omit per-rule tls: blocks (single wildcard
cert served via --default-ssl-certificate). nginx-ingress's normal ssl-redirect
setting only forces HTTPS 301 for hosts with an explicit TLS block on their own
Ingress resource - since none of ours do, plain http://*.riotpiao.com requests
were served directly instead of redirected, exposing every client-facing
console (forgejo, authentik, argocd, grafana, vault, etc.) over plaintext HTTP.
force-ssl-redirect forces the redirect globally regardless of per-ingress TLS
block presence. Verified fix works (tested via manual patch then reverted -
confirmed 308 redirects to https:// on forgejo/authentik/argocd) before
committing via GitOps.
Root cause of widespread 'Unknown' sync status / Skipping auto-sync across
almost every Application: CoreDNS rewrites forgejo.riotpiao.com to the nginx
ingress controller service (rewrite name forgejo.riotpiao.com -> ingress-nginx-
controller...), which only listens on 80/443, not 3000. Every git fetch from
argocd-repo-server to the :3000 repoURL was timing out (context deadline
exceeded), so ArgoCD couldn't compare desired vs live state for any app.
Fix: use https://forgejo.riotpiao.com (no port, TLS via nginx + wildcard cert)
consistent with the 'all external endpoints HTTPS' requirement. Verified git
smart-http response 200 on the new URL before committing.
- minio console ingress: minio-console -> minio-cluster-console:9090 (service renamed by operator)
- minio-api ingress: point to minio:9000 (restored once requestAutoCert disabled)
- minio tenant: requestAutoCert: false (MinIO was TLS-only internally, breaking
plain-HTTP clients like Vault's S3 backend - this was the real cause of the
Vault S3 hang)
- argocd ingress: moved from namespace cicd -> argocd (service lives in argocd
namespace; ingress in wrong namespace can never route, was returning 503)
- removed duplicate kmsvc ingress (sqs namespace already has management-service
ingress with proper TLS block for same host/backend)
Audit method: cross-checked every ingress backend.service.{name,port} against
actual Service objects in cluster. Found 3 broken backends out of 15 ingresses.
Wildcard cert was left on letsencrypt-staging; staging root is not
browser-trusted so HTTPS to *.riotpiao.com fails cert validation.
Switch issuerRef to letsencrypt-prod to issue a trusted wildcard.
authentik/temporal DB users+databases were never provisioned (old helmfile hook
gone; db-init-job only made schemas in shared app DB). Adds managed.roles
(authentik/temporal login roles, passwords from basic-auth secrets) + Database CRs
(dedicated DBs owned by each role). Role secrets applied out-of-band (SOPS), not in
kustomize resources so data-schemas app doesn't choke on ciphertext.
- Create SOPS-encrypted cloudflared-secrets.enc.yaml with tunnel token
- Add Cloudflare vars to .env.example (CLOUDFLARE_CONNECTOR_TOKEN, ACCOUNT_ID, TUNNEL_ID, API_TOKEN)
- Document Phase 0 cloudflared-token Secret creation in BOOTSTRAP.md (manual step until CMP plugin wires it)
- Note: Cloudflare-side TCP routing (cp1.homelab -> 192.168.1.213:6443, etc.) must be configured manually in Zero Trust dashboard
Tunnel already deployed as ArgoCD Application in k8s/argocd/apps/60-applications.yaml (wave 8); this closes the missing Secret gap and documents the bootstrap path.
Add MinIO Policies and Users via CRD alongside Buckets.
Resources now declarative:
- Bucket: riotpiao-models (versioning enabled)
- Policy: policy-ollama (scoped bucket access)
- User: user-ollama (service account for Ollama/LLM)
Access keys can be overridden via SOPS or kustomize overlays.
All MinIO resource creation now git-tracked and version controlled.
Phase 0 continuation: enable ArgoCD to decrypt SOPS-encrypted secrets on sync.
1. Update ArgoCD Helm values (terraform/argocd-bootstrap.tf):
- Add SOPS_AGE_KEY_FILE env var to repoServer
- Mount sops-age K8s Secret at /home/argocd/.sops
- Add ConfigManagementPlugin for SOPS (detects *.enc.yaml files)
2. Update encrypted secrets with real values:
- k8s/base/secrets.enc.yaml: encrypted with actual service credentials
- All secret values encrypted at rest in git
- ArgoCD decrypts on sync using K8s Secret + AGE key
Prerequisites:
- K8s Secret created: kubectl create secret generic sops-age -n argocd --from-file=keys.txt=/Users/rockliang/.sops/key.txt
- SOPS_AGE_KEY_FILE env var set in ArgoCD repoServer (done above)
Next: Phase 1 — migrate 9 hookless releases to ArgoCD + create Applications that reference encrypted secrets.
Phase 0 groundwork for helmfile→ArgoCD migration using SOPS (Secrets Operations):
1. Install SOPS + AGE encryption
- AGE key generated and stored locally at ~/.sops/key.txt
- Public key embedded in .sops.yaml for file encryption rules
2. Create K8s Secret for AGE private key
- kubectl: create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt
- ArgoCD will use this key to decrypt secrets at sync time
3. Encrypt initial secrets
- k8s/base/secrets.enc.yaml: AES256_GCM encrypted secrets for all services
- Placeholder values (will be replaced with real values per environment)
- Secrets never visible in git (encrypted at rest)
4. Configure SOPS
- .sops.yaml: creation rules for k8s/*/secrets.enc.yaml files
- All future secret files auto-encrypt on edit (sops -e)
Setup: Store AGE key as K8s Secret in argocd namespace:
export KUBECONFIG=cluster-config/kubeconfig
kubectl create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt
Next: Configure ArgoCD Helm plugin to decrypt secrets on sync (Phase 0 continuation).
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
Story Crater Bot