fix(temporal): actually enable PostgreSQL persistence (chart schema mismatch)

Root cause: pinned to temporalio/helm-charts @ 0.74.0, which uses the OLD flat persistence schema (server.config.persistence.<store>.driver/.sql), NOT the datastores:-wrapped schema shown in the CURRENT chart's values/values.postgresql.yaml example (that key was introduced in a later major version). Our old values.yaml used the datastores: key, which doesn't exist in 0.74.0 - Helm doesn't validate unknown keys, so it was silently a no-op. persistence.default.driver / persistence.visibility.driver stayed at their chart default ("cassandra", with empty hosts: []) the entire time, regardless of anything nested under datastores:. Verified before writing this fix: cloned temporalio/helm-charts, checked out tag temporal-0.74.0 (exact pin), ran + against our actual values.yaml - confirmed the rendered schema-setup Job used CASSANDRA_HOST/temporal-cassandra-tool the whole time. Re-rendered with the corrected flat schema - zero Cassandra references, correct postgres12 pluginName/connectAddr wired to ddb-cluster-rw. Also fixed two compounding no-ops found the same way: - -> real keys are schema.setup.enabled / schema.update.enabled / schema.createDatabase.enabled (jobs.autoSetup doesn't exist anywhere in this chart's templates or values.yaml). - cassandra.enabled was never actually set to false (stayed at chart default true) - now explicitly false, along with mysql/elasticsearch/ prometheus/grafana (none of which we want). Password wiring: existingSecret: temporal-db-role + secretKey: password, pointing at the CNPG-generated Secret - avoids storing the DB password as plaintext in this values file. Added a new temporal-db-secret-sync Application (sync-wave 7, one before temporal's wave 8) with a PreSync hook Job that copies that Secret from the ddb namespace into temporal (Secrets are namespace-scoped; CNPG creates it in ddb, but Temporal's pods run in temporal). Deliberately a standalone directory/Application rather than folded into temporal/'s own kustomization.yaml, which has a The Temporal CLI manages, monitors, and debugs Temporal apps. It lets you run a local Temporal Service, start Workflow Executions, pass messages to running Workflows, inspect state, and more. * Start a local development service: `temporal server start-dev` * View help: pass `--help` to any command: `temporal activity complete --help` Usage: temporal [command] Available Commands: activity Operate on Activity Executions batch Manage running batch jobs completion Generate the autocompletion script for the specified shell config Manage config files (EXPERIMENTAL) env Manage environments help Help about any command operator Manage Temporal deployments schedule Perform operations on Schedules server Run Temporal Server task-queue Manage Task Queues worker Read or update Worker state workflow Start, list, and operate on Workflows Flags: --client-connect-timeout duration The client connection timeout. 0s means no timeout. (default 0s) --color string Output coloring. Accepted values: always, never, auto. (default "auto") --command-timeout duration The command execution timeout. 0s means no timeout. (default 0s) --config-file $CONFIG_PATH/temporalio/temporal.toml File path to read TOML config from, defaults to $CONFIG_PATH/temporalio/temporal.toml where `$CONFIG_PATH` is defined as `$HOME/.config` on Unix, `$HOME/Library/Application Support` on macOS, and `%AppData%` on Windows. --disable-config-env If set, disables loading environment config from environment variables. --disable-config-file If set, disables loading environment config from config file. --env ENV Active environment name (ENV). (default "default") --env-file $HOME/.config/temporalio/temporal.yaml Path to environment settings file. Defaults to $HOME/.config/temporalio/temporal.yaml. -h, --help help for temporal --log-format string Log format. Accepted values: text, json. (default "text") --log-level string Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. (default "never") --no-json-shorthand-payloads Raw payload output, even if the JSON option was used. -o, --output string Non-logging data output format. Accepted values: text, json, jsonl, none. (default "text") --profile string Profile to use for config file. --time-format string Time format. Accepted values: relative, iso, raw. (default "relative") -v, --version version for temporal Use "temporal [command] --help" for more information about a command. transformer that would silently rewrite the copy-job's ddb-scoped RoleBinding back to temporal (same class of bug just fixed in k8s/security/iam/kustomization.yaml).
2026-08-18 15:08:02 -07:00
parent 6b4f72ef1c
commit 3c1342cef9
3 changed files with 214 additions and 54 deletions
@@ -0,0 +1,104 @@
+# Copies the CNPG-generated temporal-db-role Secret from the ddb namespace
+# into the temporal namespace, as a plain (non-SOPS) k8s Secret with the same
+# keys. Kubernetes Secrets are strictly namespace-scoped - a Deployment in
+# `temporal` cannot reference a Secret living in `ddb` via secretKeyRef, and
+# temporal-values.yaml's server.config.persistence.*.sql.existingSecret:
+# temporal-db-role expects to find it in ITS OWN namespace (temporal).
+#
+# Deliberately a standalone directory (no kustomization.yaml) applied as its
+# own small Application - avoids the k8s/applications/temporal/kustomization.yaml
+# `namespace: temporal` transformer, which would silently force-rewrite this
+# Job's ddb-scoped RoleBinding back to temporal (same class of bug fixed
+# earlier in k8s/security/iam/kustomization.yaml - see that file's comments).
+#
+# PreSync + BeforeHookCreation: reruns on every ArgoCD sync of the temporal
+# app group, so it re-copies the password if CNPG ever rotates it. Runs
+# before the main `temporal` Application (sync-wave 8) since this app is
+# registered at sync-wave 7.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: temporal-db-secret-sync
+  namespace: temporal
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: temporal-db-secret-sync
+rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "list", "create", "update", "patch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: temporal-db-secret-sync
+  namespace: ddb
+subjects:
+  - kind: ServiceAccount
+    name: temporal-db-secret-sync
+    namespace: temporal
+roleRef:
+  kind: ClusterRole
+  name: temporal-db-secret-sync
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: temporal-db-secret-sync
+  namespace: temporal
+subjects:
+  - kind: ServiceAccount
+    name: temporal-db-secret-sync
+    namespace: temporal
+roleRef:
+  kind: ClusterRole
+  name: temporal-db-secret-sync
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: temporal-db-secret-sync
+  namespace: temporal
+  annotations:
+    argocd.argoproj.io/hook: PreSync
+    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
+spec:
+  ttlSecondsAfterFinished: 600
+  backoffLimit: 5
+  template:
+    spec:
+      serviceAccountName: temporal-db-secret-sync
+      restartPolicy: Never
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: copy
+          image: bitnami/kubectl:1.30
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+          command:
+            - /bin/sh
+            - -c
+            - |
+              set -e
+              echo "waiting for ddb/temporal-db-role..."
+              until kubectl -n ddb get secret temporal-db-role >/dev/null 2>&1; do
+                echo "  not ready yet, retrying..."
+                sleep 5
+              done
+              USERNAME=$(kubectl -n ddb get secret temporal-db-role -o jsonpath='{.data.username}' | base64 -d)
+              PASSWORD=$(kubectl -n ddb get secret temporal-db-role -o jsonpath='{.data.password}' | base64 -d)
+              kubectl -n temporal create secret generic temporal-db-role \
+                --from-literal=username="$USERNAME" \
+                --from-literal=password="$PASSWORD" \
+                --dry-run=client -o yaml | kubectl apply -f -
+              echo "synced temporal-db-role -> temporal namespace"
@@ -1,35 +1,63 @@
 # k8s/temporal/temporal-values.yaml
 # Temporal — workflow engine
 # Uses external CNPG PostgreSQL for persistence (ddb-cluster)
-# Visibility via same PostgreSQL database
+# Visibility via same PostgreSQL instance, separate database.
+#
+# IMPORTANT — chart schema note (root-caused after Postgres never actually
+# taking effect despite looking configured):
+# We're pinned to temporalio/helm-charts @ 0.74.0 (see targetRevision in
+# k8s/argocd/apps/60-applications.yaml), which uses the OLD flat persistence
+# schema:
+#   server.config.persistence.<default|visibility>.driver: "sql"|"cassandra"
+#   server.config.persistence.<default|visibility>.sql: {...}
+# NOT the newer `datastores:`-wrapped schema
+# (server.config.persistence.datastores.<store>.sql) shown in the current
+# chart's values/values.postgresql.yaml example - that key was introduced in
+# a later major version and doesn't exist in 0.74.0. Helm doesn't validate
+# unknown keys, so a `datastores:` block here is silently a no-op: Temporal
+# would keep defaulting to Cassandra (with empty hosts: []) regardless of
+# anything nested inside it. Verified via `helm template` against the actual
+# 0.74.0 chart before writing this file - see chat history for the
+# side-by-side proof (rendered manifest showed CASSANDRA_HOST env vars and
+# temporal-cassandra-tool commands using the old datastores:-based values).
+#
+# Likewise `schema.setup.enabled` / `schema.update.enabled` /
+# `schema.createDatabase.enabled` are the real toggles for the schema-setup
+# Job (all default true) - there is no `jobs.autoSetup` key in this chart.

-# ── Disable embedded databases ────
+# ── Disable every bundled/optional sub-chart ─────────────────────────────────
+# postgresql/mysql: never enable - we never want the chart to deploy its own
+# DB, only to know how to talk to our external CNPG instance (which happens
+# via server.config.persistence.*.sql below, independent of these flags).
 postgresql:
  enabled: false
-
+mysql:
+  enabled: false
 cassandra:
-  enabled: true
-  persistence:
-    enabled: false
-  image:
-    repo: cassandra
-    tag: 3.11.3
-  config:
-    cluster_size: 1
-    ports:
-      cql: 9042
-  service:
-    type: ClusterIP
-
+  enabled: false
 elasticsearch:
  enabled: false
+prometheus:
+  enabled: false
+grafana:
+  enabled: false

-# ── Disable schema auto-setup ─────
-jobs:
-  autoSetup:
+# ── Schema setup/update Jobs ──────────────────────────────────────────────────
+# Disabled: ddb-cluster's seed job (k8s/data/db-init-job.yaml) already creates
+# the `temporal` and `temporal_visibility` databases and runs the Temporal
+# schema migrations out of band. Leaving these at their chart default (true)
+# would spin up a schema Job on every sync that tries to wait-for-cassandra
+# and run cassandra-tool commands (see note above) - pointless for us even
+# once correctly pointed at Postgres, since schema is already seeded.
+schema:
+  createDatabase:
+    enabled: false
+  setup:
+    enabled: false
+  update:
    enabled: false

-# ── Temporal server config (PostgreSQL persistence) ──────────────────────────────
+# ── Temporal server config (PostgreSQL persistence) ──────────────────────────
 server:
  replicaCount: 1
  jobService:
@@ -49,35 +77,40 @@ server:
      defaultStore: default
      visibilityStore: visibility
      numHistoryShards: 512
-      datastores:
-        default:
-          # PostgreSQL for workflow history and events
-          driver: sql
-          sql:
-            driver: postgres12
-            host: ddb-cluster-rw.ddb.svc.cluster.local
-            port: 5432
-            database: temporal
-            user: temporal
-            password: ""
-            maxConns: 20
-            maxIdleConns: 10
-            maxConnLifetime: "1h"
-            connectAttributes:
-              tx_isolation: "READ-COMMITTED"
-        visibility:
-          # PostgreSQL for visibility store (workflow queries)
-          driver: sql
-          sql:
-            driver: postgres12
-            host: ddb-cluster-rw.ddb.svc.cluster.local
-            port: 5432
-            database: temporal_visibility
-            user: temporal
-            password: ""
-            maxConns: 20
-            maxIdleConns: 10
-            maxConnLifetime: "1h"
+      default:
+        driver: "sql"
+        sql:
+          driver: "postgres12"
+          host: "ddb-cluster-rw.ddb.svc.cluster.local"
+          port: 5432
+          database: "temporal"
+          user: "temporal"
+          # existingSecret + secretKey: point directly at the CNPG-generated
+          # Secret (kubernetes.io/basic-auth, keys: username/password/...)
+          # rather than duplicating the password in git as plaintext. When
+          # existingSecret is set the chart's own server-secret.yaml Secret
+          # template is skipped entirely (see templates/server-secret.yaml:
+          # `not $driverConfig.existingSecret` guards its creation).
+          existingSecret: "temporal-db-role"
+          secretKey: "password"
+          maxConns: 20
+          maxIdleConns: 10
+          maxConnLifetime: "1h"
+          connectAttributes:
+            tx_isolation: "READ-COMMITTED"
+      visibility:
+        driver: "sql"
+        sql:
+          driver: "postgres12"
+          host: "ddb-cluster-rw.ddb.svc.cluster.local"
+          port: 5432
+          database: "temporal_visibility"
+          user: "temporal"
+          existingSecret: "temporal-db-role"
+          secretKey: "password"
+          maxConns: 20
+          maxIdleConns: 10
+          maxConnLifetime: "1h"
  service:
    type: ClusterIP

@@ -90,7 +123,3 @@ web:
 # ── Ingress ────────────────────────────────────────────────────────
 ingress:
  enabled: false
-
-# ── Monitoring ────────────────────────────────────────────────────────
-prometheus:
-  enabled: false
@@ -2,6 +2,33 @@
 # helpers (cloudflared tunnel, duckdns updater) that are already running.
 # Experimental dirs (llm, forge, dev-tools, shadowsocks) are intentionally
 # NOT included yet — add them here once they're production-ready.
+# Syncs the CNPG-generated temporal-db-role Secret from ddb -> temporal ns
+# (see db-secret-sync/copy-job.yaml for why this is a separate Application
+# rather than folded into temporal/'s own kustomization). Runs one wave
+# before `temporal` so the Secret exists before the server pods start.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: temporal-db-secret-sync
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "7"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    targetRevision: main
+    path: k8s/applications/temporal/db-secret-sync
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: temporal
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata: