fix(iam): stop the setup hook running after first boot
A PreSync setup run is not harmless once the instance exists: it replays every repeatable migration on every sync, and interrupting one leaves a system.migration.started event with no matching done. ZITADEL then waits forever on 'migration already started'. That deadlocked this instance twice (projections.projects4, then repeatable_delete_stale_org_fields). Move the setup job's hook from pre-install,pre-upgrade to post-delete so ArgoCD sync never runs it. Re-enable deliberately for a schema upgrade.
This commit is contained in:
@@ -118,12 +118,27 @@ initJob:
|
|||||||
activeDeadlineSeconds: 600
|
activeDeadlineSeconds: 600
|
||||||
|
|
||||||
# The setup job runs the full first-instance creation: schema migrations, the
|
# The setup job runs the full first-instance creation: schema migrations, the
|
||||||
# system org, the iam-admin machine key, and 13 projection replays. The chart
|
# system org, the iam-admin machine key, and 13 projection replays. 20 minutes
|
||||||
# default killed it at ~5.5 minutes on 2026-10-25 while it was still replaying
|
# leaves room for a cold database without masking a genuine hang.
|
||||||
# projections.projects4, which failed the whole sync. 20 minutes leaves room for
|
#
|
||||||
# a cold database without masking a genuine hang.
|
# DONE, 2026-10-26 — first boot completed. The annotations are moved from
|
||||||
|
# pre-install/pre-upgrade to post-delete because a PreSync setup run is not
|
||||||
|
# harmless after boot:
|
||||||
|
# - it re-runs every repeatable migration on every sync;
|
||||||
|
# - interrupting one (a sync aborted, or a human deleting the Job) leaves a
|
||||||
|
# `system.migration.started` event with no matching `done`, and ZITADEL then
|
||||||
|
# waits forever on `migration already started`. That is exactly how this
|
||||||
|
# instance deadlocked on projections.projects4 and later on
|
||||||
|
# repeatable_delete_stale_org_fields.
|
||||||
|
# A post-delete hook is never executed by ArgoCD sync, so setup runs only on
|
||||||
|
# first install. Re-enable it deliberately (and only with initJob too) for a
|
||||||
|
# schema upgrade, then revert.
|
||||||
setupJob:
|
setupJob:
|
||||||
activeDeadlineSeconds: 1200
|
activeDeadlineSeconds: 1200
|
||||||
|
annotations:
|
||||||
|
helm.sh/hook: post-delete
|
||||||
|
helm.sh/hook-delete-policy: hook-succeeded
|
||||||
|
helm.sh/hook-weight: "2"
|
||||||
|
|
||||||
# Two API replicas: the read path survives one node drain, and both share the
|
# Two API replicas: the read path survives one node drain, and both share the
|
||||||
# same masterkey and database. The login UI stays at one replica — a session
|
# same masterkey and database. The login UI stays at one replica — a session
|
||||||
|
|||||||
Reference in New Issue
Block a user