rock
cb1219bf96
Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam. Authentik keeps serving; no consumer is moved and no Authentik object changes. - k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances - k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat - k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3, automated sync deliberately absent for the first sync (chart PreSync hook re-run risk) - k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS - k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI, crash-loop and PreSync-job-failed rules - terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1, authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and kube-state-metrics has 1925 restarts. Two manual steps still gate the first sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned). The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it to sealed storage and delete it before the first sync. Losing it makes every encrypted ZITADEL value unreadable.
69 lines
3.8 KiB
YAML
69 lines
3.8 KiB
YAML
# ZITADEL alert parity with the Authentik rules it will replace.
|
|
#
|
|
# Only two signals are ported at this point. The metrics endpoint
|
|
# (/debug/metrics, scraped by the chart's own ServiceMonitor in ns iam) exposes
|
|
# no request or error counters that this cluster has verified, so no error-rate
|
|
# rule is written here. Writing one against a guessed series name would produce
|
|
# a permanently-firing or permanently-silent alert — both worse than none.
|
|
# AuthentikOutpostDown is deliberately not ported: it monitors an outpost that
|
|
# carries zero providers and is deleted with the migration, not migrated.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: zitadel-rules
|
|
namespace: iam
|
|
spec:
|
|
groups:
|
|
- name: zitadel.rules
|
|
interval: 30s
|
|
rules:
|
|
- alert: ZitadelDown
|
|
# absent() is required: equality alone cannot fire when the series is
|
|
# gone (ServiceMonitor deleted, Service selector changed, pods scaled
|
|
# to zero). Those are exactly the outages a target-down rule exists for.
|
|
expr: absent(up{job="zitadel"}) or up{job="zitadel"} == 0
|
|
for: 10m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "ZITADEL API target is down"
|
|
description: "Prometheus cannot scrape the ZITADEL API for 10 minutes. Token issuance and every OIDC login depend on it. Check the zitadel pods and the zitadel-db cluster in namespace iam. Value: {{ $value }}"
|
|
|
|
- alert: ZitadelLoginUIDown
|
|
expr: absent(kube_deployment_status_replicas_available{namespace="iam",deployment="zitadel-login"}) or kube_deployment_status_replicas_available{namespace="iam",deployment="zitadel-login"} == 0
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "ZITADEL login UI has no available replica"
|
|
description: "The login UI serves /ui/v2/login. With zero available replicas a browser login completes at the IdP but cannot render. Value: {{ $value }}"
|
|
|
|
- alert: ZitadelRestarting
|
|
# pod!~"zitadel-db-.*" matters: the CNPG database pods share the
|
|
# `zitadel` name prefix, and a database restart must not page as an
|
|
# IdP crash loop.
|
|
expr: absent(kube_pod_container_status_restarts_total{namespace="iam",pod=~"zitadel.*",pod!~"zitadel-db-.*"}) or increase(kube_pod_container_status_restarts_total{namespace="iam",pod=~"zitadel.*",pod!~"zitadel-db-.*"}[1h]) > 3
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "ZITADEL pod restarts more than 3 times per hour"
|
|
description: "A crash loop on the IdP is a cluster-wide login outage. Check the pod logs and the masterkey/database secret references. Value: {{ $value }}"
|
|
|
|
# The chart's zitadel-init and zitadel-setup Jobs are PreSync hooks with
|
|
# restartPolicy: Never, so their restart counter stays 0. Without this
|
|
# rule a failed schema init or first-instance setup reaches nobody but
|
|
# the person who runs `kubectl -n iam get job`.
|
|
# No absent() guard here on purpose: the chart ships only
|
|
# `before-hook-creation`, so a successful hook Job is replaced at the
|
|
# next PreSync. Once the jobs are retired, absent() would fire on a
|
|
# healthy cluster.
|
|
- alert: ZitadelSetupJobFailed
|
|
expr: kube_job_status_failed{namespace="iam",job_name=~"zitadel-(init|setup).*"} > 0
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "ZITADEL init/setup job failed"
|
|
description: "A PreSync job failed, so ArgoCD aborted the sync before the Deployment was applied. Read the job pod logs, then delete the Job object to let the next sync retry. Value: {{ $value }}"
|