Commit Graph
5 Commits
Author SHA1 Message Date
Story Crater Bot 2499cc241f fix(ddb): add PostSync Job for per-database schema permissions
ROOT CAUSE: CNPG Database CR creates databases but doesn't grant schema
permissions to the owner role. Bootstrap DB owner 'app' retains CREATE
privilege on public schema, blocking authentik/temporal from creating tables.

SECURITY FIX: Removed insecure 'GRANT TO PUBLIC' from postInitApplicationSQL.

SOLUTION: PostSync Job connects as 'app' (DB owner) and grants schema
permissions to named roles (authentik, temporal) in their respective databases.
Runs after Database CRs reconcile, survives CNPG database recreation.

Pattern: Per-database grants via PostSync, not cluster-wide PUBLIC grants.
2026-07-23 09:02:19 -07:00
Story Crater Bot 1c98628417 fix(ddb): grant universal schema permissions to all roles
Adds SQL to postInitApplicationSQL granting schema permissions to PUBLIC.
Allows any role (authentik, temporal, etc) to create tables in databases.

For existing cluster: run SQL manually (done).
For future bootstrap: automatic via initdb.

Pattern for apps: Database CR + app-specific init Job optional (co-located).
2026-07-23 08:07:34 -07:00
Story Crater Bot ec046cccde fix(ddb): use app user credentials in db-permissions Job
ddb-cluster-superuser secret doesn't exist (not configured).
Use ddb-cluster-app secret instead - app is DB owner, can grant permissions.
2026-07-23 08:05:03 -07:00
Story Crater Bot 9d485d1238 fix(ddb): add PostSync Job for database schema permissions
CNPG Database CR creates DBs but doesn't grant schema permissions properly.
Database owner is 'app' instead of specified role (authentik, temporal).

PostSync Job grants ALL on schema public to both app and named roles,
ensuring applications can create tables. Runs after Database CRs reconcile.

Fixes: authentik InsufficientPrivilege error on migration.
2026-07-23 08:02:44 -07:00
Story Crater Bot dafccd5d72 feat: complete GitOps migration, storage HA verification, and cluster fixes
Major accomplishments from comprehensive cluster review:

## Storage HA (answering "are volumes replicated?")
- Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas
- Fixed CLAUDE.md contradiction (sole node → 3-node HA)
- Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer)
- Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static)

## GitOps Infrastructure Cleanup
- Eliminated resource duplication (ddb-cluster single source of truth)
- Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps)
- Updated data-schemas app to point to k8s/data/schemas/ (wave 6)
- Archived old k8s/argocd/bootstrap/ → bootstrap.archived/

## Bootstrap Dependencies Fixed
- Added 05-wait-for-databases.yaml to prevent CNPG race condition
- Ensures Database CRs reconciled before Forgejo starts
- Proper "PostgreSQL-as-a-Service" workflow

## Longhorn CSI Plugin Fixed
- Added patch-csi-tolerations-job.yaml (GitOps PostSync hook)
- CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3)
- Fixes volume attachment on tainted control-plane nodes

## Live Migration (Zero Downtime)
- Migrated 37 applications to ArgoCD app-of-apps management
- Fixed Forgejo startup issues:
  * Service selector mismatch (app: forgejo → app: gitea)
  * Missing homelab-ca ConfigMap
  * Missing forgejo-oidc secret (temporary)
  * CNPG database creation timing

## Documentation (10 comprehensive files)
- WHATS-NEXT.md - Daily GitOps workflow
- MIGRATION-STATUS.md - Cluster health report
- REVIEW-SUMMARY.md - Session overview
- GITOPS-REBUILD-PLAN.md - Architecture reference
- DDB-REVIEW.md - PostgreSQL optimization guide
- STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation
- BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix
- STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale
- IMPLEMENTATION-CHECKLIST.md - Migration checklist
- bootstrap.sh - Automated bootstrap script

## Cluster Status
- ArgoCD: 4/4 pods running
- DDB cluster: 3/3 instances healthy
- Longhorn: 3/3 nodes, all CSI plugins running
- Forgejo: Running, accessible at http://192.168.1.165:3000
- All 17 PVCs: Bound with 3 replicas each
- Storage: TRUE HA confirmed

All future changes via git push only (100% GitOps).
2026-07-22 23:56:34 -07:00