Commit Graph
7 Commits
Author SHA1 Message Date
Story Crater Bot c2bcda58d9 refactor(cnpg): adopt simple pattern - all apps use 'app' user
ARCHITECTURAL CHANGE: Align with CNPG design intent

BEFORE (Complex, broken):
  - Per-app roles (authentik, temporal) with Database CR owner field
  - Database CR doesn't transfer ownership properly
  - Needed manual permission grants (PostSync Job)
  - Apps couldn't create tables without grants from 'app' role

AFTER (Simple, works):
  - All apps use shared 'app' bootstrap user
  - Database CRs: owner: app (matches actual ownership)
  - No permission grants needed (owner has full rights)
  - Isolation via separate database names only

CHANGES:
  - Database CRs: owner changed from app-specific to 'app'
  - ddb-cluster.yaml: removed managed.roles section
  - Deleted grant-schema-permissions PostSync Job
  - Follows Forgejo pattern (already working this way)

MANUAL STEPS REQUIRED:
  1. Update authentik-secrets: AUTHENTIK_POSTGRESQL__USER=app
  2. Update temporal secrets: similar change
  3. Recreate databases with app as owner
  4. Restart applications

Benefits:
  - Simpler architecture
  - No permission grant complexity
  - Aligns with CNPG single-cluster design
  - Matches working Forgejo implementation
2026-07-23 09:59:06 -07:00
Story Crater Bot ce019f5f3a fix(ddb): add database-level CREATE privilege for schema creation
Authentik migrations need to CREATE SCHEMA (not just tables in public schema).
This requires GRANT CREATE ON DATABASE, not just schema-level permissions.

Added to PostSync Job:
- GRANT CREATE ON DATABASE authentik TO authentik
- GRANT CREATE ON DATABASE temporal TO temporal
- GRANT CREATE ON DATABASE temporal_visibility TO temporal

App user can grant these (it owns the databases).
2026-07-23 09:36:23 -07:00
Story Crater Bot 2499cc241f fix(ddb): add PostSync Job for per-database schema permissions
ROOT CAUSE: CNPG Database CR creates databases but doesn't grant schema
permissions to the owner role. Bootstrap DB owner 'app' retains CREATE
privilege on public schema, blocking authentik/temporal from creating tables.

SECURITY FIX: Removed insecure 'GRANT TO PUBLIC' from postInitApplicationSQL.

SOLUTION: PostSync Job connects as 'app' (DB owner) and grants schema
permissions to named roles (authentik, temporal) in their respective databases.
Runs after Database CRs reconcile, survives CNPG database recreation.

Pattern: Per-database grants via PostSync, not cluster-wide PUBLIC grants.
2026-07-23 09:02:19 -07:00
Story Crater Bot 1c98628417 fix(ddb): grant universal schema permissions to all roles
Adds SQL to postInitApplicationSQL granting schema permissions to PUBLIC.
Allows any role (authentik, temporal, etc) to create tables in databases.

For existing cluster: run SQL manually (done).
For future bootstrap: automatic via initdb.

Pattern for apps: Database CR + app-specific init Job optional (co-located).
2026-07-23 08:07:34 -07:00
Story Crater Bot ec046cccde fix(ddb): use app user credentials in db-permissions Job
ddb-cluster-superuser secret doesn't exist (not configured).
Use ddb-cluster-app secret instead - app is DB owner, can grant permissions.
2026-07-23 08:05:03 -07:00
Story Crater Bot 9d485d1238 fix(ddb): add PostSync Job for database schema permissions
CNPG Database CR creates DBs but doesn't grant schema permissions properly.
Database owner is 'app' instead of specified role (authentik, temporal).

PostSync Job grants ALL on schema public to both app and named roles,
ensuring applications can create tables. Runs after Database CRs reconcile.

Fixes: authentik InsufficientPrivilege error on migration.
2026-07-23 08:02:44 -07:00
Story Crater Bot dafccd5d72 feat: complete GitOps migration, storage HA verification, and cluster fixes
Major accomplishments from comprehensive cluster review:

## Storage HA (answering "are volumes replicated?")
- Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas
- Fixed CLAUDE.md contradiction (sole node → 3-node HA)
- Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer)
- Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static)

## GitOps Infrastructure Cleanup
- Eliminated resource duplication (ddb-cluster single source of truth)
- Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps)
- Updated data-schemas app to point to k8s/data/schemas/ (wave 6)
- Archived old k8s/argocd/bootstrap/ → bootstrap.archived/

## Bootstrap Dependencies Fixed
- Added 05-wait-for-databases.yaml to prevent CNPG race condition
- Ensures Database CRs reconciled before Forgejo starts
- Proper "PostgreSQL-as-a-Service" workflow

## Longhorn CSI Plugin Fixed
- Added patch-csi-tolerations-job.yaml (GitOps PostSync hook)
- CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3)
- Fixes volume attachment on tainted control-plane nodes

## Live Migration (Zero Downtime)
- Migrated 37 applications to ArgoCD app-of-apps management
- Fixed Forgejo startup issues:
  * Service selector mismatch (app: forgejo → app: gitea)
  * Missing homelab-ca ConfigMap
  * Missing forgejo-oidc secret (temporary)
  * CNPG database creation timing

## Documentation (10 comprehensive files)
- WHATS-NEXT.md - Daily GitOps workflow
- MIGRATION-STATUS.md - Cluster health report
- REVIEW-SUMMARY.md - Session overview
- GITOPS-REBUILD-PLAN.md - Architecture reference
- DDB-REVIEW.md - PostgreSQL optimization guide
- STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation
- BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix
- STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale
- IMPLEMENTATION-CHECKLIST.md - Migration checklist
- bootstrap.sh - Automated bootstrap script

## Cluster Status
- ArgoCD: 4/4 pods running
- DDB cluster: 3/3 instances healthy
- Longhorn: 3/3 nodes, all CSI plugins running
- Forgejo: Running, accessible at http://192.168.1.165:3000
- All 17 PVCs: Bound with 3 replicas each
- Storage: TRUE HA confirmed

All future changes via git push only (100% GitOps).
2026-07-22 23:56:34 -07:00