Story Crater Bot
ec046cccde
fix(ddb): use app user credentials in db-permissions Job
...
ddb-cluster-superuser secret doesn't exist (not configured).
Use ddb-cluster-app secret instead - app is DB owner, can grant permissions.
2026-07-23 08:05:03 -07:00
Story Crater Bot
9d485d1238
fix(ddb): add PostSync Job for database schema permissions
...
CNPG Database CR creates DBs but doesn't grant schema permissions properly.
Database owner is 'app' instead of specified role (authentik, temporal).
PostSync Job grants ALL on schema public to both app and named roles,
ensuring applications can create tables. Runs after Database CRs reconcile.
Fixes: authentik InsufficientPrivilege error on migration.
2026-07-23 08:02:44 -07:00
Story Crater Bot
dafccd5d72
feat: complete GitOps migration, storage HA verification, and cluster fixes
...
Major accomplishments from comprehensive cluster review:
## Storage HA (answering "are volumes replicated?")
- Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas
- Fixed CLAUDE.md contradiction (sole node → 3-node HA)
- Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer)
- Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static)
## GitOps Infrastructure Cleanup
- Eliminated resource duplication (ddb-cluster single source of truth)
- Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps)
- Updated data-schemas app to point to k8s/data/schemas/ (wave 6)
- Archived old k8s/argocd/bootstrap/ → bootstrap.archived/
## Bootstrap Dependencies Fixed
- Added 05-wait-for-databases.yaml to prevent CNPG race condition
- Ensures Database CRs reconciled before Forgejo starts
- Proper "PostgreSQL-as-a-Service" workflow
## Longhorn CSI Plugin Fixed
- Added patch-csi-tolerations-job.yaml (GitOps PostSync hook)
- CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3)
- Fixes volume attachment on tainted control-plane nodes
## Live Migration (Zero Downtime)
- Migrated 37 applications to ArgoCD app-of-apps management
- Fixed Forgejo startup issues:
* Service selector mismatch (app: forgejo → app: gitea)
* Missing homelab-ca ConfigMap
* Missing forgejo-oidc secret (temporary)
* CNPG database creation timing
## Documentation (10 comprehensive files)
- WHATS-NEXT.md - Daily GitOps workflow
- MIGRATION-STATUS.md - Cluster health report
- REVIEW-SUMMARY.md - Session overview
- GITOPS-REBUILD-PLAN.md - Architecture reference
- DDB-REVIEW.md - PostgreSQL optimization guide
- STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation
- BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix
- STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale
- IMPLEMENTATION-CHECKLIST.md - Migration checklist
- bootstrap.sh - Automated bootstrap script
## Cluster Status
- ArgoCD: 4/4 pods running
- DDB cluster: 3/3 instances healthy
- Longhorn: 3/3 nodes, all CSI plugins running
- Forgejo: Running, accessible at http://192.168.1.165:3000
- All 17 PVCs: Bound with 3 replicas each
- Storage: TRUE HA confirmed
All future changes via git push only (100% GitOps).
2026-07-22 23:56:34 -07:00
Story Crater Bot
46ec0caf5d
fix(temporal): provision schema via CNPG temporal_visibility Database CR + enable chart schema setup/update jobs — both DBs had zero tables so server died on 'no usable database connection'
2026-07-21 21:09:03 -07:00
Story Crater Bot
5ce0b92186
feat(data): add CNPG managed roles + Database CRs for authentik/temporal — replaces missing helmfile post-sync user creation
...
authentik/temporal DB users+databases were never provisioned (old helmfile hook
gone; db-init-job only made schemas in shared app DB). Adds managed.roles
(authentik/temporal login roles, passwords from basic-auth secrets) + Database CRs
(dedicated DBs owned by each role). Role secrets applied out-of-band (SOPS), not in
kustomize resources so data-schemas app doesn't choke on ciphertext.
2026-07-20 23:36:21 -07:00
Story Crater Bot
a207c56637
fix(k8s,docs): scale ddb-cluster to single instance, pin minio to storage namespace, document 3-CP topology in USAGE
2026-07-20 08:22:53 -07:00
Story Crater Bot
578a707867
feat(gitops): migrate domain to riotpiao.com, add CNPG + Forgejo HA on Redis/Postgres, wire ArgoCD apps — enables cluster rebuild after etcd wipe and unblocks the git-source chicken-egg via standalone Helm-source Applications
2026-07-19 09:29:17 -07:00
Story Crater Bot
e2e17ae0fb
feat(data): Add CNPG cluster + database schema initialization
...
Create production PostgreSQL cluster via CNPG (3-node HA, Longhorn storage).
Schema initialization Job creates schemas for:
- Authentik (identity provider)
- Temporal (workflow engine)
- Vault (secrets management)
- App (generic application databases)
Database layer now captures complete IaC for stateful infrastructure.
Services find ready schemas when deployed.
2026-07-16 14:53:59 -07:00
Story Crater Bot
2d7330798b
refactor(k8s): Reorganize into 5-layer structure with production kustomizations
2026-07-16 14:28:19 -07:00
Story Crater Bot