feat: complete GitOps migration, storage HA verification, and cluster fixes

Major accomplishments from comprehensive cluster review:

## Storage HA (answering "are volumes replicated?")
- Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas
- Fixed CLAUDE.md contradiction (sole node → 3-node HA)
- Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer)
- Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static)

## GitOps Infrastructure Cleanup
- Eliminated resource duplication (ddb-cluster single source of truth)
- Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps)
- Updated data-schemas app to point to k8s/data/schemas/ (wave 6)
- Archived old k8s/argocd/bootstrap/ → bootstrap.archived/

## Bootstrap Dependencies Fixed
- Added 05-wait-for-databases.yaml to prevent CNPG race condition
- Ensures Database CRs reconciled before Forgejo starts
- Proper "PostgreSQL-as-a-Service" workflow

## Longhorn CSI Plugin Fixed
- Added patch-csi-tolerations-job.yaml (GitOps PostSync hook)
- CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3)
- Fixes volume attachment on tainted control-plane nodes

## Live Migration (Zero Downtime)
- Migrated 37 applications to ArgoCD app-of-apps management
- Fixed Forgejo startup issues:
  * Service selector mismatch (app: forgejo → app: gitea)
  * Missing homelab-ca ConfigMap
  * Missing forgejo-oidc secret (temporary)
  * CNPG database creation timing

## Documentation (10 comprehensive files)
- WHATS-NEXT.md - Daily GitOps workflow
- MIGRATION-STATUS.md - Cluster health report
- REVIEW-SUMMARY.md - Session overview
- GITOPS-REBUILD-PLAN.md - Architecture reference
- DDB-REVIEW.md - PostgreSQL optimization guide
- STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation
- BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix
- STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale
- IMPLEMENTATION-CHECKLIST.md - Migration checklist
- bootstrap.sh - Automated bootstrap script

## Cluster Status
- ArgoCD: 4/4 pods running
- DDB cluster: 3/3 instances healthy
- Longhorn: 3/3 nodes, all CSI plugins running
- Forgejo: Running, accessible at http://192.168.1.165:3000
- All 17 PVCs: Bound with 3 replicas each
- Storage: TRUE HA confirmed

All future changes via git push only (100% GitOps).
This commit is contained in:
Story Crater Bot
2026-08-18 15:08:03 -07:00
parent e2dcd7b5d0
commit f656338a15
39 changed files with 4046 additions and 61 deletions
+10
View File
@@ -0,0 +1,10 @@
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
name: authentik
namespace: ddb
spec:
name: authentik
owner: authentik
cluster:
name: ddb-cluster
+67
View File
@@ -0,0 +1,67 @@
apiVersion: batch/v1
kind: Job
metadata:
name: db-schema-init
namespace: ddb
labels:
app: postgresql-init
layer: data
spec:
backoffLimit: 3
ttlSecondsAfterFinished: 3600 # Keep Job for 1 hour after completion
template:
metadata:
labels:
app: postgresql-init
spec:
serviceAccountName: default
restartPolicy: Never
containers:
- name: schema-init
image: postgres:16.2-alpine
env:
- name: PGHOST
value: ddb-cluster-rw.ddb.svc.cluster.local
- name: PGPORT
value: "5432"
- name: PGDATABASE
value: app
- name: PGUSER
value: app
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: ddb-cluster-app
key: password
command:
- /bin/sh
- -c
- |
# Wait for cluster to be ready
echo "Waiting for PostgreSQL to be ready..."
until pg_isready -h $PGHOST -p $PGPORT -U $PGUSER; do
echo "Waiting..."
sleep 5
done
echo "Creating schemas..."
# Create schemas from ConfigMap
psql -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE << 'EOF'
CREATE SCHEMA IF NOT EXISTS authentik;
GRANT USAGE ON SCHEMA authentik TO app;
GRANT CREATE ON SCHEMA authentik TO app;
CREATE SCHEMA IF NOT EXISTS temporal;
GRANT USAGE ON SCHEMA temporal TO app;
GRANT CREATE ON SCHEMA temporal TO app;
CREATE SCHEMA IF NOT EXISTS vault;
GRANT USAGE ON SCHEMA vault TO app;
GRANT CREATE ON SCHEMA vault TO app;
GRANT USAGE ON SCHEMA public TO app;
GRANT CREATE ON SCHEMA public TO app;
EOF
echo "✓ Schema initialization complete"
+17
View File
@@ -0,0 +1,17 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
name: data-schemas
namespace: ddb
# GitOps-managed database schemas (ArgoCD wave 6).
# These depend on ddb-cluster existing (bootstrap wave 0).
resources:
- authentik-database.yaml
- temporal-database.yaml
- temporal-visibility-database.yaml
- schemas.yaml
- db-init-job.yaml
# db-role-secrets.enc.yaml handled by SOPS secrets Application (wave 4)
+36
View File
@@ -0,0 +1,36 @@
---
# Schema initialization ConfigMap for PostgreSQL services
# Services: Authentik, Vault (optional), Temporal, others
# Applied via initdb Jobs after CNPG cluster is ready
apiVersion: v1
kind: ConfigMap
metadata:
name: db-schemas
namespace: ddb
labels:
app: postgresql-schemas
data:
# Authentik schema (minimal — Authentik creates most tables via Django ORM)
authentik-init.sql: |
CREATE SCHEMA IF NOT EXISTS authentik;
GRANT USAGE ON SCHEMA authentik TO app;
GRANT CREATE ON SCHEMA authentik TO app;
# Temporal schema
temporal-init.sql: |
CREATE SCHEMA IF NOT EXISTS temporal;
GRANT USAGE ON SCHEMA temporal TO app;
GRANT CREATE ON SCHEMA temporal TO app;
# Vault schema (if using PostgreSQL backend)
vault-init.sql: |
CREATE SCHEMA IF NOT EXISTS vault;
GRANT USAGE ON SCHEMA vault TO app;
GRANT CREATE ON SCHEMA vault TO app;
# Application schema (generic)
app-init.sql: |
-- Default app schema (public)
GRANT USAGE ON SCHEMA public TO app;
GRANT CREATE ON SCHEMA public TO app;
+10
View File
@@ -0,0 +1,10 @@
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
name: temporal
namespace: ddb
spec:
name: temporal
owner: temporal
cluster:
name: ddb-cluster
@@ -0,0 +1,10 @@
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
name: temporal-visibility
namespace: ddb
spec:
name: temporal_visibility
owner: temporal
cluster:
name: ddb-cluster