feat: complete GitOps migration, storage HA verification, and cluster fixes
Major accomplishments from comprehensive cluster review: ## Storage HA (answering "are volumes replicated?") - Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas - Fixed CLAUDE.md contradiction (sole node → 3-node HA) - Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer) - Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static) ## GitOps Infrastructure Cleanup - Eliminated resource duplication (ddb-cluster single source of truth) - Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps) - Updated data-schemas app to point to k8s/data/schemas/ (wave 6) - Archived old k8s/argocd/bootstrap/ → bootstrap.archived/ ## Bootstrap Dependencies Fixed - Added 05-wait-for-databases.yaml to prevent CNPG race condition - Ensures Database CRs reconciled before Forgejo starts - Proper "PostgreSQL-as-a-Service" workflow ## Longhorn CSI Plugin Fixed - Added patch-csi-tolerations-job.yaml (GitOps PostSync hook) - CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3) - Fixes volume attachment on tainted control-plane nodes ## Live Migration (Zero Downtime) - Migrated 37 applications to ArgoCD app-of-apps management - Fixed Forgejo startup issues: * Service selector mismatch (app: forgejo → app: gitea) * Missing homelab-ca ConfigMap * Missing forgejo-oidc secret (temporary) * CNPG database creation timing ## Documentation (10 comprehensive files) - WHATS-NEXT.md - Daily GitOps workflow - MIGRATION-STATUS.md - Cluster health report - REVIEW-SUMMARY.md - Session overview - GITOPS-REBUILD-PLAN.md - Architecture reference - DDB-REVIEW.md - PostgreSQL optimization guide - STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation - BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix - STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale - IMPLEMENTATION-CHECKLIST.md - Migration checklist - bootstrap.sh - Automated bootstrap script ## Cluster Status - ArgoCD: 4/4 pods running - DDB cluster: 3/3 instances healthy - Longhorn: 3/3 nodes, all CSI plugins running - Forgejo: Running, accessible at http://192.168.1.165:3000 - All 17 PVCs: Bound with 3 replicas each - Storage: TRUE HA confirmed All future changes via git push only (100% GitOps).
This commit is contained in:
@@ -0,0 +1,10 @@
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Database
|
||||
metadata:
|
||||
name: authentik
|
||||
namespace: ddb
|
||||
spec:
|
||||
name: authentik
|
||||
owner: authentik
|
||||
cluster:
|
||||
name: ddb-cluster
|
||||
@@ -0,0 +1,67 @@
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: db-schema-init
|
||||
namespace: ddb
|
||||
labels:
|
||||
app: postgresql-init
|
||||
layer: data
|
||||
spec:
|
||||
backoffLimit: 3
|
||||
ttlSecondsAfterFinished: 3600 # Keep Job for 1 hour after completion
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: postgresql-init
|
||||
spec:
|
||||
serviceAccountName: default
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: schema-init
|
||||
image: postgres:16.2-alpine
|
||||
env:
|
||||
- name: PGHOST
|
||||
value: ddb-cluster-rw.ddb.svc.cluster.local
|
||||
- name: PGPORT
|
||||
value: "5432"
|
||||
- name: PGDATABASE
|
||||
value: app
|
||||
- name: PGUSER
|
||||
value: app
|
||||
- name: PGPASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: ddb-cluster-app
|
||||
key: password
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
# Wait for cluster to be ready
|
||||
echo "Waiting for PostgreSQL to be ready..."
|
||||
until pg_isready -h $PGHOST -p $PGPORT -U $PGUSER; do
|
||||
echo "Waiting..."
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "Creating schemas..."
|
||||
|
||||
# Create schemas from ConfigMap
|
||||
psql -h $PGHOST -p $PGPORT -U $PGUSER -d $PGDATABASE << 'EOF'
|
||||
CREATE SCHEMA IF NOT EXISTS authentik;
|
||||
GRANT USAGE ON SCHEMA authentik TO app;
|
||||
GRANT CREATE ON SCHEMA authentik TO app;
|
||||
|
||||
CREATE SCHEMA IF NOT EXISTS temporal;
|
||||
GRANT USAGE ON SCHEMA temporal TO app;
|
||||
GRANT CREATE ON SCHEMA temporal TO app;
|
||||
|
||||
CREATE SCHEMA IF NOT EXISTS vault;
|
||||
GRANT USAGE ON SCHEMA vault TO app;
|
||||
GRANT CREATE ON SCHEMA vault TO app;
|
||||
|
||||
GRANT USAGE ON SCHEMA public TO app;
|
||||
GRANT CREATE ON SCHEMA public TO app;
|
||||
EOF
|
||||
|
||||
echo "✓ Schema initialization complete"
|
||||
@@ -0,0 +1,17 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: data-schemas
|
||||
|
||||
namespace: ddb
|
||||
|
||||
# GitOps-managed database schemas (ArgoCD wave 6).
|
||||
# These depend on ddb-cluster existing (bootstrap wave 0).
|
||||
resources:
|
||||
- authentik-database.yaml
|
||||
- temporal-database.yaml
|
||||
- temporal-visibility-database.yaml
|
||||
- schemas.yaml
|
||||
- db-init-job.yaml
|
||||
|
||||
# db-role-secrets.enc.yaml handled by SOPS secrets Application (wave 4)
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
# Schema initialization ConfigMap for PostgreSQL services
|
||||
# Services: Authentik, Vault (optional), Temporal, others
|
||||
# Applied via initdb Jobs after CNPG cluster is ready
|
||||
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: db-schemas
|
||||
namespace: ddb
|
||||
labels:
|
||||
app: postgresql-schemas
|
||||
data:
|
||||
# Authentik schema (minimal — Authentik creates most tables via Django ORM)
|
||||
authentik-init.sql: |
|
||||
CREATE SCHEMA IF NOT EXISTS authentik;
|
||||
GRANT USAGE ON SCHEMA authentik TO app;
|
||||
GRANT CREATE ON SCHEMA authentik TO app;
|
||||
|
||||
# Temporal schema
|
||||
temporal-init.sql: |
|
||||
CREATE SCHEMA IF NOT EXISTS temporal;
|
||||
GRANT USAGE ON SCHEMA temporal TO app;
|
||||
GRANT CREATE ON SCHEMA temporal TO app;
|
||||
|
||||
# Vault schema (if using PostgreSQL backend)
|
||||
vault-init.sql: |
|
||||
CREATE SCHEMA IF NOT EXISTS vault;
|
||||
GRANT USAGE ON SCHEMA vault TO app;
|
||||
GRANT CREATE ON SCHEMA vault TO app;
|
||||
|
||||
# Application schema (generic)
|
||||
app-init.sql: |
|
||||
-- Default app schema (public)
|
||||
GRANT USAGE ON SCHEMA public TO app;
|
||||
GRANT CREATE ON SCHEMA public TO app;
|
||||
@@ -0,0 +1,10 @@
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Database
|
||||
metadata:
|
||||
name: temporal
|
||||
namespace: ddb
|
||||
spec:
|
||||
name: temporal
|
||||
owner: temporal
|
||||
cluster:
|
||||
name: ddb-cluster
|
||||
@@ -0,0 +1,10 @@
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Database
|
||||
metadata:
|
||||
name: temporal-visibility
|
||||
namespace: ddb
|
||||
spec:
|
||||
name: temporal_visibility
|
||||
owner: temporal
|
||||
cluster:
|
||||
name: ddb-cluster
|
||||
Reference in New Issue
Block a user