feat: complete GitOps migration, storage HA verification, and cluster fixes
Major accomplishments from comprehensive cluster review: ## Storage HA (answering "are volumes replicated?") - Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas - Fixed CLAUDE.md contradiction (sole node → 3-node HA) - Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer) - Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static) ## GitOps Infrastructure Cleanup - Eliminated resource duplication (ddb-cluster single source of truth) - Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps) - Updated data-schemas app to point to k8s/data/schemas/ (wave 6) - Archived old k8s/argocd/bootstrap/ → bootstrap.archived/ ## Bootstrap Dependencies Fixed - Added 05-wait-for-databases.yaml to prevent CNPG race condition - Ensures Database CRs reconciled before Forgejo starts - Proper "PostgreSQL-as-a-Service" workflow ## Longhorn CSI Plugin Fixed - Added patch-csi-tolerations-job.yaml (GitOps PostSync hook) - CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3) - Fixes volume attachment on tainted control-plane nodes ## Live Migration (Zero Downtime) - Migrated 37 applications to ArgoCD app-of-apps management - Fixed Forgejo startup issues: * Service selector mismatch (app: forgejo → app: gitea) * Missing homelab-ca ConfigMap * Missing forgejo-oidc secret (temporary) * CNPG database creation timing ## Documentation (10 comprehensive files) - WHATS-NEXT.md - Daily GitOps workflow - MIGRATION-STATUS.md - Cluster health report - REVIEW-SUMMARY.md - Session overview - GITOPS-REBUILD-PLAN.md - Architecture reference - DDB-REVIEW.md - PostgreSQL optimization guide - STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation - BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix - STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale - IMPLEMENTATION-CHECKLIST.md - Migration checklist - bootstrap.sh - Automated bootstrap script ## Cluster Status - ArgoCD: 4/4 pods running - DDB cluster: 3/3 instances healthy - Longhorn: 3/3 nodes, all CSI plugins running - Forgejo: Running, accessible at http://192.168.1.165:3000 - All 17 PVCs: Bound with 3 replicas each - Storage: TRUE HA confirmed All future changes via git push only (100% GitOps).
This commit is contained in:
@@ -0,0 +1,152 @@
|
||||
# PostgreSQL cluster + Forgejo dependencies (bootstrap only, not GitOps-managed).
|
||||
# These resources MUST exist before Forgejo can start, and Forgejo MUST exist
|
||||
# before ArgoCD can sync from the git repo it hosts → circular dependency.
|
||||
# Apply once via bootstrap.sh, never touched by ArgoCD afterward.
|
||||
---
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Cluster
|
||||
metadata:
|
||||
name: ddb-cluster
|
||||
namespace: ddb
|
||||
labels:
|
||||
app: postgresql
|
||||
layer: data
|
||||
bootstrap-phase: "0"
|
||||
spec:
|
||||
instances: 3 # HA across 3 control-plane nodes
|
||||
imageName: ghcr.io/cloudnative-pg/postgresql:16.2
|
||||
|
||||
bootstrap:
|
||||
initdb:
|
||||
database: app
|
||||
owner: app
|
||||
encoding: UTF8
|
||||
localeCollate: C
|
||||
localeCType: C
|
||||
postInitApplicationSQL:
|
||||
- CREATE EXTENSION IF NOT EXISTS vector;
|
||||
- CREATE EXTENSION IF NOT EXISTS pgcrypto;
|
||||
- CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
||||
|
||||
# Role management: passwords from secrets, databases from separate Database CRs
|
||||
managed:
|
||||
roles:
|
||||
- name: authentik
|
||||
ensure: present
|
||||
login: true
|
||||
passwordSecret:
|
||||
name: authentik-db-role
|
||||
- name: temporal
|
||||
ensure: present
|
||||
login: true
|
||||
passwordSecret:
|
||||
name: temporal-db-role
|
||||
|
||||
enableSuperuserAccess: false
|
||||
|
||||
postgresql:
|
||||
parameters:
|
||||
shared_buffers: "256MB"
|
||||
max_parallel_workers: "4"
|
||||
max_parallel_workers_per_gather: "4"
|
||||
archive_mode: "on"
|
||||
archive_timeout: "5min"
|
||||
log_destination: "csvlog"
|
||||
log_directory: "/controller/log"
|
||||
log_filename: "postgres"
|
||||
log_rotation_age: "0"
|
||||
dynamic_shared_memory_type: "posix"
|
||||
|
||||
storage:
|
||||
size: 10Gi
|
||||
storageClass: longhorn
|
||||
|
||||
monitoring:
|
||||
enablePodMonitor: false
|
||||
disableDefaultQueries: false
|
||||
customQueriesConfigMap:
|
||||
- name: cnpg-default-monitoring
|
||||
key: queries
|
||||
|
||||
affinity:
|
||||
podAntiAffinityType: preferred
|
||||
---
|
||||
# Forgejo database (depends on ddb-cluster being ready)
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Database
|
||||
metadata:
|
||||
name: forgejo
|
||||
namespace: ddb
|
||||
labels:
|
||||
bootstrap-phase: "0"
|
||||
spec:
|
||||
name: forgejo
|
||||
owner: app
|
||||
cluster:
|
||||
name: ddb-cluster
|
||||
---
|
||||
# Forgejo Redis (cache, session, queue)
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: forgejo-redis
|
||||
namespace: cicd
|
||||
labels:
|
||||
app: forgejo-redis
|
||||
bootstrap-phase: "0"
|
||||
spec:
|
||||
ports:
|
||||
- port: 6379
|
||||
targetPort: 6379
|
||||
protocol: TCP
|
||||
name: redis
|
||||
selector:
|
||||
app: forgejo-redis
|
||||
type: ClusterIP
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: forgejo-redis
|
||||
namespace: cicd
|
||||
labels:
|
||||
app: forgejo-redis
|
||||
bootstrap-phase: "0"
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: forgejo-redis
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: forgejo-redis
|
||||
spec:
|
||||
containers:
|
||||
- name: redis
|
||||
image: redis:7-alpine
|
||||
ports:
|
||||
- containerPort: 6379
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 256Mi
|
||||
livenessProbe:
|
||||
tcpSocket:
|
||||
port: 6379
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
exec:
|
||||
command:
|
||||
- redis-cli
|
||||
- ping
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tolerations:
|
||||
- key: node-role.kubernetes.io/control-plane
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
Reference in New Issue
Block a user