Story Crater Bot
dafccd5d72
Major accomplishments from comprehensive cluster review: ## Storage HA (answering "are volumes replicated?") - Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas - Fixed CLAUDE.md contradiction (sole node → 3-node HA) - Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer) - Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static) ## GitOps Infrastructure Cleanup - Eliminated resource duplication (ddb-cluster single source of truth) - Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps) - Updated data-schemas app to point to k8s/data/schemas/ (wave 6) - Archived old k8s/argocd/bootstrap/ → bootstrap.archived/ ## Bootstrap Dependencies Fixed - Added 05-wait-for-databases.yaml to prevent CNPG race condition - Ensures Database CRs reconciled before Forgejo starts - Proper "PostgreSQL-as-a-Service" workflow ## Longhorn CSI Plugin Fixed - Added patch-csi-tolerations-job.yaml (GitOps PostSync hook) - CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3) - Fixes volume attachment on tainted control-plane nodes ## Live Migration (Zero Downtime) - Migrated 37 applications to ArgoCD app-of-apps management - Fixed Forgejo startup issues: * Service selector mismatch (app: forgejo → app: gitea) * Missing homelab-ca ConfigMap * Missing forgejo-oidc secret (temporary) * CNPG database creation timing ## Documentation (10 comprehensive files) - WHATS-NEXT.md - Daily GitOps workflow - MIGRATION-STATUS.md - Cluster health report - REVIEW-SUMMARY.md - Session overview - GITOPS-REBUILD-PLAN.md - Architecture reference - DDB-REVIEW.md - PostgreSQL optimization guide - STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation - BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix - STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale - IMPLEMENTATION-CHECKLIST.md - Migration checklist - bootstrap.sh - Automated bootstrap script ## Cluster Status - ArgoCD: 4/4 pods running - DDB cluster: 3/3 instances healthy - Longhorn: 3/3 nodes, all CSI plugins running - Forgejo: Running, accessible at http://192.168.1.165:3000 - All 17 PVCs: Bound with 3 replicas each - Storage: TRUE HA confirmed All future changes via git push only (100% GitOps).
33 lines
706 B
Bash
Executable File
33 lines
706 B
Bash
Executable File
#!/bin/bash
|
|
set -e
|
|
|
|
export SOPS_AGE_KEY_FILE=/sops-age/keys.txt
|
|
|
|
# Decrypt grafana secrets once
|
|
GRAFANA_SECRETS=$(sops -d k8s/platform/logging/grafana-secrets.enc.yaml)
|
|
|
|
# Stage 0: grafana
|
|
OIDC_SECRET=$(echo "$GRAFANA_SECRETS" | yq -r '.env.GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET')
|
|
ADMIN_PASSWORD=$(echo "$GRAFANA_SECRETS" | yq -r '.adminPassword')
|
|
|
|
cat <<EOF
|
|
---
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: grafana-oidc
|
|
namespace: logging
|
|
type: Opaque
|
|
data:
|
|
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: $(echo -n "$OIDC_SECRET" | base64 -w0)
|
|
---
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: grafana-admin
|
|
namespace: logging
|
|
type: Opaque
|
|
data:
|
|
admin-password: $(echo -n "$ADMIN_PASSWORD" | base64 -w0)
|
|
EOF
|