Story Crater Bot
f656338a15
Major accomplishments from comprehensive cluster review: ## Storage HA (answering "are volumes replicated?") - Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas - Fixed CLAUDE.md contradiction (sole node → 3-node HA) - Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer) - Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static) ## GitOps Infrastructure Cleanup - Eliminated resource duplication (ddb-cluster single source of truth) - Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps) - Updated data-schemas app to point to k8s/data/schemas/ (wave 6) - Archived old k8s/argocd/bootstrap/ → bootstrap.archived/ ## Bootstrap Dependencies Fixed - Added 05-wait-for-databases.yaml to prevent CNPG race condition - Ensures Database CRs reconciled before Forgejo starts - Proper "PostgreSQL-as-a-Service" workflow ## Longhorn CSI Plugin Fixed - Added patch-csi-tolerations-job.yaml (GitOps PostSync hook) - CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3) - Fixes volume attachment on tainted control-plane nodes ## Live Migration (Zero Downtime) - Migrated 37 applications to ArgoCD app-of-apps management - Fixed Forgejo startup issues: * Service selector mismatch (app: forgejo → app: gitea) * Missing homelab-ca ConfigMap * Missing forgejo-oidc secret (temporary) * CNPG database creation timing ## Documentation (10 comprehensive files) - WHATS-NEXT.md - Daily GitOps workflow - MIGRATION-STATUS.md - Cluster health report - REVIEW-SUMMARY.md - Session overview - GITOPS-REBUILD-PLAN.md - Architecture reference - DDB-REVIEW.md - PostgreSQL optimization guide - STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation - BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix - STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale - IMPLEMENTATION-CHECKLIST.md - Migration checklist - bootstrap.sh - Automated bootstrap script ## Cluster Status - ArgoCD: 4/4 pods running - DDB cluster: 3/3 instances healthy - Longhorn: 3/3 nodes, all CSI plugins running - Forgejo: Running, accessible at http://192.168.1.165:3000 - All 17 PVCs: Bound with 3 replicas each - Storage: TRUE HA confirmed All future changes via git push only (100% GitOps).
85 lines
2.4 KiB
YAML
85 lines
2.4 KiB
YAML
# PostSync hook to patch longhorn-csi-plugin DaemonSet with control-plane tolerations
|
|
# This runs after longhorn-config Application syncs, ensuring CSI plugin can run on all nodes
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: longhorn-patch-csi-tolerations
|
|
namespace: longhorn-system
|
|
annotations:
|
|
argocd.argoproj.io/hook: PostSync
|
|
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
|
|
spec:
|
|
backoffLimit: 3
|
|
template:
|
|
metadata:
|
|
name: patch-csi-tolerations
|
|
spec:
|
|
restartPolicy: Never
|
|
serviceAccountName: longhorn-patch-csi-tolerations
|
|
containers:
|
|
- name: patch
|
|
image: bitnami/kubectl:latest
|
|
command:
|
|
- /bin/bash
|
|
- -c
|
|
- |
|
|
set -euo pipefail
|
|
|
|
echo "Patching longhorn-csi-plugin DaemonSet with control-plane tolerations..."
|
|
|
|
kubectl patch daemonset longhorn-csi-plugin -n longhorn-system --type=json -p='[
|
|
{
|
|
"op": "add",
|
|
"path": "/spec/template/spec/tolerations/-",
|
|
"value": {
|
|
"key": "node-role.kubernetes.io/control-plane",
|
|
"operator": "Exists",
|
|
"effect": "NoSchedule"
|
|
}
|
|
}
|
|
]'
|
|
|
|
echo "✓ Patch applied successfully"
|
|
|
|
echo ""
|
|
echo "Waiting for CSI plugin pods to roll out to all nodes..."
|
|
kubectl rollout status daemonset/longhorn-csi-plugin -n longhorn-system --timeout=120s
|
|
|
|
echo ""
|
|
echo "Final status:"
|
|
kubectl get daemonset longhorn-csi-plugin -n longhorn-system
|
|
kubectl get pods -n longhorn-system -l app=longhorn-csi-plugin -o wide
|
|
---
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: longhorn-patch-csi-tolerations
|
|
namespace: longhorn-system
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: Role
|
|
metadata:
|
|
name: longhorn-patch-csi-tolerations
|
|
namespace: longhorn-system
|
|
rules:
|
|
- apiGroups: ["apps"]
|
|
resources: ["daemonsets"]
|
|
verbs: ["get", "patch"]
|
|
- apiGroups: [""]
|
|
resources: ["pods"]
|
|
verbs: ["list", "get"]
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: RoleBinding
|
|
metadata:
|
|
name: longhorn-patch-csi-tolerations
|
|
namespace: longhorn-system
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: Role
|
|
name: longhorn-patch-csi-tolerations
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: longhorn-patch-csi-tolerations
|
|
namespace: longhorn-system
|