rock/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ingress): add TLS configuration for Forgejo Ingress

Browse Source 
- Add explicit tls block with riotpiao-com-tls secret
- Enables HTTPS access to https://forgejo.riotpiao.com
- Matches wildcard certificate (*.riotpiao.com)

The file comment mentioned TLS should be handled via default-ssl-certificate,
but explicit TLS blocks are needed for proper HTTPS routing.
This commit is contained in:
Story Crater Bot
2026-07-23 07:20:51 -07:00
parent a831c4d3db
commit 4e7a7b065e
2 changed files with 4 additions and 459 deletions
Download Patch File Download Diff File Expand all files Collapse all files
GITOPS-REBUILD-PLAN.md
-459
View File
@@ -1,459 +0,0 @@
# GitOps Infrastructure Rebuild Plan
**Goal:** Bootstrap cluster from local source + GitOps-managed future state with zero resource duplication.
---
## 📊 Current vs Proposed Architecture
### Current Issues ❌
1. **Circular dependencies** - CNPG operator, ddb-cluster, Forgejo scattered between bootstrap/, data/, argocd/bootstrap/
2. **Resource duplication** - ddb-cluster.yaml in k8s/data/ deployed by wave 4 app, but should be bootstrap-only
3. **Manual bootstrap steps** - Scattered across BOOTSTRAP.md, requires copying secrets between namespaces
4. **SOPS secrets** - db-role-secrets.enc.yaml excluded from kustomization (out-of-band)
5. **No single source of truth** - Same resources defined in multiple places
### Proposed Architecture ✅
```
┌─────────────────────────────────────────────────────────────┐
│ Phase 0: Local Bootstrap (One-time, k8s/bootstrap-local/) │
│ ─────────────────────────────────────────────────────────── │
│ • ArgoCD + SOPS plugin │
│ • CloudNativePG operator │
│ • ddb-cluster (PostgreSQL 3 replicas) │
│ • Forgejo + dependencies (DB, Redis) │
│ │
│ Script: ./bootstrap.sh (automated) │
└─────────────────────────────────────────────────────────────┘
↓
git push to Forgejo
↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 1: GitOps (100% automated, ArgoCD app-of-apps) │
│ ─────────────────────────────────────────────────────────── │
│ Wave 0: Substrate (cert-manager, nginx, reloader) │
│ Wave 1: Networking (Cilium policies, CoreDNS) │
│ Wave 2: Storage/Observability (MinIO, Longhorn, Prometheus) │
│ Wave 3: Logging (Loki, Grafana, Promtail) │
│ Wave 4: Secrets (ALL *.enc.yaml via SOPS) │
│ Wave 5: IAM (Vault, Authentik, Forgejo runner) │
│ Wave 6: Data Schemas (DB init, per-app databases) │
│ Wave 7: Messaging (Kafka, Redis, SQS) │
│ Wave 8: Applications (Temporal, Portainer, etc.) │
│ │
│ All changes: git commit → push → ArgoCD auto-sync │
└─────────────────────────────────────────────────────────────┘
```
---
## 🗂️ Directory Structure (Single Source of Truth)
```
homelab/
├── bootstrap.sh # Automated bootstrap script
├── GITOPS-REBUILD-PLAN.md # This file
├── DDB-REVIEW.md # DDB configuration review
│
├── k8s/
│ ├── bootstrap-local/ # 🔴 Phase 0: Apply once, never touched by ArgoCD
│ │ ├── kustomization.yaml # Orchestrates 00-04
│ │ ├── 00-namespaces.yaml # All namespaces with PodSecurity labels
│ │ ├── 01-argocd.yaml # ArgoCD ConfigMaps
│ │ ├── 02-cnpg-operator.yaml # CNPG operator Application
│ │ ├── 03-ddb-bootstrap.yaml # PostgreSQL cluster + Forgejo DB + Redis
│ │ └── 04-forgejo.yaml # Forgejo Application (inline values)
│ │
│ ├── argocd/
│ │ ├── root/
│ │ │ ├── kustomization.yaml
│ │ │ └── homelab-root.yaml # 🟢 App-of-apps (Phase 1 entry point)
│ │ ├── projects/
│ │ │ └── homelab-project.yaml
│ │ └── apps/ # Wave-based applications
│ │ ├── 00-substrate.yaml # cert-manager, nginx, reloader
│ │ ├── 01-networking.yaml # RENAMED from 05-networking
│ │ ├── 02-storage-obs.yaml # RENAMED from 10-storage-observability
│ │ ├── 03-logging.yaml # RENAMED from 20-logging
│ │ ├── 04-secrets.yaml # RENAMED from 00-secrets (after logging for Grafana)
│ │ ├── 05-iam.yaml # RENAMED from 30-security
│ │ ├── 06-data.yaml # RENAMED from 40-data (schemas ONLY, not cluster)
│ │ ├── 07-messaging.yaml # RENAMED from 50-messaging
│ │ └── 08-applications.yaml # RENAMED from 60-applications
│ │
│ ├── data/
│ │ ├── cluster/ # 🔴 Bootstrap-only (duplicated in bootstrap-local/)
│ │ │ ├── ddb-cluster.yaml # NOT deployed by ArgoCD
│ │ │ ├── forgejo-database.yaml # NOT deployed by ArgoCD
│ │ │ └── kustomization.yaml # Reference only
│ │ └── schemas/ # 🟢 GitOps-managed (wave 6)
│ │ ├── authentik-database.yaml
│ │ ├── temporal-database.yaml
│ │ ├── temporal-visibility-database.yaml
│ │ ├── schemas.yaml
│ │ ├── db-init-job.yaml
│ │ └── kustomization.yaml
│ │
│ ├── security/
│ │ ├── sops-secrets/
│ │ │ └── (all *.enc.yaml handled by wave 4 SOPS app)
│ │ └── ...
│ │
│ └── [other directories unchanged]
│
└── terraform/ # Talos configs ONLY (no k8s resources)
├── main.tf
├── variables.tf
└── provider.tf
```
---
## 🚀 Execution Plan
### **Step 1: Prepare Local Environment**
```bash
# 1. Ensure prerequisites
export KUBECONFIG=~/.kube/homelab-config
kubectl cluster-info # Verify cluster reachable
# 2. Verify SOPS age key exists
ls ~/.sops/homelab-age.key
# 3. Install ArgoCD CLI (if not present)
brew install argocd # macOS
# or: curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
```
### **Step 2: Backup Current State (Safety)**
```bash
# Export all current resources for rollback
kubectl get applications -n argocd -o yaml > backup-argocd-apps.yaml
kubectl get cluster ddb-cluster -n ddb -o yaml > backup-ddb-cluster.yaml
kubectl get all -n cicd -o yaml > backup-forgejo.yaml
```
### **Step 3: Clean Shutdown (Optional - for fresh rebuild)**
⚠️ **ONLY if doing a complete rebuild**. Skip if incrementally migrating.
```bash
# Delete all ArgoCD Applications (keeps resources, just removes ArgoCD management)
kubectl delete applications --all -n argocd
# Optionally delete the entire cluster and re-provision (nuclear option)
# talosctl reset --nodes <nodes> --graceful --reboot
```
### **Step 4: Run Bootstrap Script**
```bash
# Execute automated bootstrap (creates ArgoCD, CNPG, DDB, Forgejo)
./bootstrap.sh
# Expected output:
# ✅ ArgoCD installed
# ✅ SOPS age secret created
# ✅ CNPG operator ready
# ✅ DDB cluster healthy
# ✅ Forgejo ready at https://forgejo.riotpiao.com
```
### **Step 5: Push to Forgejo**
```bash
# Add Forgejo remote (if not already added)
git remote add forgejo https://forgejo.riotpiao.com/riotpiao.com/homelab.git
# Push entire repo
git push forgejo main
# Verify via Forgejo UI: https://forgejo.riotpiao.com
```
### **Step 6: Deploy App-of-Apps Root**
```bash
# Apply homelab-project + homelab-root
kubectl apply -f k8s/argocd/projects/homelab-project.yaml
kubectl apply -k k8s/argocd/root
# Sync everything (waves 0-8)
argocd app sync homelab-root --prune
# Watch the sync (Ctrl+C to exit)
watch -n 2 'kubectl get applications -n argocd'
```
### **Step 7: Verify All Applications Healthy**
```bash
# Check ArgoCD Applications
argocd app list
# Expected output (all Synced + Healthy):
# NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH
# homelab-root https://kubernetes.default.svc argocd homelab Synced Healthy
# cert-manager https://kubernetes.default.svc cert-manager homelab Synced Healthy
# ingress-nginx https://kubernetes.default.svc ingress-nginx homelab Synced Healthy
# data-schemas https://kubernetes.default.svc ddb homelab Synced Healthy
# temporal https://kubernetes.default.svc temporal homelab Synced Healthy
# ... (all other apps)
# Check all pods
kubectl get pods --all-namespaces | grep -v Running
# (should be empty or only Completed jobs)
```
---
## 🔐 DDB Cluster - ArgoCD Application
**Status:** DDB cluster is **bootstrap-only** (NOT managed by ArgoCD).
**Reason:** Circular dependency - Forgejo (which hosts the git repo) depends on ddb-cluster → ddb-cluster cannot be in the git repo Forgejo hosts.
⚠️ **CRITICAL: Storage HA Status Unknown**
- CLAUDE.md claims single-node storage (cp-1 only) = NO HA
- Longhorn manifests show 3-node config (cp-1, cp-2, cp-3) = HA
- **VERIFY BEFORE PROCEEDING:** See `STORAGE-ARCHITECTURE-CLARIFICATION.md`
- If single-node: Deploy `k8s/infrastructure/longhorn/` first
**Current approach (recommended):**
- ddb-cluster deployed via `k8s/bootstrap-local/03-ddb-bootstrap.yaml`
- Applied once during bootstrap, never touched afterward
- Schema changes (Database CRs) are GitOps-managed via wave 6 `data-schemas` app
**Alternative (if Forgejo is decoupled):**
If you later move Forgejo to an external git host (GitHub, GitLab), you could create:
```yaml
# k8s/argocd/apps/03-database.yaml (NOT currently used)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ddb-cluster
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "3"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
targetRevision: main
path: k8s/data/cluster
destination:
server: https://kubernetes.default.svc
namespace: ddb
syncPolicy:
automated:
prune: false # Safety: never auto-delete database
selfHeal: true
```
**DDB Configuration Review:** See `DDB-REVIEW.md` for:
- Current vs recommended configuration
- Resource limits, connection pooling, backups
- Monitoring and performance tuning
---
## 📋 Resource Mapping (No Duplication)
| Resource | Bootstrap (k8s/bootstrap-local/) | GitOps (ArgoCD apps/) | Terraform |
|----------|----------------------------------|-----------------------|-----------|
| ArgoCD | ✅ 01-argocd.yaml | ❌ Not managed | ❌ |
| CNPG operator | ✅ 02-cnpg-operator.yaml (as Application) | ❌ | ❌ |
| ddb-cluster | ✅ 03-ddb-bootstrap.yaml | ❌ | ❌ |
| forgejo-database | ✅ 03-ddb-bootstrap.yaml | ❌ | ❌ |
| Forgejo | ✅ 04-forgejo.yaml | ❌ (manual sync only) | ❌ |
| Forgejo Redis | ✅ 03-ddb-bootstrap.yaml | ❌ | ❌ |
| cert-manager | ❌ | ✅ Wave 0 (00-substrate) | ❌ |
| ingress-nginx | ❌ | ✅ Wave 0 (00-substrate) | ❌ |
| Prometheus | ❌ | ✅ Wave 2 (02-storage-obs) | ❌ |
| Loki | ❌ | ✅ Wave 3 (03-logging) | ❌ |
| SOPS secrets | ❌ | ✅ Wave 4 (04-secrets) | ❌ |
| Authentik | ❌ | ✅ Wave 5 (05-iam) | ❌ |
| authentik-database | ❌ | ✅ Wave 6 (06-data schemas) | ❌ |
| temporal-database | ❌ | ✅ Wave 6 (06-data schemas) | ❌ |
| Temporal | ❌ | ✅ Wave 8 (08-applications) | ❌ |
| Talos configs | ❌ | ❌ | ✅ terraform/ |
**Rule:** Each resource has exactly ONE source of truth. No overlap.
---
## 🧪 Testing the Rebuild
### **Dry-Run (Safe)**
```bash
# 1. Test bootstrap script without actually applying
./bootstrap.sh --dry-run # (modify script to support this flag)
# 2. Test kustomization builds
kubectl kustomize k8s/bootstrap-local/
kubectl kustomize k8s/data/schemas/
# 3. Test ArgoCD app rendering
argocd app diff homelab-root --local k8s/argocd/apps/
```
### **Incremental Migration (Safer than Full Rebuild)**
If cluster is currently running, migrate incrementally:
```bash
# 1. Add bootstrap-local/ resources alongside existing (won't conflict)
kubectl apply -k k8s/bootstrap-local/ --dry-run=client
# 2. Update ArgoCD apps one wave at a time
kubectl apply -f k8s/argocd/apps/00-substrate.yaml
argocd app sync cert-manager ingress-nginx reloader
# (verify healthy, then proceed to next wave)
# 3. Delete old bootstrap Applications
kubectl delete application cnpg-operator -n argocd # Now in bootstrap-local
kubectl delete application forgejo -n argocd # Now manual-sync-only
```
---
## 🎯 Success Criteria
✅ **Bootstrap completes in <10 minutes** (`./bootstrap.sh`)
✅ **All namespaces pre-created with correct PodSecurity labels**
✅ **ddb-cluster healthy (3/3 replicas)**
✅ **Forgejo accessible at https://forgejo.riotpiao.com**
✅ **ArgoCD app-of-apps sync completes (all waves)**
✅ **All Applications show Synced + Healthy**
✅ **No duplicate resources** (each resource managed by exactly one mechanism)
✅ **SOPS secrets decrypted successfully** (all *.enc.yaml)
✅ **Temporal connects to PostgreSQL** (via temporal-database)
✅ **Authentik connects to PostgreSQL** (via authentik-database)
✅ **Future changes via git push only** (no manual kubectl apply)
---
## 🔄 Day-2 Operations (Post-Bootstrap)
### **Making Changes**
```bash
# 1. Edit manifests locally
vim k8s/applications/temporal/temporal-values.yaml
# 2. Commit + push
git add -A
git commit -m "fix(temporal): increase replicas to 3"
git push forgejo main
# 3. ArgoCD auto-syncs within 3 minutes (or manual)
argocd app sync temporal
```
### **Adding a New Service**
```bash
# 1. Create manifests
mkdir k8s/applications/myapp
cat > k8s/applications/myapp/kustomization.yaml << EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: myapp
resources:
- deployment.yaml
- service.yaml
EOF
# 2. Create ArgoCD Application
cat > k8s/argocd/apps/08-applications.yaml << 'EOF'
# (append to existing file)
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: myapp
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "8"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
targetRevision: main
path: k8s/applications/myapp
destination:
server: https://kubernetes.default.svc
namespace: myapp
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
EOF
# 3. Push
git add -A && git commit -m "feat(myapp): add new application" && git push
# ArgoCD picks it up automatically
```
### **Updating DDB Cluster (Bootstrap Resources)**
Since ddb-cluster is bootstrap-only, changes require manual apply:
```bash
# 1. Edit k8s/bootstrap-local/03-ddb-bootstrap.yaml
vim k8s/bootstrap-local/03-ddb-bootstrap.yaml
# 2. Apply changes (will patch existing cluster)
kubectl apply -f k8s/bootstrap-local/03-ddb-bootstrap.yaml
# 3. Commit to git (for record-keeping)
git add k8s/bootstrap-local/03-ddb-bootstrap.yaml
git commit -m "chore(ddb): increase shared_buffers to 512MB"
git push
```
---
## 📚 Related Documents
- **CLAUDE.md** - Cluster topology, hard rules, GitOps gotchas
- **DDB-REVIEW.md** - PostgreSQL configuration review and recommendations
- **USAGE.md** - (STALE - helmfile era, ignore deployment procedures)
- **TROUBLESHOOTING.md** - Generic Kubernetes debugging methodology
- **k8s/argocd/bootstrap/BOOTSTRAP.md** - (OLD - replaced by this plan)
---
## ❓ FAQ
**Q: Why is ddb-cluster not managed by ArgoCD?**
A: Circular dependency - Forgejo needs DDB, ArgoCD syncs from Forgejo. Breaking the cycle requires one manual bootstrap step.
**Q: Can I move Forgejo to ArgoCD management?**
A: Only if you move the git repo to an external host (GitHub, GitLab). The `04-forgejo.yaml` Application already exists, just change `syncPolicy.automated` to enable it.
**Q: What if bootstrap.sh fails halfway?**
A: Re-run it. All commands are idempotent (`kubectl apply`, `--dry-run=client -o yaml | kubectl apply`).
**Q: How do I rollback a bad GitOps change?**
A: `git revert <commit>` → `git push`. ArgoCD auto-syncs the rollback.
**Q: Why rename wave numbers (00 → 0, 05 → 1, etc.)?**
A: Cleaner sequential naming (0-8). Old helmfile convention used gaps (00, 05, 10, ...) which are unnecessary in pure ArgoCD.
**Q: Can I delete k8s/argocd/bootstrap/ after migration?**
A: Yes, once bootstrap-local/ is working. Keep BOOTSTRAP.md for historical reference if needed.
---
**Last Updated:** 2025-01-XX
**Status:** Ready for implementation
k8s/bootstrap/ingress/ingress.yaml
+4
View File
@@ -209,6 +209,10 @@ metadata:
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
ingressClassName: nginx
tls:
- hosts:
- forgejo.riotpiao.com
secretName: riotpiao-com-tls
rules:
- host: forgejo.riotpiao.com
http: