rock/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
168 Commits 3 Branches 0 Tags
616660cebeeed100c043ac2243452be0bf457c67
Commit Graph
168 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Story Crater Bot ab76e40d05 revert(phase4): Remove Pod Job approach for Terraform apply 
Reverting Phase 4 Pod Job implementation in favor of CI runner (Forgejo Actions).

Deleted:
- k8s/argocd/apps/phase4-terraform-0.yaml
- k8s/hooks/phase4/ (terraform-apply-hook.yaml, terraform-rbac.yaml, terraform-s3-secrets.enc.yaml)

Reason: Pod Job approach had limitations (eviction, timeouts, pod security policies).
Next: Implement Forgejo Actions CI workflow for terraform apply.
 2026-07-15 18:07:54 -07:00
Story Crater Bot b568c015e2 feat(phase4): ArgoCD-driven Terraform with PVC imports 
Phase 4 implementation (true IaC):
- ArgoCD Application: terraform-apply (PostSync Hook Job)
- Hook Job runs: terraform init && terraform apply -auto-approve
- ServiceAccount + ClusterRole for cluster-admin
- SOPS-encrypted S3 credentials (terraform-s3-secrets.enc.yaml)
- Pre-commit hook blocks local 'terraform apply'
- In-cluster kubeconfig for Kubernetes provider
- AWS credentials file with minio profile

State imports:
- Imported kubernetes_persistent_volume_claim.portainer (dashboard/portainer)
- Imported kubernetes_persistent_volume_claim.grafana (logging/grafana)
- Imported kubernetes_persistent_volume_claim.loki (logging/storage-loki-0)

Workflow:
1. Edit terraform/*.tf files
2. git push to main
3. ArgoCD detects changes in k8s/hooks/phase4
4. Hook Job automatically runs terraform apply
5. No manual 'terraform apply' needed ever again
 2026-07-15 17:48:18 -07:00
Story Crater Bot e71c7ad37e feat(phase4): ArgoCD-driven Terraform apply via PostSync Hook Job 
- Create Phase 4 ArgoCD Application (terraform-apply)
- PostSync Hook Job runs: terraform init && terraform apply -auto-approve
- ServiceAccount + ClusterRole for cluster-admin RBAC
- S3 credentials encrypted with SOPS (terraform-s3-secrets.enc.yaml)
- Pre-commit hook blocks local 'terraform apply' — all changes via git push
- True IaC: modify terraform/*.tf → git push → ArgoCD applies automatically
 2026-07-15 16:39:27 -07:00
Story Crater Bot 842360288d docs(terraform): add state management script and best practices guide 2026-07-15 16:31:55 -07:00
Story Crater Bot f18f96eb5b chore(phase4): stub helmfile — all releases managed by Terraform + ArgoCD 2026-07-15 16:27:13 -07:00
Story Crater Bot d655726eca feat(helmfile): remove phase3 releases (authentik, vault, story-crater, ollama) — ArgoCD-managed. Keep temporal 2026-07-15 16:26:30 -07:00
Story Crater Bot ff22027c7a refactor(argocd): phase3 reduced to authentik only (remove vault, temporal, ollama, story-crater) 2026-07-15 16:24:45 -07:00
Story Crater Bot f158512261 feat(argocd): create phase3 Applications (authentik, vault, temporal, ollama, story-crater) with SOPS secrets and Hook Jobs 2026-07-15 16:23:30 -07:00
Story Crater Bot 69d2240cf4 fix(argocd): use homelab-ca wildcard TLS instead of --insecure mode 2026-07-15 16:21:31 -07:00
Story Crater Bot 543105bf46 feat(helmfile): remove phase2 releases (cloudnative-pg, loki, grafana, prometheus, forgejo, forgejo-runner) — ArgoCD-managed 2026-07-15 16:07:42 -07:00
Story Crater Bot 7d1eb09486 feat(argocd): add phase2 Hook Jobs (CNPG, Prometheus, Forgejo-Runner) and update Applications to multi-source 2026-07-15 16:05:34 -07:00
Story Crater Bot 0dddf15dc8 feat(argocd): add SOPS-encrypted secrets for phase2 releases (loki, grafana, forgejo) 2026-07-15 16:02:20 -07:00
Story Crater Bot 37ee3dc5b1 feat(argocd): create phase2 Applications (prometheus, cloudnative-pg, loki, grafana, forgejo, forgejo-runner) 2026-07-15 15:12:36 -07:00
Story Crater Bot f864e3dc51 feat(argocd): remove claude-terminal and blackbox-exporter from phase1 2026-07-15 15:11:30 -07:00
Story Crater Bot 96e40916e3 feat(phase1): remove 9 hookless releases from helmfile — now ArgoCD-managed 
Removed releases (all now managed via ArgoCD Applications):
- strimzi-operator
- kafka-cluster
- kmsvc-redis
- queue-crd
- management-service
- promtail
- blackbox-exporter
- portainer
- claude-terminal

Helmfile now contains only hook-heavy releases (phases 2-3) + argocd (self-referential, never migrates).

Next: Verify helmfile diff is clean, then confirm all 9 apps Synced/Healthy in ArgoCD.
 2026-07-15 15:07:39 -07:00
Story Crater Bot 04100232ef feat(phase1): create ArgoCD Applications for 9 hookless releases 
Created Applications for Phase 1 migration (no presync/postsync hooks):
- strimzi-operator (strimzi/strimzi-kafka-operator v0.46.0)
- kmsvc-redis (bitnami/redis v20.6.0)
- kafka-cluster (local chart k8s/sqs/charts/kafka-cluster)
- queue-crd (local chart k8s/sqs/charts/queue-crd)
- management-service (local chart k8s/sqs/charts/management-service)
- promtail (grafana/promtail)
- blackbox-exporter (prometheus-community/prometheus-blackbox-exporter ~11)
- portainer (portainer/portainer)
- claude-terminal (local chart k8s/dev-tools)

Organized by sync wave: 0 (bootstrap), 1 (messaging/observability), 3 (dashboards/tools).
All configured with auto-sync, CreateNamespace, prune, selfHeal.

Next: Remove corresponding release blocks from helmfile.yaml.gotmpl per migration guide
(one release at a time, verify helmfile diff is clean).

Applications applied to cluster; awaiting helmfile cleanup to finalize migration.
 2026-07-15 15:06:02 -07:00
Story Crater Bot 09980aa41d docs(phase1): create migration guide for 9 hookless releases 
Detailed Phase 1 workflow:
- Template Application spec (Helm source, values, sync policy)
- Per-release migration pattern (create → test → remove → commit)
- Helmfile ↔ ArgoCD mapping table
- Local chart handling (source.path vs source.chart)
- Verification checklist
- Rollback instructions

Reference: execute one release at a time, verify before next.
 2026-07-15 15:05:12 -07:00
Story Crater Bot 5144ab732d feat(phase0): configure ArgoCD SOPS decryption + update encrypted secrets 
Phase 0 continuation: enable ArgoCD to decrypt SOPS-encrypted secrets on sync.

1. Update ArgoCD Helm values (terraform/argocd-bootstrap.tf):
   - Add SOPS_AGE_KEY_FILE env var to repoServer
   - Mount sops-age K8s Secret at /home/argocd/.sops
   - Add ConfigManagementPlugin for SOPS (detects *.enc.yaml files)

2. Update encrypted secrets with real values:
   - k8s/base/secrets.enc.yaml: encrypted with actual service credentials
   - All secret values encrypted at rest in git
   - ArgoCD decrypts on sync using K8s Secret + AGE key

Prerequisites:
  - K8s Secret created: kubectl create secret generic sops-age -n argocd --from-file=keys.txt=/Users/rockliang/.sops/key.txt
  - SOPS_AGE_KEY_FILE env var set in ArgoCD repoServer (done above)

Next: Phase 1 — migrate 9 hookless releases to ArgoCD + create Applications that reference encrypted secrets.
 2026-07-15 15:04:26 -07:00
Story Crater BotandClaude Haiku 4.5 4379f3cb21 feat(phase0): setup SOPS for encrypted secret management 
Phase 0 groundwork for helmfile→ArgoCD migration using SOPS (Secrets Operations):

1. Install SOPS + AGE encryption
   - AGE key generated and stored locally at ~/.sops/key.txt
   - Public key embedded in .sops.yaml for file encryption rules

2. Create K8s Secret for AGE private key
   - kubectl: create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt
   - ArgoCD will use this key to decrypt secrets at sync time

3. Encrypt initial secrets
   - k8s/base/secrets.enc.yaml: AES256_GCM encrypted secrets for all services
   - Placeholder values (will be replaced with real values per environment)
   - Secrets never visible in git (encrypted at rest)

4. Configure SOPS
   - .sops.yaml: creation rules for k8s/*/secrets.enc.yaml files
   - All future secret files auto-encrypt on edit (sops -e)

Setup: Store AGE key as K8s Secret in argocd namespace:
  export KUBECONFIG=cluster-config/kubeconfig
  kubectl create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt

Next: Configure ArgoCD Helm plugin to decrypt secrets on sync (Phase 0 continuation).

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
 2026-07-15 15:00:06 -07:00
Story Crater Bot d2f4b3c7e4 Revert "feat(phase0): bootstrap External Secrets Operator and fix helmfile dual-ownership" 
This reverts commit e7f3409d0f.
 2026-07-15 14:59:54 -07:00
Story Crater BotandClaude Haiku 4.5 e7f3409d0f feat(phase0): bootstrap External Secrets Operator and fix helmfile dual-ownership 
Phase 0 groundwork for helmfile→ArgoCD migration:

1. Remove 3 bootstrap releases from helmfile (cert-manager, reloader, ingress-nginx)
   — already managed by terraform/bootstrap-releases.tf; eliminates dual-ownership

2. Bootstrap ESO (External Secrets Operator) as TF-managed release
   — required for all ExternalSecret resources in phases 1-3
   — added to bootstrap-releases.tf + helm-repositories.tf

3. Create ClusterSecretStore connecting ESO to Vault (K8s auth)
   — enables per-namespace/per-release secret injection
   — vault config documented in docs/PHASE0-ESO-VAULT-SETUP.md (manual setup)

4. Fix argocd-bootstrap.tf CA cert copy: use jq instead of sed for cleaner metadata handling

Changes:
- helmfile.yaml.gotmpl: remove cert-manager/reloader/ingress-nginx blocks
- terraform/bootstrap-releases.tf: add external-secrets release
- terraform/helm-repositories.tf: add external-secrets Helm repo
- k8s/external-secrets/clustersecretstore.yaml: ESO→Vault ClusterSecretStore
- k8s/argocd/apps/0-wave-0.yaml: stub wave 0 applications (schema fix, rewrite pending Phase 1)
- docs/PHASE0-ESO-VAULT-SETUP.md: manual ESO-Vault auth setup procedure

Next: Phase 1 will incrementally rewrite ArgoCD Applications + migrate helmfile releases.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
 2026-07-15 14:53:16 -07:00
Story Crater BotandClaude Haiku 4.5 23ec31bd6d feat(terraform): import Longhorn StorageClasses and app PVCs to Terraform state 
- Phase 1: longhorn, longhorn-kafka StorageClasses (cluster-wide defaults)
- Phase 2 pilot: grafana, loki, portainer, forgejo PVCs
- All imports protected by lifecycle.prevent_destroy
- Removes Helm annotations (meta.helm.sh/*) to prevent dual-ownership conflicts
- Remote state backend (MinIO S3) syncs automatically on plan/apply
- Import-only approach: zero data loss, existing volumes untouched
- See terraform/LONGHORN_PVC_IMPORT.md for execution record

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
 2026-07-15 12:22:35 -07:00
Story Crater Bot 421086f845 docs(iac): enforce single source of truth for infrastructure 
Add IaC practice section to coding-standards.md:
- All infrastructure state via Terraform or Helm (never ad-hoc scripts)
- Clear division: Terraform owns helm releases/namespaces/storage/state
- Anti-pattern: split bucket definitions across multiple files
- Bootstrap-only exception: document one-time setup with rationale

Rationale: prevents state drift, credential duplication, and unclear ownership.
 2026-07-14 23:37:54 -07:00
Story Crater Bot bd00bca9bf chore: remove terraform cache from git tracking 2026-07-14 23:34:01 -07:00
Story Crater Bot 397edf632f fix: correct gitignore patterns for terraform state and cache 
Remove malformed line and clarify rules:
- terraform/.terraform/ (local provider cache)
- terraform/*.tfstate* (local state backups)
- skills-lock.json (lock file)

All TF state now remote (MinIO S3), local files safe to exclude.
 2026-07-14 23:33:50 -07:00
Story Crater Bot 13557184c6 chore: update terraform dependencies and config 
terraform.lock.hcl updated with provider versions (goauthentik 2024.12.1).
Regenerated from current provider blocks.
 2026-07-14 23:33:16 -07:00
Story Crater Bot 6d554961c2 feat(terraform): enable S3 remote state backend (MinIO) 
Migrate terraform state from local file to MinIO S3 bucket (terraform-state).
Backend config: https://minio-api.riotpiao.homelab.com (external endpoint).
State now persisted remotely, shared across team, safe for cluster rebuild.

Also added terraform-state bucket to MinIO managed buckets.
 2026-07-14 23:30:45 -07:00
Story Crater Bot 3eccf9f653 test(argocd): add label to vault app to verify GitOps flow 
Add test-gitops=true label to vault Application to demonstrate end-to-end
GitOps sync: commit push → ArgoCD detects change → applies label to live app.
Tests that root-app watches k8s/argocd/apps/ and propagates changes.
 2026-07-14 23:24:12 -07:00
Story Crater Bot 52cb895cda feat(minio): add loki storage buckets (chunks/ruler/admin/index) 
Move loki bucket creation from helmfile post-hook to TF-managed buckets array.
Now all MinIO buckets (6 total) declared in terraform/minio.tf for IaC completeness.
 2026-07-14 17:00:09 -07:00
Story Crater Bot bca247a763 feat(authentik): import 24 resources to TF; chore(bootstrap): add cilium to TF 
Import all live authentik resources (groups, users, oauth2 providers, applications)
into terraform state via authentik-generated.tf. Provider config in authentik-config.tf.
Resources are drift-free and match live cluster.

Add cilium CNI to bootstrap helm_release.for_each (1.19.5, kube-system).
Cilium was unmanaged (helmfile-only); now IaC-owned. Critical path for
cluster rebuild recovery. Adds cilium repo to helm-repositories.tf.
 2026-07-14 16:36:12 -07:00
Story Crater Bot 9e3781a069 fix(minio): migrate to official chart, TF-owned 
Bitnami wiped Docker Hub catalog (bitnami/minio: 0 tags), chart 14.1.0
dead on ImagePullBackOff. Move to minio/minio 5.4.0 (quay.io) as one TF
helm_release. Add longhorn-xfs SC: default SC ext4 mkfs on 100Gi exceeds
kubelet mount timeout, xfs near-instant. Drop minio ArgoCD Apps (TF owns
now, kills dual-controller conflict). Fix double base64 on OIDC secret.
 2026-07-14 16:07:03 -07:00
Story Crater Bot ebeb4948d4 Add: minio-operator TF management (v4.5.8 downgrade) - WIP due to helm conflicts 2026-07-14 15:04:50 -07:00
Story Crater Bot a3190abe50 Revert: Use minio-creds secret with MINIO_ROOT_* keys (operator expected format) 2026-07-14 14:37:41 -07:00
Story Crater Bot dddc7a524a Fix: Tenant credentials secret reference from minio-creds to minio 2026-07-14 14:34:49 -07:00
Story Crater Bot ed7be6f229 TF: Add minio-operator Helm repo to ArgoCD config + AppProject sourceRepos 2026-07-14 14:32:45 -07:00
Story Crater Bot 418ab7bfc2 Fix: minio-operator uses official MinIO Operator Helm chart 2026-07-14 14:26:25 -07:00
Story Crater Bot ca8525c625 Add minio-operator Application to deploy operator before Tenant 2026-07-14 14:26:01 -07:00
Story Crater Bot 8a3a892cbd Fix: Inject homelab-ca cert into ArgoCD repo-server 
- Mount homelab-ca-secret for TLS verification
- Allows repo-server to reach forgejo.riotpiao.homelab.com
- Fixes x509 certificate verification error
 2026-07-14 14:01:14 -07:00
Story Crater Bot cdacdd8d11 Re-enable cert-manager manifests for TF import 
- ClusterIssuers + Certificates now back in TF
- Will import existing live resources
 2026-07-14 13:55:08 -07:00
Story Crater Bot 263a48a22d Fix: ArgoCD AppProject sourceRepos for correct forgejo URL 
- Changed from forgejo.forge.riotpiao.homelab.com/rock/* to forgejo.riotpiao.homelab.com/riotpiao.com/*
- Allows homelab root app to access workload app manifests
 2026-07-14 13:52:53 -07:00
Story Crater Bot 47c0301a43 Step 2: ArgoCD app-of-apps manifests for 19 workloads 
Wave 0: minio, strimzi-operator, kmsvc-redis, prometheus
Wave 1: vault, loki, cloudnative-pg, authentik (manual-sync), temporal, kafka-cluster
Wave 2: queue-crd, management-service, grafana, promtail, forgejo
Wave 3: forgejo-runner, portainer

All auto-sync except authentik (manual-sync only for IAM safety)
 2026-07-14 13:52:18 -07:00
Story Crater Bot edd4ea7fe4 Fix: set ingress-nginx to privileged PodSecurity level 
- privileged level allows hostPort (80/443) required for nginx
- Other namespaces remain at baseline for security
- Cleaner than exempting namespace entirely
 2026-07-14 13:43:56 -07:00
Story Crater Bot 086ad9f9a9 Fix: exempt ingress-nginx from PodSecurity policy 
- restricted policy forbids hostPort (80/443) — broke nginx
- Remove pod-security labels from ingress-nginx namespace entirely
- Other namespaces remain at baseline level
 2026-07-14 13:41:36 -07:00
Story Crater Bot 3eabb847fd Re-add ingress-nginx to TF bootstrap (PodSecurity policy fixed) 
- ingress-nginx now has restricted policy level (allows hostPort)
- Previous timeout was due to policy blocking pod deployment
- Re-importing helm release to TF management
 2026-07-14 13:37:22 -07:00
Story Crater Bot 6109477bf9 Skip TF management of ingress-nginx (helm timeout issues) 
- ingress-nginx already deployed and working in cluster
- Helm updates timeout repeatedly (5+ min with context deadline exceeded)
- Remove from bootstrap releases; manage separately via helm/kubectl
- cert-manager + reloader continue via TF
 2026-07-14 13:32:40 -07:00
Story Crater Bot 20c634fd03 Fix: ingress-nginx PodSecurity policy enforcement level 
- ingress-nginx requires hostPort (80/443) which is forbidden at baseline level
- Change to restricted enforcement level to allow hostPort
- Other namespaces remain at baseline for security
 2026-07-14 13:31:16 -07:00
Story Crater Bot 074b43e1f2 Temp: disable kubernetes_manifest cert-manager resources (already live) 
- Will import separately after helm issues resolved
- Avoids re-create conflicts during bootstrap apply
 2026-07-14 13:26:26 -07:00
Story Crater Bot c2e084c7c2 Fix: downgrade ArgoCD to 7.3.3, ignore helm metadata drift 
- ArgoCD 7.9.1 -> 7.3.3 (match live cluster)
- Ignore helm release metadata in lifecycle rules
- Prevents unnecessary upgrade attempts
 2026-07-14 13:16:55 -07:00
Story Crater Bot dd608d3231 Step 1 complete: Bootstrap layer with ArgoCD, cert-manager, namespaces imported to TF 
- ArgoCD migrated to argocd namespace
- Cert-manager issuers/certs created
- 20 namespaces imported with pod-security labels
- S3 backend temporarily offline (MinIO), using local backup
- Pending: Remove metadata drift from helm releases, re-apply
 2026-07-14 13:14:46 -07:00
Story Crater Bot 9a4d486b86 feat: Terraform foundation for cluster & app bootstrap 
Phase 1 infrastructure-as-code setup:
- Core providers (kubernetes, helm, null)
- 15 Helm repositories (grafana, minio, prometheus, etc.)
- Namespace scaffolding (15 namespaces with pod-security labels)
- Storage classes (longhorn, longhorn-kafka with prevent_destroy)
- TLS certificate bootstrap (selfsigned, CA, wildcard cert)
- Remote state backend config (local for now, S3/GCS TODO)
- Variable definitions for all secrets/OIDC clients

Tested: terraform plan passes with no changes (bootstrap infrastructure ready)
Next: Create 25 helm_release resources (Phase 2-4)

Kept helmfile intact; network/Cilium managed via helmfile (no config risk)
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
 2026-07-14 09:27:24 -07:00