Commit Graph
41 Commits
Author SHA1 Message Date
Story Crater Bot 6a2aacc4e6 feat(terraform): add per-node Cloudflare Tunnel cert SANs to controlplane certSANs — remote talosctl/kubectl over tunnel pass TLS verification
Adds optional cloudflare_talos_sans (machine.certSANs, talos API :50000) and
cloudflare_apiserver_sans (cluster.apiServer.certSANs, kube-apiserver :6443) per
control-plane node. cp-1 gets cp1.homelab + cp1-talos.homelab; cp-2/cp-3 get
their cpN-talos.homelab. Values set in gitignored tfvars.
2026-07-21 08:02:24 -07:00
Story Crater Bot 15b1ec6ad4 feat(terraform): restructure control planes into a 3-node map with LAN etcd advertise and live machine CA — enables talos-cp-1/2/3 HA and drops worker configs 2026-07-20 08:22:53 -07:00
Story Crater Bot 578a707867 feat(gitops): migrate domain to riotpiao.com, add CNPG + Forgejo HA on Redis/Postgres, wire ArgoCD apps — enables cluster rebuild after etcd wipe and unblocks the git-source chicken-egg via standalone Helm-source Applications 2026-07-19 09:29:17 -07:00
Story Crater Bot c6493f14ae feat(ci,iac): Consolidate Forgejo CI workflows and add Talos Terraform IaC
Consolidate three separate Forgejo Actions (argocd-sync, security-scan, validate-k8s) into single cluster-ci workflow for cleaner CI/CD pipeline with proper job sequencing and reduced auth overhead.

Add Terraform configuration for Talos cluster machine configs:
- Provider setup for Talos
- Centralized variables for CP and worker configs
- Template-based config generation for controlplane.yaml and worker-*.yaml
- Sensitive data separated in terraform.tfvars (gitignored)
- Local state tracking for infrastructure
2026-07-17 23:44:08 -07:00
Story Crater Bot a860de94da refactor: remove terraform entirely, migrate to pure GitOps (ArgoCD)
Delete entire terraform/ directory.

Architecture: Terraform + ArgoCD → ArgoCD only
- Single tool: ArgoCD manages all infrastructure and applications
- Source of truth: git only (k8s/ directory)
- Continuous reconciliation: no manual apply needed
- Simpler state: no tfstate backend, no state files

Next: Migrate all Terraform resources to k8s/ YAML manifests
and ArgoCD Applications (namespaces, storage classes, Helm releases,
RBAC, network policies, Authentik config).
2026-07-16 11:04:52 -07:00
Story Crater Bot 94a2bd648c fix: remove kubeconfig references, use try() for pod runtime files
- main.tf: remove kubeconfig_path local (no longer used with direct auth)
- providers.tf: wrap file() with try() to handle plan-time on non-pod systems

try() allows terraform plan to work locally; at runtime in pod, files exist and are used.
2026-07-15 21:36:05 -07:00
Story Crater Bot be2e5c321f fix: use direct in-cluster kubernetes auth instead of kubeconfig file
- providers.tf: use host + token + ca_crt from mounted service account secrets
- workflow: remove kubeconfig generation step (no longer needed)
- variables.tf: remove unused kubeconfig_path variable

This is the standard pattern for running terraform inside k8s pods.
2026-07-15 21:26:09 -07:00
Story Crater Bot 9d1d79b774 fix: kubeconfig path default for runner container — use /tmp/kubeconfig not local path 2026-07-15 21:03:11 -07:00
Story Crater Bot 72ab6b6973 fix: terraform fmt — normalize formatting across all files 2026-07-15 21:01:55 -07:00
Story Crater Bot 7114fc8fc9 fix:Update the home lab repo 2026-07-15 20:40:02 -07:00
Story Crater Bot fe7b749951 fix: terraform init backend config — use endpoint with inline credentials 2026-07-15 19:18:17 -07:00
Story Crater Bot 694350634b fix: use endpoints.s3 for S3 backend (endpoint deprecated in TF 1.8+) 2026-07-15 19:16:13 -07:00
Story Crater Bot ab103f00f0 fix: core-cli OAuth2 + S3 backend + admin group
- Link core-cli app to OAuth2 provider (was hardcoded to 0)
- Add core-cli user to authentik_admins for CI access
- Fix terraform init: use 'endpoint' not 'endpoints.s3' for S3 backend
  (Terraform 1.9.4 compatibility, matches state.tf config)
2026-07-15 19:09:59 -07:00
Story Crater Bot e4d645eae9 feat: Terraform CI via Forgejo Actions + MinIO S3 state backend
- ArgoCD manages MinIO (phase 0), Terraform manages infrastructure
- Runner workflow: pulls state from S3, validates, plans, applies
- 34 resources imported to state, S3 backend operational
- Fixed AppProject repos, S3 endpoint deprecation, runner package manager
2026-07-15 18:48:32 -07:00
Story Crater Bot e71c7ad37e feat(phase4): ArgoCD-driven Terraform apply via PostSync Hook Job
- Create Phase 4 ArgoCD Application (terraform-apply)
- PostSync Hook Job runs: terraform init && terraform apply -auto-approve
- ServiceAccount + ClusterRole for cluster-admin RBAC
- S3 credentials encrypted with SOPS (terraform-s3-secrets.enc.yaml)
- Pre-commit hook blocks local 'terraform apply' — all changes via git push
- True IaC: modify terraform/*.tf → git push → ArgoCD applies automatically
2026-07-15 16:39:27 -07:00
Story Crater Bot 69d2240cf4 fix(argocd): use homelab-ca wildcard TLS instead of --insecure mode 2026-07-15 16:21:31 -07:00
Story Crater Bot 5144ab732d feat(phase0): configure ArgoCD SOPS decryption + update encrypted secrets
Phase 0 continuation: enable ArgoCD to decrypt SOPS-encrypted secrets on sync.

1. Update ArgoCD Helm values (terraform/argocd-bootstrap.tf):
   - Add SOPS_AGE_KEY_FILE env var to repoServer
   - Mount sops-age K8s Secret at /home/argocd/.sops
   - Add ConfigManagementPlugin for SOPS (detects *.enc.yaml files)

2. Update encrypted secrets with real values:
   - k8s/base/secrets.enc.yaml: encrypted with actual service credentials
   - All secret values encrypted at rest in git
   - ArgoCD decrypts on sync using K8s Secret + AGE key

Prerequisites:
  - K8s Secret created: kubectl create secret generic sops-age -n argocd --from-file=keys.txt=/Users/rockliang/.sops/key.txt
  - SOPS_AGE_KEY_FILE env var set in ArgoCD repoServer (done above)

Next: Phase 1 — migrate 9 hookless releases to ArgoCD + create Applications that reference encrypted secrets.
2026-07-15 15:04:26 -07:00
Story Crater Bot d2f4b3c7e4 Revert "feat(phase0): bootstrap External Secrets Operator and fix helmfile dual-ownership"
This reverts commit e7f3409d0f.
2026-07-15 14:59:54 -07:00
Story Crater BotandClaude Haiku 4.5 e7f3409d0f feat(phase0): bootstrap External Secrets Operator and fix helmfile dual-ownership
Phase 0 groundwork for helmfile→ArgoCD migration:

1. Remove 3 bootstrap releases from helmfile (cert-manager, reloader, ingress-nginx)
   — already managed by terraform/bootstrap-releases.tf; eliminates dual-ownership

2. Bootstrap ESO (External Secrets Operator) as TF-managed release
   — required for all ExternalSecret resources in phases 1-3
   — added to bootstrap-releases.tf + helm-repositories.tf

3. Create ClusterSecretStore connecting ESO to Vault (K8s auth)
   — enables per-namespace/per-release secret injection
   — vault config documented in docs/PHASE0-ESO-VAULT-SETUP.md (manual setup)

4. Fix argocd-bootstrap.tf CA cert copy: use jq instead of sed for cleaner metadata handling

Changes:
- helmfile.yaml.gotmpl: remove cert-manager/reloader/ingress-nginx blocks
- terraform/bootstrap-releases.tf: add external-secrets release
- terraform/helm-repositories.tf: add external-secrets Helm repo
- k8s/external-secrets/clustersecretstore.yaml: ESO→Vault ClusterSecretStore
- k8s/argocd/apps/0-wave-0.yaml: stub wave 0 applications (schema fix, rewrite pending Phase 1)
- docs/PHASE0-ESO-VAULT-SETUP.md: manual ESO-Vault auth setup procedure

Next: Phase 1 will incrementally rewrite ArgoCD Applications + migrate helmfile releases.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
2026-07-15 14:53:16 -07:00
Story Crater BotandClaude Haiku 4.5 23ec31bd6d feat(terraform): import Longhorn StorageClasses and app PVCs to Terraform state
- Phase 1: longhorn, longhorn-kafka StorageClasses (cluster-wide defaults)
- Phase 2 pilot: grafana, loki, portainer, forgejo PVCs
- All imports protected by lifecycle.prevent_destroy
- Removes Helm annotations (meta.helm.sh/*) to prevent dual-ownership conflicts
- Remote state backend (MinIO S3) syncs automatically on plan/apply
- Import-only approach: zero data loss, existing volumes untouched
- See terraform/LONGHORN_PVC_IMPORT.md for execution record

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
2026-07-15 12:22:35 -07:00
Story Crater Bot bd00bca9bf chore: remove terraform cache from git tracking 2026-07-14 23:34:01 -07:00
Story Crater Bot 13557184c6 chore: update terraform dependencies and config
terraform.lock.hcl updated with provider versions (goauthentik 2024.12.1).
Regenerated from current provider blocks.
2026-07-14 23:33:16 -07:00
Story Crater Bot 6d554961c2 feat(terraform): enable S3 remote state backend (MinIO)
Migrate terraform state from local file to MinIO S3 bucket (terraform-state).
Backend config: https://minio-api.riotpiao.homelab.com (external endpoint).
State now persisted remotely, shared across team, safe for cluster rebuild.

Also added terraform-state bucket to MinIO managed buckets.
2026-07-14 23:30:45 -07:00
Story Crater Bot 52cb895cda feat(minio): add loki storage buckets (chunks/ruler/admin/index)
Move loki bucket creation from helmfile post-hook to TF-managed buckets array.
Now all MinIO buckets (6 total) declared in terraform/minio.tf for IaC completeness.
2026-07-14 17:00:09 -07:00
Story Crater Bot bca247a763 feat(authentik): import 24 resources to TF; chore(bootstrap): add cilium to TF
Import all live authentik resources (groups, users, oauth2 providers, applications)
into terraform state via authentik-generated.tf. Provider config in authentik-config.tf.
Resources are drift-free and match live cluster.

Add cilium CNI to bootstrap helm_release.for_each (1.19.5, kube-system).
Cilium was unmanaged (helmfile-only); now IaC-owned. Critical path for
cluster rebuild recovery. Adds cilium repo to helm-repositories.tf.
2026-07-14 16:36:12 -07:00
Story Crater Bot 9e3781a069 fix(minio): migrate to official chart, TF-owned
Bitnami wiped Docker Hub catalog (bitnami/minio: 0 tags), chart 14.1.0
dead on ImagePullBackOff. Move to minio/minio 5.4.0 (quay.io) as one TF
helm_release. Add longhorn-xfs SC: default SC ext4 mkfs on 100Gi exceeds
kubelet mount timeout, xfs near-instant. Drop minio ArgoCD Apps (TF owns
now, kills dual-controller conflict). Fix double base64 on OIDC secret.
2026-07-14 16:07:03 -07:00
Story Crater Bot ebeb4948d4 Add: minio-operator TF management (v4.5.8 downgrade) - WIP due to helm conflicts 2026-07-14 15:04:50 -07:00
Story Crater Bot ed7be6f229 TF: Add minio-operator Helm repo to ArgoCD config + AppProject sourceRepos 2026-07-14 14:32:45 -07:00
Story Crater Bot ca8525c625 Add minio-operator Application to deploy operator before Tenant 2026-07-14 14:26:01 -07:00
Story Crater Bot 8a3a892cbd Fix: Inject homelab-ca cert into ArgoCD repo-server
- Mount homelab-ca-secret for TLS verification
- Allows repo-server to reach forgejo.riotpiao.homelab.com
- Fixes x509 certificate verification error
2026-07-14 14:01:14 -07:00
Story Crater Bot cdacdd8d11 Re-enable cert-manager manifests for TF import
- ClusterIssuers + Certificates now back in TF
- Will import existing live resources
2026-07-14 13:55:08 -07:00
Story Crater Bot 263a48a22d Fix: ArgoCD AppProject sourceRepos for correct forgejo URL
- Changed from forgejo.forge.riotpiao.homelab.com/rock/* to forgejo.riotpiao.homelab.com/riotpiao.com/*
- Allows homelab root app to access workload app manifests
2026-07-14 13:52:53 -07:00
Story Crater Bot edd4ea7fe4 Fix: set ingress-nginx to privileged PodSecurity level
- privileged level allows hostPort (80/443) required for nginx
- Other namespaces remain at baseline for security
- Cleaner than exempting namespace entirely
2026-07-14 13:43:56 -07:00
Story Crater Bot 086ad9f9a9 Fix: exempt ingress-nginx from PodSecurity policy
- restricted policy forbids hostPort (80/443) — broke nginx
- Remove pod-security labels from ingress-nginx namespace entirely
- Other namespaces remain at baseline level
2026-07-14 13:41:36 -07:00
Story Crater Bot 3eabb847fd Re-add ingress-nginx to TF bootstrap (PodSecurity policy fixed)
- ingress-nginx now has restricted policy level (allows hostPort)
- Previous timeout was due to policy blocking pod deployment
- Re-importing helm release to TF management
2026-07-14 13:37:22 -07:00
Story Crater Bot 6109477bf9 Skip TF management of ingress-nginx (helm timeout issues)
- ingress-nginx already deployed and working in cluster
- Helm updates timeout repeatedly (5+ min with context deadline exceeded)
- Remove from bootstrap releases; manage separately via helm/kubectl
- cert-manager + reloader continue via TF
2026-07-14 13:32:40 -07:00
Story Crater Bot 20c634fd03 Fix: ingress-nginx PodSecurity policy enforcement level
- ingress-nginx requires hostPort (80/443) which is forbidden at baseline level
- Change to restricted enforcement level to allow hostPort
- Other namespaces remain at baseline for security
2026-07-14 13:31:16 -07:00
Story Crater Bot 074b43e1f2 Temp: disable kubernetes_manifest cert-manager resources (already live)
- Will import separately after helm issues resolved
- Avoids re-create conflicts during bootstrap apply
2026-07-14 13:26:26 -07:00
Story Crater Bot c2e084c7c2 Fix: downgrade ArgoCD to 7.3.3, ignore helm metadata drift
- ArgoCD 7.9.1 -> 7.3.3 (match live cluster)
- Ignore helm release metadata in lifecycle rules
- Prevents unnecessary upgrade attempts
2026-07-14 13:16:55 -07:00
Story Crater Bot dd608d3231 Step 1 complete: Bootstrap layer with ArgoCD, cert-manager, namespaces imported to TF
- ArgoCD migrated to argocd namespace
- Cert-manager issuers/certs created
- 20 namespaces imported with pod-security labels
- S3 backend temporarily offline (MinIO), using local backup
- Pending: Remove metadata drift from helm releases, re-apply
2026-07-14 13:14:46 -07:00
Story Crater Bot 9a4d486b86 feat: Terraform foundation for cluster & app bootstrap
Phase 1 infrastructure-as-code setup:
- Core providers (kubernetes, helm, null)
- 15 Helm repositories (grafana, minio, prometheus, etc.)
- Namespace scaffolding (15 namespaces with pod-security labels)
- Storage classes (longhorn, longhorn-kafka with prevent_destroy)
- TLS certificate bootstrap (selfsigned, CA, wildcard cert)
- Remote state backend config (local for now, S3/GCS TODO)
- Variable definitions for all secrets/OIDC clients

Tested: terraform plan passes with no changes (bootstrap infrastructure ready)
Next: Create 25 helm_release resources (Phase 2-4)

Kept helmfile intact; network/Cilium managed via helmfile (no config risk)
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
2026-07-14 09:27:24 -07:00