rock/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
286 Commits 3 Branches 0 Tags
225866945a5765ed1379593676d336764a7bda9e
Commit Graph
56 Commits
Author SHA1 Message Date
Story Crater Bot 225866945a fix(agent-pod): remote tui session for multi-agent 2026-08-18 14:23:55 -07:00
Story Crater Bot a2e97e8cd3 fix(cilium): restrict L2 announcement to control-plane nodes — GPU worker lacks eno1 (Mellanox enp28s0f*), so when it won the .160 lease it couldn't ARP the VIP, black-holing all ingress (flapped on reboots) 2026-08-13 07:58:12 -07:00
Story Crater Bot 063f9bcd23 fix(homarr): add AUTH_OIDC_URI + email account linking — homarr hides the Authentik sign-in button unless AUTH_OIDC_URI (authorize endpoint) is set alongside AUTH_OIDC_ISSUER (per authentik/homarr SSO docs); was the missing var 2026-08-13 07:26:59 -07:00
Story Crater Bot 3a91c19b5c feat(monitoring): enable Alertmanager (null receiver, longhorn PVC, az-a) + fix forgejo-rules ns forgejo->cicd — alerting delivery was disabled; forgejo PrometheusRule targeted a nonexistent namespace 2026-08-13 07:10:03 -07:00
Story Crater Bot 09873aa275 fix(coredns): own Corefile+hostname rewrites via Talos inlineManifest (single-source terraform/files/coredns/Corefile), drop ArgoCD coredns-config app — in-cluster *.riotpiao.com now resolves to nginx ingress so MinIO/OIDC discovery works; update cp-2 IP .213->.214 2026-08-12 16:16:44 -07:00
Story Crater Bot 36dfaa4ddd fix(terraform): switch NVIDIA extensions to LTS channel (580.xx) — Tesla V100/Volta is Legacy-tier, production channel (595.xx) silently ignores the GPU 2026-08-11 12:12:26 -07:00
Story Crater Bot 9188be39c6 fix(terraform): correct NVIDIA extension names to nonfree-kmod-nvidia-production/nvidia-container-toolkit-production, add required nvidia kernel modules to worker config 2026-08-11 10:31:22 -07:00
Story Crater Bot d4508afc07 feat(terraform): add gpu-node role label to worker node config, persists across reinstalls 2026-08-10 23:20:49 -07:00
Story Crater Bot 3f5d44d6fa fix(terraform): cap EPHEMERAL volume size to reserve disk space for swap partition on worker nodes 2026-08-10 22:22:20 -07:00
Story Crater Bot 209df7558e fix(terraform): parameterize worker network interface, use nvme diskSelector instead of raw path, add configurable swap partition support 2026-08-10 22:05:53 -07:00
Story Crater Bot d81c57f860 chore(terraform): enable disk wipe on install for all nodes (controlplane and worker) 2026-08-10 21:10:02 -07:00
Story Crater Bot e7b526b1d0 fix(worker): correct interface name to enp28s0f0np0 for proper network routing, add kubernetes CA to worker config 2026-08-10 21:07:00 -07:00
Story Crater Bot 8589c40b44 feat(terraform): add GPU-enabled Talos schematic and worker node template support 2026-08-10 19:44:18 -07:00
Story Crater Bot bf67d2d9de fix: patch bootstrap cluster with correct config 2026-07-25 07:09:45 -07:00
Story Crater Bot 8fda8c50d3 refactor(argocd): remove orphaned infrastructure Applications - bootstrap is source of truth 
REMOVED ORPHANED APPLICATIONS:
  - cnpg-operator (OutOfSync, conflicted with bootstrap)
  - forgejo (OutOfSync, conflicted with bootstrap)
  - ingress-nginx-bootstrap (orphaned, no ownerReferences)

ARCHITECTURE NOW CLEAN:
  ✅ Bootstrap: 7 manifests (infrastructure base for regional deployment)
     - ArgoCD, CNPG operator, DDB, Forgejo, ingress-nginx, namespaces, wait-for-databases
  ✅ ArgoCD: 32 Applications (all services/apps)
  ✅ No duplicate management

DEPLOYMENT FLOW:
  1. kubectl apply -k k8s/bootstrap-local/ (infrastructure)
  2. kubectl apply -k k8s/argocd/root/ (app-of-apps)
  3. ArgoCD auto-syncs from Forgejo (applications)

CLEANUP:
  - Archived old bootstrap configs (k8s/argocd/bootstrap.archived/)
  - Deleted orphaned Applications (ArgoCD tracking only, resources untouched)

Bootstrap remains single source of truth for infrastructure.
ArgoCD manages all applications and services.
 2026-07-23 10:29:10 -07:00
Story Crater Bot 6a2aacc4e6 feat(terraform): add per-node Cloudflare Tunnel cert SANs to controlplane certSANs — remote talosctl/kubectl over tunnel pass TLS verification 
Adds optional cloudflare_talos_sans (machine.certSANs, talos API :50000) and
cloudflare_apiserver_sans (cluster.apiServer.certSANs, kube-apiserver :6443) per
control-plane node. cp-1 gets cp1.homelab + cp1-talos.homelab; cp-2/cp-3 get
their cpN-talos.homelab. Values set in gitignored tfvars.
 2026-07-21 08:02:24 -07:00
Story Crater Bot 15b1ec6ad4 feat(terraform): restructure control planes into a 3-node map with LAN etcd advertise and live machine CA — enables talos-cp-1/2/3 HA and drops worker configs 2026-07-20 08:22:53 -07:00
Story Crater Bot 578a707867 feat(gitops): migrate domain to riotpiao.com, add CNPG + Forgejo HA on Redis/Postgres, wire ArgoCD apps — enables cluster rebuild after etcd wipe and unblocks the git-source chicken-egg via standalone Helm-source Applications 2026-07-19 09:29:17 -07:00
Story Crater Bot c6493f14ae feat(ci,iac): Consolidate Forgejo CI workflows and add Talos Terraform IaC 
Consolidate three separate Forgejo Actions (argocd-sync, security-scan, validate-k8s) into single cluster-ci workflow for cleaner CI/CD pipeline with proper job sequencing and reduced auth overhead.

Add Terraform configuration for Talos cluster machine configs:
- Provider setup for Talos
- Centralized variables for CP and worker configs
- Template-based config generation for controlplane.yaml and worker-*.yaml
- Sensitive data separated in terraform.tfvars (gitignored)
- Local state tracking for infrastructure
 2026-07-17 23:44:08 -07:00
Story Crater Bot a860de94da refactor: remove terraform entirely, migrate to pure GitOps (ArgoCD) 
Delete entire terraform/ directory.

Architecture: Terraform + ArgoCD → ArgoCD only
- Single tool: ArgoCD manages all infrastructure and applications
- Source of truth: git only (k8s/ directory)
- Continuous reconciliation: no manual apply needed
- Simpler state: no tfstate backend, no state files

Next: Migrate all Terraform resources to k8s/ YAML manifests
and ArgoCD Applications (namespaces, storage classes, Helm releases,
RBAC, network policies, Authentik config).
 2026-07-16 11:04:52 -07:00
Story Crater Bot 94a2bd648c fix: remove kubeconfig references, use try() for pod runtime files 
- main.tf: remove kubeconfig_path local (no longer used with direct auth)
- providers.tf: wrap file() with try() to handle plan-time on non-pod systems

try() allows terraform plan to work locally; at runtime in pod, files exist and are used.
 2026-07-15 21:36:05 -07:00
Story Crater Bot be2e5c321f fix: use direct in-cluster kubernetes auth instead of kubeconfig file 
- providers.tf: use host + token + ca_crt from mounted service account secrets
- workflow: remove kubeconfig generation step (no longer needed)
- variables.tf: remove unused kubeconfig_path variable

This is the standard pattern for running terraform inside k8s pods.
 2026-07-15 21:26:09 -07:00
Story Crater Bot 9d1d79b774 fix: kubeconfig path default for runner container — use /tmp/kubeconfig not local path 2026-07-15 21:03:11 -07:00
Story Crater Bot 72ab6b6973 fix: terraform fmt — normalize formatting across all files 2026-07-15 21:01:55 -07:00
Story Crater Bot 7114fc8fc9 fix:Update the home lab repo 2026-07-15 20:40:02 -07:00
Story Crater Bot fe7b749951 fix: terraform init backend config — use endpoint with inline credentials 2026-07-15 19:18:17 -07:00
Story Crater Bot 694350634b fix: use endpoints.s3 for S3 backend (endpoint deprecated in TF 1.8+) 2026-07-15 19:16:13 -07:00
Story Crater Bot ab103f00f0 fix: core-cli OAuth2 + S3 backend + admin group 
- Link core-cli app to OAuth2 provider (was hardcoded to 0)
- Add core-cli user to authentik_admins for CI access
- Fix terraform init: use 'endpoint' not 'endpoints.s3' for S3 backend
  (Terraform 1.9.4 compatibility, matches state.tf config)
 2026-07-15 19:09:59 -07:00
Story Crater Bot e4d645eae9 feat: Terraform CI via Forgejo Actions + MinIO S3 state backend 
- ArgoCD manages MinIO (phase 0), Terraform manages infrastructure
- Runner workflow: pulls state from S3, validates, plans, applies
- 34 resources imported to state, S3 backend operational
- Fixed AppProject repos, S3 endpoint deprecation, runner package manager
 2026-07-15 18:48:32 -07:00
Story Crater Bot e71c7ad37e feat(phase4): ArgoCD-driven Terraform apply via PostSync Hook Job 
- Create Phase 4 ArgoCD Application (terraform-apply)
- PostSync Hook Job runs: terraform init && terraform apply -auto-approve
- ServiceAccount + ClusterRole for cluster-admin RBAC
- S3 credentials encrypted with SOPS (terraform-s3-secrets.enc.yaml)
- Pre-commit hook blocks local 'terraform apply' — all changes via git push
- True IaC: modify terraform/*.tf → git push → ArgoCD applies automatically
 2026-07-15 16:39:27 -07:00
Story Crater Bot 69d2240cf4 fix(argocd): use homelab-ca wildcard TLS instead of --insecure mode 2026-07-15 16:21:31 -07:00
Story Crater Bot 5144ab732d feat(phase0): configure ArgoCD SOPS decryption + update encrypted secrets 
Phase 0 continuation: enable ArgoCD to decrypt SOPS-encrypted secrets on sync.

1. Update ArgoCD Helm values (terraform/argocd-bootstrap.tf):
   - Add SOPS_AGE_KEY_FILE env var to repoServer
   - Mount sops-age K8s Secret at /home/argocd/.sops
   - Add ConfigManagementPlugin for SOPS (detects *.enc.yaml files)

2. Update encrypted secrets with real values:
   - k8s/base/secrets.enc.yaml: encrypted with actual service credentials
   - All secret values encrypted at rest in git
   - ArgoCD decrypts on sync using K8s Secret + AGE key

Prerequisites:
  - K8s Secret created: kubectl create secret generic sops-age -n argocd --from-file=keys.txt=/Users/rockliang/.sops/key.txt
  - SOPS_AGE_KEY_FILE env var set in ArgoCD repoServer (done above)

Next: Phase 1 — migrate 9 hookless releases to ArgoCD + create Applications that reference encrypted secrets.
 2026-07-15 15:04:26 -07:00
Story Crater Bot d2f4b3c7e4 Revert "feat(phase0): bootstrap External Secrets Operator and fix helmfile dual-ownership" 
This reverts commit e7f3409d0f.
 2026-07-15 14:59:54 -07:00
Story Crater BotandClaude Haiku 4.5 e7f3409d0f feat(phase0): bootstrap External Secrets Operator and fix helmfile dual-ownership 
Phase 0 groundwork for helmfile→ArgoCD migration:

1. Remove 3 bootstrap releases from helmfile (cert-manager, reloader, ingress-nginx)
   — already managed by terraform/bootstrap-releases.tf; eliminates dual-ownership

2. Bootstrap ESO (External Secrets Operator) as TF-managed release
   — required for all ExternalSecret resources in phases 1-3
   — added to bootstrap-releases.tf + helm-repositories.tf

3. Create ClusterSecretStore connecting ESO to Vault (K8s auth)
   — enables per-namespace/per-release secret injection
   — vault config documented in docs/PHASE0-ESO-VAULT-SETUP.md (manual setup)

4. Fix argocd-bootstrap.tf CA cert copy: use jq instead of sed for cleaner metadata handling

Changes:
- helmfile.yaml.gotmpl: remove cert-manager/reloader/ingress-nginx blocks
- terraform/bootstrap-releases.tf: add external-secrets release
- terraform/helm-repositories.tf: add external-secrets Helm repo
- k8s/external-secrets/clustersecretstore.yaml: ESO→Vault ClusterSecretStore
- k8s/argocd/apps/0-wave-0.yaml: stub wave 0 applications (schema fix, rewrite pending Phase 1)
- docs/PHASE0-ESO-VAULT-SETUP.md: manual ESO-Vault auth setup procedure

Next: Phase 1 will incrementally rewrite ArgoCD Applications + migrate helmfile releases.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
 2026-07-15 14:53:16 -07:00
Story Crater BotandClaude Haiku 4.5 23ec31bd6d feat(terraform): import Longhorn StorageClasses and app PVCs to Terraform state 
- Phase 1: longhorn, longhorn-kafka StorageClasses (cluster-wide defaults)
- Phase 2 pilot: grafana, loki, portainer, forgejo PVCs
- All imports protected by lifecycle.prevent_destroy
- Removes Helm annotations (meta.helm.sh/*) to prevent dual-ownership conflicts
- Remote state backend (MinIO S3) syncs automatically on plan/apply
- Import-only approach: zero data loss, existing volumes untouched
- See terraform/LONGHORN_PVC_IMPORT.md for execution record

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
 2026-07-15 12:22:35 -07:00
Story Crater Bot bd00bca9bf chore: remove terraform cache from git tracking 2026-07-14 23:34:01 -07:00
Story Crater Bot 13557184c6 chore: update terraform dependencies and config 
terraform.lock.hcl updated with provider versions (goauthentik 2024.12.1).
Regenerated from current provider blocks.
 2026-07-14 23:33:16 -07:00
Story Crater Bot 6d554961c2 feat(terraform): enable S3 remote state backend (MinIO) 
Migrate terraform state from local file to MinIO S3 bucket (terraform-state).
Backend config: https://minio-api.riotpiao.homelab.com (external endpoint).
State now persisted remotely, shared across team, safe for cluster rebuild.

Also added terraform-state bucket to MinIO managed buckets.
 2026-07-14 23:30:45 -07:00
Story Crater Bot 52cb895cda feat(minio): add loki storage buckets (chunks/ruler/admin/index) 
Move loki bucket creation from helmfile post-hook to TF-managed buckets array.
Now all MinIO buckets (6 total) declared in terraform/minio.tf for IaC completeness.
 2026-07-14 17:00:09 -07:00
Story Crater Bot bca247a763 feat(authentik): import 24 resources to TF; chore(bootstrap): add cilium to TF 
Import all live authentik resources (groups, users, oauth2 providers, applications)
into terraform state via authentik-generated.tf. Provider config in authentik-config.tf.
Resources are drift-free and match live cluster.

Add cilium CNI to bootstrap helm_release.for_each (1.19.5, kube-system).
Cilium was unmanaged (helmfile-only); now IaC-owned. Critical path for
cluster rebuild recovery. Adds cilium repo to helm-repositories.tf.
 2026-07-14 16:36:12 -07:00
Story Crater Bot 9e3781a069 fix(minio): migrate to official chart, TF-owned 
Bitnami wiped Docker Hub catalog (bitnami/minio: 0 tags), chart 14.1.0
dead on ImagePullBackOff. Move to minio/minio 5.4.0 (quay.io) as one TF
helm_release. Add longhorn-xfs SC: default SC ext4 mkfs on 100Gi exceeds
kubelet mount timeout, xfs near-instant. Drop minio ArgoCD Apps (TF owns
now, kills dual-controller conflict). Fix double base64 on OIDC secret.
 2026-07-14 16:07:03 -07:00
Story Crater Bot ebeb4948d4 Add: minio-operator TF management (v4.5.8 downgrade) - WIP due to helm conflicts 2026-07-14 15:04:50 -07:00
Story Crater Bot ed7be6f229 TF: Add minio-operator Helm repo to ArgoCD config + AppProject sourceRepos 2026-07-14 14:32:45 -07:00
Story Crater Bot ca8525c625 Add minio-operator Application to deploy operator before Tenant 2026-07-14 14:26:01 -07:00
Story Crater Bot 8a3a892cbd Fix: Inject homelab-ca cert into ArgoCD repo-server 
- Mount homelab-ca-secret for TLS verification
- Allows repo-server to reach forgejo.riotpiao.homelab.com
- Fixes x509 certificate verification error
 2026-07-14 14:01:14 -07:00
Story Crater Bot cdacdd8d11 Re-enable cert-manager manifests for TF import 
- ClusterIssuers + Certificates now back in TF
- Will import existing live resources
 2026-07-14 13:55:08 -07:00
Story Crater Bot 263a48a22d Fix: ArgoCD AppProject sourceRepos for correct forgejo URL 
- Changed from forgejo.forge.riotpiao.homelab.com/rock/* to forgejo.riotpiao.homelab.com/riotpiao.com/*
- Allows homelab root app to access workload app manifests
 2026-07-14 13:52:53 -07:00
Story Crater Bot edd4ea7fe4 Fix: set ingress-nginx to privileged PodSecurity level 
- privileged level allows hostPort (80/443) required for nginx
- Other namespaces remain at baseline for security
- Cleaner than exempting namespace entirely
 2026-07-14 13:43:56 -07:00
Story Crater Bot 086ad9f9a9 Fix: exempt ingress-nginx from PodSecurity policy 
- restricted policy forbids hostPort (80/443) — broke nginx
- Remove pod-security labels from ingress-nginx namespace entirely
- Other namespaces remain at baseline level
 2026-07-14 13:41:36 -07:00
Story Crater Bot 3eabb847fd Re-add ingress-nginx to TF bootstrap (PodSecurity policy fixed) 
- ingress-nginx now has restricted policy level (allows hostPort)
- Previous timeout was due to policy blocking pod deployment
- Re-importing helm release to TF management
 2026-07-14 13:37:22 -07:00