Commit Graph
73 Commits
Author SHA1 Message Date
Story Crater Bot b1be1d0d2c fix(forgejo-runner): point at in-cluster forgejo Service :3000 not public :443 — runner i/o timeout, forgejo serves 3000 not 443 2026-08-18 15:08:02 -07:00
Story Crater Bot 0a6568491b fix(minio,loki): declare loki-chunks/ruler/admin buckets in minio Tenant — loki failed with NoSuchBucket 2026-08-18 15:08:02 -07:00
Story Crater Bot 63fc502359 fix(loki,vault,iam): loki minio endpoint :80 not :9000, emit vault-minio-creds via CMP, drop redundant broken authentik-migrations job 2026-08-18 15:08:02 -07:00
Story Crater Bot 7ab39280c2 fix(ingress): add homelab-ingress ArgoCD app to apply orphaned ingress.yaml — services had no Ingress object, unreachable via LAN ingress .160 2026-08-18 15:08:02 -07:00
Story Crater Bot 97330d780c feat(terraform): add per-node Cloudflare Tunnel cert SANs to controlplane certSANs — remote talosctl/kubectl over tunnel pass TLS verification
Adds optional cloudflare_talos_sans (machine.certSANs, talos API :50000) and
cloudflare_apiserver_sans (cluster.apiServer.certSANs, kube-apiserver :6443) per
control-plane node. cp-1 gets cp1.homelab + cp1-talos.homelab; cp-2/cp-3 get
their cpN-talos.homelab. Values set in gitignored tfvars.
2026-08-18 15:08:02 -07:00
Story Crater Bot 2b89981c28 chore(ci): add SOPS-encrypted runner-token secret record for forgejo-runner registration 2026-08-18 15:08:02 -07:00
Story Crater Bot 384548b424 fix(storage): add longhorn-wffc SC + pin portainer/forgejo-runner to az-a — fixes PVC attach 2026-08-18 15:08:02 -07:00
Story Crater Bot cda75eeb7b fix(authentik): drop redundant authentik-migrate init container — server entrypoint migrates; old-image manage migrate tripped version-history precheck on empty DB 2026-08-18 15:08:02 -07:00
Story Crater Bot 79118de8d3 feat(data): add CNPG managed roles + Database CRs for authentik/temporal — replaces missing helmfile post-sync user creation
authentik/temporal DB users+databases were never provisioned (old helmfile hook
gone; db-init-job only made schemas in shared app DB). Adds managed.roles
(authentik/temporal login roles, passwords from basic-auth secrets) + Database CRs
(dedicated DBs owned by each role). Role secrets applied out-of-band (SOPS), not in
kustomize resources so data-schemas app doesn't choke on ciphertext.
2026-08-18 15:08:02 -07:00
Story Crater Bot 48f3dd1db9 feat(argocd): wire SOPS CMP sidecar + fix loki/grafana/authentik secret resolution 2026-08-18 15:08:02 -07:00
Story Crater Bot d9ae0a6c44 feat(substrate): deploy cert-manager/ingress-nginx/reloader + privileged PodSecurity for ingress-nginx 2026-08-18 15:08:02 -07:00
Story Crater Bot a6465f7158 fix(minio): correct operator chart source + rewrite Tenant to v5 schema + config.env creds — tenant now boots 2026-08-18 15:08:02 -07:00
Story Crater Bot 2623eecfca feat(argocd): SOPS CMP plugin decryption for Stage 0 secrets (simplify to directory source) 2026-08-18 15:08:02 -07:00
Story Crater Bot dca0e7cb9a feat(cloudflared): wire tunnel token secret and document bootstrap
- Create SOPS-encrypted cloudflared-secrets.enc.yaml with tunnel token
- Add Cloudflare vars to .env.example (CLOUDFLARE_CONNECTOR_TOKEN, ACCOUNT_ID, TUNNEL_ID, API_TOKEN)
- Document Phase 0 cloudflared-token Secret creation in BOOTSTRAP.md (manual step until CMP plugin wires it)
- Note: Cloudflare-side TCP routing (cp1.homelab -> 192.168.1.213:6443, etc.) must be configured manually in Zero Trust dashboard

Tunnel already deployed as ArgoCD Application in k8s/argocd/apps/60-applications.yaml (wave 8); this closes the missing Secret gap and documents the bootstrap path.
2026-08-18 15:08:02 -07:00
Story Crater Bot 1168dc8417 fix(k8s,docs): scale ddb-cluster to single instance, pin minio to storage namespace, document 3-CP topology in USAGE 2026-08-18 15:08:02 -07:00
Story Crater Bot 8f86a03828 refactor(argocd): replace wave/layer/phase schemes with two-phase bootstrap + app-of-apps and document both CD scopes — fixes self-hosted-git chicken-egg and stale paths 2026-08-18 15:08:02 -07:00
Story Crater Bot 5b34e71111 feat(terraform): restructure control planes into a 3-node map with LAN etcd advertise and live machine CA — enables talos-cp-1/2/3 HA and drops worker configs 2026-08-18 15:08:02 -07:00
Story Crater Bot 9956379f5f chore: remove scratch planning docs — not meant for the repo 2026-08-18 15:08:02 -07:00
Story Crater Bot 54bfb5ade6 feat(gitops): migrate domain to riotpiao.com, add CNPG + Forgejo HA on Redis/Postgres, wire ArgoCD apps — enables cluster rebuild after etcd wipe and unblocks the git-source chicken-egg via standalone Helm-source Applications 2026-08-18 15:08:02 -07:00
Story Crater Bot 491e88e493 feat(ci,iac): Consolidate Forgejo CI workflows and add Talos Terraform IaC
Consolidate three separate Forgejo Actions (argocd-sync, security-scan, validate-k8s) into single cluster-ci workflow for cleaner CI/CD pipeline with proper job sequencing and reduced auth overhead.

Add Terraform configuration for Talos cluster machine configs:
- Provider setup for Talos
- Centralized variables for CP and worker configs
- Template-based config generation for controlplane.yaml and worker-*.yaml
- Sensitive data separated in terraform.tfvars (gitignored)
- Local state tracking for infrastructure
2026-08-18 15:08:01 -07:00
Story Crater Bot e48580adb9 fix(ci): Forgejo Actions auth + kustomize cleanup in validate-k8s workflow 2026-08-18 15:08:01 -07:00
Story Crater Bot 517d823f77 feat(minio): Expand CRDs to include Policies and Users — full YAML-driven resource creation
Add MinIO Policies and Users via CRD alongside Buckets.

Resources now declarative:
- Bucket: riotpiao-models (versioning enabled)
- Policy: policy-ollama (scoped bucket access)
- User: user-ollama (service account for Ollama/LLM)

Access keys can be overridden via SOPS or kustomize overlays.
All MinIO resource creation now git-tracked and version controlled.
2026-08-18 15:08:01 -07:00
Story Crater Bot 5c259237d7 feat(data): Add CNPG cluster + database schema initialization
Create production PostgreSQL cluster via CNPG (3-node HA, Longhorn storage).

Schema initialization Job creates schemas for:
- Authentik (identity provider)
- Temporal (workflow engine)
- Vault (secrets management)
- App (generic application databases)

Database layer now captures complete IaC for stateful infrastructure.
Services find ready schemas when deployed.
2026-08-18 15:08:01 -07:00
Story Crater Bot c12fbcf45e feat(minio): Add MinIO Bucket CRD for riotpiao-models — replaces shell script setup 2026-08-18 15:08:01 -07:00
Story Crater Bot 0f30d77288 refactor(k8s): Reorganize into 5-layer structure with production kustomizations 2026-08-18 15:08:01 -07:00
Story Crater Bot 563f720d09 refactor: retire Terraform, migrate to pure ArgoCD GitOps + CI validation 2026-08-18 15:08:01 -07:00
Story Crater Bot adbad3d97e fix(ci): use direct in-cluster Kubernetes auth for CI runner — drop kubeconfig file dependency 2026-08-18 15:08:01 -07:00
Story Crater Bot 3833c0d119 fix: terraform fmt — normalize formatting across all files 2026-08-18 15:08:01 -07:00
Story Crater Bot df9d9845c0 fix(ci): correct core-cli auth + S3 backend config for CI runner (iterate) 2026-08-18 15:08:01 -07:00
Story Crater Bot f16f439feb feat: Terraform CI via Forgejo Actions + MinIO S3 state backend
- ArgoCD manages MinIO (phase 0), Terraform manages infrastructure
- Runner workflow: pulls state from S3, validates, plans, applies
- 34 resources imported to state, S3 backend operational
- Fixed AppProject repos, S3 endpoint deprecation, runner package manager
2026-08-18 15:08:01 -07:00
Story Crater Bot 292146bce4 feat(phase4): ArgoCD-driven Terraform apply via PVC imports (Pod Job approach tried and reverted) 2026-08-18 15:08:01 -07:00
Story Crater Bot 74729f1c59 docs(terraform): add state management script and best practices guide 2026-08-18 15:08:01 -07:00
Story Crater Bot 31a266ee58 chore(phase4): stub helmfile — all releases managed by Terraform + ArgoCD 2026-08-18 15:08:01 -07:00
Story Crater Bot 5e887b4da3 feat(argocd): migrate phase3 (authentik) to ArgoCD, keep temporal on helmfile 2026-08-18 15:08:01 -07:00
Story Crater Bot 0b4a79b5ca fix(argocd): use homelab-ca wildcard TLS instead of --insecure mode 2026-08-18 15:08:01 -07:00
Story Crater Bot 2060d4c493 feat(argocd): migrate phase2 releases (CNPG/Loki/Grafana/Forgejo/Forgejo-Runner) to ArgoCD 2026-08-18 15:08:01 -07:00
Story Crater Bot 3e0a1c86ba feat(argocd): migrate phase1 hookless releases to ArgoCD 2026-08-18 15:08:01 -07:00
Story Crater Bot 8eab299b30 docs(phase1): create migration guide for 9 hookless releases
Detailed Phase 1 workflow:
- Template Application spec (Helm source, values, sync policy)
- Per-release migration pattern (create → test → remove → commit)
- Helmfile ↔ ArgoCD mapping table
- Local chart handling (source.path vs source.chart)
- Verification checklist
- Rollback instructions

Reference: execute one release at a time, verify before next.
2026-08-18 15:08:01 -07:00
Story Crater Bot 08a3fd9bcb feat(phase0): configure ArgoCD SOPS decryption + update encrypted secrets
Phase 0 continuation: enable ArgoCD to decrypt SOPS-encrypted secrets on sync.

1. Update ArgoCD Helm values (terraform/argocd-bootstrap.tf):
   - Add SOPS_AGE_KEY_FILE env var to repoServer
   - Mount sops-age K8s Secret at /home/argocd/.sops
   - Add ConfigManagementPlugin for SOPS (detects *.enc.yaml files)

2. Update encrypted secrets with real values:
   - k8s/base/secrets.enc.yaml: encrypted with actual service credentials
   - All secret values encrypted at rest in git
   - ArgoCD decrypts on sync using K8s Secret + AGE key

Prerequisites:
  - K8s Secret created: kubectl create secret generic sops-age -n argocd --from-file=keys.txt=/Users/rockliang/.sops/key.txt
  - SOPS_AGE_KEY_FILE env var set in ArgoCD repoServer (done above)

Next: Phase 1 — migrate 9 hookless releases to ArgoCD + create Applications that reference encrypted secrets.
2026-08-18 15:08:01 -07:00
Story Crater BotandClaude Haiku 4.5 ee385e59a1 feat(phase0): setup SOPS for encrypted secret management
Phase 0 groundwork for helmfile→ArgoCD migration using SOPS (Secrets Operations):

1. Install SOPS + AGE encryption
   - AGE key generated and stored locally at ~/.sops/key.txt
   - Public key embedded in .sops.yaml for file encryption rules

2. Create K8s Secret for AGE private key
   - kubectl: create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt
   - ArgoCD will use this key to decrypt secrets at sync time

3. Encrypt initial secrets
   - k8s/base/secrets.enc.yaml: AES256_GCM encrypted secrets for all services
   - Placeholder values (will be replaced with real values per environment)
   - Secrets never visible in git (encrypted at rest)

4. Configure SOPS
   - .sops.yaml: creation rules for k8s/*/secrets.enc.yaml files
   - All future secret files auto-encrypt on edit (sops -e)

Setup: Store AGE key as K8s Secret in argocd namespace:
  export KUBECONFIG=cluster-config/kubeconfig
  kubectl create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt

Next: Configure ArgoCD Helm plugin to decrypt secrets on sync (Phase 0 continuation).

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
2026-08-18 15:08:01 -07:00
Story Crater BotandClaude Haiku 4.5 f2b6ad1c60 feat(terraform): import Longhorn StorageClasses and app PVCs to Terraform state
- Phase 1: longhorn, longhorn-kafka StorageClasses (cluster-wide defaults)
- Phase 2 pilot: grafana, loki, portainer, forgejo PVCs
- All imports protected by lifecycle.prevent_destroy
- Removes Helm annotations (meta.helm.sh/*) to prevent dual-ownership conflicts
- Remote state backend (MinIO S3) syncs automatically on plan/apply
- Import-only approach: zero data loss, existing volumes untouched
- See terraform/LONGHORN_PVC_IMPORT.md for execution record

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
2026-08-18 15:08:01 -07:00
Story Crater Bot 3a7471fc9b docs(iac): enforce single source of truth for infrastructure
Add IaC practice section to coding-standards.md:
- All infrastructure state via Terraform or Helm (never ad-hoc scripts)
- Clear division: Terraform owns helm releases/namespaces/storage/state
- Anti-pattern: split bucket definitions across multiple files
- Bootstrap-only exception: document one-time setup with rationale

Rationale: prevents state drift, credential duplication, and unclear ownership.
2026-08-18 15:08:01 -07:00
Story Crater Bot 29d5ba3e6e chore(terraform): clean up TF state git-tracking (gitignore, drop cached state, dep bump) 2026-08-18 15:08:01 -07:00
Story Crater Bot 49eabd0e46 feat(terraform): enable S3 remote state backend (MinIO)
Migrate terraform state from local file to MinIO S3 bucket (terraform-state).
Backend config: https://minio-api.riotpiao.homelab.com (external endpoint).
State now persisted remotely, shared across team, safe for cluster rebuild.

Also added terraform-state bucket to MinIO managed buckets.
2026-08-18 15:08:01 -07:00
Story Crater Bot 22e18c6f33 feat(minio): add loki storage buckets (chunks/ruler/admin/index) 2026-08-18 15:08:01 -07:00
Story Crater Bot acfc9cfc04 feat(authentik): import 24 resources to TF; chore(bootstrap): add cilium to TF
Import all live authentik resources (groups, users, oauth2 providers, applications)
into terraform state via authentik-generated.tf. Provider config in authentik-config.tf.
Resources are drift-free and match live cluster.

Add cilium CNI to bootstrap helm_release.for_each (1.19.5, kube-system).
Cilium was unmanaged (helmfile-only); now IaC-owned. Critical path for
cluster rebuild recovery. Adds cilium repo to helm-repositories.tf.
2026-08-18 15:08:01 -07:00
Story Crater Bot 80e146e106 feat(minio): migrate to official MinIO Operator chart, TF-owned 2026-08-18 15:08:01 -07:00
Story Crater Bot 07b785e5fc Fix: Inject homelab-ca cert into ArgoCD repo-server
- Mount homelab-ca-secret for TLS verification
- Allows repo-server to reach forgejo.riotpiao.homelab.com
- Fixes x509 certificate verification error
2026-08-18 15:08:01 -07:00
Story Crater Bot bb1239ab0d Re-enable cert-manager manifests for TF import
- ClusterIssuers + Certificates now back in TF
- Will import existing live resources
2026-08-18 15:08:01 -07:00
Story Crater Bot 68dc426e53 feat(argocd): phase2 app-of-apps for 19 workloads + AppProject sourceRepos fix 2026-08-18 15:08:01 -07:00