rock/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
133 Commits 3 Branches 0 Tags
b2191509fb4ebf29c3a868c40994a45b1453fc9e
Commit Graph
85 Commits
Author SHA1 Message Date
Story Crater Bot e71c7ad37e feat(phase4): ArgoCD-driven Terraform apply via PostSync Hook Job 
- Create Phase 4 ArgoCD Application (terraform-apply)
- PostSync Hook Job runs: terraform init && terraform apply -auto-approve
- ServiceAccount + ClusterRole for cluster-admin RBAC
- S3 credentials encrypted with SOPS (terraform-s3-secrets.enc.yaml)
- Pre-commit hook blocks local 'terraform apply' — all changes via git push
- True IaC: modify terraform/*.tf → git push → ArgoCD applies automatically
 2026-07-15 16:39:27 -07:00
Story Crater Bot ff22027c7a refactor(argocd): phase3 reduced to authentik only (remove vault, temporal, ollama, story-crater) 2026-07-15 16:24:45 -07:00
Story Crater Bot f158512261 feat(argocd): create phase3 Applications (authentik, vault, temporal, ollama, story-crater) with SOPS secrets and Hook Jobs 2026-07-15 16:23:30 -07:00
Story Crater Bot 7d1eb09486 feat(argocd): add phase2 Hook Jobs (CNPG, Prometheus, Forgejo-Runner) and update Applications to multi-source 2026-07-15 16:05:34 -07:00
Story Crater Bot 0dddf15dc8 feat(argocd): add SOPS-encrypted secrets for phase2 releases (loki, grafana, forgejo) 2026-07-15 16:02:20 -07:00
Story Crater Bot 37ee3dc5b1 feat(argocd): create phase2 Applications (prometheus, cloudnative-pg, loki, grafana, forgejo, forgejo-runner) 2026-07-15 15:12:36 -07:00
Story Crater Bot f864e3dc51 feat(argocd): remove claude-terminal and blackbox-exporter from phase1 2026-07-15 15:11:30 -07:00
Story Crater Bot 04100232ef feat(phase1): create ArgoCD Applications for 9 hookless releases 
Created Applications for Phase 1 migration (no presync/postsync hooks):
- strimzi-operator (strimzi/strimzi-kafka-operator v0.46.0)
- kmsvc-redis (bitnami/redis v20.6.0)
- kafka-cluster (local chart k8s/sqs/charts/kafka-cluster)
- queue-crd (local chart k8s/sqs/charts/queue-crd)
- management-service (local chart k8s/sqs/charts/management-service)
- promtail (grafana/promtail)
- blackbox-exporter (prometheus-community/prometheus-blackbox-exporter ~11)
- portainer (portainer/portainer)
- claude-terminal (local chart k8s/dev-tools)

Organized by sync wave: 0 (bootstrap), 1 (messaging/observability), 3 (dashboards/tools).
All configured with auto-sync, CreateNamespace, prune, selfHeal.

Next: Remove corresponding release blocks from helmfile.yaml.gotmpl per migration guide
(one release at a time, verify helmfile diff is clean).

Applications applied to cluster; awaiting helmfile cleanup to finalize migration.
 2026-07-15 15:06:02 -07:00
Story Crater Bot 5144ab732d feat(phase0): configure ArgoCD SOPS decryption + update encrypted secrets 
Phase 0 continuation: enable ArgoCD to decrypt SOPS-encrypted secrets on sync.

1. Update ArgoCD Helm values (terraform/argocd-bootstrap.tf):
   - Add SOPS_AGE_KEY_FILE env var to repoServer
   - Mount sops-age K8s Secret at /home/argocd/.sops
   - Add ConfigManagementPlugin for SOPS (detects *.enc.yaml files)

2. Update encrypted secrets with real values:
   - k8s/base/secrets.enc.yaml: encrypted with actual service credentials
   - All secret values encrypted at rest in git
   - ArgoCD decrypts on sync using K8s Secret + AGE key

Prerequisites:
  - K8s Secret created: kubectl create secret generic sops-age -n argocd --from-file=keys.txt=/Users/rockliang/.sops/key.txt
  - SOPS_AGE_KEY_FILE env var set in ArgoCD repoServer (done above)

Next: Phase 1 — migrate 9 hookless releases to ArgoCD + create Applications that reference encrypted secrets.
 2026-07-15 15:04:26 -07:00
Story Crater BotandClaude Haiku 4.5 4379f3cb21 feat(phase0): setup SOPS for encrypted secret management 
Phase 0 groundwork for helmfile→ArgoCD migration using SOPS (Secrets Operations):

1. Install SOPS + AGE encryption
   - AGE key generated and stored locally at ~/.sops/key.txt
   - Public key embedded in .sops.yaml for file encryption rules

2. Create K8s Secret for AGE private key
   - kubectl: create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt
   - ArgoCD will use this key to decrypt secrets at sync time

3. Encrypt initial secrets
   - k8s/base/secrets.enc.yaml: AES256_GCM encrypted secrets for all services
   - Placeholder values (will be replaced with real values per environment)
   - Secrets never visible in git (encrypted at rest)

4. Configure SOPS
   - .sops.yaml: creation rules for k8s/*/secrets.enc.yaml files
   - All future secret files auto-encrypt on edit (sops -e)

Setup: Store AGE key as K8s Secret in argocd namespace:
  export KUBECONFIG=cluster-config/kubeconfig
  kubectl create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt

Next: Configure ArgoCD Helm plugin to decrypt secrets on sync (Phase 0 continuation).

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
 2026-07-15 15:00:06 -07:00
Story Crater Bot d2f4b3c7e4 Revert "feat(phase0): bootstrap External Secrets Operator and fix helmfile dual-ownership" 
This reverts commit e7f3409d0f.
 2026-07-15 14:59:54 -07:00
Story Crater BotandClaude Haiku 4.5 e7f3409d0f feat(phase0): bootstrap External Secrets Operator and fix helmfile dual-ownership 
Phase 0 groundwork for helmfile→ArgoCD migration:

1. Remove 3 bootstrap releases from helmfile (cert-manager, reloader, ingress-nginx)
   — already managed by terraform/bootstrap-releases.tf; eliminates dual-ownership

2. Bootstrap ESO (External Secrets Operator) as TF-managed release
   — required for all ExternalSecret resources in phases 1-3
   — added to bootstrap-releases.tf + helm-repositories.tf

3. Create ClusterSecretStore connecting ESO to Vault (K8s auth)
   — enables per-namespace/per-release secret injection
   — vault config documented in docs/PHASE0-ESO-VAULT-SETUP.md (manual setup)

4. Fix argocd-bootstrap.tf CA cert copy: use jq instead of sed for cleaner metadata handling

Changes:
- helmfile.yaml.gotmpl: remove cert-manager/reloader/ingress-nginx blocks
- terraform/bootstrap-releases.tf: add external-secrets release
- terraform/helm-repositories.tf: add external-secrets Helm repo
- k8s/external-secrets/clustersecretstore.yaml: ESO→Vault ClusterSecretStore
- k8s/argocd/apps/0-wave-0.yaml: stub wave 0 applications (schema fix, rewrite pending Phase 1)
- docs/PHASE0-ESO-VAULT-SETUP.md: manual ESO-Vault auth setup procedure

Next: Phase 1 will incrementally rewrite ArgoCD Applications + migrate helmfile releases.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
 2026-07-15 14:53:16 -07:00
Story Crater Bot 3eccf9f653 test(argocd): add label to vault app to verify GitOps flow 
Add test-gitops=true label to vault Application to demonstrate end-to-end
GitOps sync: commit push → ArgoCD detects change → applies label to live app.
Tests that root-app watches k8s/argocd/apps/ and propagates changes.
 2026-07-14 23:24:12 -07:00
Story Crater Bot 9e3781a069 fix(minio): migrate to official chart, TF-owned 
Bitnami wiped Docker Hub catalog (bitnami/minio: 0 tags), chart 14.1.0
dead on ImagePullBackOff. Move to minio/minio 5.4.0 (quay.io) as one TF
helm_release. Add longhorn-xfs SC: default SC ext4 mkfs on 100Gi exceeds
kubelet mount timeout, xfs near-instant. Drop minio ArgoCD Apps (TF owns
now, kills dual-controller conflict). Fix double base64 on OIDC secret.
 2026-07-14 16:07:03 -07:00
Story Crater Bot a3190abe50 Revert: Use minio-creds secret with MINIO_ROOT_* keys (operator expected format) 2026-07-14 14:37:41 -07:00
Story Crater Bot dddc7a524a Fix: Tenant credentials secret reference from minio-creds to minio 2026-07-14 14:34:49 -07:00
Story Crater Bot 418ab7bfc2 Fix: minio-operator uses official MinIO Operator Helm chart 2026-07-14 14:26:25 -07:00
Story Crater Bot ca8525c625 Add minio-operator Application to deploy operator before Tenant 2026-07-14 14:26:01 -07:00
Story Crater Bot 47c0301a43 Step 2: ArgoCD app-of-apps manifests for 19 workloads 
Wave 0: minio, strimzi-operator, kmsvc-redis, prometheus
Wave 1: vault, loki, cloudnative-pg, authentik (manual-sync), temporal, kafka-cluster
Wave 2: queue-crd, management-service, grafana, promtail, forgejo
Wave 3: forgejo-runner, portainer

All auto-sync except authentik (manual-sync only for IAM safety)
 2026-07-14 13:52:18 -07:00
Story Crater Bot dd608d3231 Step 1 complete: Bootstrap layer with ArgoCD, cert-manager, namespaces imported to TF 
- ArgoCD migrated to argocd namespace
- Cert-manager issuers/certs created
- 20 namespaces imported with pod-security labels
- S3 backend temporarily offline (MinIO), using local backup
- Pending: Remove metadata drift from helm releases, re-apply
 2026-07-14 13:14:46 -07:00
Story Crater Bot 4d32e2765b feat: three-tier log aggregation for Loki 
Critical services (iam/monitoring/temporal/cicd) keep 100% logs.
Others get 50% sampling + selective drops (health/debug noise).
Balances log volume (40-50% reduction) with error visibility.
 2026-07-13 17:52:59 -07:00
Story Crater Bot 0d4e98d88b feat: track full CoreDNS Deployment manifest, add topologySpreadConstraints 
CoreDNS is Talos-bootstrapped and previously untracked except for its
ConfigMap. Pull the full live spec into one file as the single source of
truth, add topologySpreadConstraints so the 2 replicas don't land on the
same node. ScheduleAnyway (not DoNotSchedule) to avoid blocking scheduling
if a node is briefly unavailable.
 2026-07-13 16:51:49 -07:00
Story Crater Bot ebf97f573e feat: spread management-service pods across nodes via topologySpreadConstraints 
3-9 replicas (HPA) previously relied on implicit scheduler spreading.
ScheduleAnyway (not DoNotSchedule) so pods still get scheduled if a node
is briefly unavailable, just less evenly.
 2026-07-13 16:43:58 -07:00
Story Crater Bot b33fdde5b7 fix: temporal service config - add explicit ClusterIP services for history/matching 2026-07-13 13:20:50 -07:00
Story Crater Bot cd1c5691e8 feat: point SQS charts to public GHCR image 
Forgejo registry unreachable from worker nodes (network isolation +
host-to-ClusterIP routing gaps). Both management-service and queue-operator
now ship from the same public GHCR image, with queue-operator selected via
command override.
 2026-07-13 10:31:22 -07:00
Story Crater Bot e6d7626fb6 remove: strip all oauth2-proxy deployments 
- Delete oauth2-proxy helm releases from helmfile (temporal, kmsvc, longhorn, portainer)
- Remove oauth2-proxy manifests and ingress redirects
- Add direct ingress for kmsvc management service
- Update temporal/portainer/longhorn ingress comments to reflect direct service exposure

Services now accessible without oauth2-proxy layer.
 2026-07-11 19:36:31 -07:00
Story Crater Bot e1d0cfd70b k8s/aux: add cert-manager longhorn dashboard forge dev-tools and shadowsocks 
- cert-manager ClusterIssuers (LetsEncrypt + homelab-ca)
- Longhorn storage dashboard
- Portainer dashboard config
- Forgejo git service
- Claude terminal remote access
- Shadowsocks tunnel for remote access
 2026-07-11 19:19:59 -07:00
Story Crater Bot 18f2f94f8e k8s/cilium: add lb-ipam pool configuration 
- Cilium LB-IPAM pool (192.168.1.160-192.168.1.170)
- Fixed IP assignment for LoadBalancer services
 2026-07-11 19:19:44 -07:00
Story Crater Bot 6d5a0ba205 k8s/services: add ingress networking portainer llm and project guides 
- Nginx ingress + TLS termination (homelab-ca)
- Portainer container UI
- CoreDNS internal DNS rewrites
- DuckDNS DDNS updater
- Ollama LLM inference
- 8 project-usage guides (team reference)
 2026-07-11 19:17:54 -07:00
Story Crater Bot 1c02e2b831 k8s/messaging: add kafka kmsvc and temporal workflows 
- Kafka 3-broker cluster (RF=3, min-ISR=2)
- kmsvc SQS-like API on Kafka
- Redis dedup (standalone, can extend to HA)
- Temporal workflow orchestration (Cassandra backend)
 2026-07-11 19:17:42 -07:00
Story Crater Bot 4ab596196e k8s/ci-cd: add forgejo gitops and argocd deployment 
- Forgejo git forge + OCI registry
- Argo CD pull-based GitOps
- Private CA TLS (self-signed 10-year cert)
- Machine credentials scoped to repositories
 2026-07-11 19:17:34 -07:00
Story Crater Bot 63d7256b9e k8s/monitoring: add prometheus grafana loki observability 
- Loki log aggregation (MinIO backed, 10-day retention)
- Promtail daemonset (pod + talos journal logs)
- Prometheus + kube-state-metrics
- Grafana dashboards (6-row template per service)
 2026-07-11 19:17:28 -07:00
Story Crater Bot 674c8f0d66 k8s/iam: add cloudnativepg postgres and vault + authentik 
- PostgreSQL 3-replica HA with pgvector
- Vault S3 storage backend (MinIO)
- Authentik federated OIDC provider
- Vault auto-unseal via postStart hook
 2026-07-11 19:17:22 -07:00
Story Crater Bot 36aea89e47 k8s/storage: add minio s3 with 3-way replication and oidc 
- MinIO 3-node site replication (az-a/b/c)
- S3 backend for Loki chunks (10-day retention)
- OIDC integration with Authentik
- envFrom for secret injection
 2026-07-11 19:16:56 -07:00
Story Crater Bot 11c26f3f29 k8s: add base namespace and pod disruption budgets 
- Namespace setup script with PSP/RBAC
- PodDisruptionBudgets for all services (zero-downtime drain)
 2026-07-11 19:16:50 -07:00