Phase 0 continuation: enable ArgoCD to decrypt SOPS-encrypted secrets on sync.
1. Update ArgoCD Helm values (terraform/argocd-bootstrap.tf):
- Add SOPS_AGE_KEY_FILE env var to repoServer
- Mount sops-age K8s Secret at /home/argocd/.sops
- Add ConfigManagementPlugin for SOPS (detects *.enc.yaml files)
2. Update encrypted secrets with real values:
- k8s/base/secrets.enc.yaml: encrypted with actual service credentials
- All secret values encrypted at rest in git
- ArgoCD decrypts on sync using K8s Secret + AGE key
Prerequisites:
- K8s Secret created: kubectl create secret generic sops-age -n argocd --from-file=keys.txt=/Users/rockliang/.sops/key.txt
- SOPS_AGE_KEY_FILE env var set in ArgoCD repoServer (done above)
Next: Phase 1 — migrate 9 hookless releases to ArgoCD + create Applications that reference encrypted secrets.
Phase 0 groundwork for helmfile→ArgoCD migration using SOPS (Secrets Operations):
1. Install SOPS + AGE encryption
- AGE key generated and stored locally at ~/.sops/key.txt
- Public key embedded in .sops.yaml for file encryption rules
2. Create K8s Secret for AGE private key
- kubectl: create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt
- ArgoCD will use this key to decrypt secrets at sync time
3. Encrypt initial secrets
- k8s/base/secrets.enc.yaml: AES256_GCM encrypted secrets for all services
- Placeholder values (will be replaced with real values per environment)
- Secrets never visible in git (encrypted at rest)
4. Configure SOPS
- .sops.yaml: creation rules for k8s/*/secrets.enc.yaml files
- All future secret files auto-encrypt on edit (sops -e)
Setup: Store AGE key as K8s Secret in argocd namespace:
export KUBECONFIG=cluster-config/kubeconfig
kubectl create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt
Next: Configure ArgoCD Helm plugin to decrypt secrets on sync (Phase 0 continuation).
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
Add IaC practice section to coding-standards.md:
- All infrastructure state via Terraform or Helm (never ad-hoc scripts)
- Clear division: Terraform owns helm releases/namespaces/storage/state
- Anti-pattern: split bucket definitions across multiple files
- Bootstrap-only exception: document one-time setup with rationale
Rationale: prevents state drift, credential duplication, and unclear ownership.
Remove malformed line and clarify rules:
- terraform/.terraform/ (local provider cache)
- terraform/*.tfstate* (local state backups)
- skills-lock.json (lock file)
All TF state now remote (MinIO S3), local files safe to exclude.
Migrate terraform state from local file to MinIO S3 bucket (terraform-state).
Backend config: https://minio-api.riotpiao.homelab.com (external endpoint).
State now persisted remotely, shared across team, safe for cluster rebuild.
Also added terraform-state bucket to MinIO managed buckets.
Move loki bucket creation from helmfile post-hook to TF-managed buckets array.
Now all MinIO buckets (6 total) declared in terraform/minio.tf for IaC completeness.
Import all live authentik resources (groups, users, oauth2 providers, applications)
into terraform state via authentik-generated.tf. Provider config in authentik-config.tf.
Resources are drift-free and match live cluster.
Add cilium CNI to bootstrap helm_release.for_each (1.19.5, kube-system).
Cilium was unmanaged (helmfile-only); now IaC-owned. Critical path for
cluster rebuild recovery. Adds cilium repo to helm-repositories.tf.
- privileged level allows hostPort (80/443) required for nginx
- Other namespaces remain at baseline for security
- Cleaner than exempting namespace entirely
- ingress-nginx now has restricted policy level (allows hostPort)
- Previous timeout was due to policy blocking pod deployment
- Re-importing helm release to TF management
- ingress-nginx already deployed and working in cluster
- Helm updates timeout repeatedly (5+ min with context deadline exceeded)
- Remove from bootstrap releases; manage separately via helm/kubectl
- cert-manager + reloader continue via TF
- ingress-nginx requires hostPort (80/443) which is forbidden at baseline level
- Change to restricted enforcement level to allow hostPort
- Other namespaces remain at baseline for security
CoreDNS is Talos-bootstrapped and previously untracked except for its
ConfigMap. Pull the full live spec into one file as the single source of
truth, add topologySpreadConstraints so the 2 replicas don't land on the
same node. ScheduleAnyway (not DoNotSchedule) to avoid blocking scheduling
if a node is briefly unavailable.
3-9 replicas (HPA) previously relied on implicit scheduler spreading.
ScheduleAnyway (not DoNotSchedule) so pods still get scheduled if a node
is briefly unavailable, just less evenly.
Forgejo registry unreachable from worker nodes (network isolation +
host-to-ClusterIP routing gaps). Both management-service and queue-operator
now ship from the same public GHCR image, with queue-operator selected via
command override.
- Delete oauth2-proxy helm releases from helmfile (temporal, kmsvc, longhorn, portainer)
- Remove oauth2-proxy manifests and ingress redirects
- Add direct ingress for kmsvc management service
- Update temporal/portainer/longhorn ingress comments to reflect direct service exposure
Services now accessible without oauth2-proxy layer.
Story Crater Bot