Correct CMP setup (prior attempt used unsupported config): repoServer.initContainers
fetches sops v3.9.0 + yq v4.44.3 into a shared volume; repoServer.extraContainers
runs argocd-cmp-server with plugin.yaml from the sops-cmp-plugin ConfigMap, age key
from sops-age Secret. Plugin emits authentik/loki-s3-creds/grafana-admin/grafana-oidc
Secrets from decrypted enc files. sops-secrets Application (wave 0) uses the plugin at
repo root. Unblocks authentik/loki/grafana which were Degraded on missing secrets.
Substrate had no owner since Terraform was deleted (Pure GitOps). Adds 5 wave-0/1
Applications: cert-manager v1.21.0 (installCRDs, CP tolerations), ingress-nginx
4.15.1 (LB 192.168.1.160), reloader 2.2.14 at wave 0; LE ClusterIssuers +
*.riotpiao.com wildcard cert at wave 1 (DNS-01 via Cloudflare). Adds 3 chart
repos to AppProject sourceRepos and SOPS-encrypted cloudflare-api-token secret.
Cert starts on letsencrypt-staging; flip to prod after clean issue.
Switch Tenant from credsSecret to configuration field (v5 pods read config.env
shell exports); pin image to RELEASE.2025-07-23 (the old 2024-06 tag was pulled
from Docker Hub, ErrImagePull); drop prometheusOperator:true (made operator fail
reconcile hunting Prometheus in ns default). MinIO now serves S3, 4/4 drives OK,
root auth works. Operator's cosmetic 'empty tenant credentials' health-log is
harmless (documented inline).
Old Tenant used unknown v2 fields (pools[].size/storageClass, spec.console/metrics/ingress)
and referenced nonexistent minio.min.io/v1alpha1 Bucket/Policy/User kinds, so the app
never synced. Rewrites to valid v2: single erasure-coded pool (4 vols) pinned to
talos-cp-1/az-a (only schedulable+Longhorn node per 3-CP topology), spec.buckets +
spec.users declarative provisioning, prometheusOperator ServiceMonitor, features.domains.
Drops hand-rolled minio-service (operator owns it), dead multi-site replication job,
and legacy alias. Adds mc-based PostSync job for the ollama scoped policy, and
SOPS-encrypted minio-creds/oidc/user secrets for IaC record.
The operator chart moved to https://operator.min.io/; https://charts.min.io/ only
ships the standalone minio chart, causing 'chart operator not found in index'.
Pin to 5.0.18 (v5.x schema matches minio-operator-values.yaml operator.image.tag v5.0.0);
targetRevision '*' was fragile. Unblocks minio-tenant (needs operator CRDs).
Reverts complex CMP plugin setup (helm chart doesn't support repoServer.extraContainers).
Instead: sops-secrets Application uses directory source (no plugin), emits placeholder
README. Manually-created Secrets (grafana-admin) live in target namespaces.
Full CMP plugin work deferred to future stage. Grafana values still wired to
admin.existingSecret (no-op until Secret exists, which it now does).
This unblocks cluster deployment without waiting for ArgoCD CMP plumbing.
- Create SOPS-encrypted cloudflared-secrets.enc.yaml with tunnel token
- Add Cloudflare vars to .env.example (CLOUDFLARE_CONNECTOR_TOKEN, ACCOUNT_ID, TUNNEL_ID, API_TOKEN)
- Document Phase 0 cloudflared-token Secret creation in BOOTSTRAP.md (manual step until CMP plugin wires it)
- Note: Cloudflare-side TCP routing (cp1.homelab -> 192.168.1.213:6443, etc.) must be configured manually in Zero Trust dashboard
Tunnel already deployed as ArgoCD Application in k8s/argocd/apps/60-applications.yaml (wave 8); this closes the missing Secret gap and documents the bootstrap path.
Consolidate three separate Forgejo Actions (argocd-sync, security-scan, validate-k8s) into single cluster-ci workflow for cleaner CI/CD pipeline with proper job sequencing and reduced auth overhead.
Add Terraform configuration for Talos cluster machine configs:
- Provider setup for Talos
- Centralized variables for CP and worker configs
- Template-based config generation for controlplane.yaml and worker-*.yaml
- Sensitive data separated in terraform.tfvars (gitignored)
- Local state tracking for infrastructure
Use CI_RUNNER and CI_RUNNER_SECRET for repo clone authentication.
Embed credentials in git clone URL: https://user:token@host/repo.git
Removes dependency on GITHUB_TOKEN (GitHub-specific) and improves Forgejo compatibility.
Recreate ingress-nginx-controller LoadBalancer service that was deleted.
Add to k8s/bootstrap/ingress/ kustomization for ArgoCD management.
LoadBalancer assigned IP: 192.168.1.160 (via MetalLB)
ArgoCD now accessible via: https://192.168.1.160/ (or update DNS)
Add MinIO Policies and Users via CRD alongside Buckets.
Resources now declarative:
- Bucket: riotpiao-models (versioning enabled)
- Policy: policy-ollama (scoped bucket access)
- User: user-ollama (service account for Ollama/LLM)
Access keys can be overridden via SOPS or kustomize overlays.
All MinIO resource creation now git-tracked and version controlled.
- main.tf: remove kubeconfig_path local (no longer used with direct auth)
- providers.tf: wrap file() with try() to handle plan-time on non-pod systems
try() allows terraform plan to work locally; at runtime in pod, files exist and are used.
- providers.tf: use host + token + ca_crt from mounted service account secrets
- workflow: remove kubeconfig generation step (no longer needed)
- variables.tf: remove unused kubeconfig_path variable
This is the standard pattern for running terraform inside k8s pods.
- Link core-cli app to OAuth2 provider (was hardcoded to 0)
- Add core-cli user to authentik_admins for CI access
- Fix terraform init: use 'endpoint' not 'endpoints.s3' for S3 backend
(Terraform 1.9.4 compatibility, matches state.tf config)
Reverting Phase 4 Pod Job implementation in favor of CI runner (Forgejo Actions).
Deleted:
- k8s/argocd/apps/phase4-terraform-0.yaml
- k8s/hooks/phase4/ (terraform-apply-hook.yaml, terraform-rbac.yaml, terraform-s3-secrets.enc.yaml)
Reason: Pod Job approach had limitations (eviction, timeouts, pod security policies).
Next: Implement Forgejo Actions CI workflow for terraform apply.
Phase 0 continuation: enable ArgoCD to decrypt SOPS-encrypted secrets on sync.
1. Update ArgoCD Helm values (terraform/argocd-bootstrap.tf):
- Add SOPS_AGE_KEY_FILE env var to repoServer
- Mount sops-age K8s Secret at /home/argocd/.sops
- Add ConfigManagementPlugin for SOPS (detects *.enc.yaml files)
2. Update encrypted secrets with real values:
- k8s/base/secrets.enc.yaml: encrypted with actual service credentials
- All secret values encrypted at rest in git
- ArgoCD decrypts on sync using K8s Secret + AGE key
Prerequisites:
- K8s Secret created: kubectl create secret generic sops-age -n argocd --from-file=keys.txt=/Users/rockliang/.sops/key.txt
- SOPS_AGE_KEY_FILE env var set in ArgoCD repoServer (done above)
Next: Phase 1 — migrate 9 hookless releases to ArgoCD + create Applications that reference encrypted secrets.
Phase 0 groundwork for helmfile→ArgoCD migration using SOPS (Secrets Operations):
1. Install SOPS + AGE encryption
- AGE key generated and stored locally at ~/.sops/key.txt
- Public key embedded in .sops.yaml for file encryption rules
2. Create K8s Secret for AGE private key
- kubectl: create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt
- ArgoCD will use this key to decrypt secrets at sync time
3. Encrypt initial secrets
- k8s/base/secrets.enc.yaml: AES256_GCM encrypted secrets for all services
- Placeholder values (will be replaced with real values per environment)
- Secrets never visible in git (encrypted at rest)
4. Configure SOPS
- .sops.yaml: creation rules for k8s/*/secrets.enc.yaml files
- All future secret files auto-encrypt on edit (sops -e)
Setup: Store AGE key as K8s Secret in argocd namespace:
export KUBECONFIG=cluster-config/kubeconfig
kubectl create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt
Next: Configure ArgoCD Helm plugin to decrypt secrets on sync (Phase 0 continuation).
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
Add IaC practice section to coding-standards.md:
- All infrastructure state via Terraform or Helm (never ad-hoc scripts)
- Clear division: Terraform owns helm releases/namespaces/storage/state
- Anti-pattern: split bucket definitions across multiple files
- Bootstrap-only exception: document one-time setup with rationale
Rationale: prevents state drift, credential duplication, and unclear ownership.
Remove malformed line and clarify rules:
- terraform/.terraform/ (local provider cache)
- terraform/*.tfstate* (local state backups)
- skills-lock.json (lock file)
All TF state now remote (MinIO S3), local files safe to exclude.
Migrate terraform state from local file to MinIO S3 bucket (terraform-state).
Backend config: https://minio-api.riotpiao.homelab.com (external endpoint).
State now persisted remotely, shared across team, safe for cluster rebuild.
Also added terraform-state bucket to MinIO managed buckets.
Move loki bucket creation from helmfile post-hook to TF-managed buckets array.
Now all MinIO buckets (6 total) declared in terraform/minio.tf for IaC completeness.
Import all live authentik resources (groups, users, oauth2 providers, applications)
into terraform state via authentik-generated.tf. Provider config in authentik-config.tf.
Resources are drift-free and match live cluster.
Add cilium CNI to bootstrap helm_release.for_each (1.19.5, kube-system).
Cilium was unmanaged (helmfile-only); now IaC-owned. Critical path for
cluster rebuild recovery. Adds cilium repo to helm-repositories.tf.
- privileged level allows hostPort (80/443) required for nginx
- Other namespaces remain at baseline for security
- Cleaner than exempting namespace entirely
- ingress-nginx now has restricted policy level (allows hostPort)
- Previous timeout was due to policy blocking pod deployment
- Re-importing helm release to TF management
- ingress-nginx already deployed and working in cluster
- Helm updates timeout repeatedly (5+ min with context deadline exceeded)
- Remove from bootstrap releases; manage separately via helm/kubectl
- cert-manager + reloader continue via TF
- ingress-nginx requires hostPort (80/443) which is forbidden at baseline level
- Change to restricted enforcement level to allow hostPort
- Other namespaces remain at baseline for security
CoreDNS is Talos-bootstrapped and previously untracked except for its
ConfigMap. Pull the full live spec into one file as the single source of
truth, add topologySpreadConstraints so the 2 replicas don't land on the
same node. ScheduleAnyway (not DoNotSchedule) to avoid blocking scheduling
if a node is briefly unavailable.
3-9 replicas (HPA) previously relied on implicit scheduler spreading.
ScheduleAnyway (not DoNotSchedule) so pods still get scheduled if a node
is briefly unavailable, just less evenly.
Forgejo registry unreachable from worker nodes (network isolation +
host-to-ClusterIP routing gaps). Both management-service and queue-operator
now ship from the same public GHCR image, with queue-operator selected via
command override.
- Delete oauth2-proxy helm releases from helmfile (temporal, kmsvc, longhorn, portainer)
- Remove oauth2-proxy manifests and ingress redirects
- Add direct ingress for kmsvc management service
- Update temporal/portainer/longhorn ingress comments to reflect direct service exposure
Services now accessible without oauth2-proxy layer.
Story Crater Bot