Loki keys are under .loki.storage.s3 not .loki.s3 (returned null). Emit a separate
authentik-secrets Secret (not 'authentik', which the Helm chart owns) and merge it
via envFrom on server/worker/migrate. Remove discover fileName (caused MatchRepository
timeouts; app names the plugin explicitly).
Correct CMP setup (prior attempt used unsupported config): repoServer.initContainers
fetches sops v3.9.0 + yq v4.44.3 into a shared volume; repoServer.extraContainers
runs argocd-cmp-server with plugin.yaml from the sops-cmp-plugin ConfigMap, age key
from sops-age Secret. Plugin emits authentik/loki-s3-creds/grafana-admin/grafana-oidc
Secrets from decrypted enc files. sops-secrets Application (wave 0) uses the plugin at
repo root. Unblocks authentik/loki/grafana which were Degraded on missing secrets.
Substrate had no owner since Terraform was deleted (Pure GitOps). Adds 5 wave-0/1
Applications: cert-manager v1.21.0 (installCRDs, CP tolerations), ingress-nginx
4.15.1 (LB 192.168.1.160), reloader 2.2.14 at wave 0; LE ClusterIssuers +
*.riotpiao.com wildcard cert at wave 1 (DNS-01 via Cloudflare). Adds 3 chart
repos to AppProject sourceRepos and SOPS-encrypted cloudflare-api-token secret.
Cert starts on letsencrypt-staging; flip to prod after clean issue.
Switch Tenant from credsSecret to configuration field (v5 pods read config.env
shell exports); pin image to RELEASE.2025-07-23 (the old 2024-06 tag was pulled
from Docker Hub, ErrImagePull); drop prometheusOperator:true (made operator fail
reconcile hunting Prometheus in ns default). MinIO now serves S3, 4/4 drives OK,
root auth works. Operator's cosmetic 'empty tenant credentials' health-log is
harmless (documented inline).
Old Tenant used unknown v2 fields (pools[].size/storageClass, spec.console/metrics/ingress)
and referenced nonexistent minio.min.io/v1alpha1 Bucket/Policy/User kinds, so the app
never synced. Rewrites to valid v2: single erasure-coded pool (4 vols) pinned to
talos-cp-1/az-a (only schedulable+Longhorn node per 3-CP topology), spec.buckets +
spec.users declarative provisioning, prometheusOperator ServiceMonitor, features.domains.
Drops hand-rolled minio-service (operator owns it), dead multi-site replication job,
and legacy alias. Adds mc-based PostSync job for the ollama scoped policy, and
SOPS-encrypted minio-creds/oidc/user secrets for IaC record.
The operator chart moved to https://operator.min.io/; https://charts.min.io/ only
ships the standalone minio chart, causing 'chart operator not found in index'.
Pin to 5.0.18 (v5.x schema matches minio-operator-values.yaml operator.image.tag v5.0.0);
targetRevision '*' was fragile. Unblocks minio-tenant (needs operator CRDs).
Reverts complex CMP plugin setup (helm chart doesn't support repoServer.extraContainers).
Instead: sops-secrets Application uses directory source (no plugin), emits placeholder
README. Manually-created Secrets (grafana-admin) live in target namespaces.
Full CMP plugin work deferred to future stage. Grafana values still wired to
admin.existingSecret (no-op until Secret exists, which it now does).
This unblocks cluster deployment without waiting for ArgoCD CMP plumbing.
- Create SOPS-encrypted cloudflared-secrets.enc.yaml with tunnel token
- Add Cloudflare vars to .env.example (CLOUDFLARE_CONNECTOR_TOKEN, ACCOUNT_ID, TUNNEL_ID, API_TOKEN)
- Document Phase 0 cloudflared-token Secret creation in BOOTSTRAP.md (manual step until CMP plugin wires it)
- Note: Cloudflare-side TCP routing (cp1.homelab -> 192.168.1.213:6443, etc.) must be configured manually in Zero Trust dashboard
Tunnel already deployed as ArgoCD Application in k8s/argocd/apps/60-applications.yaml (wave 8); this closes the missing Secret gap and documents the bootstrap path.
Recreate ingress-nginx-controller LoadBalancer service that was deleted.
Add to k8s/bootstrap/ingress/ kustomization for ArgoCD management.
LoadBalancer assigned IP: 192.168.1.160 (via MetalLB)
ArgoCD now accessible via: https://192.168.1.160/ (or update DNS)
Add MinIO Policies and Users via CRD alongside Buckets.
Resources now declarative:
- Bucket: riotpiao-models (versioning enabled)
- Policy: policy-ollama (scoped bucket access)
- User: user-ollama (service account for Ollama/LLM)
Access keys can be overridden via SOPS or kustomize overlays.
All MinIO resource creation now git-tracked and version controlled.
Reverting Phase 4 Pod Job implementation in favor of CI runner (Forgejo Actions).
Deleted:
- k8s/argocd/apps/phase4-terraform-0.yaml
- k8s/hooks/phase4/ (terraform-apply-hook.yaml, terraform-rbac.yaml, terraform-s3-secrets.enc.yaml)
Reason: Pod Job approach had limitations (eviction, timeouts, pod security policies).
Next: Implement Forgejo Actions CI workflow for terraform apply.
Phase 0 continuation: enable ArgoCD to decrypt SOPS-encrypted secrets on sync.
1. Update ArgoCD Helm values (terraform/argocd-bootstrap.tf):
- Add SOPS_AGE_KEY_FILE env var to repoServer
- Mount sops-age K8s Secret at /home/argocd/.sops
- Add ConfigManagementPlugin for SOPS (detects *.enc.yaml files)
2. Update encrypted secrets with real values:
- k8s/base/secrets.enc.yaml: encrypted with actual service credentials
- All secret values encrypted at rest in git
- ArgoCD decrypts on sync using K8s Secret + AGE key
Prerequisites:
- K8s Secret created: kubectl create secret generic sops-age -n argocd --from-file=keys.txt=/Users/rockliang/.sops/key.txt
- SOPS_AGE_KEY_FILE env var set in ArgoCD repoServer (done above)
Next: Phase 1 — migrate 9 hookless releases to ArgoCD + create Applications that reference encrypted secrets.
Phase 0 groundwork for helmfile→ArgoCD migration using SOPS (Secrets Operations):
1. Install SOPS + AGE encryption
- AGE key generated and stored locally at ~/.sops/key.txt
- Public key embedded in .sops.yaml for file encryption rules
2. Create K8s Secret for AGE private key
- kubectl: create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt
- ArgoCD will use this key to decrypt secrets at sync time
3. Encrypt initial secrets
- k8s/base/secrets.enc.yaml: AES256_GCM encrypted secrets for all services
- Placeholder values (will be replaced with real values per environment)
- Secrets never visible in git (encrypted at rest)
4. Configure SOPS
- .sops.yaml: creation rules for k8s/*/secrets.enc.yaml files
- All future secret files auto-encrypt on edit (sops -e)
Setup: Store AGE key as K8s Secret in argocd namespace:
export KUBECONFIG=cluster-config/kubeconfig
kubectl create secret generic sops-age -n argocd --from-file=keys.txt=~/.sops/key.txt
Next: Configure ArgoCD Helm plugin to decrypt secrets on sync (Phase 0 continuation).
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
CoreDNS is Talos-bootstrapped and previously untracked except for its
ConfigMap. Pull the full live spec into one file as the single source of
truth, add topologySpreadConstraints so the 2 replicas don't land on the
same node. ScheduleAnyway (not DoNotSchedule) to avoid blocking scheduling
if a node is briefly unavailable.
3-9 replicas (HPA) previously relied on implicit scheduler spreading.
ScheduleAnyway (not DoNotSchedule) so pods still get scheduled if a node
is briefly unavailable, just less evenly.
Story Crater Bot