Commit Graph
125 Commits
Author SHA1 Message Date
Story Crater Bot 88f8a764de fix(prometheus): set monitoring ns privileged via managedNamespaceMetadata — node-exporter hostNetwork/hostPID/hostPath blocked by baseline PSS 2026-07-21 11:05:04 -07:00
Story Crater Bot 34e996475f fix(promtail): set logging ns privileged via managedNamespaceMetadata — promtail hostPath/privileged/DAC_READ_SEARCH blocked by baseline PSS, DaemonSet created 0 pods 2026-07-21 11:03:58 -07:00
Story Crater Bot 3f4653ac56 fix(argocd): raise repo-server memory 512Mi->1Gi — OOMKilled under CMP+Helm rendering caused chronic restarts, not-ready endpoint, and cluster-wide sync 'no route to host' failures 2026-07-21 10:01:11 -07:00
Story Crater Bot 1dc6a2025f fix(kmsvc-redis): use bitnamilegacy/redis mirror + allowInsecureImages — docker.io/bitnami pulled version-pinned tags, ImagePullBackOff blocked redis + queue-operator 2026-07-21 09:47:19 -07:00
Story Crater Bot da925f3101 fix(forgejo-runner): add fsGroup 1000 so runner user can write /data/.runner — register hit permission denied on root-owned Longhorn PVC 2026-07-21 09:40:59 -07:00
Story Crater Bot 2443708abb chore(ci): refresh forgejo runner registration token — prior token invalid/expired 2026-07-21 09:38:15 -07:00
Story Crater Bot 9a34c12068 fix(forgejo-runner): point at in-cluster forgejo Service :3000 not public :443 — runner i/o timeout, forgejo serves 3000 not 443 2026-07-21 09:35:32 -07:00
Story Crater Bot f646bb06fd fix(minio,loki): declare loki-chunks/ruler/admin buckets in minio Tenant — loki failed with NoSuchBucket 2026-07-21 09:31:52 -07:00
Story Crater Bot 4363739d59 fix(loki,vault,iam): loki minio endpoint :80 not :9000, emit vault-minio-creds via CMP, drop redundant broken authentik-migrations job 2026-07-21 09:24:52 -07:00
Story Crater Bot 2bf543bba1 fix(ingress): add homelab-ingress ArgoCD app to apply orphaned ingress.yaml — services had no Ingress object, unreachable via LAN ingress .160 2026-07-21 09:15:58 -07:00
Story Crater Bot 6a2aacc4e6 feat(terraform): add per-node Cloudflare Tunnel cert SANs to controlplane certSANs — remote talosctl/kubectl over tunnel pass TLS verification
Adds optional cloudflare_talos_sans (machine.certSANs, talos API :50000) and
cloudflare_apiserver_sans (cluster.apiServer.certSANs, kube-apiserver :6443) per
control-plane node. cp-1 gets cp1.homelab + cp1-talos.homelab; cp-2/cp-3 get
their cpN-talos.homelab. Values set in gitignored tfvars.
2026-07-21 08:02:24 -07:00
Story Crater Bot 0471177250 chore(ci): add SOPS-encrypted runner-token secret record for forgejo-runner registration 2026-07-21 07:55:28 -07:00
Story Crater Bot 7fb73d6a4c fix(scheduling): pin portainer+forgejo-runner to az-a, add nodeSelector to runner chart template — WFFC alone insufficient with single Longhorn node (cp-1 only) 2026-07-20 23:51:46 -07:00
Story Crater Bot e0b24c83d0 fix(storage): add longhorn-wffc WaitForFirstConsumer default SC, repoint portainer/forgejo-runner — Immediate binding placed PVCs on non-storage nodes (cp-2/cp-3), attach failed 2026-07-20 23:49:01 -07:00
Story Crater Bot 9117fd777a fix(authentik): drop redundant authentik-migrate init container — server entrypoint migrates; old-image manage migrate tripped version-history precheck on empty DB 2026-07-20 23:40:08 -07:00
Story Crater Bot 5ce0b92186 feat(data): add CNPG managed roles + Database CRs for authentik/temporal — replaces missing helmfile post-sync user creation
authentik/temporal DB users+databases were never provisioned (old helmfile hook
gone; db-init-job only made schemas in shared app DB). Adds managed.roles
(authentik/temporal login roles, passwords from basic-auth secrets) + Database CRs
(dedicated DBs owned by each role). Role secrets applied out-of-band (SOPS), not in
kustomize resources so data-schemas app doesn't choke on ciphertext.
2026-07-20 23:36:21 -07:00
Story Crater Bot 89fa87f7c1 fix(sops-cmp): grafana-admin secret needs admin-user key too — chart existingSecret requires both user and password 2026-07-20 23:07:51 -07:00
Story Crater Bot 42cd0204fa fix(logging): pin grafana + loki to az-a (talos-cp-1) — sole Longhorn node, PVC fails to attach on cp-2/cp-3 2026-07-20 23:04:19 -07:00
Story Crater Bot 3a95f57b8f fix(loki): wire S3 creds from loki-s3-creds Secret via expand-env + extraEnvFrom — replaces empty helmfile-injected access keys 2026-07-20 23:00:51 -07:00
Story Crater Bot 2ec6eba9d2 fix(sops-cmp): correct loki s3 path (.loki.storage.s3), emit authentik-secrets separately, drop broken discover — merge via server/worker/migrate envFrom
Loki keys are under .loki.storage.s3 not .loki.s3 (returned null). Emit a separate
authentik-secrets Secret (not 'authentik', which the Helm chart owns) and merge it
via envFrom on server/worker/migrate. Remove discover fileName (caused MatchRepository
timeouts; app names the plugin explicitly).
2026-07-20 22:55:23 -07:00
Story Crater Bot d6f5b9ed69 feat(argocd): wire SOPS ConfigManagementPlugin properly — initContainer installs sops/yq, sidecar decrypts *.enc.yaml into app Secrets
Correct CMP setup (prior attempt used unsupported config): repoServer.initContainers
fetches sops v3.9.0 + yq v4.44.3 into a shared volume; repoServer.extraContainers
runs argocd-cmp-server with plugin.yaml from the sops-cmp-plugin ConfigMap, age key
from sops-age Secret. Plugin emits authentik/loki-s3-creds/grafana-admin/grafana-oidc
Secrets from decrypted enc files. sops-secrets Application (wave 0) uses the plugin at
repo root. Unblocks authentik/loki/grafana which were Degraded on missing secrets.
2026-07-20 13:13:47 -07:00
Story Crater Bot ce1fc4e296 fix(ingress-nginx): set privileged PodSecurity via managedNamespaceMetadata — hostPort 80/443 blocked by default baseline enforce, makes label permanent in IaC 2026-07-20 12:56:05 -07:00
Story Crater Bot d841bbdb95 feat(substrate): deploy cert-manager, ingress-nginx, reloader + LE staging issuers via app-of-apps — restores substrate ownership after Terraform removal
Substrate had no owner since Terraform was deleted (Pure GitOps). Adds 5 wave-0/1
Applications: cert-manager v1.21.0 (installCRDs, CP tolerations), ingress-nginx
4.15.1 (LB 192.168.1.160), reloader 2.2.14 at wave 0; LE ClusterIssuers +
*.riotpiao.com wildcard cert at wave 1 (DNS-01 via Cloudflare). Adds 3 chart
repos to AppProject sourceRepos and SOPS-encrypted cloudflare-api-token secret.
Cert starts on letsencrypt-staging; flip to prod after clean issue.
2026-07-20 12:50:02 -07:00
Story Crater Bot d9d2e34558 fix(minio): use configuration secret (config.env) for root creds, valid image tag — tenant now boots and authenticates
Switch Tenant from credsSecret to configuration field (v5 pods read config.env
shell exports); pin image to RELEASE.2025-07-23 (the old 2024-06 tag was pulled
from Docker Hub, ErrImagePull); drop prometheusOperator:true (made operator fail
reconcile hunting Prometheus in ns default). MinIO now serves S3, 4/4 drives OK,
root auth works. Operator's cosmetic 'empty tenant credentials' health-log is
harmless (documented inline).
2026-07-20 12:42:15 -07:00
Story Crater Bot 4cfac71a73 fix(minio): rewrite Tenant to operator-v5 schema, single-node pool, declarative buckets/users — removes invalid Bucket/Policy/User CRs and dead multi-site replication
Old Tenant used unknown v2 fields (pools[].size/storageClass, spec.console/metrics/ingress)
and referenced nonexistent minio.min.io/v1alpha1 Bucket/Policy/User kinds, so the app
never synced. Rewrites to valid v2: single erasure-coded pool (4 vols) pinned to
talos-cp-1/az-a (only schedulable+Longhorn node per 3-CP topology), spec.buckets +
spec.users declarative provisioning, prometheusOperator ServiceMonitor, features.domains.
Drops hand-rolled minio-service (operator owns it), dead multi-site replication job,
and legacy alias. Adds mc-based PostSync job for the ollama scoped policy, and
SOPS-encrypted minio-creds/oidc/user secrets for IaC record.
2026-07-20 12:27:44 -07:00
Story Crater Bot 7728f20d2b fix(argocd): allow operator.min.io in homelab AppProject sourceRepos — minio operator chart repo was blocked by allowlist 2026-07-20 12:18:45 -07:00
Story Crater Bot cea1a78a37 fix(minio): correct operator chart repoURL and pin version — charts.min.io lacks operator chart, use [email protected]
The operator chart moved to https://operator.min.io/; https://charts.min.io/ only
ships the standalone minio chart, causing 'chart operator not found in index'.
Pin to 5.0.18 (v5.x schema matches minio-operator-values.yaml operator.image.tag v5.0.0);
targetRevision '*' was fragile. Unblocks minio-tenant (needs operator CRDs).
2026-07-20 12:10:38 -07:00
Story Crater Bot abaea8823b refactor(argocd): simplify secrets approach — use directory source, manual Secrets for Stage 0
Reverts complex CMP plugin setup (helm chart doesn't support repoServer.extraContainers).
Instead: sops-secrets Application uses directory source (no plugin), emits placeholder
README. Manually-created Secrets (grafana-admin) live in target namespaces.

Full CMP plugin work deferred to future stage. Grafana values still wired to
admin.existingSecret (no-op until Secret exists, which it now does).

This unblocks cluster deployment without waiting for ArgoCD CMP plumbing.
2026-07-20 11:49:23 -07:00
Story Crater BotandClaude Haiku 4.5 d282ae1aa0 feat(argocd): deploy SOPS CMP plugin for secret decryption — Stage 0 grafana
Adds ConfigManagementPlugin (CMP) sidecar to argocd-repoServer. Plugin decrypts
*.enc.yaml files with age key from sops-age Secret, emits plain Kubernetes Secrets.

Stage 0: grafana only (2 Secrets: grafana-oidc + new grafana-admin). Updates
grafana-values.yaml to wire admin.existingSecret (chart-native support).

CMP Application (00-secrets.yaml) syncs at wave 0 before grafana/loki/authentik.
Decryption happens on-demand during sync, no pre-built Secret commits. Stages 1-4
(loki/authentik/forgejo/temporal) extend plugin script incrementally after
verification.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
2026-07-20 11:30:06 -07:00
Story Crater Bot 063308308f feat(cloudflared): wire tunnel token secret and document bootstrap
- Create SOPS-encrypted cloudflared-secrets.enc.yaml with tunnel token
- Add Cloudflare vars to .env.example (CLOUDFLARE_CONNECTOR_TOKEN, ACCOUNT_ID, TUNNEL_ID, API_TOKEN)
- Document Phase 0 cloudflared-token Secret creation in BOOTSTRAP.md (manual step until CMP plugin wires it)
- Note: Cloudflare-side TCP routing (cp1.homelab -> 192.168.1.213:6443, etc.) must be configured manually in Zero Trust dashboard

Tunnel already deployed as ArgoCD Application in k8s/argocd/apps/60-applications.yaml (wave 8); this closes the missing Secret gap and documents the bootstrap path.
2026-07-20 10:50:20 -07:00
Story Crater Bot a207c56637 fix(k8s,docs): scale ddb-cluster to single instance, pin minio to storage namespace, document 3-CP topology in USAGE 2026-07-20 08:22:53 -07:00
Story Crater Bot c759481ea6 refactor(argocd): replace wave/layer/phase schemes with two-phase bootstrap + app-of-apps and document both CD scopes — fixes self-hosted-git chicken-egg and stale paths 2026-07-20 08:22:53 -07:00
Story Crater Bot 15b1ec6ad4 feat(terraform): restructure control planes into a 3-node map with LAN etcd advertise and live machine CA — enables talos-cp-1/2/3 HA and drops worker configs 2026-07-20 08:22:53 -07:00
Story Crater Bot f7a8df0514 chore: remove GITOPS_ARCHITECTURE.md — scratch planning doc, not meant for the repo 2026-07-19 09:30:48 -07:00
Story Crater Bot 32281ee923 chore: remove scratch planning docs — not meant for the repo 2026-07-19 09:30:25 -07:00
Story Crater Bot 578a707867 feat(gitops): migrate domain to riotpiao.com, add CNPG + Forgejo HA on Redis/Postgres, wire ArgoCD apps — enables cluster rebuild after etcd wipe and unblocks the git-source chicken-egg via standalone Helm-source Applications 2026-07-19 09:29:17 -07:00
Story Crater Bot c6493f14ae feat(ci,iac): Consolidate Forgejo CI workflows and add Talos Terraform IaC
Consolidate three separate Forgejo Actions (argocd-sync, security-scan, validate-k8s) into single cluster-ci workflow for cleaner CI/CD pipeline with proper job sequencing and reduced auth overhead.

Add Terraform configuration for Talos cluster machine configs:
- Provider setup for Talos
- Centralized variables for CP and worker configs
- Template-based config generation for controlplane.yaml and worker-*.yaml
- Sensitive data separated in terraform.tfvars (gitignored)
- Local state tracking for infrastructure
2026-07-17 23:44:08 -07:00
Story Crater Bot 7437078f23 fix(ci): Correct Forgejo Actions template syntax for git clone auth
Use proper Forgejo variables: gitea.server_url, gitea.repository
Construct CLONE_URL correctly: https://user:token@host/repo.git
Use bash parameter expansion to strip https:// prefix

Removes invalid Forgejo filter syntax (| replace)
2026-07-17 12:27:40 -07:00
Story Crater Bot f8b19d9f55 fix(ci): Replace GitHub actions/checkout with Forgejo auth
Use CI_RUNNER and CI_RUNNER_SECRET for repo clone authentication.
Embed credentials in git clone URL: https://user:token@host/repo.git

Removes dependency on GITHUB_TOKEN (GitHub-specific) and improves Forgejo compatibility.
2026-07-17 12:19:23 -07:00
Story Crater Bot c000ddb402 fix(ci): Remove stale kustomize before reinstall in validate-k8s workflow
Prevent 'kustomize exists' error when downloading tools in CI runner.
Use -f flag on mv commands to force overwrite.
2026-07-17 11:31:44 -07:00
Story Crater Bot 888c4f5493 fix(ingress): Add nginx LoadBalancer service to GitOps — removes 503 error
Remove manual nginx-controller-svc.yaml (duplicate with Helm-managed service).
Helm chart creates LoadBalancer service automatically. Manual manifest caused conflicts.

Ingress controller now solely managed by Helm chart values.
2026-07-17 00:46:27 -07:00
Story Crater Bot 403e495fe9 fix(ingress): Add nginx LoadBalancer service to GitOps — fixes 503 error
Recreate ingress-nginx-controller LoadBalancer service that was deleted.
Add to k8s/bootstrap/ingress/ kustomization for ArgoCD management.

LoadBalancer assigned IP: 192.168.1.160 (via MetalLB)
ArgoCD now accessible via: https://192.168.1.160/ (or update DNS)
2026-07-16 23:35:02 -07:00
Story Crater Bot f30771a78a feat(minio): Expand CRDs to include Policies and Users — full YAML-driven resource creation
Add MinIO Policies and Users via CRD alongside Buckets.

Resources now declarative:
- Bucket: riotpiao-models (versioning enabled)
- Policy: policy-ollama (scoped bucket access)
- User: user-ollama (service account for Ollama/LLM)

Access keys can be overridden via SOPS or kustomize overlays.
All MinIO resource creation now git-tracked and version controlled.
2026-07-16 14:55:50 -07:00
Story Crater Bot e2e17ae0fb feat(data): Add CNPG cluster + database schema initialization
Create production PostgreSQL cluster via CNPG (3-node HA, Longhorn storage).

Schema initialization Job creates schemas for:
- Authentik (identity provider)
- Temporal (workflow engine)
- Vault (secrets management)
- App (generic application databases)

Database layer now captures complete IaC for stateful infrastructure.
Services find ready schemas when deployed.
2026-07-16 14:53:59 -07:00
Story Crater Bot c230b3ee45 feat(minio): Add MinIO Bucket CRD for riotpiao-models — replaces shell script setup 2026-07-16 14:31:33 -07:00
Story Crater Bot 2d7330798b refactor(k8s): Reorganize into 5-layer structure with production kustomizations 2026-07-16 14:28:19 -07:00
Story Crater Bot a81b9b6169 refactor(ci-cd): Replace Terraform pipeline with GitOps validation and ArgoCD sync
Delete old terraform-apply.yml (terraform fmt/init/validate/plan/apply).

Create new GitOps CI/CD:
- validate-k8s.yaml: YAML lint, kubeval, kustomize build, ArgoCD validation
- argocd-sync.yaml: Auto-sync homelab-root on main branch
- security-scan.yaml: Trivy, Polaris, secret detection
- .yamllint.yaml: YAML linting configuration

Add documentation (.forgejo/CI-CD.md) and architecture guides.

Git is now single source of truth. CI validates, ArgoCD deploys.
2026-07-16 12:52:25 -07:00
Story Crater Bot a860de94da refactor: remove terraform entirely, migrate to pure GitOps (ArgoCD)
Delete entire terraform/ directory.

Architecture: Terraform + ArgoCD → ArgoCD only
- Single tool: ArgoCD manages all infrastructure and applications
- Source of truth: git only (k8s/ directory)
- Continuous reconciliation: no manual apply needed
- Simpler state: no tfstate backend, no state files

Next: Migrate all Terraform resources to k8s/ YAML manifests
and ArgoCD Applications (namespaces, storage classes, Helm releases,
RBAC, network policies, Authentik config).
2026-07-16 11:04:52 -07:00
Story Crater Bot 94a2bd648c fix: remove kubeconfig references, use try() for pod runtime files
- main.tf: remove kubeconfig_path local (no longer used with direct auth)
- providers.tf: wrap file() with try() to handle plan-time on non-pod systems

try() allows terraform plan to work locally; at runtime in pod, files exist and are used.
2026-07-15 21:36:05 -07:00
Story Crater Bot be2e5c321f fix: use direct in-cluster kubernetes auth instead of kubeconfig file
- providers.tf: use host + token + ca_crt from mounted service account secrets
- workflow: remove kubeconfig generation step (no longer needed)
- variables.tf: remove unused kubeconfig_path variable

This is the standard pattern for running terraform inside k8s pods.
2026-07-15 21:26:09 -07:00