Commit Graph
75 Commits
Author SHA1 Message Date
rock 449c2a9109 gitops: add secret-rotation controller ArgoCD Application
- Syncs k8s/apps/secret-rotation-controller/ kustomization
- Auto-prune and self-heal enabled
- Creates secret-rotation namespace
- ArgoCD will deploy CRD, RBAC, ExternalSecret, Deployment
2026-09-04 13:51:26 -07:00
rock 87786d8733 messaging: remove queue-crd/management-service (moved to kmsvc-manage)
- Applications now managed by kmsvc-root from kmsvc-manage.git
- Added ServerSideApply to homelab-root for proper annotation sync
- Avoids duplicate Application conflicts with Image Updater
2026-09-03 08:28:38 -07:00
rock 17a98afa3f image-updater: filter to SHA tags only (skip :latest)
allow-tags: regexp:^[0-9a-f]{7}$ ensures newest-build strategy
compares commit SHA tags, not the stale :latest tag
2026-09-03 08:07:19 -07:00
rock c6032cc354 argocd: add Image Updater for auto-deploy on image push
- Install argocd-image-updater via Helm (wave 1)
- Configure Forgejo registry (anonymous pulls)
- Annotate apps for auto-update: api-gw, portfolio, management-service, queue-crd
- Uses newest-build strategy for commit SHA tags
2026-09-02 20:31:09 -07:00
rock 1af9232613 feat: add tempo and otel-collector for distributed tracing 2026-08-31 15:02:15 -07:00
rock be55d68571 feat(argocd): add portfolio app, explicit sourceRepos, coredns rewrite for riotpiao.com
- Add Application for rock/riotpiao.com repo (portfolio site)
- Replace wildcard sourceRepos with explicit repo list
- Add CoreDNS rewrite for root domain riotpiao.com
2026-08-31 14:18:57 -07:00
rock 2038253d48 feat(security): add Kyverno for image scanning and Pod security policies
- Install Kyverno policy engine for admission control
- Add ClusterPolicies:
  * Disallow 'latest' tags (require explicit versions)
  * Restrict to trusted registries (docker.io, ghcr.io, quay.io, etc.)
  * Require non-root containers
  * Drop all Linux capabilities by default
  * Require securityContext on all containers
  * Require read-only root filesystem (audit only)
  * Require resource requests/limits (prevent starvation)
- All policies in audit mode initially (failurePolicy: ignore)
- Ready to graduate to enforce after testing
- Fixes: missing image scanning from security audit
2026-08-31 11:43:04 -07:00
rock 136bfaf0f2 feat(ops): add cluster-wide stale job/pod cleanup CronJob
Daily 04:00 UTC sweeper in kube-system:
- Delete failed Jobs older than 24h (any namespace)
- Delete completed standalone Jobs older than 72h (no CronJob owner)
- Delete orphan Error/Evicted pods older than 1h
- Self-cleans via ttlSecondsAfterFinished
2026-08-30 07:27:50 -07:00
Story Crater Bot 61a1975669 feat: deploy Immich with Authentik OIDC, rock as admin
Self-hosted photo backup (Google Photos replacement) - raw manifests,
no Helm chart, self-contained under k8s/apps/immich including its own
CNPG Postgres. Media PVC shares the cp-3 HDD 2TB/2TB with
paperless-media.

Postgres is pg18, not this repo's usual 16.2: CNPG's official pgvector
extension image (ghcr.io/cloudnative-pg/pgvector) is only published
for pg18, loaded via CNPG's ImageVolume extension mechanism (operator
1.30.0 / k8s 1.36.1 both support it). Immich auto-manages CREATE
EXTENSION itself at startup.

OIDC via a new "immich_role" Authentik scope mapping (homelab-admins/
immich-admins -> "admin" claim, else "user"), consumed by Immich's
OAuth roleClaim setting which re-syncs isAdmin on every login - more
reliable than Immich's racy first-user-is-admin fallback. Config
composed into an immich-oidc Secret and mounted as IMMICH_CONFIG_FILE,
matching the paperless-oidc pattern. k8s RBAC (immich-operator Role +
oidc:immich-admins binding) mirrors paperless/rbac.yaml.

immich namespace pre-created in k8s/infra/databases/namespaces.yaml
(not just immich's own CreateNamespace=true) since the iam PostSync
job's RoleBinding needs it to exist before wave 8.
2026-08-25 18:21:28 -07:00
Story Crater Bot 89f01b6f2e feat: add homelab-wide Authentik RBAC model and k8s OIDC auth wiring
Adds permissions claim + per-service admin groups in Authentik, scoped
Role/RoleBinding per service, public PKCE kubernetes OAuth2 client, and
kube-apiserver OIDC extraArgs. Also fixes paperless OIDC signup permissions
via adapter override and adds CoreDNS rewrite for authentik.riotpiao.com.
2026-08-25 15:03:44 -07:00
Story Crater Bot bea76eeba6 fix: repoint ArgoCD Applications from GitHub to in-cluster Forgejo
homelab-root and every child Application still tracked github.com/Riotpiaole/riotpiao.homelab.com, which had diverged from origin (Forgejo) for a while - pushes to Forgejo were never picked up by ArgoCD. Repointed to forgejo.riotpiao.com/rock/homelab.git, already covered by the AppProject's rock/* wildcard.
2026-08-25 11:20:26 -07:00
Story Crater Bot e8e5acfb13 feat: add paperless-ngx with OIDC, CNPG db, cp-3 HDD media, MinIO backup
Fixes controlplane.tftpl's install.wipe:true (should be false, live CPs already run false) and syncs coredns Corefile back to what's actually deployed (drops an unrolled-out, stale Kong-era rewrite).
2026-08-25 11:11:37 -07:00
Story Crater Bot 9eb5c8ea1a Remove separate memory app, bundle into wave 2 databases 2026-08-22 23:16:40 -07:00
Story Crater Bot aa6b2ae9c3 Add Poimen Memory to ArgoCD wave 2 deployment (namespace: poimen) 2026-08-22 23:09:02 -07:00
Story Crater Bot 06855b3768 fix(argocd): update poimen repoURL after rock/poiman rename to rock/poimen
The poiman repo was renamed to poimen on Forgejo; the stale repoURL made
poimen-root fail with a 301 redirect ComparisonError (ArgoCD's git
client doesn't follow redirects on smart-HTTP fetch), blocking sync for
poimen-root and everything under it.
2026-08-21 21:57:36 -07:00
Story Crater Bot 2f5793899b fix(argocd): poimen-root point to k8s/argocd directory 2026-08-21 21:28:18 -07:00
Story Crater Bot 2a1cb77443 fix(argocd): poimen-root point to k8s/argocd/apps like kmsvc-root 2026-08-21 21:27:38 -07:00
Story Crater Bot 9415a30309 fix(argocd): poimen-root use single source from poiman, remove workflows 2026-08-21 21:25:55 -07:00
Story Crater Bot a63838bae3 feat(argocd): enable poimen-root Application for poiman orchestration 2026-08-21 21:24:12 -07:00
Story Crater Bot 0eb0dd89f6 chore(argocd): track main branch instead of prod for auto-sync on every commit 2026-08-21 20:55:20 -07:00
Story Crater Bot 7a7e0fe813 chore(argocd): update api-gateway to track homelab-frontend prod branch 2026-08-21 20:47:38 -07:00
Story Crater Bot 9e5e51733d chore(argocd): add poimen application placeholder 2026-08-21 20:44:49 -07:00
Story Crater Bot 544fa371f6 chore(argocd): add kmsvc-manage bootstrap application 2026-08-21 20:44:35 -07:00
Story Crater Bot dbc4a55b02 feat(forgejo-runner): split into golang/node/rust runners, retire generic docker runner 2026-08-21 16:49:26 -07:00
Story Crater Bot 130746e6a1 Add Temporal worker for production task queue 2026-08-21 16:44:59 -07:00
Story Crater Bot bedf062906 chore(forgejo-runner): arm for cascading delete ahead of 3-runner migration 2026-08-21 16:35:01 -07:00
Story Crater Bot c938a58544 stage1: A1-A2 AppProject and projects Application
A1: Replace per-repo Forgejo entries with https://forgejo.riotpiao.com/rock/*
    wildcard so onboarding never requires touching AppProject.

A2: Add wave -1 Application for k8s/argocd/projects/ so it syncs before
    any Application references the AppProject.

Also add kustomization.yaml to k8s/argocd/projects/ to make it renderable.

Enabled by Stage 1 (A1, A2).
2026-08-20 21:31:05 -07:00
Story Crater Bot 7e40a4baf6 feat: cut api.riotpiao.com over to the Go gateway and retire Kong
Ingress api/api now backs onto api-gateway:8080; the kong Application, its
Helm values, plugins and llm-routes are removed. Gateway image v0.0.0 is in
the Forgejo registry and the pull secret is in the api namespace.
2026-08-19 22:51:53 -07:00
Story Crater Bot f8136f4e1e fix: ignore Reloader's injected env var on the Forgejo Deployment
Argo would otherwise strip STAKATER_* on each sync and fight Reloader for it,
recreating the forge pod every reconcile.
2026-08-19 22:43:29 -07:00
Story Crater Bot df5e623347 feat: manage Forgejo with Argo instead of the bootstrap Helm release
Values changes were inert as a bootstrap release, so the proxy-body-size fix
never reached the live Ingress. First sync is manual — the chart owns the
Forgejo PVC.
2026-08-19 22:37:56 -07:00
Story Crater Bot afb9b35292 fix(agent-pod): remote tui session for multi-agent 2026-08-18 15:08:04 -07:00
Story Crater Bot 5f16f16f0f fix(argocd): clone the public GitHub seed anonymously over HTTPS and delete the SSH deploy-key Secret — its private half had been committed in plaintext to a public remote, and a public repo needs no credential at all 2026-08-18 15:08:04 -07:00
Story Crater Bot c3ffc611f4 fix(homarr): add AUTH_OIDC_URI + email account linking — homarr hides the Authentik sign-in button unless AUTH_OIDC_URI (authorize endpoint) is set alongside AUTH_OIDC_ISSUER (per authentik/homarr SSO docs); was the missing var 2026-08-18 15:08:04 -07:00
Story Crater Bot fc10a9871a refactor(ingress): drop redundant ArgoCD ingress-nginx app — chart 4.15.1 was double-managed by both the helm-bootstrap release and this ArgoCD app (same chart), fighting over the controller/LB service (ingress-config drift). ingress-nginx is bootstrap-critical (ArgoCD's own reachability path), so helm-bootstrap is the single owner 2026-08-18 15:08:04 -07:00
Story Crater Bot 71fb7e9826 feat(sms): add BlueBubbles iMessage delivery (Docker-OSX macOS VM pinned to worker-2) + ArgoCD app + dedicated longhorn-imessage-local SC — default longhorn SC can't schedule a 3-replica 200Gi volume (only worker-1 has 200Gi free at 100% over-provisioning) and Immediate binding would pin the qcow2 to the wrong node
- namespace: PodSecurity privileged, needed for /dev/kvm + privileged QEMU
- storageclass: 1 replica, strict-local, WaitForFirstConsumer
- deployment: nodeSelector workload=imessage + matching NoSchedule toleration,
  Recreate strategy (two QEMU procs on one qcow2 corrupts it), no readiness
  probe (guest install is interactive and takes many minutes)
- services: ClusterIP only; VNC is an unauthenticated console, reach it with
  port-forward, never an Ingress
- networkpolicy: default-deny, opt-in via sms-client=true on port 1234
2026-08-18 15:08:04 -07:00
Story Crater Bot 61b906cce1 feat(monitoring): enable Alertmanager (null receiver, longhorn PVC, az-a) + fix forgejo-rules ns forgejo->cicd — alerting delivery was disabled; forgejo PrometheusRule targeted a nonexistent namespace 2026-08-18 15:08:04 -07:00
Story Crater Bot 650fcf1b61 fix(coredns): own Corefile+hostname rewrites via Talos inlineManifest (single-source terraform/files/coredns/Corefile), drop ArgoCD coredns-config app — in-cluster *.riotpiao.com now resolves to nginx ingress so MinIO/OIDC discovery works; update cp-2 IP .213->.214 2026-08-18 15:08:04 -07:00
Story Crater Bot 78c9946cf6 feat(reloader): enable autoReloadAll + reloadOnCreate — watch all workloads without per-Deployment annotations (charts like homarr don't expose them); auto-restart pods when ksops secrets are created/rotated 2026-08-18 15:08:04 -07:00
Story Crater Bot 51d938d13a chore(duckdns): remove duckdns updater entirely — superseded by cloudflared tunnel; drop app-def, manifests, kube-system Deployment 2026-08-18 15:08:04 -07:00
Story Crater Bot 13ebfe158a fix(cert-manager): cert-manager-issuers directory.include renders empty — switch to explicit resources list, restore automated sync 2026-08-18 15:08:04 -07:00
Story Crater Bot 8ff3060ff0 refactor(argocd): replace SOPS CMP with ksops kustomize generator, rotate age key — CMP discover glob silently shadowed kustomize rendering of any app whose path held a .enc.yaml (MinIO Tenant/cloudflared/authentik jobs never applied); centralize 8 Secret manifests under k8s/argocd/secrets, defer 4 helm-values fragments 2026-08-18 15:08:04 -07:00
Story Crater Bot b99eeac0e8 fix(argocd): use comma-separated include list, not brace expansion
ArgoCD directory.include uses Go filepath.Match glob syntax, not shell
brace expansion - {a,b,c} silently matched nothing, only the original 2
files stayed tracked.
2026-08-18 15:08:04 -07:00
Story Crater Bot 9257525b38 feat(cert-manager): add self-signed homelab-ca ClusterIssuer + trust bundle, fix grafana-oidc secret
homelab-ca was referenced by 6 manifests (authentik, forgejo-runner,
blackbox-exporter, management-service) as a CA trust ConfigMap but never
existed anywhere - not in git, not live in cluster. Generated a new
10-year self-signed root CA, wired it as a ClusterIssuer (cert-manager
namespace) and distributed the public cert as a ConfigMap to every
consuming namespace (iam, cicd, monitoring, sqs). Private key lives only
in the encrypted Secret. Widened cert-manager-issuers' directory include
glob rather than creating a new Application - destination.namespace is
just a fallback default on a plain directory source, not a transformer,
so it doesn't fight with each ConfigMap's own explicit namespace.

Also adds grafana-oidc secret (GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET),
same pre-existing gap as grafana-admin - was meant to come from a deleted
manual script, value already available in .env.
2026-08-18 15:08:04 -07:00
Story Crater Bot efb9389093 feat(argocd): migrate all applications from Forgejo to GitHub
- Replace all forgejo.riotpiao.com repo URLs with [email protected] SSH URLs
- Enables immediate GitOps sync without waiting for Forgejo mirror setup
- Includes ingress-nginx now fully ArgoCD-managed (wave 0)
- SOPS secrets can now sync and decrypt TLS certificates
2026-08-18 15:08:04 -07:00
Story Crater Bot 820702c748 feat(argocd): migrate ingress-nginx to full GitOps management
- Create ArgoCD Application for ingress-nginx controller (wave 0)
- Source: GitHub repo + Helm chart with local values file
- Adopts existing bootstrap Helm release (no downtime)
- Enables automated sync and self-heal for nginx configuration
2026-08-18 15:08:03 -07:00
Story Crater Bot 5b3307ffee fix(bootstrap): correct cluster config + complete Phase4 ArgoCD bootstrap permanent fixes 2026-08-18 15:08:03 -07:00
Story Crater Bot 41f5b05395 refactor(k8s): consolidate to infra/+apps/ single-source tree, dedicated per-app CNPG (authentik-db/temporal-db), wire monitoring-config, forgejo→cicd ns, drop orphan/stale (data-schemas, ollama, story-crater, sqs/argocd, key-rotation) 2026-08-18 15:08:03 -07:00
Story Crater Bot 20bd4dcf5e refactor(temporal): adopt unified CNPG pattern - use 'app' user
CHANGES:
  - temporal-values.yaml: user 'app', existingSecret 'ddb-cluster-app'
  - bootstrap.sh: Copy ddb-cluster-app to temporal namespace
  - Removed db-secret-sync directory (obsolete PostSync Job)
  - 60-applications.yaml: Removed db-secret-sync source from temporal Application

PATTERN (same as Forgejo/Authentik):
  1. Database CR: owner app
  2. bootstrap.sh: Copy ddb-cluster-app to temporal namespace
  3. App values: Reference ddb-cluster-app secret
  4. No PostSync Jobs needed

FIXES:
  - Temporal schema CrashLoopBackOff (wrong credentials)
  - Dropped/recreated databases with app owner (clean state)

Following CLAUDE.md CNPG pattern documentation.
2026-08-18 15:08:03 -07:00
Story Crater Bot e5209e3794 refactor(argocd): consolidate Applications (39→35)
Merge related Applications using multi-source pattern and PostSync hooks:

1. ingress-config ← wildcard-cert + homelab-ingress (2→1)
   - Both in k8s/bootstrap/ingress/, now use kustomization
   - Certificate deployed before Ingresses (wave 1)

2. homarr ← homarr + homarr-patches (2→1)
   - Added PostSync hook source (fix-probes-job.yaml)
   - Patches run after Helm chart deployment

3. temporal ← temporal + temporal-db-secret-sync (2→1)
   - Added PostSync hook source (copy-job.yaml)
   - DB secret sync runs after Temporal deployment

4. Removed duplicate: ingress-nginx Application
   - ingress-nginx-bootstrap (bootstrap) is working
   - Removed redundant ArgoCD-managed ingress-nginx
   - Eliminated duplicate DaemonSet

Skipped: cert-manager + cert-manager-issuers
  - Wave separation needed (CRDs before Issuers)
  - Keep separate for safety

Result: 39 → 35 Applications (-4, -10.3%)

Files:
- k8s/bootstrap/ingress/kustomization.yaml (updated)
- k8s/argocd/apps/00-substrate.yaml (merges + removal)
- k8s/argocd/apps/60-applications.yaml (merges)
- CONSOLIDATION-RESULTS.md (documentation)
- APPLICATION-CONSOLIDATION-PLAN.md (analysis)
- GITOPS-STATUS.md (updated inventory)
2026-08-18 15:08:03 -07:00
Story Crater Bot f656338a15 feat: complete GitOps migration, storage HA verification, and cluster fixes
Major accomplishments from comprehensive cluster review:

## Storage HA (answering "are volumes replicated?")
- Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas
- Fixed CLAUDE.md contradiction (sole node → 3-node HA)
- Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer)
- Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static)

## GitOps Infrastructure Cleanup
- Eliminated resource duplication (ddb-cluster single source of truth)
- Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps)
- Updated data-schemas app to point to k8s/data/schemas/ (wave 6)
- Archived old k8s/argocd/bootstrap/ → bootstrap.archived/

## Bootstrap Dependencies Fixed
- Added 05-wait-for-databases.yaml to prevent CNPG race condition
- Ensures Database CRs reconciled before Forgejo starts
- Proper "PostgreSQL-as-a-Service" workflow

## Longhorn CSI Plugin Fixed
- Added patch-csi-tolerations-job.yaml (GitOps PostSync hook)
- CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3)
- Fixes volume attachment on tainted control-plane nodes

## Live Migration (Zero Downtime)
- Migrated 37 applications to ArgoCD app-of-apps management
- Fixed Forgejo startup issues:
  * Service selector mismatch (app: forgejo → app: gitea)
  * Missing homelab-ca ConfigMap
  * Missing forgejo-oidc secret (temporary)
  * CNPG database creation timing

## Documentation (10 comprehensive files)
- WHATS-NEXT.md - Daily GitOps workflow
- MIGRATION-STATUS.md - Cluster health report
- REVIEW-SUMMARY.md - Session overview
- GITOPS-REBUILD-PLAN.md - Architecture reference
- DDB-REVIEW.md - PostgreSQL optimization guide
- STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation
- BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix
- STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale
- IMPLEMENTATION-CHECKLIST.md - Migration checklist
- bootstrap.sh - Automated bootstrap script

## Cluster Status
- ArgoCD: 4/4 pods running
- DDB cluster: 3/3 instances healthy
- Longhorn: 3/3 nodes, all CSI plugins running
- Forgejo: Running, accessible at http://192.168.1.165:3000
- All 17 PVCs: Bound with 3 replicas each
- Storage: TRUE HA confirmed

All future changes via git push only (100% GitOps).
2026-08-18 15:08:03 -07:00