rock/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
249 Commits 3 Branches 0 Tags
9eb5c8ea1a57eba980326998a490410d8d0d4096
Commit Graph
75 Commits
Author SHA1 Message Date
Story Crater Bot 9eb5c8ea1a Remove separate memory app, bundle into wave 2 databases 2026-08-22 23:16:40 -07:00
Story Crater Bot aa6b2ae9c3 Add Poimen Memory to ArgoCD wave 2 deployment (namespace: poimen) 2026-08-22 23:09:02 -07:00
Story Crater Bot 06855b3768 fix(argocd): update poimen repoURL after rock/poiman rename to rock/poimen 
The poiman repo was renamed to poimen on Forgejo; the stale repoURL made
poimen-root fail with a 301 redirect ComparisonError (ArgoCD's git
client doesn't follow redirects on smart-HTTP fetch), blocking sync for
poimen-root and everything under it.
 2026-08-21 21:57:36 -07:00
Story Crater Bot 2f5793899b fix(argocd): poimen-root point to k8s/argocd directory 2026-08-21 21:28:18 -07:00
Story Crater Bot 2a1cb77443 fix(argocd): poimen-root point to k8s/argocd/apps like kmsvc-root 2026-08-21 21:27:38 -07:00
Story Crater Bot 9415a30309 fix(argocd): poimen-root use single source from poiman, remove workflows 2026-08-21 21:25:55 -07:00
Story Crater Bot a63838bae3 feat(argocd): enable poimen-root Application for poiman orchestration 2026-08-21 21:24:12 -07:00
Story Crater Bot 0eb0dd89f6 chore(argocd): track main branch instead of prod for auto-sync on every commit 2026-08-21 20:55:20 -07:00
Story Crater Bot 7a7e0fe813 chore(argocd): update api-gateway to track homelab-frontend prod branch 2026-08-21 20:47:38 -07:00
Story Crater Bot 9e5e51733d chore(argocd): add poimen application placeholder 2026-08-21 20:44:49 -07:00
Story Crater Bot 544fa371f6 chore(argocd): add kmsvc-manage bootstrap application 2026-08-21 20:44:35 -07:00
Story Crater Bot dbc4a55b02 feat(forgejo-runner): split into golang/node/rust runners, retire generic docker runner 2026-08-21 16:49:26 -07:00
Story Crater Bot 130746e6a1 Add Temporal worker for production task queue 2026-08-21 16:44:59 -07:00
Story Crater Bot bedf062906 chore(forgejo-runner): arm for cascading delete ahead of 3-runner migration 2026-08-21 16:35:01 -07:00
Story Crater Bot c938a58544 stage1: A1-A2 AppProject and projects Application 
A1: Replace per-repo Forgejo entries with https://forgejo.riotpiao.com/rock/*
    wildcard so onboarding never requires touching AppProject.

A2: Add wave -1 Application for k8s/argocd/projects/ so it syncs before
    any Application references the AppProject.

Also add kustomization.yaml to k8s/argocd/projects/ to make it renderable.

Enabled by Stage 1 (A1, A2).
 2026-08-20 21:31:05 -07:00
Story Crater Bot e669ee0ec2 pi-models: fix baseUrl to match homelab-frontend gateway contract 
Kong was retired 2026-08-19, replaced by the rock/homelab-frontend Go
gateway (single /v1/chat/completions endpoint, model routed via the
request body's "model" field per API.md). Old per-model baseUrls
(/v1/ornith, /v1/reasoning, /v1/qwen) all 404 against the new gateway.
Also flipping reasoning's supportsTools to true -- confirmed working via
live test now that reasoning runs Qwen3-32B instead of DeepSeek-R1.
 2026-08-20 00:26:52 -07:00
Story Crater Bot 05b7a847e3 chore: drop the Kong key-auth credential secret, unused now that Kong is gone 2026-08-19 23:40:50 -07:00
Story Crater Bot 7e40a4baf6 feat: cut api.riotpiao.com over to the Go gateway and retire Kong 
Ingress api/api now backs onto api-gateway:8080; the kong Application, its
Helm values, plugins and llm-routes are removed. Gateway image v0.0.0 is in
the Forgejo registry and the pull secret is in the api namespace.
 2026-08-19 22:51:53 -07:00
Story Crater Bot f8136f4e1e fix: ignore Reloader's injected env var on the Forgejo Deployment 
Argo would otherwise strip STAKATER_* on each sync and fight Reloader for it,
recreating the forge pod every reconcile.
 2026-08-19 22:43:29 -07:00
Story Crater Bot df5e623347 feat: manage Forgejo with Argo instead of the bootstrap Helm release 
Values changes were inert as a bootstrap release, so the proxy-body-size fix
never reached the live Ingress. First sync is manual — the chart owns the
Forgejo PVC.
 2026-08-19 22:37:56 -07:00
Story Crater Bot 720181c900 feat: let the runner build and the cluster pull from the Forgejo registry 
- Runner egress: allow 192.168.1.160/32:443. forgejo.riotpiao.com resolves to
  the ingress LB, inside the 192.168.1.0/24 block the NetworkPolicy denies, so
  docker push hung until timeout.
- dind CA: also mount homelab-ca at /etc/docker/certs.d/forgejo.riotpiao.com/,
  the path dockerd actually reads for per-registry trust.
- Pull secret: dockerconfigjson for the api namespace; /v2/ answers 401.
- AppProject: allow the Forgejo repo as a source for api-gw.
 2026-08-19 21:48:01 -07:00
Story Crater Bot afb9b35292 fix(agent-pod): remote tui session for multi-agent 2026-08-18 15:08:04 -07:00
Story Crater Bot 5f16f16f0f fix(argocd): clone the public GitHub seed anonymously over HTTPS and delete the SSH deploy-key Secret — its private half had been committed in plaintext to a public remote, and a public repo needs no credential at all 2026-08-18 15:08:04 -07:00
Story Crater Bot c3ffc611f4 fix(homarr): add AUTH_OIDC_URI + email account linking — homarr hides the Authentik sign-in button unless AUTH_OIDC_URI (authorize endpoint) is set alongside AUTH_OIDC_ISSUER (per authentik/homarr SSO docs); was the missing var 2026-08-18 15:08:04 -07:00
Story Crater Bot fc10a9871a refactor(ingress): drop redundant ArgoCD ingress-nginx app — chart 4.15.1 was double-managed by both the helm-bootstrap release and this ArgoCD app (same chart), fighting over the controller/LB service (ingress-config drift). ingress-nginx is bootstrap-critical (ArgoCD's own reachability path), so helm-bootstrap is the single owner 2026-08-18 15:08:04 -07:00
Story Crater Bot 71fb7e9826 feat(sms): add BlueBubbles iMessage delivery (Docker-OSX macOS VM pinned to worker-2) + ArgoCD app + dedicated longhorn-imessage-local SC — default longhorn SC can't schedule a 3-replica 200Gi volume (only worker-1 has 200Gi free at 100% over-provisioning) and Immediate binding would pin the qcow2 to the wrong node 
- namespace: PodSecurity privileged, needed for /dev/kvm + privileged QEMU
- storageclass: 1 replica, strict-local, WaitForFirstConsumer
- deployment: nodeSelector workload=imessage + matching NoSchedule toleration,
  Recreate strategy (two QEMU procs on one qcow2 corrupts it), no readiness
  probe (guest install is interactive and takes many minutes)
- services: ClusterIP only; VNC is an unauthenticated console, reach it with
  port-forward, never an Ingress
- networkpolicy: default-deny, opt-in via sms-client=true on port 1234
 2026-08-18 15:08:04 -07:00
Story Crater Bot 61b906cce1 feat(monitoring): enable Alertmanager (null receiver, longhorn PVC, az-a) + fix forgejo-rules ns forgejo->cicd — alerting delivery was disabled; forgejo PrometheusRule targeted a nonexistent namespace 2026-08-18 15:08:04 -07:00
Story Crater Bot 6401652aa8 feat(forgejo): add runner-token Secret via ksops — forgejo-runner register initContainer needs the registration token (from gitea actions generate-runner-token); was missing so runner deploy stuck 0/1 2026-08-18 15:08:04 -07:00
Story Crater Bot 650fcf1b61 fix(coredns): own Corefile+hostname rewrites via Talos inlineManifest (single-source terraform/files/coredns/Corefile), drop ArgoCD coredns-config app — in-cluster *.riotpiao.com now resolves to nginx ingress so MinIO/OIDC discovery works; update cp-2 IP .213->.214 2026-08-18 15:08:04 -07:00
Story Crater Bot 78c9946cf6 feat(reloader): enable autoReloadAll + reloadOnCreate — watch all workloads without per-Deployment annotations (charts like homarr don't expose them); auto-restart pods when ksops secrets are created/rotated 2026-08-18 15:08:04 -07:00
Story Crater Bot 1a1edcfc27 fix(homarr): add auth-oidc-secret + db-encryption Secrets via ksops — homarr chart's envSecrets expect these exact names (oidc-client-id/secret, db-encryption-key); were never created so homarr CreateContainerConfigError 2026-08-18 15:08:04 -07:00
Story Crater Bot 51d938d13a chore(duckdns): remove duckdns updater entirely — superseded by cloudflared tunnel; drop app-def, manifests, kube-system Deployment 2026-08-18 15:08:04 -07:00
Story Crater Bot 6057b64509 fix(cert-manager): regenerate homelab-ca cert with basicConstraints CA:TRUE — old self-signed cert lacked CA:TRUE so the homelab-ca ClusterIssuer rejected it ('certificate is not a CA'); regen keypair Secret + trust-bundle ConfigMaps (4 ns) with matching CA cert 2026-08-18 15:08:04 -07:00
Story Crater Bot b66c5f4916 fix: deploy authentik/loki/vault Secrets via ksops (were dead helm-values fragments, causing CreateContainerConfigError) 2026-08-18 15:08:04 -07:00
Story Crater Bot 13ebfe158a fix(cert-manager): cert-manager-issuers directory.include renders empty — switch to explicit resources list, restore automated sync 2026-08-18 15:08:04 -07:00
Story Crater Bot 8ff3060ff0 refactor(argocd): replace SOPS CMP with ksops kustomize generator, rotate age key — CMP discover glob silently shadowed kustomize rendering of any app whose path held a .enc.yaml (MinIO Tenant/cloudflared/authentik jobs never applied); centralize 8 Secret manifests under k8s/argocd/secrets, defer 4 helm-values fragments 2026-08-18 15:08:04 -07:00
Story Crater Bot b99eeac0e8 fix(argocd): use comma-separated include list, not brace expansion 
ArgoCD directory.include uses Go filepath.Match glob syntax, not shell
brace expansion - {a,b,c} silently matched nothing, only the original 2
files stayed tracked.
 2026-08-18 15:08:04 -07:00
Story Crater Bot 9257525b38 feat(cert-manager): add self-signed homelab-ca ClusterIssuer + trust bundle, fix grafana-oidc secret 
homelab-ca was referenced by 6 manifests (authentik, forgejo-runner,
blackbox-exporter, management-service) as a CA trust ConfigMap but never
existed anywhere - not in git, not live in cluster. Generated a new
10-year self-signed root CA, wired it as a ClusterIssuer (cert-manager
namespace) and distributed the public cert as a ConfigMap to every
consuming namespace (iam, cicd, monitoring, sqs). Private key lives only
in the encrypted Secret. Widened cert-manager-issuers' directory include
glob rather than creating a new Application - destination.namespace is
just a fallback default on a plain directory source, not a transformer,
so it doesn't fight with each ConfigMap's own explicit namespace.

Also adds grafana-oidc secret (GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET),
same pre-existing gap as grafana-admin - was meant to come from a deleted
manual script, value already available in .env.
 2026-08-18 15:08:04 -07:00
Story Crater Bot efb9389093 feat(argocd): migrate all applications from Forgejo to GitHub 
- Replace all forgejo.riotpiao.com repo URLs with [email protected] SSH URLs
- Enables immediate GitOps sync without waiting for Forgejo mirror setup
- Includes ingress-nginx now fully ArgoCD-managed (wave 0)
- SOPS secrets can now sync and decrypt TLS certificates
 2026-08-18 15:08:04 -07:00
Story Crater Bot 820702c748 feat(argocd): migrate ingress-nginx to full GitOps management 
- Create ArgoCD Application for ingress-nginx controller (wave 0)
- Source: GitHub repo + Helm chart with local values file
- Adopts existing bootstrap Helm release (no downtime)
- Enables automated sync and self-heal for nginx configuration
 2026-08-18 15:08:03 -07:00
Story Crater Bot 5b3307ffee fix(bootstrap): correct cluster config + complete Phase4 ArgoCD bootstrap permanent fixes 2026-08-18 15:08:03 -07:00
Story Crater Bot 41f5b05395 refactor(k8s): consolidate to infra/+apps/ single-source tree, dedicated per-app CNPG (authentik-db/temporal-db), wire monitoring-config, forgejo→cicd ns, drop orphan/stale (data-schemas, ollama, story-crater, sqs/argocd, key-rotation) 2026-08-18 15:08:03 -07:00
Story Crater Bot 20bd4dcf5e refactor(temporal): adopt unified CNPG pattern - use 'app' user 
CHANGES:
  - temporal-values.yaml: user 'app', existingSecret 'ddb-cluster-app'
  - bootstrap.sh: Copy ddb-cluster-app to temporal namespace
  - Removed db-secret-sync directory (obsolete PostSync Job)
  - 60-applications.yaml: Removed db-secret-sync source from temporal Application

PATTERN (same as Forgejo/Authentik):
  1. Database CR: owner app
  2. bootstrap.sh: Copy ddb-cluster-app to temporal namespace
  3. App values: Reference ddb-cluster-app secret
  4. No PostSync Jobs needed

FIXES:
  - Temporal schema CrashLoopBackOff (wrong credentials)
  - Dropped/recreated databases with app owner (clean state)

Following CLAUDE.md CNPG pattern documentation.
 2026-08-18 15:08:03 -07:00
Story Crater Bot fef07c058a refactor(argocd): remove orphaned infrastructure Applications - bootstrap is source of truth 
REMOVED ORPHANED APPLICATIONS:
  - cnpg-operator (OutOfSync, conflicted with bootstrap)
  - forgejo (OutOfSync, conflicted with bootstrap)
  - ingress-nginx-bootstrap (orphaned, no ownerReferences)

ARCHITECTURE NOW CLEAN:
  ✅ Bootstrap: 7 manifests (infrastructure base for regional deployment)
     - ArgoCD, CNPG operator, DDB, Forgejo, ingress-nginx, namespaces, wait-for-databases
  ✅ ArgoCD: 32 Applications (all services/apps)
  ✅ No duplicate management

DEPLOYMENT FLOW:
  1. kubectl apply -k k8s/bootstrap-local/ (infrastructure)
  2. kubectl apply -k k8s/argocd/root/ (app-of-apps)
  3. ArgoCD auto-syncs from Forgejo (applications)

CLEANUP:
  - Archived old bootstrap configs (k8s/argocd/bootstrap.archived/)
  - Deleted orphaned Applications (ArgoCD tracking only, resources untouched)

Bootstrap remains single source of truth for infrastructure.
ArgoCD manages all applications and services.
 2026-08-18 15:08:03 -07:00
Story Crater Bot e5209e3794 refactor(argocd): consolidate Applications (39→35) 
Merge related Applications using multi-source pattern and PostSync hooks:

1. ingress-config ← wildcard-cert + homelab-ingress (2→1)
   - Both in k8s/bootstrap/ingress/, now use kustomization
   - Certificate deployed before Ingresses (wave 1)

2. homarr ← homarr + homarr-patches (2→1)
   - Added PostSync hook source (fix-probes-job.yaml)
   - Patches run after Helm chart deployment

3. temporal ← temporal + temporal-db-secret-sync (2→1)
   - Added PostSync hook source (copy-job.yaml)
   - DB secret sync runs after Temporal deployment

4. Removed duplicate: ingress-nginx Application
   - ingress-nginx-bootstrap (bootstrap) is working
   - Removed redundant ArgoCD-managed ingress-nginx
   - Eliminated duplicate DaemonSet

Skipped: cert-manager + cert-manager-issuers
  - Wave separation needed (CRDs before Issuers)
  - Keep separate for safety

Result: 39 → 35 Applications (-4, -10.3%)

Files:
- k8s/bootstrap/ingress/kustomization.yaml (updated)
- k8s/argocd/apps/00-substrate.yaml (merges + removal)
- k8s/argocd/apps/60-applications.yaml (merges)
- CONSOLIDATION-RESULTS.md (documentation)
- APPLICATION-CONSOLIDATION-PLAN.md (analysis)
- GITOPS-STATUS.md (updated inventory)
 2026-08-18 15:08:03 -07:00
Story Crater Bot f656338a15 feat: complete GitOps migration, storage HA verification, and cluster fixes 
Major accomplishments from comprehensive cluster review:

## Storage HA (answering "are volumes replicated?")
- Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas
- Fixed CLAUDE.md contradiction (sole node → 3-node HA)
- Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer)
- Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static)

## GitOps Infrastructure Cleanup
- Eliminated resource duplication (ddb-cluster single source of truth)
- Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps)
- Updated data-schemas app to point to k8s/data/schemas/ (wave 6)
- Archived old k8s/argocd/bootstrap/ → bootstrap.archived/

## Bootstrap Dependencies Fixed
- Added 05-wait-for-databases.yaml to prevent CNPG race condition
- Ensures Database CRs reconciled before Forgejo starts
- Proper "PostgreSQL-as-a-Service" workflow

## Longhorn CSI Plugin Fixed
- Added patch-csi-tolerations-job.yaml (GitOps PostSync hook)
- CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3)
- Fixes volume attachment on tainted control-plane nodes

## Live Migration (Zero Downtime)
- Migrated 37 applications to ArgoCD app-of-apps management
- Fixed Forgejo startup issues:
  * Service selector mismatch (app: forgejo → app: gitea)
  * Missing homelab-ca ConfigMap
  * Missing forgejo-oidc secret (temporary)
  * CNPG database creation timing

## Documentation (10 comprehensive files)
- WHATS-NEXT.md - Daily GitOps workflow
- MIGRATION-STATUS.md - Cluster health report
- REVIEW-SUMMARY.md - Session overview
- GITOPS-REBUILD-PLAN.md - Architecture reference
- DDB-REVIEW.md - PostgreSQL optimization guide
- STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation
- BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix
- STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale
- IMPLEMENTATION-CHECKLIST.md - Migration checklist
- bootstrap.sh - Automated bootstrap script

## Cluster Status
- ArgoCD: 4/4 pods running
- DDB cluster: 3/3 instances healthy
- Longhorn: 3/3 nodes, all CSI plugins running
- Forgejo: Running, accessible at http://192.168.1.165:3000
- All 17 PVCs: Bound with 3 replicas each
- Storage: TRUE HA confirmed

All future changes via git push only (100% GitOps).
 2026-08-18 15:08:03 -07:00
Story Crater Bot d394efc78a fix(homarr): probe tuning via dedicated PostSync patch-job Application (chart-values/Kustomize-patch/controller.probes attempts superseded) 2026-08-18 15:08:03 -07:00
Story Crater Bot ab2f1eeeb3 feat(homarr): bring up chart (repo/sourceRepos, image tag, minimal values schema) 2026-08-18 15:08:03 -07:00
Story Crater Bot 86f94f96fd feat(homarr): complete wiring for landing page deployment 
Adds Homarr landing page with Authentik SSO:
- k8s/argocd/apps/60-applications.yaml: multi-source Application (homarr
  chart from homarr-labs + in-repo values), ns dashboard, wave 8
- k8s/bootstrap/ingress/ingress.yaml: homarr.riotpiao.com → dashboard/homarr:3000
- k8s/bootstrap/coredns/coredns-configmap.yaml: rewrite homarr.riotpiao.com
  to ingress controller
- k8s/security/iam/scripts/authentik-provision.py: added 'homarr' to SERVICES
  (generates OAuth provider/app + homarr-oidc secret with client-id/secret)
- k8s/security/iam/rbac-dashboard-rolebinding.yaml: grants authentik-provisioner
  SA access to dashboard ns for secret management
- k8s/security/iam/kustomization.yaml: includes new RoleBinding

Homarr now fully wired:
- Ingress: https://homarr.riotpiao.com
- SSO: redirects to Authentik, login as rock
- Persistence: 5Gi RWO on longhorn-wffc (3-replica HA)
- Tile config: UI-managed (saved to PVC)
 2026-08-18 15:08:03 -07:00
Story Crater Bot 2458da8e91 fix(forgejo): remove nodeSelector now that Longhorn runs on all nodes 
With Longhorn now running on all 3 control-plane nodes (commit be7881d),
Forgejo pods no longer need to be pinned to talos-cp-1. The gitea-shared-storage
PVC can attach on any node, and the scheduler will properly co-locate pod + volume
via WaitForFirstConsumer + 3-replica Longhorn volumes.

Removes the kubernetes.io/hostname: talos-cp-1 nodeSelector added in commit
dde4b60 (which was a workaround for single-node storage).
 2026-08-18 15:08:03 -07:00