Commit Graph
338 Commits
Author SHA1 Message Date
rock 28a9ef88a0 feat: add Tekton Pipelines for CI/CD orchestration
Install Tekton Pipelines via ArgoCD for Kubernetes-native CI/CD:

COMPONENTS:
- Tekton Pipelines: CNCF-standard test orchestration
- Task/Pipeline CRDs: Reusable workflow definitions
- PipelineRun: Ephemeral execution instances
- ArgoCD Application: GitOps-managed installation

INTEGRATION:
- homelab-frontend CI triggers PipelineRun via Kubernetes API
- Tests run in cluster against actual services
- Results flow back to CI for pass/fail decisions
- Image promotion only on test success

FILES:
- k8s/infra/tekton/: Tekton infrastructure setup
  - namespace.yaml: tekton-pipelines namespace
  - kustomization.yaml: Release manifest reference
- k8s/argocd/apps/06-ci-cd.yaml: ArgoCD Application
- k8s/argocd/projects/homelab-project.yaml: Added Tekton repos

WAVE ORDERING:
Wave 06 (CI/CD) is deployed after:
  - Wave 05 (Networking)
  - Wave 04 (Core components)
But before Wave 40+ (Applications)

BENEFITS:
✓ Kubernetes-native (no external CI runners)
✓ GitOps-managed (everything in git via ArgoCD)
✓ CNCF-standard (industry-proven Tekton project)
✓ Pre-merge testing (tests must pass before deploy)
✓ Observable (logs, status, results tracking)
✓ Secure (non-root, resource limits, RBAC)
2026-09-13 14:46:00 +09:00
rock cda7153da2 fix: add PAPERLESS_USERNAME env var to paperless-ai (#45)
Required by paperless-ai to find its own user ID for scanning.
Without this, scanning aborts with "Failed to get own user ID".Reviewed-on: #45

Co-authored-by: rock <[email protected]>
2026-09-13 04:58:44 +00:00
rock 0aaa45edb0 fix(comfyui): disable k8s service links to fix COMFYUI_PORT env collision
Root cause: k8s Service named 'comfyui' auto-injects COMFYUI_PORT=tcp://...
into pod env, clobbering ai-dock's own COMFYUI_PORT variable which expects
a port number. This broke caddy's proxy config, leaving port 8188 dead.

Fix: enableServiceLinks: false, revert to port 8188 (ai-dock default).
Also bump startup probe failureThreshold 60->120 (20min budget for model loading).
2026-09-13 09:29:14 +09:00
rock e4f6f03a8c fix(comfyui): correct port 8188->18188, add TLS ingress at comfyui.riotpiao.com (#44)
## Problem
ComfyUI pod restarting 273+ times over 45h. Startup probe failed every time.

## Root Cause
ai-dock image runs ComfyUI on port 18188 (adds 10000 to configured port), not 8188. Caddy does NOT proxy 8188->18188.

## Fix
- deployment: containerPort, probes -> 18188
- service: targetPort -> 18188
- ingress: TLS at comfyui.riotpiao.com, WebSocket headers
- CoreDNS: rewrite comfyui.riotpiao.com

## Post-merge
CoreDNS rewrite needs terraform apply + make apply-cp.
Cloudflare DNS: add CNAME comfyui.riotpiao.com -> tunnel.Reviewed-on: #44

Co-authored-by: rock <[email protected]>
2026-09-13 00:18:54 +00:00
rock 682138c664 fix(paperless-ai): increase memory limit 512Mi -> 2Gi (OOMKilled) 2026-09-13 08:35:55 +09:00
rock f8de5f506b feat(paperless): add paperless-ai with local LLM for auto-tagging
- paperless-ai deployment using clusterzx/paperless-ai
- LLM via local api-gateway (reasoning model, no auth - phase 3 pending)
- Paperless API token SOPS-encrypted
- Auto-tags new documents, 5min scan interval
- Adds 'ai-processed' tag to classified documents
2026-09-13 06:04:49 +09:00
rock 899e2aed2a fix: authentik probe timeouts blocking DB migration on fresh cluster
Liveness probe was firing after 60s with failureThreshold:6, killing
the server container before it finished applying 200+ DB migrations.
The startup probe (20min timeout) never got a chance to complete.

Root cause: Authentik health checks fail during long DB bootstrap.
Both liveness and startup probes run in parallel. Liveness killed the
pod at 60s; migrations need 2-3min minimum.

Solution: Add initialDelaySeconds:300 to liveness/readiness probes so
they don't fire until 5min have passed (migrations definitely complete).
Worker gets same treatment since it depends on server's DB bootstrap.
2026-09-13 06:02:28 +09:00
rock cd126d5339 feat(forgejo): enable SMTP email notifications via Gmail
- mailer config: smtp+starttls to smtp.gmail.com:587
- SMTP creds via forgejo-smtp secret (SOPS-encrypted, Gmail App Password)
- Env var injection: GITEA__MAILER__USER/PASSWD from secret
- Enables CI completion emails, password reset, repo notifications
2026-09-13 06:01:05 +09:00
rock d2fb52e425 fix(smtp): update gotify-smtp with real Gmail App Password 2026-09-13 05:57:21 +09:00
rock 03044614b7 chore: add encrypted backups and rotation schedule
- database-passwords-backup.enc.yaml: SOPS-encrypted DB credentials
- memory-agent-oidc.enc.yaml: SOPS-encrypted Poimen OIDC credentials
- oauth2-credentials.enc.yaml: SOPS-encrypted all OAuth2 secrets (6 providers)
- rotate-secrets.sh: 90-day rotation schedule (next: 2026-12-11)

These files enable full credential recovery and rotation management.
All SOPS-encrypted with cluster key for in-cluster decryption only.
Manual decryption requires ~/.sops.yaml configuration + GPG key.
2026-09-12 23:52:34 +09:00
rock 7613b4fbf2 feat(iam): rock user with Forgejo email, email password recovery, encrypt SMTP creds
- provision-rbac.py: create rock user ([email protected]) matching Forgejo
- Email recovery flow: identification -> email stage -> password reset
- SMTP via Gmail (gotify-smtp secret, SOPS-encrypted)
- Recovery flow bound to brand for login page reset link
- minio-provision-paperless: add bucket creation, use quay.io/minio/mc
2026-09-12 23:44:26 +09:00
rock 0d55e77bc4 chore: remove unused obsidian-vault PVC
No pods mount it. Obsidian integration was retired in favor of memory graph.
2026-09-12 23:18:18 +09:00
rock 92a8a9ef20 fix(ha): scale all CNPG clusters to 3 replicas, raise overprovisioning to 200%
- authentik-db: 2 → 3 instances
- gotify-db: 1 → 3 instances
- longhorn storageOverProvisioningPercentage: 100 → 200
  (actual disk usage ~44/474 GB on cp-1; thin provisioning is safe)
  Unblocks 3rd Longhorn replica scheduling on cp-1 which had DiskPressure
2026-09-12 19:02:47 +09:00
rock ad3b4f40b3 fix(longhorn): remove invalid ext4 mount options uid=26,gid=26 from longhorn-cnpg
uid/gid are FAT/NTFS/FUSE mount options, not ext4. Caused mount exit 32
on every newly provisioned CNPG PVC, blocking authentik-db initdb.
CNPG handles postgres ownership via its own init container.
2026-09-12 18:49:21 +09:00
rock e4f75bde90 fix(databases): Scale authentik-db to 2 replicas (production-safe HA) 2026-09-12 18:18:41 +09:00
rock 29ab21efff fix(databases): Scale authentik-db to 3 replicas for HA (fresh clean cluster) 2026-09-12 18:10:12 +09:00
rock 8e6ebed984 restore: authentik-db cluster (was accidentally deleted) 2026-09-12 18:07:57 +09:00
rock afc023ff2b temp: remove authentik-db from kustomization (corruption recovery) 2026-09-12 18:04:38 +09:00
rock 707271fab0 temp: remove authentik-db cluster (corruption recovery) 2026-09-12 18:03:14 +09:00
rock 1d47234f12 fix(databases): Scale authentik-db and gotify-db to 1 replica (Longhorn corruption recovery) 2026-09-12 17:27:45 +09:00
rock 43fec41ec4 feat: scale memory-db to 3 replicas for HA
Update CNPG Cluster instances from 2 to 3 for high availability.
Ensures quorum majority for failover and maintains consistency.
2026-09-12 04:52:18 +09:00
rock 2e99c3cc7e Merge branch 'feature/37-vllm-prometheus-scraping' 2026-09-12 04:52:14 +09:00
rock 4b63e809e2 fix(argocd-image-updater): remove duplicate ARGOCD_GRPC_WEB env var
Fixes sync error: 'may not be specified when value is not empty'
The Helm chart already defines ARGOCD_GRPC_WEB, so remove from extraEnv.
2026-09-11 11:06:05 +09:00
rock 179e12b9a9 feat: vLLM Prometheus metrics scraping (#37)
Add ServiceMonitor for llm-serving namespace. Wire prometheus.io annotations to all vLLM pods (reasoning, ornith, embeddings, reranker). Scrape /metrics@8080 every 30s with proper relabeling.
2026-09-11 10:46:15 +09:00
rock 7bd9f83fa7 fix(cnpg): scale paperless-db, immich-db, gotify-db to 3 instances (#30)
Co-authored-by: rock <[email protected]>
2026-09-10 22:56:41 +00:00
rock 9d3a669dfe fix(gotify): move SOPS secrets to ksops generator
ArgoCD couldn't decrypt secrets.yaml because it was listed as a plain
kustomize resource. Move the 3 secrets (gotify-admin, gotify-tokens,
gotify-smtp) to k8s/argocd/secrets/ as .enc.yaml files processed by
the ksops generator, matching the repo convention.

Fixes ComparisonError: 'Object Kind is missing' (SOPS ciphertext
parsed as raw YAML).
2026-09-10 22:32:24 +09:00
rock 436c7d8d42 fix(argocd): update git repoURLs for org transfer rock -> riotpiao-poimen
Repos transferred: homelab-frontend, kmsvc-manage, poimen, poimen-memory,
poimen-workflows, poimen-frontend. Old URLs return 301 which ArgoCD
doesn't follow.

NOT changed: container image registry paths (rock/ is correct for registry),
riotpiao.com (still under rock org).

Also adds poimen-frontend to AppProject sourceRepos allowlist.
2026-09-10 10:49:30 +09:00
rock b571d518e0 fix(argocd): update repoURL after org transfer (#29)
Co-authored-by: rock <[email protected]>
2026-09-10 01:41:37 +00:00
rock 128e76ce2d feat(gotify): push notification server + SMTP email relay (#19)
Co-authored-by: rock <[email protected]>
2026-09-10 01:35:59 +00:00
rock 8790de6038 feat: add ComfyUI + rebalance GPU allocation (#17)
## GPU Rebalance (4× V100 32GB)

| Pod | Before | After |
|-----|--------|-------|
| reasoning (PP=2) | 2 GPU | 2 GPU |
| ornith | 2 GPU (2 replicas) | 1 GPU (1 replica) |
| comfyui | — | 1 GPU (**new**) |
| qwen-cpu | — | CPU on cp-2 (**new**) |
| embeddings/reranker | CPU | CPU |

## Changes

- `ornith.yaml`: scale 2→1, remove qwen2.5 co-loading, MAX_LOADED_MODELS=1
- `qwen-cpu.yaml`: new Ollama deployment on talos-cp-2 (144GB RAM), 5Gi PVC
- `k8s/apps/comfyui/`: new ComfyUI deployment (1 GPU, 50Gi model PVC, ingress)
- `58-comfyui.yaml`: ArgoCD Application (wave 8)

Gateway route update in separate PR (homelab-frontend).Reviewed-on: #17

Co-authored-by: rock <[email protected]>
2026-09-09 02:11:28 +00:00
rock 95c67f9437 P3.8: Restrict llm-serving ingress to api-gateway only (#15)
## Summary

Replaces hand-applied `llm-serving-default-deny` NetworkPolicy with a git-managed, namespace-scoped policy that only allows traffic from the api-gateway.

**Issue:** #13

Co-authored-by: rock <[email protected]>
2026-09-08 16:56:36 +00:00
rock 1630704f8b feat: switch image-updater to digest-based :latest tracking
All image-updater annotations now use update-strategy: digest with
allow-tags: ^latest$ and write-back-method: argocd. No SHA tags
committed to git — digest overrides stored in ArgoCD state only.

- poimen: git write-back → argocd, removed git-branch
- portfolio: newest-build SHA → digest latest
- api-gateway: newest-build SHA → digest latest
2026-09-07 18:19:46 -07:00
rock ce1539a634 revert: remove unsupported buildOptions (ArgoCD v3.4.5 doesn't support it)
- buildOptions field not available in ArgoCD v3.4.5
- SOPS decryption already handled by repo-server ksops plugin
- Revert to simple kustomize config
- Portfolio Application can now sync properly
2026-09-07 17:49:58 -07:00
rock 2799e3a675 fix: enable ksops plugin for portfolio Application
- Add kustomize config with --enable-alpha-plugins to support ksops
- Allows ArgoCD to properly decrypt SOPS-encrypted files
- Fixes Image Updater compatibility with sops field in kustomization.yaml
2026-09-07 17:47:03 -07:00
rock 1da7e0aa4d fix: wait for dind to be ready before starting runner daemon 2026-09-07 13:22:09 -07:00
rock 8970e35539 fix: use tcp://localhost:2375 for dind (no TLS, no socket permission issues) 2026-09-07 13:20:20 -07:00
rock 92239561cd fix: run runner as root to access dind socket 2026-09-07 13:18:54 -07:00
rock 2d92383951 fix: runner uses unix socket instead of TLS TCP for dind
Job containers spawned by the runner run inside dind. With TCP+TLS
(tcp://localhost:2376), localhost inside those containers doesn't
reach the dind sidecar. Unix socket at /run/docker.sock works because
both runner and dind share the /run emptyDir.

Also disables DOCKER_TLS_CERTDIR so dind creates the socket instead
of only listening on TLS TCP.
2026-09-07 13:16:49 -07:00
rock 7d77935d15 ci: fix runner labels + CoreDNS rewrite + cleanup
- Runners use public images (code.forgejo.org/forgejo/runner:6)
- Labels pull from Docker Hub: golang:1.26, node:22, rust:1-bookworm
- Add CoreDNS api.riotpiao.com rewrite
- Fix runner re-registration to keep labels in sync
- Add unified CI pattern docs to CLAUDE.example.md
- Remove dead .forgejo/ workflow dir (Forgejo uses .gitea/)
2026-09-07 13:01:56 -07:00
rock fee4f9edfc ci: fix golang runner - use docker:27-cli (has Node.js + golang + git)
Previous image (golang:1.26-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'

Solution:
- Change golang runner image from golang:1.26-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, Go toolchain, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓

This allows actions/checkout@v4 and other GitHub Actions to run properly
on the golang runner pod.

Note: node:22-bookworm runner already has Node.js, no change needed.
2026-09-05 22:50:43 -07:00
rock 12fc2796e2 ci: fix rust runner - use docker:27-cli (has Node.js + git + docker)
Previous image (rust:1.83-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'

Solution:
- Change rust runner image from rust:1.83-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓

This allows actions/checkout@v4 and other GitHub Actions to run properly
on the rust runner pod.
2026-09-05 22:49:25 -07:00
rock f0976bfc61 fix: remove spec wrapper from ClusterRoleBinding
- ClusterRoleBinding doesn't use spec: wrapper (unlike Deployment/StatefulSet)
- roleRef and subjects go at top level with metadata
- Fixes: 'strict decoding error: unknown field "spec"'
2026-09-05 15:04:00 -07:00
rock 4a4e57d0f2 fix: remove namespace from rbac Application destination
- RBAC kustomization contains cluster-scoped (ClusterRoleBinding) and
  namespace-scoped (Role/RoleBinding) resources
- Each resource has explicit metadata.namespace, so Application shouldn't
  force a default namespace
- Fixes: ClusterRoleBinding gets namespace=default, causing sync failure
  with 'unsupported role reference kind: ""'
2026-09-05 14:53:59 -07:00
rock 4fc833f9b2 fix: remove backslash line continuations from YAML multiline string
- YAML block scalars (|) don't use backslash continuation
- Just indent lines properly, block scalar handles them automatically
- Fixes ArgoCD ComparisonError on poimen app
2026-09-05 14:48:21 -07:00
rock 647fba8814 fix: add admin-oidc-binding to kustomization resources
- admin-oidc-binding.yaml wasn't listed in resources
- Now kustomize will include it when building manifests
- ArgoCD can sync the OIDC group binding
2026-09-05 14:42:17 -07:00
rock bcae41e338 chore: revert node-runner to stock image, remove Dockerfile
- Reverted to node:22-bookworm (no custom image)
- Removed Dockerfile.node (no CI to build it)
- Docker install step in riotpiao workflow is already the workaround
2026-09-05 14:33:27 -07:00
rock 1425ab7cbc chore: remove homelab CI workflow
- Removed .gitea/workflows/build-runner-node.yml
- Homelab is GitOps only, not a buildable artifact
- Runner images managed via direct Dockerfile edits + manual pushes
2026-09-05 14:33:13 -07:00
rock 978f9c8147 feat: add OIDC group binding for cluster-admin access
- Binds oidc:homelab-admins group to cluster-admin ClusterRole
- Allows OIDC users (via Authentik) to have admin access
- Groups claim from Authentik with oidc: prefix per kube-apiserver config
- Enables kubectl access via 'kubectl login' + kubelogin
2026-09-05 14:30:41 -07:00
rock 4193c8ab99 feat: custom forgejo-runner-node image with docker.io pre-installed
- Dockerfile.node extends node:22-bookworm + docker.io
- CI workflow builds and pushes to forgejo registry on changes
- values-node.yaml references custom image instead of stock node
- Removes need to install docker in every workflow using node runner
2026-09-05 14:20:42 -07:00
rock 2c011e08e2 feat: Image Updater git write-back for multi-source poimen Application
- write-back-method: git (commits image updates back to repos)
- git-branch: main
- Mounts ArgoCD SSH credentials for git pushes
- Image Updater commits new SHAs → repos → ArgoCD syncs
2026-09-05 13:56:40 -07:00