Commit Graph
353 Commits
Author SHA1 Message Date
rockandpoimen 39f5fe3683 feat: add ComfyUI, rebalance GPU allocation
GPU rebalance (4× V100 32GB on worker-1):
- reasoning: 2 GPU (unchanged, PP=2 for Qwen3-32B)
- ornith: 2→1 GPU (scale to 1 replica, ornith:35b only)
- comfyui: 0→1 GPU (new)
- embeddings/reranker: 0 GPU (CPU, unchanged)

qwen2.5:3b-instruct moved to CPU on talos-cp-2 (144GB RAM).
Separate Ollama deployment + 5Gi PVC, pulls model on first start.

Gateway config updated in homelab-frontend (separate commit).

Co-authored-by: poimen <[email protected]>
2026-09-08 18:26:07 -07:00
rock c5aadd98f8 chore: gitignore IAM provisioning scripts 2026-09-08 15:29:42 -07:00
rock 95c67f9437 P3.8: Restrict llm-serving ingress to api-gateway only (#15)
## Summary

Replaces hand-applied `llm-serving-default-deny` NetworkPolicy with a git-managed, namespace-scoped policy that only allows traffic from the api-gateway.

**Issue:** #13

Co-authored-by: rock <[email protected]>
2026-09-08 16:56:36 +00:00
rock 1630704f8b feat: switch image-updater to digest-based :latest tracking
All image-updater annotations now use update-strategy: digest with
allow-tags: ^latest$ and write-back-method: argocd. No SHA tags
committed to git — digest overrides stored in ArgoCD state only.

- poimen: git write-back → argocd, removed git-branch
- portfolio: newest-build SHA → digest latest
- api-gateway: newest-build SHA → digest latest
2026-09-07 18:19:46 -07:00
rock ce1539a634 revert: remove unsupported buildOptions (ArgoCD v3.4.5 doesn't support it)
- buildOptions field not available in ArgoCD v3.4.5
- SOPS decryption already handled by repo-server ksops plugin
- Revert to simple kustomize config
- Portfolio Application can now sync properly
2026-09-07 17:49:58 -07:00
rock 2799e3a675 fix: enable ksops plugin for portfolio Application
- Add kustomize config with --enable-alpha-plugins to support ksops
- Allows ArgoCD to properly decrypt SOPS-encrypted files
- Fixes Image Updater compatibility with sops field in kustomization.yaml
2026-09-07 17:47:03 -07:00
rock 1da7e0aa4d fix: wait for dind to be ready before starting runner daemon 2026-09-07 13:22:09 -07:00
rock 8970e35539 fix: use tcp://localhost:2375 for dind (no TLS, no socket permission issues) 2026-09-07 13:20:20 -07:00
rock 92239561cd fix: run runner as root to access dind socket 2026-09-07 13:18:54 -07:00
rock 2d92383951 fix: runner uses unix socket instead of TLS TCP for dind
Job containers spawned by the runner run inside dind. With TCP+TLS
(tcp://localhost:2376), localhost inside those containers doesn't
reach the dind sidecar. Unix socket at /run/docker.sock works because
both runner and dind share the /run emptyDir.

Also disables DOCKER_TLS_CERTDIR so dind creates the socket instead
of only listening on TLS TCP.
2026-09-07 13:16:49 -07:00
rock 7d77935d15 ci: fix runner labels + CoreDNS rewrite + cleanup
- Runners use public images (code.forgejo.org/forgejo/runner:6)
- Labels pull from Docker Hub: golang:1.26, node:22, rust:1-bookworm
- Add CoreDNS api.riotpiao.com rewrite
- Fix runner re-registration to keep labels in sync
- Add unified CI pattern docs to CLAUDE.example.md
- Remove dead .forgejo/ workflow dir (Forgejo uses .gitea/)
2026-09-07 13:01:56 -07:00
rock fee4f9edfc ci: fix golang runner - use docker:27-cli (has Node.js + golang + git)
Previous image (golang:1.26-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'

Solution:
- Change golang runner image from golang:1.26-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, Go toolchain, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓

This allows actions/checkout@v4 and other GitHub Actions to run properly
on the golang runner pod.

Note: node:22-bookworm runner already has Node.js, no change needed.
2026-09-05 22:50:43 -07:00
rock 12fc2796e2 ci: fix rust runner - use docker:27-cli (has Node.js + git + docker)
Previous image (rust:1.83-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'

Solution:
- Change rust runner image from rust:1.83-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓

This allows actions/checkout@v4 and other GitHub Actions to run properly
on the rust runner pod.
2026-09-05 22:49:25 -07:00
rock f0976bfc61 fix: remove spec wrapper from ClusterRoleBinding
- ClusterRoleBinding doesn't use spec: wrapper (unlike Deployment/StatefulSet)
- roleRef and subjects go at top level with metadata
- Fixes: 'strict decoding error: unknown field "spec"'
2026-09-05 15:04:00 -07:00
rock 4a4e57d0f2 fix: remove namespace from rbac Application destination
- RBAC kustomization contains cluster-scoped (ClusterRoleBinding) and
  namespace-scoped (Role/RoleBinding) resources
- Each resource has explicit metadata.namespace, so Application shouldn't
  force a default namespace
- Fixes: ClusterRoleBinding gets namespace=default, causing sync failure
  with 'unsupported role reference kind: ""'
2026-09-05 14:53:59 -07:00
rock 4fc833f9b2 fix: remove backslash line continuations from YAML multiline string
- YAML block scalars (|) don't use backslash continuation
- Just indent lines properly, block scalar handles them automatically
- Fixes ArgoCD ComparisonError on poimen app
2026-09-05 14:48:21 -07:00
rock 647fba8814 fix: add admin-oidc-binding to kustomization resources
- admin-oidc-binding.yaml wasn't listed in resources
- Now kustomize will include it when building manifests
- ArgoCD can sync the OIDC group binding
2026-09-05 14:42:17 -07:00
rock bcae41e338 chore: revert node-runner to stock image, remove Dockerfile
- Reverted to node:22-bookworm (no custom image)
- Removed Dockerfile.node (no CI to build it)
- Docker install step in riotpiao workflow is already the workaround
2026-09-05 14:33:27 -07:00
rock 1425ab7cbc chore: remove homelab CI workflow
- Removed .gitea/workflows/build-runner-node.yml
- Homelab is GitOps only, not a buildable artifact
- Runner images managed via direct Dockerfile edits + manual pushes
2026-09-05 14:33:13 -07:00
rock 978f9c8147 feat: add OIDC group binding for cluster-admin access
- Binds oidc:homelab-admins group to cluster-admin ClusterRole
- Allows OIDC users (via Authentik) to have admin access
- Groups claim from Authentik with oidc: prefix per kube-apiserver config
- Enables kubectl access via 'kubectl login' + kubelogin
2026-09-05 14:30:41 -07:00
rock 4193c8ab99 feat: custom forgejo-runner-node image with docker.io pre-installed
- Dockerfile.node extends node:22-bookworm + docker.io
- CI workflow builds and pushes to forgejo registry on changes
- values-node.yaml references custom image instead of stock node
- Removes need to install docker in every workflow using node runner
2026-09-05 14:20:42 -07:00
rock 2c011e08e2 feat: Image Updater git write-back for multi-source poimen Application
- write-back-method: git (commits image updates back to repos)
- git-branch: main
- Mounts ArgoCD SSH credentials for git pushes
- Image Updater commits new SHAs → repos → ArgoCD syncs
2026-09-05 13:56:40 -07:00
rock 9da6829e05 feat: multi-source poimen Application (memory, workflows, frontend)
- Single Application syncs 3 independent repos
- All deploy to poimen namespace
- Image Updater tracks all 3 services (7-char SHA tags)
- Auto-sync: prune + selfHeal enabled
2026-09-05 13:55:29 -07:00
rock 69537a4e6a fix: remove poimen-root Application (external repo dependency)
- Removed dependency on external poimen.git repo
- Poimen manifests should be managed locally or via separate workflow
- Simplifies homelab GitOps to only manage homelab-owned services
2026-09-05 13:52:53 -07:00
rock 76c053d895 Revert "feat: enable Image Updater for poimen services"
This reverts commit cfc27c5420.
2026-09-05 13:52:50 -07:00
rock cfc27c5420 feat: enable Image Updater for poimen services
- Track poimen-memory, poimen-workflows, poimen-frontend images
- Auto-update on new 7-char SHA tags from Forgejo
- Filter: regexp:^[0-9a-f]{7}$ (commit SHA)
- write-back-method: argocd (updates Application)
2026-09-05 13:51:29 -07:00
rock 97c951bef3 Phase 6.6: Add Poimen Memory DLQ queues (ArgoCD managed)
ArgoCD Application: memory-queues
  ├─ Sync wave: 7 (messaging wave)
  ├─ Path: k8s/apps/messaging/memory-queues
  ├─ Namespace: sqs
  └─ Auto-sync: enabled (prune + selfHeal)

Helm Chart: memory-queues
  ├─ Chart.yaml: v0.1.0
  ├─ values.yaml: Queue config
  └─ templates/queues.yaml: Queue CRD resources

Queues Created:

1. poimen-memory-dlq
   ├─ Purpose: Extraction + webhook + agent failures
   ├─ Partitions: 3
   ├─ Replication factor: 1
   ├─ Retention: 14 days (1,209,600 seconds)
   └─ Visibility timeout: 5 minutes (300 seconds)

2. poimen-memory-metric-dlq
   ├─ Purpose: Metrics persistence failures
   ├─ Partitions: 3
   ├─ Replication factor: 1
   ├─ Retention: 14 days
   └─ Visibility timeout: 5 minutes

Resource: Queue CRD (kmsvc.io/v1alpha1)
  └─ Managed by: queue-operator (already running in sqs ns)

Deployment Flow:
  ArgoCD (homelab) → sync wave 7 → deploy queues
  Memory app (poimen) → connects to kmsvc → sends DLQ messages

Files:
  ├─ k8s/apps/messaging/memory-queues/Chart.yaml (new)
  ├─ k8s/apps/messaging/memory-queues/values.yaml (new)
  ├─ k8s/apps/messaging/memory-queues/templates/queues.yaml (new)
  └─ k8s/argocd/apps/50-memory-queues.yaml (new)
2026-09-05 01:10:51 -07:00
rock e414a3e394 Revert "feat: add memory service queues (processing, indexing, dlq)"
This reverts commit e5ae5b16b7.
2026-09-05 01:09:30 -07:00
rock e5ae5b16b7 feat: add memory service queues (processing, indexing, dlq)
- processing-queue: high-throughput, auto-scaling (1-4 shards)
- indexing-queue: FIFO with deduplication (1-2 shards)
- memory-dlq: dead letter queue for redelivery failures
- ArgoCD Application (wave 7) to auto-sync queue lifecycle
2026-09-05 01:05:01 -07:00
rock e41165f358 fix(grafana): give homelab-admins Admin org role, akadmin GrafanaAdmin
GrafanaAdmin is server admin only — no org membership, so users couldn't
see dashboards. Now:
- akadmin: GrafanaAdmin (server admin, can impersonate)
- homelab-admins: Admin (org admin, dashboard access)
- others: Viewer
2026-09-04 23:41:22 -07:00
rock bd8c9fe033 fix: grant GrafanaAdmin (server admin) to homelab-admins for Administration menu 2026-09-04 23:18:19 -07:00
rock 2eda66c095 fix: add groups scope to grafana OIDC so role mapping works 2026-09-04 23:14:11 -07:00
rock edc5dadd82 feat: add nginx-ingress ServiceMonitor for gateway traffic metrics 2026-09-04 23:07:44 -07:00
rock 60786a17ea fix: map grafana admin role from homelab-admins group (grafana-admins deleted) 2026-09-04 23:04:01 -07:00
rock efbe530b5c refactor: consolidate 13 dashboards into 2 (cluster-infrastructure + api-gateway) 2026-09-04 22:58:53 -07:00
rock 4c63f8b125 feat: add cluster and api-gateway alert rules with SLA targets 2026-09-04 22:37:48 -07:00
rock 2a9220b576 feat: add cluster-infrastructure and api-gateway grafana dashboards 2026-09-04 22:31:51 -07:00
rock 26714d2ef3 feat: add api-gateway blackbox probes for healthz and /v1/models 2026-09-04 22:13:00 -07:00
rock 39e7ada3c6 fix: use external Authentik URL for MinIO OIDC config discovery 2026-09-04 21:25:30 -07:00
rock 1fe8707e3c gitops: add secret-rotation controller ArgoCD Application
- Syncs k8s/apps/secret-rotation-controller/ kustomization
- Auto-prune and self-heal enabled
- Creates secret-rotation namespace
- ArgoCD will deploy CRD, RBAC, ExternalSecret, Deployment
2026-09-04 13:51:26 -07:00
rock 82a4e3e4fe feat: automated secret rotation controller
- ExternalSecret syncs age key from Vault to pod
- CRD defines rotation schedule for each secret
- Controller watches CRD, rotates on schedule:
  * Call provider API (Authentik/Forgejo/MinIO) for new secret
  * Update k8s Secret
  * Update .enc.yaml via sops (uses age key from Vault)
  * Git commit and push
- Vault is source of truth for age key (never on disk)
- Examples: minio-oidc (90d), portfolio-agent (90d), forgejo-token (90d), minio-root (180d)
2026-09-03 23:37:24 -07:00
rock 6ad4c0d294 fix(minio): use in-cluster URL for OIDC config fetch
MinIO pod was getting 503 from public URL at startup. Use in-cluster
authentik-server.iam.svc for metadata fetch; browser redirects still
use public URLs from OIDC metadata response.
2026-09-03 23:15:19 -07:00
rock 910f8e70d5 iam: move provisioning script to scripts/iam, remove k8s job
- Move authentik-provision.py to scripts/iam/ (manual-only)
- Remove job/RBAC resources (not needed for local runs)
- Use public URL directly (no sed substitution needed)
- Add app password support via set_key endpoint
- Support both password grant and client_credentials
2026-09-03 19:03:58 -07:00
rock 20513c8b3b iam: add memory scope, service accounts, manual provisioning
- Add 'memory' scope property mapping (memory_projects, memory_visibility, memory_role)
- Add capability groups: llm-users, memory-users, memory-writers
- Add service account provisioning for portfolio-agent, memory-agent
- Fix sops-secrets kustomization (generatorOptions)
- Add RoleBindings for portfolio, poimen, dashboard namespaces
- Remove PostSync hook - IAM provisioning is now manual-only
2026-09-03 18:07:22 -07:00
rock 87786d8733 messaging: remove queue-crd/management-service (moved to kmsvc-manage)
- Applications now managed by kmsvc-root from kmsvc-manage.git
- Added ServerSideApply to homelab-root for proper annotation sync
- Avoids duplicate Application conflicts with Image Updater
2026-09-03 08:28:38 -07:00
rock 17a98afa3f image-updater: filter to SHA tags only (skip :latest)
allow-tags: regexp:^[0-9a-f]{7}$ ensures newest-build strategy
compares commit SHA tags, not the stale :latest tag
2026-09-03 08:07:19 -07:00
rock 7e9ef86826 appproject: allow argo-helm repo for image-updater 2026-09-02 20:32:28 -07:00
rock c6032cc354 argocd: add Image Updater for auto-deploy on image push
- Install argocd-image-updater via Helm (wave 1)
- Configure Forgejo registry (anonymous pulls)
- Annotate apps for auto-update: api-gw, portfolio, management-service, queue-crd
- Uses newest-build strategy for commit SHA tags
2026-09-02 20:31:09 -07:00
rock f21cc4721a kmsvc: migrate image from GHCR to Forgejo registry
Consolidate all internal images to forgejo.riotpiao.com for ArgoCD Image Updater
2026-09-02 20:12:35 -07:00
rock 51c66299e9 forgejo-runner: gc every 30min instead of daily
- Fix template to use .Values.gc.schedule instead of hardcoded cron
- Change schedule from daily 03:00 UTC to every 30 minutes
- Prevents DinD PVC fill-up (was at 93% before manual prune)
2026-09-02 19:52:41 -07:00