riotpiao-poimen/homelab
2
0
Fork 0
Code Issues 23 Pull Requests 2 Actions Packages Projects 1 Releases Wiki Activity
389 Commits 8 Branches 0 Tags
b06ee310b551f92dd993d35765c451bdd3bb885a
Commit Graph
123 Commits
Author SHA1 Message Date
rock b06ee310b5 fix: use tektoncd/operator for proper K8s-native Tekton installation 
ROOT CAUSE:
- Previous Application pointed to storage bucket (not valid ArgoCD source)
- ArgoCD couldn't sync manifests from non-git/non-helm source
- tektoncd/operator is the official way to install Tekton

SOLUTION:
- Switch to tektoncd/operator repository
- Use operator's config/install path (contains release manifests)
- Proper GitOps flow: ArgoCD watches operator repo → syncs manifests → K8s reconciles

BENEFITS:
✓ Official Tekton approach
✓ Proper K8s Operator pattern
✓ ArgoCD-compatible (git source)
✓ Automatic updates from upstream
✓ Full GitOps workflow
 2026-09-13 15:10:39 +09:00
poimenandrock 5f16d5c6a3 feat: add Tekton Pipelines for CI/CD orchestration (#46) 
Install Tekton Pipelines (CNCF CI/CD) via ArgoCD for pre-merge integration testing.

## What This Does

Adds Tekton Pipelines to the homelab cluster infrastructure for orchestrating CI/CD workflows:

1. **Tekton Pipelines Installation**
   - Kubernetes-native CI/CD (CNCF project)
   - Task and Pipeline CRDs for workflow definitions
   - PipelineRun for ephemeral test execution
   - Webhook support for event-driven triggers

2. **ArgoCD Management**
   - ArgoCD Application manages Tekton installation
   - Automatic updates from upstream
   - GitOps-driven (everything in git)
   - Wave 06 deployment (after networking, before apps)

3. **Integration with homelab-frontend**
   - CI creates Tekton PipelineRun
   - Tests execute in cluster
   - Results flow back to CI
   - Image promotion only on pass

## Architecture

```
Cluster Infrastructure (homelab):
  └── Tekton Pipelines (Wave 06 - CI/CD)
      ├── Task: Run integration tests
      ├── Pipeline: Orchestrate workflows
      └── PipelineRun: Execute on demand

Application: homelab-frontend
  └── CI Workflow (.gitea/workflows/ci.yaml)
      ├── Build image
      ├── Create PipelineRun
      ├── Wait for completion
      └── Promote to :latest (if pass)
```

## Files Added

- `k8s/infra/tekton/namespace.yaml` - Tekton namespace
- `k8s/infra/tekton/kustomization.yaml` - Release manifest reference
- `k8s/argocd/apps/06-ci-cd.yaml` - ArgoCD Application (Wave 06)
- `k8s/argocd/projects/homelab-project.yaml` - Added Tekton repos

## Wave Ordering

Wave 06 (CI/CD) is deployed in proper order:
- Wave 00-01: ArgoCD bootstrap
- Wave 05: Networking (ingress, etc.)
- **Wave 06: CI/CD (Tekton Pipelines)** ← NEW
- Wave 10+: Storage, logging, monitoring
- Wave 40+: Data services (databases)
- Wave 50+: Applications (API gateway, etc.)

## Benefits

✓ **Kubernetes-Native**: Uses standard K8s CRDs (Task, Pipeline, PipelineRun)
✓ **GitOps**: Everything in git, managed by ArgoCD, no manual kubectl
✓ **Pre-Merge Testing**: Tests must pass before code deploys
✓ **Observable**: Logs, status, results tracking
✓ **Secure**: Non-root containers, resource limits, RBAC
✓ **CNCF-Standard**: Industry-proven Tekton project
✓ **Scalable**: Can add more tests/tasks without complexity

## Integration with PR #25

This homelab PR works with homelab-frontend PR #25:
- homelab (this): Installs Tekton infrastructure
- homelab-frontend #25: Integrates tests with Tekton

Together they form complete GitOps CI/CD pipeline.

## Testing After Merge

1. ArgoCD syncs this repo
2. Wave 06 deployment triggered
3. Tekton Pipelines installed to cluster
4. homelab-frontend PR #25 can merge
5. First code push triggers integration tests

## Review Checklist

- [ ] Tekton namespace created properly
- [ ] ArgoCD Application configuration correct
- [ ] Wave 06 ordering makes sense
- [ ] Project repos include Tekton
- [ ] Integration with homelab-frontend understood
- [ ] No hardcoded values
- [ ] Documentation is clear

---------

Co-authored-by: rock <[email protected]>
Reviewed-on: #46
Co-authored-by: poimen <[email protected]>
 2026-09-13 05:48:06 +00:00
rock f8de5f506b feat(paperless): add paperless-ai with local LLM for auto-tagging 
- paperless-ai deployment using clusterzx/paperless-ai
- LLM via local api-gateway (reasoning model, no auth - phase 3 pending)
- Paperless API token SOPS-encrypted
- Auto-tags new documents, 5min scan interval
- Adds 'ai-processed' tag to classified documents
 2026-09-13 06:04:49 +09:00
rock cd126d5339 feat(forgejo): enable SMTP email notifications via Gmail 
- mailer config: smtp+starttls to smtp.gmail.com:587
- SMTP creds via forgejo-smtp secret (SOPS-encrypted, Gmail App Password)
- Env var injection: GITEA__MAILER__USER/PASSWD from secret
- Enables CI completion emails, password reset, repo notifications
 2026-09-13 06:01:05 +09:00
rock d2fb52e425 fix(smtp): update gotify-smtp with real Gmail App Password 2026-09-13 05:57:21 +09:00
rock 03044614b7 chore: add encrypted backups and rotation schedule 
- database-passwords-backup.enc.yaml: SOPS-encrypted DB credentials
- memory-agent-oidc.enc.yaml: SOPS-encrypted Poimen OIDC credentials
- oauth2-credentials.enc.yaml: SOPS-encrypted all OAuth2 secrets (6 providers)
- rotate-secrets.sh: 90-day rotation schedule (next: 2026-12-11)

These files enable full credential recovery and rotation management.
All SOPS-encrypted with cluster key for in-cluster decryption only.
Manual decryption requires ~/.sops.yaml configuration + GPG key.
 2026-09-12 23:52:34 +09:00
rock 7613b4fbf2 feat(iam): rock user with Forgejo email, email password recovery, encrypt SMTP creds 
- provision-rbac.py: create rock user ([email protected]) matching Forgejo
- Email recovery flow: identification -> email stage -> password reset
- SMTP via Gmail (gotify-smtp secret, SOPS-encrypted)
- Recovery flow bound to brand for login page reset link
- minio-provision-paperless: add bucket creation, use quay.io/minio/mc
 2026-09-12 23:44:26 +09:00
rock 1d47234f12 fix(databases): Scale authentik-db and gotify-db to 1 replica (Longhorn corruption recovery) 2026-09-12 17:27:45 +09:00
rock 9d3a669dfe fix(gotify): move SOPS secrets to ksops generator 
ArgoCD couldn't decrypt secrets.yaml because it was listed as a plain
kustomize resource. Move the 3 secrets (gotify-admin, gotify-tokens,
gotify-smtp) to k8s/argocd/secrets/ as .enc.yaml files processed by
the ksops generator, matching the repo convention.

Fixes ComparisonError: 'Object Kind is missing' (SOPS ciphertext
parsed as raw YAML).
 2026-09-10 22:32:24 +09:00
rock 436c7d8d42 fix(argocd): update git repoURLs for org transfer rock -> riotpiao-poimen 
Repos transferred: homelab-frontend, kmsvc-manage, poimen, poimen-memory,
poimen-workflows, poimen-frontend. Old URLs return 301 which ArgoCD
doesn't follow.

NOT changed: container image registry paths (rock/ is correct for registry),
riotpiao.com (still under rock org).

Also adds poimen-frontend to AppProject sourceRepos allowlist.
 2026-09-10 10:49:30 +09:00
rock b571d518e0 fix(argocd): update repoURL after org transfer (#29) 
Co-authored-by: rock <[email protected]>
 2026-09-10 01:41:37 +00:00
rock 128e76ce2d feat(gotify): push notification server + SMTP email relay (#19) 
Co-authored-by: rock <[email protected]>
 2026-09-10 01:35:59 +00:00
rock 8790de6038 feat: add ComfyUI + rebalance GPU allocation (#17) 
## GPU Rebalance (4× V100 32GB)

| Pod | Before | After |
|-----|--------|-------|
| reasoning (PP=2) | 2 GPU | 2 GPU |
| ornith | 2 GPU (2 replicas) | 1 GPU (1 replica) |
| comfyui | — | 1 GPU (**new**) |
| qwen-cpu | — | CPU on cp-2 (**new**) |
| embeddings/reranker | CPU | CPU |

## Changes

- `ornith.yaml`: scale 2→1, remove qwen2.5 co-loading, MAX_LOADED_MODELS=1
- `qwen-cpu.yaml`: new Ollama deployment on talos-cp-2 (144GB RAM), 5Gi PVC
- `k8s/apps/comfyui/`: new ComfyUI deployment (1 GPU, 50Gi model PVC, ingress)
- `58-comfyui.yaml`: ArgoCD Application (wave 8)

Gateway route update in separate PR (homelab-frontend).Reviewed-on: #17

Co-authored-by: rock <[email protected]>
 2026-09-09 02:11:28 +00:00
rock 1630704f8b feat: switch image-updater to digest-based :latest tracking 
All image-updater annotations now use update-strategy: digest with
allow-tags: ^latest$ and write-back-method: argocd. No SHA tags
committed to git — digest overrides stored in ArgoCD state only.

- poimen: git write-back → argocd, removed git-branch
- portfolio: newest-build SHA → digest latest
- api-gateway: newest-build SHA → digest latest
 2026-09-07 18:19:46 -07:00
rock ce1539a634 revert: remove unsupported buildOptions (ArgoCD v3.4.5 doesn't support it) 
- buildOptions field not available in ArgoCD v3.4.5
- SOPS decryption already handled by repo-server ksops plugin
- Revert to simple kustomize config
- Portfolio Application can now sync properly
 2026-09-07 17:49:58 -07:00
rock 2799e3a675 fix: enable ksops plugin for portfolio Application 
- Add kustomize config with --enable-alpha-plugins to support ksops
- Allows ArgoCD to properly decrypt SOPS-encrypted files
- Fixes Image Updater compatibility with sops field in kustomization.yaml
 2026-09-07 17:47:03 -07:00
rock 7d77935d15 ci: fix runner labels + CoreDNS rewrite + cleanup 
- Runners use public images (code.forgejo.org/forgejo/runner:6)
- Labels pull from Docker Hub: golang:1.26, node:22, rust:1-bookworm
- Add CoreDNS api.riotpiao.com rewrite
- Fix runner re-registration to keep labels in sync
- Add unified CI pattern docs to CLAUDE.example.md
- Remove dead .forgejo/ workflow dir (Forgejo uses .gitea/)
 2026-09-07 13:01:56 -07:00
rock 4a4e57d0f2 fix: remove namespace from rbac Application destination 
- RBAC kustomization contains cluster-scoped (ClusterRoleBinding) and
  namespace-scoped (Role/RoleBinding) resources
- Each resource has explicit metadata.namespace, so Application shouldn't
  force a default namespace
- Fixes: ClusterRoleBinding gets namespace=default, causing sync failure
  with 'unsupported role reference kind: ""'
 2026-09-05 14:53:59 -07:00
rock 4fc833f9b2 fix: remove backslash line continuations from YAML multiline string 
- YAML block scalars (|) don't use backslash continuation
- Just indent lines properly, block scalar handles them automatically
- Fixes ArgoCD ComparisonError on poimen app
 2026-09-05 14:48:21 -07:00
rock 2c011e08e2 feat: Image Updater git write-back for multi-source poimen Application 
- write-back-method: git (commits image updates back to repos)
- git-branch: main
- Mounts ArgoCD SSH credentials for git pushes
- Image Updater commits new SHAs → repos → ArgoCD syncs
 2026-09-05 13:56:40 -07:00
rock 9da6829e05 feat: multi-source poimen Application (memory, workflows, frontend) 
- Single Application syncs 3 independent repos
- All deploy to poimen namespace
- Image Updater tracks all 3 services (7-char SHA tags)
- Auto-sync: prune + selfHeal enabled
 2026-09-05 13:55:29 -07:00
rock 69537a4e6a fix: remove poimen-root Application (external repo dependency) 
- Removed dependency on external poimen.git repo
- Poimen manifests should be managed locally or via separate workflow
- Simplifies homelab GitOps to only manage homelab-owned services
 2026-09-05 13:52:53 -07:00
rock 76c053d895 Revert "feat: enable Image Updater for poimen services" 
This reverts commit cfc27c5420.
 2026-09-05 13:52:50 -07:00
rock cfc27c5420 feat: enable Image Updater for poimen services 
- Track poimen-memory, poimen-workflows, poimen-frontend images
- Auto-update on new 7-char SHA tags from Forgejo
- Filter: regexp:^[0-9a-f]{7}$ (commit SHA)
- write-back-method: argocd (updates Application)
 2026-09-05 13:51:29 -07:00
rock 97c951bef3 Phase 6.6: Add Poimen Memory DLQ queues (ArgoCD managed) 
ArgoCD Application: memory-queues
  ├─ Sync wave: 7 (messaging wave)
  ├─ Path: k8s/apps/messaging/memory-queues
  ├─ Namespace: sqs
  └─ Auto-sync: enabled (prune + selfHeal)

Helm Chart: memory-queues
  ├─ Chart.yaml: v0.1.0
  ├─ values.yaml: Queue config
  └─ templates/queues.yaml: Queue CRD resources

Queues Created:

1. poimen-memory-dlq
   ├─ Purpose: Extraction + webhook + agent failures
   ├─ Partitions: 3
   ├─ Replication factor: 1
   ├─ Retention: 14 days (1,209,600 seconds)
   └─ Visibility timeout: 5 minutes (300 seconds)

2. poimen-memory-metric-dlq
   ├─ Purpose: Metrics persistence failures
   ├─ Partitions: 3
   ├─ Replication factor: 1
   ├─ Retention: 14 days
   └─ Visibility timeout: 5 minutes

Resource: Queue CRD (kmsvc.io/v1alpha1)
  └─ Managed by: queue-operator (already running in sqs ns)

Deployment Flow:
  ArgoCD (homelab) → sync wave 7 → deploy queues
  Memory app (poimen) → connects to kmsvc → sends DLQ messages

Files:
  ├─ k8s/apps/messaging/memory-queues/Chart.yaml (new)
  ├─ k8s/apps/messaging/memory-queues/values.yaml (new)
  ├─ k8s/apps/messaging/memory-queues/templates/queues.yaml (new)
  └─ k8s/argocd/apps/50-memory-queues.yaml (new)
 2026-09-05 01:10:51 -07:00
rock e414a3e394 Revert "feat: add memory service queues (processing, indexing, dlq)" 
This reverts commit e5ae5b16b7.
 2026-09-05 01:09:30 -07:00
rock e5ae5b16b7 feat: add memory service queues (processing, indexing, dlq) 
- processing-queue: high-throughput, auto-scaling (1-4 shards)
- indexing-queue: FIFO with deduplication (1-2 shards)
- memory-dlq: dead letter queue for redelivery failures
- ArgoCD Application (wave 7) to auto-sync queue lifecycle
 2026-09-05 01:05:01 -07:00
rock 1fe8707e3c gitops: add secret-rotation controller ArgoCD Application 
- Syncs k8s/apps/secret-rotation-controller/ kustomization
- Auto-prune and self-heal enabled
- Creates secret-rotation namespace
- ArgoCD will deploy CRD, RBAC, ExternalSecret, Deployment
 2026-09-04 13:51:26 -07:00
rock 20513c8b3b iam: add memory scope, service accounts, manual provisioning 
- Add 'memory' scope property mapping (memory_projects, memory_visibility, memory_role)
- Add capability groups: llm-users, memory-users, memory-writers
- Add service account provisioning for portfolio-agent, memory-agent
- Fix sops-secrets kustomization (generatorOptions)
- Add RoleBindings for portfolio, poimen, dashboard namespaces
- Remove PostSync hook - IAM provisioning is now manual-only
 2026-09-03 18:07:22 -07:00
rock 87786d8733 messaging: remove queue-crd/management-service (moved to kmsvc-manage) 
- Applications now managed by kmsvc-root from kmsvc-manage.git
- Added ServerSideApply to homelab-root for proper annotation sync
- Avoids duplicate Application conflicts with Image Updater
 2026-09-03 08:28:38 -07:00
rock 17a98afa3f image-updater: filter to SHA tags only (skip :latest) 
allow-tags: regexp:^[0-9a-f]{7}$ ensures newest-build strategy
compares commit SHA tags, not the stale :latest tag
 2026-09-03 08:07:19 -07:00
rock 7e9ef86826 appproject: allow argo-helm repo for image-updater 2026-09-02 20:32:28 -07:00
rock c6032cc354 argocd: add Image Updater for auto-deploy on image push 
- Install argocd-image-updater via Helm (wave 1)
- Configure Forgejo registry (anonymous pulls)
- Annotate apps for auto-update: api-gw, portfolio, management-service, queue-crd
- Uses newest-build strategy for commit SHA tags
 2026-09-02 20:31:09 -07:00
rock 77683ec7c6 fix: add poimen-memory and poimen-workflows Forgejo repos to sourceRepos 2026-09-02 09:52:41 -07:00
rock 88c9f3047a fix: disable name suffix hash for portfolio-secrets to match deployment reference 2026-09-01 10:38:05 -07:00
rock a6c3fdf786 feat: add portfolio LLM_API_TOKEN to ksops secrets 
- portfolio-secrets.enc.env: FORGEJO_TOKEN + LLM_API_TOKEN for api.riotpiao.com
- kustomization: secretGenerator for ksops handling at deploy time
- Will be SOPS encrypted with homelab age key before merge
 2026-09-01 09:42:02 -07:00
rock 4a1a60104d feat: add portfolio SOPS secret for CI status 2026-08-31 22:33:00 -07:00
rock 1af9232613 feat: add tempo and otel-collector for distributed tracing 2026-08-31 15:02:15 -07:00
rock be55d68571 feat(argocd): add portfolio app, explicit sourceRepos, coredns rewrite for riotpiao.com 
- Add Application for rock/riotpiao.com repo (portfolio site)
- Replace wildcard sourceRepos with explicit repo list
- Add CoreDNS rewrite for root domain riotpiao.com
 2026-08-31 14:18:57 -07:00
rock 2038253d48 feat(security): add Kyverno for image scanning and Pod security policies 
- Install Kyverno policy engine for admission control
- Add ClusterPolicies:
  * Disallow 'latest' tags (require explicit versions)
  * Restrict to trusted registries (docker.io, ghcr.io, quay.io, etc.)
  * Require non-root containers
  * Drop all Linux capabilities by default
  * Require securityContext on all containers
  * Require read-only root filesystem (audit only)
  * Require resource requests/limits (prevent starvation)
- All policies in audit mode initially (failurePolicy: ignore)
- Ready to graduate to enforce after testing
- Fixes: missing image scanning from security audit
 2026-08-31 11:43:04 -07:00
rock 136bfaf0f2 feat(ops): add cluster-wide stale job/pod cleanup CronJob 
Daily 04:00 UTC sweeper in kube-system:
- Delete failed Jobs older than 24h (any namespace)
- Delete completed standalone Jobs older than 72h (no CronJob owner)
- Delete orphan Error/Evicted pods older than 1h
- Self-cleans via ttlSecondsAfterFinished
 2026-08-30 07:27:50 -07:00
Story Crater Bot aaa728bab4 fix: rotate vault unseal keys after vault-0 wipe/reinit 
Old root token and unseal keys were dead (lost access to the previous
Vault store). Wiped the S3 backend and vault-0, re-initialized fresh,
rotated these to match the new live unseal keys so ArgoCD's next sync
doesn't clobber them back to the dead ones.
 2026-08-26 16:33:35 -07:00
Story Crater Bot 61a1975669 feat: deploy Immich with Authentik OIDC, rock as admin 
Self-hosted photo backup (Google Photos replacement) - raw manifests,
no Helm chart, self-contained under k8s/apps/immich including its own
CNPG Postgres. Media PVC shares the cp-3 HDD 2TB/2TB with
paperless-media.

Postgres is pg18, not this repo's usual 16.2: CNPG's official pgvector
extension image (ghcr.io/cloudnative-pg/pgvector) is only published
for pg18, loaded via CNPG's ImageVolume extension mechanism (operator
1.30.0 / k8s 1.36.1 both support it). Immich auto-manages CREATE
EXTENSION itself at startup.

OIDC via a new "immich_role" Authentik scope mapping (homelab-admins/
immich-admins -> "admin" claim, else "user"), consumed by Immich's
OAuth roleClaim setting which re-syncs isAdmin on every login - more
reliable than Immich's racy first-user-is-admin fallback. Config
composed into an immich-oidc Secret and mounted as IMMICH_CONFIG_FILE,
matching the paperless-oidc pattern. k8s RBAC (immich-operator Role +
oidc:immich-admins binding) mirrors paperless/rbac.yaml.

immich namespace pre-created in k8s/infra/databases/namespaces.yaml
(not just immich's own CreateNamespace=true) since the iam PostSync
job's RoleBinding needs it to exist before wave 8.
 2026-08-25 18:21:28 -07:00
Story Crater Bot 89f01b6f2e feat: add homelab-wide Authentik RBAC model and k8s OIDC auth wiring 
Adds permissions claim + per-service admin groups in Authentik, scoped
Role/RoleBinding per service, public PKCE kubernetes OAuth2 client, and
kube-apiserver OIDC extraArgs. Also fixes paperless OIDC signup permissions
via adapter override and adds CoreDNS rewrite for authentik.riotpiao.com.
 2026-08-25 15:03:44 -07:00
Story Crater Bot bea76eeba6 fix: repoint ArgoCD Applications from GitHub to in-cluster Forgejo 
homelab-root and every child Application still tracked github.com/Riotpiaole/riotpiao.homelab.com, which had diverged from origin (Forgejo) for a while - pushes to Forgejo were never picked up by ArgoCD. Repointed to forgejo.riotpiao.com/rock/homelab.git, already covered by the AppProject's rock/* wildcard.
 2026-08-25 11:20:26 -07:00
Story Crater Bot e8e5acfb13 feat: add paperless-ngx with OIDC, CNPG db, cp-3 HDD media, MinIO backup 
Fixes controlplane.tftpl's install.wipe:true (should be false, live CPs already run false) and syncs coredns Corefile back to what's actually deployed (drops an unrolled-out, stale Kong-era rewrite).
 2026-08-25 11:11:37 -07:00
Story Crater Bot 9299514d6e Add Forgejo registry PAT secret (encrypted with SOPS, ksops managed) 2026-08-23 16:15:26 -07:00
Story Crater Bot 80cafed201 Track all poimen-* repos in AppProject for flexible service onboarding 2026-08-22 23:20:58 -07:00
Story Crater Bot 9eb5c8ea1a Remove separate memory app, bundle into wave 2 databases 2026-08-22 23:16:40 -07:00
Story Crater Bot aa6b2ae9c3 Add Poimen Memory to ArgoCD wave 2 deployment (namespace: poimen) 2026-08-22 23:09:02 -07:00