riotpiao-poimen/homelab
2
0
Fork 0
Code Issues 23 Pull Requests 2 Actions Packages Projects 1 Releases Wiki Activity
454 Commits 20 Branches 0 Tags
2d9e3abee805182b8e4ac06c363694a979b87c9b
Commit Graph
136 Commits
Author SHA1 Message Date
rock 5d22691c3e chore(oidc): one-shot bootstrap - restart apiservers to initialise the OIDC authenticator 
k8s/argocd/apps/99-oidc-bootstrap.yaml
 2026-10-07 03:46:01 +00:00
rock f780e31f9c Merge origin/main into fix/secrets-placeholders (rebuild ksops file list, fix #54 indent) 2026-10-06 10:05:45 +08:00
rock c93a24f786 fix(secrets): populate the empty sops placeholders and drop the unused one 
(cherry picked from commit f6207ff2b0)
 2026-10-06 10:03:04 +08:00
rock b6e317b34c feat(poimen): pi sandbox workspace volume, encrypted api keys, cluster dns rewrite (#54) 
Adds the pi coding-agent sandbox's cluster side:
- `k8s/infra/poimen/pi-agent-workspace-pv.yaml` — static hostPath PV + PVC for the sandbox /workspace (Longhorn cannot attach new volumes in this cluster; this gives real persistence and survived a pod delete/recreate test: marker file + 38 KB session.jsonl intact, the session answered from its own history afterwards).
- `k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml` — the sandbox's API-key Secret, SOPS-encrypted (recipient from .sops.yaml, payload only: `encrypted_regex ^(data|stringData)$`), registered in the ksops generator. Round-trip verified against the live Secret.
- `terraform/files/coredns/Corefile` — `rewrite name pi.riotpiao.com ingress-nginx-controller...` so the hostname resolves in-cluster.

Note: the keys currently in that Secret are the ones leaked earlier and must be rotated; re-encrypt with `sops -e -i` after rotating.Reviewed-on: #54
Co-authored-by: rock <[email protected]>
 2026-10-06 01:49:12 +00:00
rock bb24418a2e feat: remove ArgoCD Image Updater, replaced by Forgejo webhook 
Image promotion now handled by api-gw webhook endpoint
(/v1/webhooks/forgejo) which patches ArgoCD Application CRs
on Forgejo package push events (homelab-frontend PR #55).

- Delete Image Updater Application and values
- Remove argocd-image-updater.argoproj.io annotations from all apps
- Registry secrets kept (used by webhook digest queries + imagePullSecrets)
 2026-09-28 23:06:16 +08:00
rock 797e11ce11 feat(secrets): add forgejo-registry-pat for api namespace 
Encrypted SOPS secret (age) — api-gateway image promoter needs
REGISTRY_PAT to query Forgejo v2 registry for image digests.
Key: pat  Value: bdf6a1d... (registry PAT from existing forgejo-registry-pat secret)
Namespace: api  Secret name: forgejo-registry-pat
 2026-09-28 11:11:06 +08:00
rock 76a3281e80 fix(argocd): image-list comma-separated — updater v0.15.1 treats pipe block as single image name 2026-09-28 09:39:13 +08:00
rock c04ed6136b refactor(poimen): move frontend k8s manifests to poimen-frontend repo 
- Remove Deployment/Service/NetworkPolicy from homelab (wrong place)
- Add k8s/apps/poimen-frontend/ with ingress only (edge concern, same as api-gw)
- Update 70-poimen.yaml: source 4 → poimen-frontend.git k8s/
                          source 5 → homelab k8s/apps/poimen-frontend/ (ingress)
 2026-09-28 09:13:02 +08:00
rock 227839c27a feat(poimen): consolidate into single multi-source ArgoCD app 
- Merge poimen-workflows + poimen-memory + db + frontend into one app
- Delete duplicate apps: poimen-workflows, poimen-memory-app (kubectl done)
- Delete poimen-memory ArgoCD app file (replaced by multi-source source 2)
- Delete poimen-memory-database (replaced by source 3 → k8s/infra/poimen/)
- Move memory-db CNPG cluster to k8s/infra/poimen/ (isolated from other tenants)
- Remove memory-db from k8s/infra/databases/kustomization.yaml
- Add frontend manifests (Deployment, Service, Ingress) to k8s/apps/poimen/
- Add frontend NetworkPolicy: egress to api-gw (api:8080), memory (poimen:8080),
  Gotify (notifications:80), Authentik (iam:9000/9443), DNS (kube-system:53)
- Sync-wave -1 on CNPG cluster ensures DB ready before app pods (wave 0)
- Image updater: workflows + memory + frontend tracked in one annotation block
- poimen-toolkit stays separate (CRD management, independent rollback)
 2026-09-28 08:51:31 +08:00
rock 3914c78bc5 feat(argocd): poimen-memory Application — Redis + NetworkPolicy auto-deploy 
Points to riotpiao-poimen/poimen-memory.git k8s/app/ (kustomize).
ArgoCD auto-syncs Redis deployment, NetworkPolicies, ConfigMap, Service.
Image updater: digest strategy on :latest tag for rolling deployments.
Wave 7 — same wave as poimen workflow app.
 2026-09-27 19:27:05 +08:00
rock 920693df1b feat: poimen-toolkit ArgoCD app + sourceRepo 
- k8s/argocd/apps/71-poimen-toolkit.yaml: Application wave-7, points to
  poimen-toolkit repo k8s/ path, ServerSideApply for CRD schema size,
  image-updater for controller:latest digest
- k8s/argocd/projects/homelab-project.yaml: add poimen-toolkit.git to sourceRepos
 2026-09-27 18:03:42 +08:00
rock 91ce48ffae fix(sops): add LLM_AUTH_CLIENT_SECRET for paperless-ai-agent Authentik auth 2026-09-14 23:19:43 +09:00
rock 7a5d0a83d5 fix: vendor Tekton release.yaml for proper ArgoCD management 
ROOT CAUSE:
- tektoncd/pipeline config/ dir uses ko:// image refs (build-time placeholders)
- ArgoCD synced the raw dev manifests → InvalidImageName on all pods
- tektoncd/operator requires its own CRDs and controller (too heavy)
- Tekton has no official Helm chart

FIX:
- Vendor the pre-built release.yaml (v0.68.0) into k8s/infra/tekton/
- Point ArgoCD Application at our own repo (forgejo)
- Release contains real container images (ghcr.io/tektoncd/pipeline/*)
- Remove external tektoncd repo from AppProject (not needed)

TO UPGRADE TEKTON:
  1. Download new release from github.com/tektoncd/pipeline/releases
  2. Replace k8s/infra/tekton/release.yaml
  3. Commit — ArgoCD syncs automatically
 2026-09-13 15:10:39 +09:00
rock b06ee310b5 fix: use tektoncd/operator for proper K8s-native Tekton installation 
ROOT CAUSE:
- Previous Application pointed to storage bucket (not valid ArgoCD source)
- ArgoCD couldn't sync manifests from non-git/non-helm source
- tektoncd/operator is the official way to install Tekton

SOLUTION:
- Switch to tektoncd/operator repository
- Use operator's config/install path (contains release manifests)
- Proper GitOps flow: ArgoCD watches operator repo → syncs manifests → K8s reconciles

BENEFITS:
✓ Official Tekton approach
✓ Proper K8s Operator pattern
✓ ArgoCD-compatible (git source)
✓ Automatic updates from upstream
✓ Full GitOps workflow
 2026-09-13 15:10:39 +09:00
poimenandrock 5f16d5c6a3 feat: add Tekton Pipelines for CI/CD orchestration (#46) 
Install Tekton Pipelines (CNCF CI/CD) via ArgoCD for pre-merge integration testing.

## What This Does

Adds Tekton Pipelines to the homelab cluster infrastructure for orchestrating CI/CD workflows:

1. **Tekton Pipelines Installation**
   - Kubernetes-native CI/CD (CNCF project)
   - Task and Pipeline CRDs for workflow definitions
   - PipelineRun for ephemeral test execution
   - Webhook support for event-driven triggers

2. **ArgoCD Management**
   - ArgoCD Application manages Tekton installation
   - Automatic updates from upstream
   - GitOps-driven (everything in git)
   - Wave 06 deployment (after networking, before apps)

3. **Integration with homelab-frontend**
   - CI creates Tekton PipelineRun
   - Tests execute in cluster
   - Results flow back to CI
   - Image promotion only on pass

## Architecture

```
Cluster Infrastructure (homelab):
  └── Tekton Pipelines (Wave 06 - CI/CD)
      ├── Task: Run integration tests
      ├── Pipeline: Orchestrate workflows
      └── PipelineRun: Execute on demand

Application: homelab-frontend
  └── CI Workflow (.gitea/workflows/ci.yaml)
      ├── Build image
      ├── Create PipelineRun
      ├── Wait for completion
      └── Promote to :latest (if pass)
```

## Files Added

- `k8s/infra/tekton/namespace.yaml` - Tekton namespace
- `k8s/infra/tekton/kustomization.yaml` - Release manifest reference
- `k8s/argocd/apps/06-ci-cd.yaml` - ArgoCD Application (Wave 06)
- `k8s/argocd/projects/homelab-project.yaml` - Added Tekton repos

## Wave Ordering

Wave 06 (CI/CD) is deployed in proper order:
- Wave 00-01: ArgoCD bootstrap
- Wave 05: Networking (ingress, etc.)
- **Wave 06: CI/CD (Tekton Pipelines)** ← NEW
- Wave 10+: Storage, logging, monitoring
- Wave 40+: Data services (databases)
- Wave 50+: Applications (API gateway, etc.)

## Benefits

✓ **Kubernetes-Native**: Uses standard K8s CRDs (Task, Pipeline, PipelineRun)
✓ **GitOps**: Everything in git, managed by ArgoCD, no manual kubectl
✓ **Pre-Merge Testing**: Tests must pass before code deploys
✓ **Observable**: Logs, status, results tracking
✓ **Secure**: Non-root containers, resource limits, RBAC
✓ **CNCF-Standard**: Industry-proven Tekton project
✓ **Scalable**: Can add more tests/tasks without complexity

## Integration with PR #25

This homelab PR works with homelab-frontend PR #25:
- homelab (this): Installs Tekton infrastructure
- homelab-frontend #25: Integrates tests with Tekton

Together they form complete GitOps CI/CD pipeline.

## Testing After Merge

1. ArgoCD syncs this repo
2. Wave 06 deployment triggered
3. Tekton Pipelines installed to cluster
4. homelab-frontend PR #25 can merge
5. First code push triggers integration tests

## Review Checklist

- [ ] Tekton namespace created properly
- [ ] ArgoCD Application configuration correct
- [ ] Wave 06 ordering makes sense
- [ ] Project repos include Tekton
- [ ] Integration with homelab-frontend understood
- [ ] No hardcoded values
- [ ] Documentation is clear

---------

Co-authored-by: rock <[email protected]>
Reviewed-on: #46
Co-authored-by: poimen <[email protected]>
 2026-09-13 05:48:06 +00:00
rock f8de5f506b feat(paperless): add paperless-ai with local LLM for auto-tagging 
- paperless-ai deployment using clusterzx/paperless-ai
- LLM via local api-gateway (reasoning model, no auth - phase 3 pending)
- Paperless API token SOPS-encrypted
- Auto-tags new documents, 5min scan interval
- Adds 'ai-processed' tag to classified documents
 2026-09-13 06:04:49 +09:00
rock cd126d5339 feat(forgejo): enable SMTP email notifications via Gmail 
- mailer config: smtp+starttls to smtp.gmail.com:587
- SMTP creds via forgejo-smtp secret (SOPS-encrypted, Gmail App Password)
- Env var injection: GITEA__MAILER__USER/PASSWD from secret
- Enables CI completion emails, password reset, repo notifications
 2026-09-13 06:01:05 +09:00
rock d2fb52e425 fix(smtp): update gotify-smtp with real Gmail App Password 2026-09-13 05:57:21 +09:00
rock 03044614b7 chore: add encrypted backups and rotation schedule 
- database-passwords-backup.enc.yaml: SOPS-encrypted DB credentials
- memory-agent-oidc.enc.yaml: SOPS-encrypted Poimen OIDC credentials
- oauth2-credentials.enc.yaml: SOPS-encrypted all OAuth2 secrets (6 providers)
- rotate-secrets.sh: 90-day rotation schedule (next: 2026-12-11)

These files enable full credential recovery and rotation management.
All SOPS-encrypted with cluster key for in-cluster decryption only.
Manual decryption requires ~/.sops.yaml configuration + GPG key.
 2026-09-12 23:52:34 +09:00
rock 7613b4fbf2 feat(iam): rock user with Forgejo email, email password recovery, encrypt SMTP creds 
- provision-rbac.py: create rock user ([email protected]) matching Forgejo
- Email recovery flow: identification -> email stage -> password reset
- SMTP via Gmail (gotify-smtp secret, SOPS-encrypted)
- Recovery flow bound to brand for login page reset link
- minio-provision-paperless: add bucket creation, use quay.io/minio/mc
 2026-09-12 23:44:26 +09:00
rock 1d47234f12 fix(databases): Scale authentik-db and gotify-db to 1 replica (Longhorn corruption recovery) 2026-09-12 17:27:45 +09:00
rock 9d3a669dfe fix(gotify): move SOPS secrets to ksops generator 
ArgoCD couldn't decrypt secrets.yaml because it was listed as a plain
kustomize resource. Move the 3 secrets (gotify-admin, gotify-tokens,
gotify-smtp) to k8s/argocd/secrets/ as .enc.yaml files processed by
the ksops generator, matching the repo convention.

Fixes ComparisonError: 'Object Kind is missing' (SOPS ciphertext
parsed as raw YAML).
 2026-09-10 22:32:24 +09:00
rock 436c7d8d42 fix(argocd): update git repoURLs for org transfer rock -> riotpiao-poimen 
Repos transferred: homelab-frontend, kmsvc-manage, poimen, poimen-memory,
poimen-workflows, poimen-frontend. Old URLs return 301 which ArgoCD
doesn't follow.

NOT changed: container image registry paths (rock/ is correct for registry),
riotpiao.com (still under rock org).

Also adds poimen-frontend to AppProject sourceRepos allowlist.
 2026-09-10 10:49:30 +09:00
rock b571d518e0 fix(argocd): update repoURL after org transfer (#29) 
Co-authored-by: rock <[email protected]>
 2026-09-10 01:41:37 +00:00
rock 128e76ce2d feat(gotify): push notification server + SMTP email relay (#19) 
Co-authored-by: rock <[email protected]>
 2026-09-10 01:35:59 +00:00
rock 8790de6038 feat: add ComfyUI + rebalance GPU allocation (#17) 
## GPU Rebalance (4× V100 32GB)

| Pod | Before | After |
|-----|--------|-------|
| reasoning (PP=2) | 2 GPU | 2 GPU |
| ornith | 2 GPU (2 replicas) | 1 GPU (1 replica) |
| comfyui | — | 1 GPU (**new**) |
| qwen-cpu | — | CPU on cp-2 (**new**) |
| embeddings/reranker | CPU | CPU |

## Changes

- `ornith.yaml`: scale 2→1, remove qwen2.5 co-loading, MAX_LOADED_MODELS=1
- `qwen-cpu.yaml`: new Ollama deployment on talos-cp-2 (144GB RAM), 5Gi PVC
- `k8s/apps/comfyui/`: new ComfyUI deployment (1 GPU, 50Gi model PVC, ingress)
- `58-comfyui.yaml`: ArgoCD Application (wave 8)

Gateway route update in separate PR (homelab-frontend).Reviewed-on: #17

Co-authored-by: rock <[email protected]>
 2026-09-09 02:11:28 +00:00
rock 1630704f8b feat: switch image-updater to digest-based :latest tracking 
All image-updater annotations now use update-strategy: digest with
allow-tags: ^latest$ and write-back-method: argocd. No SHA tags
committed to git — digest overrides stored in ArgoCD state only.

- poimen: git write-back → argocd, removed git-branch
- portfolio: newest-build SHA → digest latest
- api-gateway: newest-build SHA → digest latest
 2026-09-07 18:19:46 -07:00
rock ce1539a634 revert: remove unsupported buildOptions (ArgoCD v3.4.5 doesn't support it) 
- buildOptions field not available in ArgoCD v3.4.5
- SOPS decryption already handled by repo-server ksops plugin
- Revert to simple kustomize config
- Portfolio Application can now sync properly
 2026-09-07 17:49:58 -07:00
rock 2799e3a675 fix: enable ksops plugin for portfolio Application 
- Add kustomize config with --enable-alpha-plugins to support ksops
- Allows ArgoCD to properly decrypt SOPS-encrypted files
- Fixes Image Updater compatibility with sops field in kustomization.yaml
 2026-09-07 17:47:03 -07:00
rock 7d77935d15 ci: fix runner labels + CoreDNS rewrite + cleanup 
- Runners use public images (code.forgejo.org/forgejo/runner:6)
- Labels pull from Docker Hub: golang:1.26, node:22, rust:1-bookworm
- Add CoreDNS api.riotpiao.com rewrite
- Fix runner re-registration to keep labels in sync
- Add unified CI pattern docs to CLAUDE.example.md
- Remove dead .forgejo/ workflow dir (Forgejo uses .gitea/)
 2026-09-07 13:01:56 -07:00
rock 4a4e57d0f2 fix: remove namespace from rbac Application destination 
- RBAC kustomization contains cluster-scoped (ClusterRoleBinding) and
  namespace-scoped (Role/RoleBinding) resources
- Each resource has explicit metadata.namespace, so Application shouldn't
  force a default namespace
- Fixes: ClusterRoleBinding gets namespace=default, causing sync failure
  with 'unsupported role reference kind: ""'
 2026-09-05 14:53:59 -07:00
rock 4fc833f9b2 fix: remove backslash line continuations from YAML multiline string 
- YAML block scalars (|) don't use backslash continuation
- Just indent lines properly, block scalar handles them automatically
- Fixes ArgoCD ComparisonError on poimen app
 2026-09-05 14:48:21 -07:00
rock 2c011e08e2 feat: Image Updater git write-back for multi-source poimen Application 
- write-back-method: git (commits image updates back to repos)
- git-branch: main
- Mounts ArgoCD SSH credentials for git pushes
- Image Updater commits new SHAs → repos → ArgoCD syncs
 2026-09-05 13:56:40 -07:00
rock 9da6829e05 feat: multi-source poimen Application (memory, workflows, frontend) 
- Single Application syncs 3 independent repos
- All deploy to poimen namespace
- Image Updater tracks all 3 services (7-char SHA tags)
- Auto-sync: prune + selfHeal enabled
 2026-09-05 13:55:29 -07:00
rock 69537a4e6a fix: remove poimen-root Application (external repo dependency) 
- Removed dependency on external poimen.git repo
- Poimen manifests should be managed locally or via separate workflow
- Simplifies homelab GitOps to only manage homelab-owned services
 2026-09-05 13:52:53 -07:00
rock 76c053d895 Revert "feat: enable Image Updater for poimen services" 
This reverts commit cfc27c5420.
 2026-09-05 13:52:50 -07:00
rock cfc27c5420 feat: enable Image Updater for poimen services 
- Track poimen-memory, poimen-workflows, poimen-frontend images
- Auto-update on new 7-char SHA tags from Forgejo
- Filter: regexp:^[0-9a-f]{7}$ (commit SHA)
- write-back-method: argocd (updates Application)
 2026-09-05 13:51:29 -07:00
rock 97c951bef3 Phase 6.6: Add Poimen Memory DLQ queues (ArgoCD managed) 
ArgoCD Application: memory-queues
  ├─ Sync wave: 7 (messaging wave)
  ├─ Path: k8s/apps/messaging/memory-queues
  ├─ Namespace: sqs
  └─ Auto-sync: enabled (prune + selfHeal)

Helm Chart: memory-queues
  ├─ Chart.yaml: v0.1.0
  ├─ values.yaml: Queue config
  └─ templates/queues.yaml: Queue CRD resources

Queues Created:

1. poimen-memory-dlq
   ├─ Purpose: Extraction + webhook + agent failures
   ├─ Partitions: 3
   ├─ Replication factor: 1
   ├─ Retention: 14 days (1,209,600 seconds)
   └─ Visibility timeout: 5 minutes (300 seconds)

2. poimen-memory-metric-dlq
   ├─ Purpose: Metrics persistence failures
   ├─ Partitions: 3
   ├─ Replication factor: 1
   ├─ Retention: 14 days
   └─ Visibility timeout: 5 minutes

Resource: Queue CRD (kmsvc.io/v1alpha1)
  └─ Managed by: queue-operator (already running in sqs ns)

Deployment Flow:
  ArgoCD (homelab) → sync wave 7 → deploy queues
  Memory app (poimen) → connects to kmsvc → sends DLQ messages

Files:
  ├─ k8s/apps/messaging/memory-queues/Chart.yaml (new)
  ├─ k8s/apps/messaging/memory-queues/values.yaml (new)
  ├─ k8s/apps/messaging/memory-queues/templates/queues.yaml (new)
  └─ k8s/argocd/apps/50-memory-queues.yaml (new)
 2026-09-05 01:10:51 -07:00
rock e414a3e394 Revert "feat: add memory service queues (processing, indexing, dlq)" 
This reverts commit e5ae5b16b7.
 2026-09-05 01:09:30 -07:00
rock e5ae5b16b7 feat: add memory service queues (processing, indexing, dlq) 
- processing-queue: high-throughput, auto-scaling (1-4 shards)
- indexing-queue: FIFO with deduplication (1-2 shards)
- memory-dlq: dead letter queue for redelivery failures
- ArgoCD Application (wave 7) to auto-sync queue lifecycle
 2026-09-05 01:05:01 -07:00
rock 1fe8707e3c gitops: add secret-rotation controller ArgoCD Application 
- Syncs k8s/apps/secret-rotation-controller/ kustomization
- Auto-prune and self-heal enabled
- Creates secret-rotation namespace
- ArgoCD will deploy CRD, RBAC, ExternalSecret, Deployment
 2026-09-04 13:51:26 -07:00
rock 20513c8b3b iam: add memory scope, service accounts, manual provisioning 
- Add 'memory' scope property mapping (memory_projects, memory_visibility, memory_role)
- Add capability groups: llm-users, memory-users, memory-writers
- Add service account provisioning for portfolio-agent, memory-agent
- Fix sops-secrets kustomization (generatorOptions)
- Add RoleBindings for portfolio, poimen, dashboard namespaces
- Remove PostSync hook - IAM provisioning is now manual-only
 2026-09-03 18:07:22 -07:00
rock 87786d8733 messaging: remove queue-crd/management-service (moved to kmsvc-manage) 
- Applications now managed by kmsvc-root from kmsvc-manage.git
- Added ServerSideApply to homelab-root for proper annotation sync
- Avoids duplicate Application conflicts with Image Updater
 2026-09-03 08:28:38 -07:00
rock 17a98afa3f image-updater: filter to SHA tags only (skip :latest) 
allow-tags: regexp:^[0-9a-f]{7}$ ensures newest-build strategy
compares commit SHA tags, not the stale :latest tag
 2026-09-03 08:07:19 -07:00
rock 7e9ef86826 appproject: allow argo-helm repo for image-updater 2026-09-02 20:32:28 -07:00
rock c6032cc354 argocd: add Image Updater for auto-deploy on image push 
- Install argocd-image-updater via Helm (wave 1)
- Configure Forgejo registry (anonymous pulls)
- Annotate apps for auto-update: api-gw, portfolio, management-service, queue-crd
- Uses newest-build strategy for commit SHA tags
 2026-09-02 20:31:09 -07:00
rock 77683ec7c6 fix: add poimen-memory and poimen-workflows Forgejo repos to sourceRepos 2026-09-02 09:52:41 -07:00
rock 88c9f3047a fix: disable name suffix hash for portfolio-secrets to match deployment reference 2026-09-01 10:38:05 -07:00
rock a6c3fdf786 feat: add portfolio LLM_API_TOKEN to ksops secrets 
- portfolio-secrets.enc.env: FORGEJO_TOKEN + LLM_API_TOKEN for api.riotpiao.com
- kustomization: secretGenerator for ksops handling at deploy time
- Will be SOPS encrypted with homelab age key before merge
 2026-09-01 09:42:02 -07:00
rock 4a1a60104d feat: add portfolio SOPS secret for CI status 2026-08-31 22:33:00 -07:00