Commit Graph
98 Commits
Author SHA1 Message Date
rock 283d30b422 fix(paperless): use correct env var PAPERLESS_SOCIAL_AUTO_SIGNUP for auto-linking
The correct env var is PAPERLESS_SOCIAL_AUTO_SIGNUP not PAPERLESS_SOCIALACCOUNT_AUTO_SIGNUP
This enables automatic linking of Authentik social accounts to existing Paperless users
2026-09-15 02:30:12 +09:00
rock 2b2ceab2d1 fix(paperless): enable OIDC auto-linking to existing users by email
- Add SOCIALACCOUNT_AUTO_SIGNUP for automatic social account creation
- Enable ACCOUNT_ALLOW_SIGNUPS for OIDC users
- Add SOCIALACCOUNT_EMAIL_AUTHENTICATION_AUTO_CONNECT to link existing users
- Fixes 'An account already exists with this email' error
- Allows Authentik login to auto-link to existing Paperless users
2026-09-15 02:00:36 +09:00
rock 6243ac40b7 feat(paperless-ai): enable intelligent timeline tagging with date extraction
- Add ENABLE_INTELLIGENT_TAGGING for advanced AI-powered tagging
- Add ENABLE_DATE_EXTRACTION for automatic date/timeline parsing
- Add ENABLE_TIMELINE_TAGGING to create temporal tags (YYYY, Q#-YYYY, Last-Month, etc)
- Set TAG_EXTRACTION_MODEL to reasoning for sophisticated analysis
- Configure timeline granularity (year, quarter, month)
- Add custom prompt for temporal context extraction
- Auto-create tags for historical and recent documents
2026-09-15 00:53:30 +09:00
rock d6fca68f33 chore(paperless-ai): switch LLM model to reasoning for document processing 2026-09-15 00:38:37 +09:00
rock 7016f764e6 infrastructure(paperless): add paperless-ai production ConfigMap to git
- Commit paperless-ai-config.yaml with production environment variables
- Configure Paperless API endpoint and token
- Enable AI processing with qwen2.5:3b model
- Set up auto-tagging, correspondent/document type extraction
- Configure 60s scan interval for document processing
- Add ConfigMap to kustomization.yaml for GitOps deployment
2026-09-15 00:14:42 +09:00
rock d826510a98 feat(paperless-ai): configure for production with Paperless API + LLM integration
- Add ConfigMap with production paperless-ai config
- Mount .env config file for app startup
- Enable auto-tagging, correspondent extraction, document type detection
- Set LLM API endpoint and token file location
- Configure 60s scan interval for document processing
2026-09-15 00:13:02 +09:00
rock 7e91262257 fix(paperless-ai): write token to /tmp as fallback (PVC mount perms) 2026-09-14 23:58:41 +09:00
rock 0edd6bc73a fix(paperless-ai): use /app/data mount path consistently for init + main container 2026-09-14 23:57:44 +09:00
rock 4ad0102260 fix(paperless-ai): fix sed token extraction to handle JSON spaces 2026-09-14 23:56:33 +09:00
rock dfbe2cc920 fix(paperless-ai): use sed for JWT token extraction (jq not available) 2026-09-14 23:55:53 +09:00
rock ae93c7ca0d fix(paperless-ai): use jq for JWT token extraction instead of grep 2026-09-14 23:54:57 +09:00
rock 39036e6ff3 fix(paperless-ai): move LLM auth client secret from plaintext to SOPS
- Remove hardcoded client secret from init container
- Reference secret from paperless-ai-config (SOPS-encrypted)
- Init container reads LLM_AUTH_CLIENT_SECRET env var
- Secret key must be added to k8s/argocd/secrets/paperless-ai-secrets.enc.yaml
  by someone with SOPS/age key access

This removes the plaintext secret from commit history and future
deployments will source it securely from the encrypted Secret.
2026-09-14 23:07:52 +09:00
rock 8fa5053aef fix: allow paperless direct LLM access, bypass gateway JWT
- Add paperless namespace to llm-serving NetworkPolicy ingress
- Point paperless-ai CUSTOM_BASE_URL to reasoning-predictor directly
- Gateway JWT auth not yet available for service accounts
- paperless-ai-agent has llm:inference role in Authentik
2026-09-14 08:09:26 +09:00
rock cda7153da2 fix: add PAPERLESS_USERNAME env var to paperless-ai (#45)
Required by paperless-ai to find its own user ID for scanning.
Without this, scanning aborts with "Failed to get own user ID".Reviewed-on: #45

Co-authored-by: rock <[email protected]>
2026-09-13 04:58:44 +00:00
rock 0aaa45edb0 fix(comfyui): disable k8s service links to fix COMFYUI_PORT env collision
Root cause: k8s Service named 'comfyui' auto-injects COMFYUI_PORT=tcp://...
into pod env, clobbering ai-dock's own COMFYUI_PORT variable which expects
a port number. This broke caddy's proxy config, leaving port 8188 dead.

Fix: enableServiceLinks: false, revert to port 8188 (ai-dock default).
Also bump startup probe failureThreshold 60->120 (20min budget for model loading).
2026-09-13 09:29:14 +09:00
rock e4f6f03a8c fix(comfyui): correct port 8188->18188, add TLS ingress at comfyui.riotpiao.com (#44)
## Problem
ComfyUI pod restarting 273+ times over 45h. Startup probe failed every time.

## Root Cause
ai-dock image runs ComfyUI on port 18188 (adds 10000 to configured port), not 8188. Caddy does NOT proxy 8188->18188.

## Fix
- deployment: containerPort, probes -> 18188
- service: targetPort -> 18188
- ingress: TLS at comfyui.riotpiao.com, WebSocket headers
- CoreDNS: rewrite comfyui.riotpiao.com

## Post-merge
CoreDNS rewrite needs terraform apply + make apply-cp.
Cloudflare DNS: add CNAME comfyui.riotpiao.com -> tunnel.Reviewed-on: #44

Co-authored-by: rock <[email protected]>
2026-09-13 00:18:54 +00:00
rock 682138c664 fix(paperless-ai): increase memory limit 512Mi -> 2Gi (OOMKilled) 2026-09-13 08:35:55 +09:00
rock f8de5f506b feat(paperless): add paperless-ai with local LLM for auto-tagging
- paperless-ai deployment using clusterzx/paperless-ai
- LLM via local api-gateway (reasoning model, no auth - phase 3 pending)
- Paperless API token SOPS-encrypted
- Auto-tags new documents, 5min scan interval
- Adds 'ai-processed' tag to classified documents
2026-09-13 06:04:49 +09:00
rock 92a8a9ef20 fix(ha): scale all CNPG clusters to 3 replicas, raise overprovisioning to 200%
- authentik-db: 2 → 3 instances
- gotify-db: 1 → 3 instances
- longhorn storageOverProvisioningPercentage: 100 → 200
  (actual disk usage ~44/474 GB on cp-1; thin provisioning is safe)
  Unblocks 3rd Longhorn replica scheduling on cp-1 which had DiskPressure
2026-09-12 19:02:47 +09:00
rock 1d47234f12 fix(databases): Scale authentik-db and gotify-db to 1 replica (Longhorn corruption recovery) 2026-09-12 17:27:45 +09:00
rock 179e12b9a9 feat: vLLM Prometheus metrics scraping (#37)
Add ServiceMonitor for llm-serving namespace. Wire prometheus.io annotations to all vLLM pods (reasoning, ornith, embeddings, reranker). Scrape /metrics@8080 every 30s with proper relabeling.
2026-09-11 10:46:15 +09:00
rock 7bd9f83fa7 fix(cnpg): scale paperless-db, immich-db, gotify-db to 3 instances (#30)
Co-authored-by: rock <[email protected]>
2026-09-10 22:56:41 +00:00
rock 9d3a669dfe fix(gotify): move SOPS secrets to ksops generator
ArgoCD couldn't decrypt secrets.yaml because it was listed as a plain
kustomize resource. Move the 3 secrets (gotify-admin, gotify-tokens,
gotify-smtp) to k8s/argocd/secrets/ as .enc.yaml files processed by
the ksops generator, matching the repo convention.

Fixes ComparisonError: 'Object Kind is missing' (SOPS ciphertext
parsed as raw YAML).
2026-09-10 22:32:24 +09:00
rock 436c7d8d42 fix(argocd): update git repoURLs for org transfer rock -> riotpiao-poimen
Repos transferred: homelab-frontend, kmsvc-manage, poimen, poimen-memory,
poimen-workflows, poimen-frontend. Old URLs return 301 which ArgoCD
doesn't follow.

NOT changed: container image registry paths (rock/ is correct for registry),
riotpiao.com (still under rock org).

Also adds poimen-frontend to AppProject sourceRepos allowlist.
2026-09-10 10:49:30 +09:00
rock 128e76ce2d feat(gotify): push notification server + SMTP email relay (#19)
Co-authored-by: rock <[email protected]>
2026-09-10 01:35:59 +00:00
rock 8790de6038 feat: add ComfyUI + rebalance GPU allocation (#17)
## GPU Rebalance (4× V100 32GB)

| Pod | Before | After |
|-----|--------|-------|
| reasoning (PP=2) | 2 GPU | 2 GPU |
| ornith | 2 GPU (2 replicas) | 1 GPU (1 replica) |
| comfyui | — | 1 GPU (**new**) |
| qwen-cpu | — | CPU on cp-2 (**new**) |
| embeddings/reranker | CPU | CPU |

## Changes

- `ornith.yaml`: scale 2→1, remove qwen2.5 co-loading, MAX_LOADED_MODELS=1
- `qwen-cpu.yaml`: new Ollama deployment on talos-cp-2 (144GB RAM), 5Gi PVC
- `k8s/apps/comfyui/`: new ComfyUI deployment (1 GPU, 50Gi model PVC, ingress)
- `58-comfyui.yaml`: ArgoCD Application (wave 8)

Gateway route update in separate PR (homelab-frontend).Reviewed-on: #17

Co-authored-by: rock <[email protected]>
2026-09-09 02:11:28 +00:00
rock 95c67f9437 P3.8: Restrict llm-serving ingress to api-gateway only (#15)
## Summary

Replaces hand-applied `llm-serving-default-deny` NetworkPolicy with a git-managed, namespace-scoped policy that only allows traffic from the api-gateway.

**Issue:** #13

Co-authored-by: rock <[email protected]>
2026-09-08 16:56:36 +00:00
rock 97c951bef3 Phase 6.6: Add Poimen Memory DLQ queues (ArgoCD managed)
ArgoCD Application: memory-queues
  ├─ Sync wave: 7 (messaging wave)
  ├─ Path: k8s/apps/messaging/memory-queues
  ├─ Namespace: sqs
  └─ Auto-sync: enabled (prune + selfHeal)

Helm Chart: memory-queues
  ├─ Chart.yaml: v0.1.0
  ├─ values.yaml: Queue config
  └─ templates/queues.yaml: Queue CRD resources

Queues Created:

1. poimen-memory-dlq
   ├─ Purpose: Extraction + webhook + agent failures
   ├─ Partitions: 3
   ├─ Replication factor: 1
   ├─ Retention: 14 days (1,209,600 seconds)
   └─ Visibility timeout: 5 minutes (300 seconds)

2. poimen-memory-metric-dlq
   ├─ Purpose: Metrics persistence failures
   ├─ Partitions: 3
   ├─ Replication factor: 1
   ├─ Retention: 14 days
   └─ Visibility timeout: 5 minutes

Resource: Queue CRD (kmsvc.io/v1alpha1)
  └─ Managed by: queue-operator (already running in sqs ns)

Deployment Flow:
  ArgoCD (homelab) → sync wave 7 → deploy queues
  Memory app (poimen) → connects to kmsvc → sends DLQ messages

Files:
  ├─ k8s/apps/messaging/memory-queues/Chart.yaml (new)
  ├─ k8s/apps/messaging/memory-queues/values.yaml (new)
  ├─ k8s/apps/messaging/memory-queues/templates/queues.yaml (new)
  └─ k8s/argocd/apps/50-memory-queues.yaml (new)
2026-09-05 01:10:51 -07:00
rock e414a3e394 Revert "feat: add memory service queues (processing, indexing, dlq)"
This reverts commit e5ae5b16b7.
2026-09-05 01:09:30 -07:00
rock e5ae5b16b7 feat: add memory service queues (processing, indexing, dlq)
- processing-queue: high-throughput, auto-scaling (1-4 shards)
- indexing-queue: FIFO with deduplication (1-2 shards)
- memory-dlq: dead letter queue for redelivery failures
- ArgoCD Application (wave 7) to auto-sync queue lifecycle
2026-09-05 01:05:01 -07:00
rock 82a4e3e4fe feat: automated secret rotation controller
- ExternalSecret syncs age key from Vault to pod
- CRD defines rotation schedule for each secret
- Controller watches CRD, rotates on schedule:
  * Call provider API (Authentik/Forgejo/MinIO) for new secret
  * Update k8s Secret
  * Update .enc.yaml via sops (uses age key from Vault)
  * Git commit and push
- Vault is source of truth for age key (never on disk)
- Examples: minio-oidc (90d), portfolio-agent (90d), forgejo-token (90d), minio-root (180d)
2026-09-03 23:37:24 -07:00
rock f21cc4721a kmsvc: migrate image from GHCR to Forgejo registry
Consolidate all internal images to forgejo.riotpiao.com for ArgoCD Image Updater
2026-09-02 20:12:35 -07:00
Story Crater Bot e4de366d2a feat: add ServiceAccounts for poimen-memory/kmsvc/immich operator Roles
Bind each service's operator Role to a ServiceAccount alongside the
existing oidc:*-admins Group, and wire serviceAccountName into the
pods we control (immich-server, immich-machine-learning,
management-service). poimen-memory's Deployment lives in its own
repo/ArgoCD app, so its SA is created here but not yet wired to a pod.
2026-08-26 19:51:05 -07:00
Story Crater Bot 7f0541a8dc fix: shrink paperless-media to 500Gi, give immich the bigger share (2000Gi) - photo libraries grow faster than scanned docs 2026-08-25 19:23:07 -07:00
Story Crater Bot 2b01e7c5f3 fix: add cube+earthdistance to postInitApplicationSQL, immich needs them for geo queries 2026-08-25 18:50:09 -07:00
Story Crater Bot abed329636 fix: add postInitApplicationSQL for pgvector, app role isn't superuser
immich-server crash-looped on "permission denied to create extension
vector" - pgvector's control file isn't marked trusted, and CNPG's
app owner role isn't superuser (enableSuperuserAccess: false).
Documents the fix for future cluster rebuilds; the live cluster
already had CREATE EXTENSION run manually via the postgres pod's
local socket.
2026-08-25 18:49:09 -07:00
Story Crater Bot 0e57444909 fix: use img.riotpiao.com instead of immich.riotpiao.com for hostname 2026-08-25 18:46:58 -07:00
Story Crater Bot d0f79871c7 fix: shrink immich-media to 1400Gi, real disk headroom smaller than assumed
2000Gi didn't schedule - "insufficient storage; tags not fulfilled".
The cp-3 HDD's real usable capacity (~3724GiB) minus paperless-media's
2000Gi and ~231GiB of other apps' default-class replicas that Longhorn
placed here anyway (tags only pull matching volumes in, don't exclude
others when the untagged pool elsewhere is full) only leaves ~1493Gi
of real scheduling headroom. 1400Gi fits with margin.
2026-08-25 18:34:22 -07:00
Story Crater Bot 61a1975669 feat: deploy Immich with Authentik OIDC, rock as admin
Self-hosted photo backup (Google Photos replacement) - raw manifests,
no Helm chart, self-contained under k8s/apps/immich including its own
CNPG Postgres. Media PVC shares the cp-3 HDD 2TB/2TB with
paperless-media.

Postgres is pg18, not this repo's usual 16.2: CNPG's official pgvector
extension image (ghcr.io/cloudnative-pg/pgvector) is only published
for pg18, loaded via CNPG's ImageVolume extension mechanism (operator
1.30.0 / k8s 1.36.1 both support it). Immich auto-manages CREATE
EXTENSION itself at startup.

OIDC via a new "immich_role" Authentik scope mapping (homelab-admins/
immich-admins -> "admin" claim, else "user"), consumed by Immich's
OAuth roleClaim setting which re-syncs isAdmin on every login - more
reliable than Immich's racy first-user-is-admin fallback. Config
composed into an immich-oidc Secret and mounted as IMMICH_CONFIG_FILE,
matching the paperless-oidc pattern. k8s RBAC (immich-operator Role +
oidc:immich-admins binding) mirrors paperless/rbac.yaml.

immich namespace pre-created in k8s/infra/databases/namespaces.yaml
(not just immich's own CreateNamespace=true) since the iam PostSync
job's RoleBinding needs it to exist before wave 8.
2026-08-25 18:21:28 -07:00
Story Crater Bot 1518ebc2dd fix: shrink paperless-media to 2TB, split cp-3 HDD with Immich
4TB disk on cp-3 was single-tenant for paperless (3500Gi). Splitting
2TB/2TB with the new Immich media PVC on the same disk/tag. Live PVC
and Longhorn volume already deleted+recreated manually (data was
outdated test uploads only, nightly MinIO backup covers it).
2026-08-25 18:09:23 -07:00
Story Crater Bot 1260921450 fix: use 2.20.15 instead of 3.0.5, direct 2.13->3.0 migration is blocked
paperless-ngx v3 refuses to migrate from anything before v2.20.15
(paperless.E002). 2.20.15 already covers the API version range the
iOS app needs, so it fixes the phone upload issue without the v3
breaking changes.
2026-08-25 17:40:42 -07:00
Story Crater Bot b73903afb1 fix: bump paperless-ngx to 3.0.5 for iOS app API version compat
Swift Paperless needs REST API v3-9; server was on 2.13 (v1-5 only),
causing 406 on /api/token/ for all phone uploads. v3 requires
PAPERLESS_DBENGINE explicit instead of inferred from PAPERLESS_DBHOST.
2026-08-25 17:37:41 -07:00
Story Crater Bot 89f01b6f2e feat: add homelab-wide Authentik RBAC model and k8s OIDC auth wiring
Adds permissions claim + per-service admin groups in Authentik, scoped
Role/RoleBinding per service, public PKCE kubernetes OAuth2 client, and
kube-apiserver OIDC extraArgs. Also fixes paperless OIDC signup permissions
via adapter override and adds CoreDNS rewrite for authentik.riotpiao.com.
2026-08-25 15:03:44 -07:00
Story Crater Bot 8b49b347b5 fix: disable email verification requirement on paperless OIDC signup
allauth defaulted to ACCOUNT_EMAIL_VERIFICATION=mandatory, and building the confirmation link 500'd with NoReverseMatch on account_confirm_email (paperless-ngx doesn't wire up that view, no SMTP configured either). Authentik already verifies identity via OIDC, so this step is redundant.
2026-08-25 13:05:50 -07:00
Story Crater Bot e7c75bb9ce fix: disable enableServiceLinks on paperless pod to stop gunicorn crash-loop
Service "paperless" made k8s inject PAPERLESS_PORT=tcp://<ip>:8000 as a legacy Docker-links env var, which paperless-ngx's own entrypoint also reads for gunicorn's bind port - collided, gunicorn crash-looped ("not a valid port number"), and the pod was 1/1 Running with nothing actually listening (nginx saw 502).
2026-08-25 12:26:16 -07:00
Story Crater Bot e8e5acfb13 feat: add paperless-ngx with OIDC, CNPG db, cp-3 HDD media, MinIO backup
Fixes controlplane.tftpl's install.wipe:true (should be false, live CPs already run false) and syncs coredns Corefile back to what's actually deployed (drops an unrolled-out, stale Kong-era rewrite).
2026-08-25 11:11:37 -07:00
Story Crater Bot 130746e6a1 Add Temporal worker for production task queue 2026-08-21 16:44:59 -07:00
Story Crater Bot 236c9e189d fix: restore YaRN rope-scaling for reasoning-predictor (GPTQ requant dropped it, checkpoint's own ceiling was 40960 not 131072) 2026-08-21 16:42:39 -07:00
Story Crater Bot b93e7e3362 fix: swap reasoning-predictor to Qwen3-32B-GPTQ-Int4, 131072 context (bnb-4bit decode too slow, GPTQ is Volta-native) 2026-08-21 16:39:33 -07:00
Story Crater Bot cfd5a96331 fix: retire one reasoning-predictor replica, run PP=2 across both V100s (Qwen3.5 MoE swap abandoned, moving to Ollama) 2026-08-21 16:23:07 -07:00