- Remove hardcoded client secret from init container
- Reference secret from paperless-ai-config (SOPS-encrypted)
- Init container reads LLM_AUTH_CLIENT_SECRET env var
- Secret key must be added to k8s/argocd/secrets/paperless-ai-secrets.enc.yaml
by someone with SOPS/age key access
This removes the plaintext secret from commit history and future
deployments will source it securely from the encrypted Secret.
- Add paperless namespace to llm-serving NetworkPolicy ingress
- Point paperless-ai CUSTOM_BASE_URL to reasoning-predictor directly
- Gateway JWT auth not yet available for service accounts
- paperless-ai-agent has llm:inference role in Authentik
Required by paperless-ai to find its own user ID for scanning.
Without this, scanning aborts with "Failed to get own user ID".Reviewed-on: #45
Co-authored-by: rock <[email protected]>
Root cause: k8s Service named 'comfyui' auto-injects COMFYUI_PORT=tcp://...
into pod env, clobbering ai-dock's own COMFYUI_PORT variable which expects
a port number. This broke caddy's proxy config, leaving port 8188 dead.
Fix: enableServiceLinks: false, revert to port 8188 (ai-dock default).
Also bump startup probe failureThreshold 60->120 (20min budget for model loading).
- paperless-ai deployment using clusterzx/paperless-ai
- LLM via local api-gateway (reasoning model, no auth - phase 3 pending)
- Paperless API token SOPS-encrypted
- Auto-tags new documents, 5min scan interval
- Adds 'ai-processed' tag to classified documents
Add ServiceMonitor for llm-serving namespace. Wire prometheus.io annotations to all vLLM pods (reasoning, ornith, embeddings, reranker). Scrape /metrics@8080 every 30s with proper relabeling.
ArgoCD couldn't decrypt secrets.yaml because it was listed as a plain
kustomize resource. Move the 3 secrets (gotify-admin, gotify-tokens,
gotify-smtp) to k8s/argocd/secrets/ as .enc.yaml files processed by
the ksops generator, matching the repo convention.
Fixes ComparisonError: 'Object Kind is missing' (SOPS ciphertext
parsed as raw YAML).
Repos transferred: homelab-frontend, kmsvc-manage, poimen, poimen-memory,
poimen-workflows, poimen-frontend. Old URLs return 301 which ArgoCD
doesn't follow.
NOT changed: container image registry paths (rock/ is correct for registry),
riotpiao.com (still under rock org).
Also adds poimen-frontend to AppProject sourceRepos allowlist.
## Summary
Replaces hand-applied `llm-serving-default-deny` NetworkPolicy with a git-managed, namespace-scoped policy that only allows traffic from the api-gateway.
**Issue:** #13
Co-authored-by: rock <[email protected]>
- ExternalSecret syncs age key from Vault to pod
- CRD defines rotation schedule for each secret
- Controller watches CRD, rotates on schedule:
* Call provider API (Authentik/Forgejo/MinIO) for new secret
* Update k8s Secret
* Update .enc.yaml via sops (uses age key from Vault)
* Git commit and push
- Vault is source of truth for age key (never on disk)
- Examples: minio-oidc (90d), portfolio-agent (90d), forgejo-token (90d), minio-root (180d)
Bind each service's operator Role to a ServiceAccount alongside the
existing oidc:*-admins Group, and wire serviceAccountName into the
pods we control (immich-server, immich-machine-learning,
management-service). poimen-memory's Deployment lives in its own
repo/ArgoCD app, so its SA is created here but not yet wired to a pod.
immich-server crash-looped on "permission denied to create extension
vector" - pgvector's control file isn't marked trusted, and CNPG's
app owner role isn't superuser (enableSuperuserAccess: false).
Documents the fix for future cluster rebuilds; the live cluster
already had CREATE EXTENSION run manually via the postgres pod's
local socket.
2000Gi didn't schedule - "insufficient storage; tags not fulfilled".
The cp-3 HDD's real usable capacity (~3724GiB) minus paperless-media's
2000Gi and ~231GiB of other apps' default-class replicas that Longhorn
placed here anyway (tags only pull matching volumes in, don't exclude
others when the untagged pool elsewhere is full) only leaves ~1493Gi
of real scheduling headroom. 1400Gi fits with margin.
Self-hosted photo backup (Google Photos replacement) - raw manifests,
no Helm chart, self-contained under k8s/apps/immich including its own
CNPG Postgres. Media PVC shares the cp-3 HDD 2TB/2TB with
paperless-media.
Postgres is pg18, not this repo's usual 16.2: CNPG's official pgvector
extension image (ghcr.io/cloudnative-pg/pgvector) is only published
for pg18, loaded via CNPG's ImageVolume extension mechanism (operator
1.30.0 / k8s 1.36.1 both support it). Immich auto-manages CREATE
EXTENSION itself at startup.
OIDC via a new "immich_role" Authentik scope mapping (homelab-admins/
immich-admins -> "admin" claim, else "user"), consumed by Immich's
OAuth roleClaim setting which re-syncs isAdmin on every login - more
reliable than Immich's racy first-user-is-admin fallback. Config
composed into an immich-oidc Secret and mounted as IMMICH_CONFIG_FILE,
matching the paperless-oidc pattern. k8s RBAC (immich-operator Role +
oidc:immich-admins binding) mirrors paperless/rbac.yaml.
immich namespace pre-created in k8s/infra/databases/namespaces.yaml
(not just immich's own CreateNamespace=true) since the iam PostSync
job's RoleBinding needs it to exist before wave 8.
4TB disk on cp-3 was single-tenant for paperless (3500Gi). Splitting
2TB/2TB with the new Immich media PVC on the same disk/tag. Live PVC
and Longhorn volume already deleted+recreated manually (data was
outdated test uploads only, nightly MinIO backup covers it).
paperless-ngx v3 refuses to migrate from anything before v2.20.15
(paperless.E002). 2.20.15 already covers the API version range the
iOS app needs, so it fixes the phone upload issue without the v3
breaking changes.
Swift Paperless needs REST API v3-9; server was on 2.13 (v1-5 only),
causing 406 on /api/token/ for all phone uploads. v3 requires
PAPERLESS_DBENGINE explicit instead of inferred from PAPERLESS_DBHOST.
Adds permissions claim + per-service admin groups in Authentik, scoped
Role/RoleBinding per service, public PKCE kubernetes OAuth2 client, and
kube-apiserver OIDC extraArgs. Also fixes paperless OIDC signup permissions
via adapter override and adds CoreDNS rewrite for authentik.riotpiao.com.
allauth defaulted to ACCOUNT_EMAIL_VERIFICATION=mandatory, and building the confirmation link 500'd with NoReverseMatch on account_confirm_email (paperless-ngx doesn't wire up that view, no SMTP configured either). Authentik already verifies identity via OIDC, so this step is redundant.
Service "paperless" made k8s inject PAPERLESS_PORT=tcp://<ip>:8000 as a legacy Docker-links env var, which paperless-ngx's own entrypoint also reads for gunicorn's bind port - collided, gunicorn crash-looped ("not a valid port number"), and the pod was 1/1 Running with nothing actually listening (nginx saw 502).
Fixes controlplane.tftpl's install.wipe:true (should be false, live CPs already run false) and syncs coredns Corefile back to what's actually deployed (drops an unrolled-out, stale Kong-era rewrite).
Gateway pods are ErrImagePull — nodes resolve forgejo.riotpiao.com to a
ClusterIP and time out, so the Service had no endpoints and the host was
returning 503. Kong is still running; this restores it.
Ingress api/api now backs onto api-gateway:8080; the kong Application, its
Helm values, plugins and llm-routes are removed. Gateway image v0.0.0 is in
the Forgejo registry and the pull secret is in the api namespace.
Old i===0 && !resuming gate meant this only ran on a fresh start -- every
run this session was a resume, so poiman's branch never got the harness
gitignore rules, and portfolio's PLAN.md stayed tracked from before the
rule existed (gitignore doesn't affect already-tracked files). Now checks
and fixes both on every phase instead of once at genesis.
Dead sessions were only caught after a full 10-min stall timeout; now
polled via agent-manager status and respawned (retry once). spawnPi had
no timeout and could hang a repo's whole pipeline forever -- bounded to
5 minutes now.
32 blocks was a ~1GB safety-valve leftover from the num_cpu_blocks=2000
hang incident, not meaningful offload capacity. This model's KV cache is
~32MB/128-token block (64 layers, 8 KV heads x 128 head_dim, fp16) --
256 blocks gives ~8GB of real DRAM offload (32,768 tokens), comfortably
under the pod's 36Gi limit alongside the ~20GB bnb-4bit weights.
DeepSeek-R1-distill's tool_choice=auto narration bug needed a real fix,
not a workaround -- Qwen3's native tool-call format (hermes-compatible
chat template) solves it at the source instead of parsing around it.
Dense Qwen3-32B avoids the MoE arch/quantization pitfalls hit by the two
prior swap attempts (Kimi-distilled Qwen3.6 MoE, AWQ Qwen3-30B-A3B) --
same bnb-4bit path already proven working on this sm70 (V100) node.
Three straight failures on worker-1: Kimi-K2.6-distilled Qwen3.6-35B-A3B
had an unrecognized model type (qwen3_5_moe); the AWQ-4bit fallback needed
compute capability 80+ (marlin INT4 kernels) but this node's GPU is sm70
(V100); on-the-fly bitsandbytes against the full-precision Qwen3-30B-A3B
kept crash-looping. Reverting to the last known-good config (596b5cb) --
tool-call narration bug on judge remains open, to revisit separately.
rock
Story Crater Bot