Author SHA1 Message Date
Story Crater Bot dd491f6f8b feat(llm-serving): offload reasoning's KV cache to CPU DRAM
vLLM 0.11.0's native OffloadingConnector -- spills KV blocks to CPU RAM on
preemption instead of discarding them, avoiding recompute. Built into vLLM
core, no extra dependency. Bumped memory request/limit (+4Gi/replica) to
give the CPU block pool real room; worker-1 had ~18Gi of request headroom
across both replicas.
2026-08-19 15:02:28 -07:00
Story Crater Bot 5b041df884 fix(agent-pod): committed progress ledger so resume skips done tasks
Resuming the phase branch alone only recovers the code -- the task loop
still walked from the first task, re-verifying every already-done one
through a full planner call before reaching the first task that actually
needed work. .agent-progress is committed (not gitignored) and appended
per completed task, so a resumed run reads it once and skips straight
past known-done tasks with zero LLM calls. Validated locally against a
throwaway repo: second run skipped both tasks instantly (resumed: true)
instead of re-running planner on them.
2026-08-19 13:16:03 -07:00
Story Crater Bot 3ea057d83e fix(agent-pod): group sessions by repo, resume phase branches, fix empty-diff bug
- agent-manager spawn now gets --group repoId, so the TUI clusters
  planner/investigator/implementer/judge under one repo heading instead of
  4 unrelated sessions.
- runPhase was called with phaseBranch where it needed the true baseBranch,
  so every per-task judge review compared phaseBranch...HEAD -- always
  empty, since HEAD is phaseBranch while checked out. Judges only produced
  real verdicts anyway because they fell back to their own git log/show.
- Every restart re-cloned baseBranch fresh and started a new phase branch,
  discarding whatever a prior run had already committed mid-phase. Now:
  fetch+resume an existing phase branch if origin has one, push after every
  task instead of only at phase-end, and delete the phase branch (local +
  origin) once its milestone squash-merges into base.
2026-08-19 11:46:53 -07:00
Story Crater Bot 43c0e1faa2 fix(agent-pod): absolute paths for every sentinel/verdict file, cwd reminder per call
A pooled session's shell cwd drifts as it explores the repo between turns.
Seen live: a repo whose internal workspace dir is one letter off from the
repo's own directory name was enough for the agent to touch its sentinel
one level off from where coordinator watches for it -- coordinator waited
out the full timeout for a file that existed, just in the wrong place.
2026-08-19 11:26:33 -07:00
Story Crater Bot 3a244577e4 fix(agent-pod): fold judgeOnly status check into planner, drop separate judge pre-check 2026-08-19 10:57:21 -07:00
Story Crater Bot 7a0d09cbe0 fix(agent-pod): install python3 and sqlite3 in the container init 2026-08-19 10:46:50 -07:00
Story Crater Bot cb52356d13 fix(agent-pod): sync coordinator.js (slug repoId), tighten compaction, route judge to reasoning model 2026-08-19 10:41:33 -07:00
Story Crater Bot d0cbfac7a4 fix(agent-pod): stateless role pool (/new per reuse), never commit PLAN.md 2026-08-19 07:59:01 -07:00
Story Crater Bot 930374a3b8 feat(agent-pod): persistent per-role agent pool, concurrency moves to repo level
coordinator.js now runs one long-lived planner/investigator/implementer/judge
session per repo (reused across every task via tmux send-keys) instead of a
fresh spawn per task per stage. Tasks within a repo run sequentially against
that pool; concurrency is now REPO_CONCURRENCY (default 3) concurrent repos
via a new --repos flag, not concurrent tasks in one repo's phase.
2026-08-18 21:30:49 -07:00
Story Crater Bot b10d1c3a25 fix(llm-serving): use hermes tool-call parser, not deepseek_v3
deepseek_v3 400s on this checkpoint: "could not locate tool call start/end tokens in the tokenizer". unsloth/DeepSeek-R1-Distill-Qwen-32B is a Qwen2.5 base distilled on R1 reasoning traces -- it kept R1's <think> format but never got DeepSeek-V3's own special tool-call tokens registered in its tokenizer. hermes parses from text patterns instead of special tokens, so it works against the underlying Qwen tokenizer.
2026-08-18 20:56:40 -07:00
Story Crater Bot 4c54674dff fix(llm-serving): enable tool calling on homelab-reasoning
pi sends tool_choice="auto" for every session (Read/Bash/etc.) -- vLLM 400s on that without --enable-auto-tool-choice and a --tool-call-parser. Verified this deployed vLLM v0.11.0's registered parsers directly; deepseek_v3 matches, same family as the deepseek_r1 reasoning-parser already set (this Qwen-base distillation still emits DeepSeek's own tool-call format).
2026-08-18 20:50:21 -07:00
Story Crater Bot 05ff12c117 fix(agent-pod): use process.exitCode not process.exit() in coordinator.js
process.exit() right after console.log() can drop buffered stdout when it's piped (not a TTY) -- exactly kubectl exec's case. Explains the silent empty-output-exit-1 failures. process.exitCode + natural exit lets the event loop drain and flush first.
2026-08-18 19:13:07 -07:00
Story Crater Bot 56b1c96fcf fix(agent-pod): clone deterministically, not through a headless LLM call
git clone is mechanical -- routing it through spawnPi meant a crash gave zero diagnostic output, just a silent exit code. Direct runGit call now, same as commitPending/the squash-merge sequence. Drops the now-unused runStageWithResolver.
2026-08-18 18:57:18 -07:00
Story Crater Bot 26a959215e fix(agent-pod): sync coordinator.js ConfigMap, was stale since auto-discovery landed
The pod's coordinator-src ConfigMap still had the pre-auto-discovery version -- --tasks was required, no task-board parsing, no self-chained stages, no judge model routing. Regenerated from the current source.
2026-08-18 18:42:45 -07:00
Story Crater Bot 193b040de6 feat(agent-pod): implementer and judge learn Playwright for UI verification
Both skills already have Bash in allowed-tools -- no new pi capability needed. For UI/frontend work, implementer screenshots/clicks through the golden path via npx playwright instead of trusting that code compiling means it renders correctly; judge does the same as review evidence, FAILing on visual defects a diff alone wouldn't show. Doesn't apply to non-UI work.
2026-08-18 18:37:46 -07:00
Story Crater Bot 12778d5576 feat(llm-serving): scale ornith to 2 replicas instead of a dedicated grm GPU
reasoning keeps its 2 GPUs untouched. verifier's freed GPU goes to a second ornith replica instead of a standalone qwen-only pod -- both replicas load ornith:35b + qwen2.5:3b-instruct, k8s Service load-balances across them, so 2 concurrent implementer-style calls get independent instances.
2026-08-18 18:25:12 -07:00
Story Crater Bot cd6c620619 feat(llm-serving): retire verifier-predictor, add grm (qwen2.5:3b)
Frees verifier's GPU from an underused vLLM PRM deployment. qwen2.5:3b-instruct moves off ornith-predictor's shared pod onto its own dedicated GPU (grm.yaml), so verification/judge traffic stops contending with ornith:35b's agent traffic. /v1/qwen/chat/completions now points at grm-predictor; path unchanged.
2026-08-18 18:18:31 -07:00
Story Crater Bot 0a87302e19 fix(api): retire Kong key-auth on model routes; agent-pod builds agent-manager fork + ships coordinator.js
Kong key-auth rejected the Authorization: Bearer header every OpenAI-SDK-compatible client sends (verified: raw apikey header works, Bearer doesn't), so it's commented out and stripped from every llm-routes.yaml annotation until there's a Bearer-compatible fix. agent-pod now clones and builds the agent-manager fork from source at container start (no prebuilt binary shipped -- wrong arch and over ConfigMap's size cap) and ships coordinator.js alongside hub.js, so multiple repos can run the pipeline concurrently in one pod via kubectl exec. hub.js keeps its existing role as the container's foreground process, unchanged.
2026-08-18 17:50:52 -07:00
Story Crater Bot c64b68a36b fix(agent-pod): remote tui session for multi-agent 2026-08-18 15:08:04 -07:00
Story Crater Bot 4146a048c9 fix(ci): make the hardcoded-secret scan blocking and close the .gitignore/.sops.yaml gaps that let a plaintext deploy key through — also untracks tfplan binaries and skills-lock.json 2026-08-18 15:08:04 -07:00
Story Crater Bot f0fa1dbd27 fix(argocd): clone the public GitHub seed anonymously over HTTPS and delete the SSH deploy-key Secret — its private half had been committed in plaintext to a public remote, and a public repo needs no credential at all 2026-08-18 15:08:04 -07:00
Story Crater Bot 2b1c4b1df4 fix(forgejo): strategy Recreate for RWO data PVC — RollingUpdate deadlocked (new pod Multi-Attach error on the RWO gitea PVC held by the old pod, stuck Init forever) 2026-08-18 15:08:04 -07:00
Story Crater Bot 479318c532 fix(authentik): label argocd oidc-secret part-of=argocd — argocd's $secret substitution only reads labelled Secrets; without it OIDC login failed with oauth2 invalid_client (empty client_secret to IdP) 2026-08-18 15:08:04 -07:00
Story Crater Bot ff216429b9 feat(argocd): wire Authentik OIDC + local rock/cicd accounts + RBAC — adds oidc.config (homelab-admins->admin SSO), url, accounts.rock (login+apiKey) and accounts.cicd (apiKey for CD pipeline token), all role:admin 2026-08-18 15:08:04 -07:00
Story Crater Bot 7441aaf9c3 fix(homarr): raise CPU limit 500m->2 + disable analytics cron — Next.js aborted with exit 134 (SIGABRT) under CPU throttle during icon-updater/analytics, self-restarting in a loop and 502ing at the ingress 2026-08-18 15:08:04 -07:00
Story Crater Bot edd739198d fix(cilium): restrict L2 announcement to control-plane nodes — GPU worker lacks eno1 (Mellanox enp28s0f*), so when it won the .160 lease it couldn't ARP the VIP, black-holing all ingress (flapped on reboots) 2026-08-18 15:08:04 -07:00
Story Crater Bot 20f8aac95d fix(api): label Kong pods llm-client=true so llm-serving NetworkPolicy admits them — chat/embeddings/rerank/score routes silently hung until the client timeout because Cilium dropped Kong's packets
llm-serving-default-deny admits port 8080 only from pods carrying
llm-client=true. Kong lacked it, so every route that actually contacts an
upstream timed out. /v1/models masked the problem: request-termination answers
inside Kong and never touches an upstream, so it returned 200 throughout.

Opting in via podLabels rather than relaxing the policy — it is a compensating
control, not hygiene, since vLLM v0.11.0 is frozen on Volta and will not receive
patches for several remote/unauthenticated advisories.

podLabels land only in the pod template, not spec.selector.matchLabels, so this
is not an immutable-field change.
2026-08-18 15:08:04 -07:00
Story Crater Bot dbd3dc7b3d feat(api): add DeepSeek-shaped LLM API on Kong — /v1/models, per-model chat completions, embeddings, rerank, score; disable Kong response buffering so stream:true actually streams
Kong matches routes on host/path/method/header, never on the request body, so a
single /v1/chat/completions dispatching on body.model is not expressible in Kong
OSS (ai-proxy-advanced, which does multi-target model routing, is Enterprise).
Model therefore goes in the path:

  GET  /v1/models                        static list (request-termination)
  POST /v1/reasoning/chat/completions     reasoning-predictor  (vLLM)
  POST /v1/ornith/chat/completions        ornith-predictor     (Ollama)
  POST /v1/qwen/chat/completions          ornith-predictor     (Ollama, same pod)
  POST /v1/embeddings                     embeddings-predictor (TEI)
  POST /v1/rerank                         reranker-predictor   (TEI)
  POST /v1/score                          verifier-predictor   (vLLM pooling)

- each chat route force-overwrites body.model via request-transformer add+replace:
  ornith:35b and qwen2.5:3b-instruct share one Ollama pod, so without this a
  client hitting /v1/qwen with "model":"ornith:35b" would silently get the 35B
- routes live in ns llm-serving, not api: an Ingress can only reference a Service
  in its own namespace, and KIC watches all namespaces
- embeddings and score need no rewrite (TEI/vLLM already serve the canonical
  paths); rerank does, since /v1/rerank 404s and only /rerank exists
- read/write timeouts 1h: Kong defaults to 60s, which a 32B model on Volta
  exceeds mid-generation and returns 504
- nginx_proxy_proxy_buffering=off: buffered responses lump or stall SSE, and both
  hops (nginx Ingress and Kong) must be unbuffered or the buffered one wins
- no auth for now, per decision; api.riotpiao.com is reachable through nginx, so
  GPU time is currently unauthenticated
2026-08-18 15:08:04 -07:00
Story Crater Bot bebe8dc31b fix(ingress): remove stale ingress-nginx-controller-alias Service — its selfHeal kept clobbering the helm LoadBalancer Service (same name, dead ingress-nginx-bootstrap selector, 0 endpoints), unannouncing LB IP .160 and taking down all ingress 2026-08-18 15:08:04 -07:00
Story Crater Bot 4c0d30ce30 fix(homarr): add AUTH_OIDC_URI + email account linking — homarr hides the Authentik sign-in button unless AUTH_OIDC_URI (authorize endpoint) is set alongside AUTH_OIDC_ISSUER (per authentik/homarr SSO docs); was the missing var 2026-08-18 15:08:04 -07:00
Story Crater Bot a17ceedcd8 refactor(ingress): drop redundant ArgoCD ingress-nginx app — chart 4.15.1 was double-managed by both the helm-bootstrap release and this ArgoCD app (same chart), fighting over the controller/LB service (ingress-config drift). ingress-nginx is bootstrap-critical (ArgoCD's own reachability path), so helm-bootstrap is the single owner 2026-08-18 15:08:04 -07:00
Story Crater Bot 9a779ccaf4 feat(sms): add BlueBubbles iMessage delivery (Docker-OSX macOS VM pinned to worker-2) + ArgoCD app + dedicated longhorn-imessage-local SC — default longhorn SC can't schedule a 3-replica 200Gi volume (only worker-1 has 200Gi free at 100% over-provisioning) and Immediate binding would pin the qcow2 to the wrong node
- namespace: PodSecurity privileged, needed for /dev/kvm + privileged QEMU
- storageclass: 1 replica, strict-local, WaitForFirstConsumer
- deployment: nodeSelector workload=imessage + matching NoSchedule toleration,
  Recreate strategy (two QEMU procs on one qcow2 corrupts it), no readiness
  probe (guest install is interactive and takes many minutes)
- services: ClusterIP only; VNC is an unauthenticated console, reach it with
  port-forward, never an Ingress
- networkpolicy: default-deny, opt-in via sms-client=true on port 1234
2026-08-18 15:08:04 -07:00
Story Crater Bot 20d0517f79 feat(monitoring): enable Alertmanager (null receiver, longhorn PVC, az-a) + fix forgejo-rules ns forgejo->cicd — alerting delivery was disabled; forgejo PrometheusRule targeted a nonexistent namespace 2026-08-18 15:08:04 -07:00
Story Crater Bot c8ea7b9190 fix(prometheus): use longhorn StorageClass, drop nonexistent longhorn-wffc — Prometheus CR requested storageClass longhorn-wffc which doesn't exist (deprecated), so operator never created the StatefulSet (Reconciled=False, no metrics server) 2026-08-18 15:08:04 -07:00
Story Crater Bot d3e2215b5c fix(homarr): tune probes via chart values, drop fragile fix-probes-job — first-boot icon updater blocks health endpoint ~50s; default 10s×3 liveness SIGTERMs the pod (247 restarts, 503); chart exposes probes so the PostSync patch-job was unnecessary and reverted on every rollout 2026-08-18 15:08:04 -07:00
Story Crater Bot c00b2d1b53 fix(authentik): add minio policy scope mapping (homelab-admins->consoleAdmin else readonly) + set rock email — MinIO CLAIM_NAME=policy got no claim (no MinIO access); empty rock email broke Grafana OIDC (GitHub-style /emails 404) 2026-08-18 15:08:04 -07:00
Story Crater Bot db6bf742da fix(grafana): add email/login/name_attribute_path for Authentik OIDC — Grafana was falling back to GitHub-style <api_url>/emails (404 'Error getting email address'), breaking OAuth login; read identity from userinfo claims instead 2026-08-18 15:08:04 -07:00
Story Crater Bot f6298086f2 fix(forgejo-runner): cicd ns PSS privileged (dind needs it) + mount homelab-ca as ConfigMap not Secret — runner RS created 0 pods under baseline PSS, then FailedMount because homelab-ca is a ConfigMap trust bundle, not a Secret 2026-08-18 15:08:04 -07:00
Story Crater Bot fbc4e55718 feat(forgejo): add runner-token Secret via ksops — forgejo-runner register initContainer needs the registration token (from gitea actions generate-runner-token); was missing so runner deploy stuck 0/1 2026-08-18 15:08:04 -07:00
Story Crater Bot 06c35fb338 fix(coredns): own Corefile+hostname rewrites via Talos inlineManifest (single-source terraform/files/coredns/Corefile), drop ArgoCD coredns-config app — in-cluster *.riotpiao.com now resolves to nginx ingress so MinIO/OIDC discovery works; update cp-2 IP .213->.214 2026-08-18 15:08:04 -07:00
Story Crater Bot 09fa9c6145 feat(reloader): enable autoReloadAll + reloadOnCreate — watch all workloads without per-Deployment annotations (charts like homarr don't expose them); auto-restart pods when ksops secrets are created/rotated 2026-08-18 15:08:04 -07:00
Story Crater Bot bd99208754 fix(homarr): add auth-oidc-secret + db-encryption Secrets via ksops — homarr chart's envSecrets expect these exact names (oidc-client-id/secret, db-encryption-key); were never created so homarr CreateContainerConfigError 2026-08-18 15:08:04 -07:00
Story Crater Bot 58605e1b5c chore(duckdns): remove duckdns updater entirely — superseded by cloudflared tunnel; drop app-def, manifests, kube-system Deployment 2026-08-18 15:08:04 -07:00
Story Crater Bot 40fcbd036c fix(cert-manager): regenerate homelab-ca cert with basicConstraints CA:TRUE — old self-signed cert lacked CA:TRUE so the homelab-ca ClusterIssuer rejected it ('certificate is not a CA'); regen keypair Secret + trust-bundle ConfigMaps (4 ns) with matching CA cert 2026-08-18 15:08:04 -07:00
Story Crater Bot 69b5fc371d fix: deploy authentik/loki/vault Secrets via ksops (were dead helm-values fragments, causing CreateContainerConfigError) 2026-08-18 15:08:04 -07:00
Story Crater Bot 582524f921 fix(cert-manager): cert-manager-issuers directory.include renders empty — switch to explicit resources list, restore automated sync 2026-08-18 15:08:04 -07:00
Story Crater Bot 828e3fb287 refactor(argocd): replace SOPS CMP with ksops kustomize generator, rotate age key — CMP discover glob silently shadowed kustomize rendering of any app whose path held a .enc.yaml (MinIO Tenant/cloudflared/authentik jobs never applied); centralize 8 Secret manifests under k8s/argocd/secrets, defer 4 helm-values fragments 2026-08-18 15:08:04 -07:00
Story Crater Bot 1d5c18d62c fix(cert-manager): add homelab-ca.crt key to homelab-ca ConfigMaps — authentik init merge-ca-certs cats /homelab-ca/homelab-ca.crt which was missing, causing Init:Error and 503 2026-08-18 15:08:04 -07:00
Story Crater Bot bc1a6d8689 fix(argocd): resolve 502 on argocd.riotpiao.com, dedupe Ingress and TLS mode mismatch
argocd-server ran --insecure (plain HTTP :8080) while its Helm-managed
Ingress set ssl-passthrough: true, which sends nginx's raw TLS handshake
straight to the pod - HTTP server can't complete a TLS handshake, nginx
logged 502 (peer closed connection in SSL handshake). Compounded by a
second, conflicting Ingress for the same host in
k8s/bootstrap/ingress/ingress.yaml - two Ingress objects on one host is
undefined nginx routing behavior. Disabled the Helm-managed Ingress
(enabled: false) so ingress.yaml's passthrough Ingress is the sole
source of truth, and set server.insecure: false so argocd-server
actually terminates TLS itself, matching passthrough's requirement.
2026-08-18 15:08:04 -07:00
Story Crater Bot e4485412b0 fix(argocd): use comma-separated include list, not brace expansion
ArgoCD directory.include uses Go filepath.Match glob syntax, not shell
brace expansion - {a,b,c} silently matched nothing, only the original 2
files stayed tracked.
2026-08-18 15:08:04 -07:00
Story Crater Bot 2022595426 feat(cert-manager): add self-signed homelab-ca ClusterIssuer + trust bundle, fix grafana-oidc secret
homelab-ca was referenced by 6 manifests (authentik, forgejo-runner,
blackbox-exporter, management-service) as a CA trust ConfigMap but never
existed anywhere - not in git, not live in cluster. Generated a new
10-year self-signed root CA, wired it as a ClusterIssuer (cert-manager
namespace) and distributed the public cert as a ConfigMap to every
consuming namespace (iam, cicd, monitoring, sqs). Private key lives only
in the encrypted Secret. Widened cert-manager-issuers' directory include
glob rather than creating a new Application - destination.namespace is
just a fallback default on a plain directory source, not a transformer,
so it doesn't fight with each ConfigMap's own explicit namespace.

Also adds grafana-oidc secret (GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET),
same pre-existing gap as grafana-admin - was meant to come from a deleted
manual script, value already available in .env.
2026-08-18 15:08:04 -07:00
Story Crater Bot f67aaa41d0 fix(portainer): pin to az-b (talos-cp-2), the real Longhorn storage node
nodeSelector still targeted az-a/talos-cp-1 from before the 3-CP topology
change. talos-cp-2 (az-b) has the dedicated Longhorn disks now, so the
pod's zone pin and the PVC's only viable replica location never matched
- ReplicaSchedulingFailure: disks are unavailable, pod stuck
ContainerCreating waiting on AttachVolume.
2026-08-18 15:08:04 -07:00
Story Crater Bot 69e8cfd6d1 fix(vault): add vault-minio-creds secret, was created by deleted helmfile presync hook
Vault's S3 storage backend needs AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY
from vault-minio-creds, previously generated by a helmfile presync hook
that no longer exists post-Terraform/helmfile removal. Sourced from the
same MINIO_ROOT_USER/PASSWORD already in .env. vault-unseal-keys still
missing separately — needs a live 'vault operator init' run, deferred.
2026-08-18 15:08:04 -07:00
Story Crater Bot db2fc9afc7 fix(portainer): correct storageClass name, longhorn-wffc never existed as a class
PVC sat Pending for 17 days — storageclass.storage.k8s.io "longhorn-wffc"
not found. Only longhorn, longhorn-cnpg, longhorn-static exist. Straight
naming drift, no such class was ever created.
2026-08-18 15:08:04 -07:00
Story Crater Bot 2b74b58ea6 fix(argocd): wire SOPS CMP sidecar + grafana-admin secret on repo-server 2026-08-18 15:08:04 -07:00
Story Crater Bot 27dbfb1bd7 feat(terraform): GPU worker node support (schematic, interface/diskSelector/swap tuning, gpu-node label, NVIDIA LTS extensions) 2026-08-18 15:08:04 -07:00
Story Crater Bot 4e67fd907a feat(argocd): migrate all applications from Forgejo to GitHub
- Replace all forgejo.riotpiao.com repo URLs with [email protected] SSH URLs
- Enables immediate GitOps sync without waiting for Forgejo mirror setup
- Includes ingress-nginx now fully ArgoCD-managed (wave 0)
- SOPS secrets can now sync and decrypt TLS certificates
2026-08-18 15:08:04 -07:00
204 changed files with 5584 additions and 11701 deletions
+460
View File
@@ -0,0 +1,460 @@
# CI/CD Pipeline: GitOps Validation & Deployment
## Overview
Pure GitOps CI/CD pipeline using Forgejo Actions (self-hosted runner).
**Principle:** Validate in CI, deploy via ArgoCD (no manual steps).
```
git push
[CI: Validate]
├─ yamllint (YAML syntax)
├─ kubeval (K8s manifests)
├─ kustomize build (all layers)
├─ argocd validation (app definitions)
└─ security scan (secrets, best practices)
[If push to main]
└─ ArgoCD auto-syncs (if enabled)
```
## Workflows
### 1. validate-k8s.yaml (Mandatory)
**Trigger:** Any push/PR with k8s/ changes
**What it does:**
1. Lints all YAML files (`yamllint`)
2. Validates K8s manifests (`kubeval`)
3. Builds all kustomization layers
4. Validates ArgoCD applications
5. Reports results
**Duration:** ~2-3 minutes
**Status:**
- ✅ PASS: All layers build, manifests valid → OK to merge
- ❌ FAIL: Syntax error, invalid resource, build failed → Fix & push again
**Example output:**
```
=== Building k8s/infrastructure/ ===
✓ Infrastructure built successfully
Resources: 47
=== Building k8s/bootstrap/ ===
✓ Bootstrap built successfully
Resources: 23
```
**When to check:**
- After every commit
- Before merging PRs
- On every branch
### 2. argocd-sync.yaml (Recommended)
**Trigger:** Push to main only (k8s/ changed)
**What it does:**
1. Authenticates with ArgoCD
2. Syncs `homelab-root` application
3. Waits for sync to complete (5 min timeout)
4. Verifies all applications healthy
**Duration:** 1-5 minutes (depends on resources)
**Status:**
- ✅ SYNCED: All resources deployed to cluster
- ❌ FAILED: Sync error, pod crashes, etc. → Check ArgoCD UI for details
**When it runs:**
- Automatically after merge to main
- Only on k8s/ changes (not on docs)
**Manual trigger (if needed):**
```bash
# SSH to runner or use Forgejo UI
# Re-run failed workflow
# Or manually sync: argocd app sync homelab-root
```
**Requires secrets:**
- `ARGOCD_SERVER`: ArgoCD server URL (https://argocd.riotpiao.com)
- `ARGOCD_AUTH_TOKEN`: ArgoCD API token (generate via ArgoCD UI)
### 3. security-scan.yaml (Optional)
**Trigger:** Any push/PR with k8s/ changes
**What it does:**
1. Scans Dockerfiles for vulnerabilities (`trivy`)
2. Scans Helm charts for security issues
3. Audits K8s manifests (`polaris`)
4. Checks for hardcoded secrets
5. Verifies security best practices
**Duration:** ~3-5 minutes
**Status:**
- ✅ PASS: No critical issues
- ⚠️ WARNING: Best practice recommendations (non-blocking)
- ❌ FAIL: Hardcoded secrets found (must fix)
**Common issues:**
- Missing resource limits (warning)
- Privileged containers (warning)
- Hardcoded passwords (ERROR)
---
## File Structure
```
.forgejo/
├── workflows/ # CI/CD workflows
│ ├── validate-k8s.yaml # Validate manifests (required)
│ ├── argocd-sync.yaml # Sync to cluster (auto on main)
│ └── security-scan.yaml # Security checks (optional)
└── CI-CD.md # This file
```
---
## Setup Instructions
### 1. Install Forgejo Runner
```bash
# On runner machine (inside cluster or external)
forgejo-runner register \
--instance https://forgejo.riotpiao.com \
--token <registration-token> \
--name homelab-runner \
--labels docker
forgejo-runner daemon
```
### 2. Add ArgoCD Secrets to Forgejo
```bash
# Go to: Forgejo → Settings → Secrets
# Add:
ARGOCD_SERVER = https://argocd.riotpiao.com
ARGOCD_AUTH_TOKEN = <token> # Generate: argocd account generate-token
```
### 3. Generate ArgoCD Token
```bash
# Inside cluster
kubectl -n argocd port-forward svc/argocd-server 8080:443
# Go to: https://localhost:8080/user-info/api-tokens
# Create new token (CI/CD)
# Copy token to Forgejo secrets
```
---
## Workflow Execution
### When developer pushes to feature branch:
```
git push origin feature/new-service
Forgejo Actions triggered
validate-k8s.yaml runs:
✓ Lints YAML
✓ Validates manifests
✓ Builds kustomizations
✓ All pass → GitHub comment: "Ready to merge"
Developer opens PR
Reviewer checks:
- Code changes (YAML)
- Workflow results
- ArgoCD impact (diff)
PR merged to main
```
### When merged to main:
```
git merge feature/new-service → main
Forgejo Actions triggered
validate-k8s.yaml runs:
✓ Same validation as above
argocd-sync.yaml runs (if enabled):
✓ Syncs homelab-root
✓ Waits for sync
✓ Verifies health
✓ Resources deployed to cluster
Cluster state = git state
(No manual kubectl apply needed!)
```
---
## Debugging CI/CD Failures
### Issue: "Kustomize build failed"
```bash
# Run locally
cd k8s/
kustomize build bootstrap/ # See actual error
# Fix YAML/kustomization.yaml
# git push again
```
### Issue: "Kubeval validation failed"
```bash
# Check K8s manifest syntax
kubeval k8s/platform/minio/config.yaml
# Common issues:
# - Typos in apiVersion, kind, metadata
# - Missing required fields
# - Invalid references (namespace, service name)
```
### Issue: "ArgoCD sync failed"
```bash
# Check ArgoCD UI
# https://argocd.riotpiao.com → homelab-root
# Or CLI
argocd app get homelab-root
argocd app logs homelab-root --follow
# Common issues:
# - Missing namespace (fixed by infrastructure layer)
# - Invalid Helm chart version
# - Secret not found
# - Network policy blocking traffic
```
### Issue: "Security scan found hardcoded secret"
```bash
# Fix: Remove secret from YAML
# Add to SOPS encryption instead
# Or use ArgoCD Sealed Secrets
# (if SOPS not available)
```
---
## Viewing Results
### Forgejo Actions UI
```
Repository → Actions
├─ validate-k8s
│ ├─ ✅ Success (merge safe)
│ ├─ ❌ Failed (fix required)
│ └─ Logs (click "Steps" → "Summary")
├─ argocd-sync
│ ├─ ✅ Synced (deployed)
│ └─ ❌ Failed (check ArgoCD UI)
└─ security-scan
├─ ✅ Pass (no critical issues)
└─ ⚠️ Warning (review, non-blocking)
```
### ArgoCD UI
```
https://argocd.riotpiao.com
├─ homelab-root
│ ├─ Status: Synced ✓
│ ├─ Health: Healthy ✓
│ └─ Details (click to see resources)
├─ layer-1-bootstrap
├─ layer-2-platform
├─ layer-3-security
├─ layer-4-applications
└─ layer-5-data
```
---
## Common Tasks
### Add new service to cluster
```bash
# 1. Create directory and kustomization.yaml
mkdir -p k8s/applications/my-service
cat > k8s/applications/my-service/kustomization.yaml << EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: my-namespace
helmCharts:
- name: my-chart
repo: https://charts.example.com
version: 1.0.0
releaseName: my-service
valuesFile: values.yaml
EOF
# 2. Add values.yaml
cp /template/values.yaml k8s/applications/my-service/
# 3. Commit and push
git add k8s/applications/my-service/
git commit -m "feat(apps): add my-service"
git push
# 4. CI validates
# 5. Merge to main
# 6. ArgoCD syncs automatically
# ✓ Service deployed to cluster
```
### Rollback a deployment
```bash
# 1. Find broken commit
git log --oneline k8s/ # Identify bad commit
# 2. Revert
git revert <commit-hash>
git push
# 3. CI validates (should pass)
# 4. Merge to main
# 5. ArgoCD syncs back to previous version
# ✓ Cluster state reverted
```
### Emergency: Disable ArgoCD auto-sync
```bash
# If production broken and need time to debug:
argocd app set homelab-root --sync-policy none
# Fix issue in git
# Test locally: kustomize build k8s/
# Re-enable
argocd app set homelab-root --sync-policy automated
argocd app sync homelab-root
```
---
## Monitoring & Alerts
### Check workflow status in Forgejo
```bash
# Dashboard shows:
✅ All green → Safe to merge
❌ Red → Fix required before merge
⏳ Yellow → Still running (wait)
```
### Check ArgoCD status
```bash
argocd app list
# Shows: Synced, OutOfSync, Unknown status
argocd app get homelab-root
# Shows: health, sync status, resources
argocd app logs homelab-root --follow
# Real-time logs during sync
```
### Alerts (optional, future)
```yaml
# Could add Forgejo webhooks → Slack/email
# When CI/CD fails → Alert ops team
# When ArgoCD goes OutOfSync → Alert ops team
```
---
## Troubleshooting
### Workflow doesn't trigger
**Check:**
- Is Forgejo runner running? `forgejo-runner daemon`
- Did you push to correct branch? (validate runs on all, argocd-sync only on main)
- Did path match filter? (must change k8s/ or .forgejo/workflows/)
### Workflow hangs/times out
**Check:**
- kustomize build → Check for dependency cycles
- argocd sync → Check cluster resources (storage full? network down?)
- security scan → Large image scan → Takes time
**Fix:**
- Increase timeout in workflow
- Optimize kustomization (remove unused resources)
- Add resource limits to pods
### ArgoCD token invalid
**Fix:**
```bash
# Regenerate token
argocd account generate-token
# Update Forgejo secret
# Settings → Secrets → ARGOCD_AUTH_TOKEN = <new-token>
```
---
## Best Practices
**DO:**
- Commit all K8s changes to git (no manual kubectl apply)
- Run validate-k8s locally before push
- Write descriptive commit messages (why this change?)
- Review workflow logs before merging
- Monitor ArgoCD sync after merge
**DON'T:**
- Push directly to main (always use PR)
- Skip workflow validation (it catches errors early)
- Ignore security scan warnings
- Manually `kubectl apply` (breaks GitOps)
- Edit resources in cluster (they revert via ArgoCD)
---
## Next Steps
1. **Setup Forgejo runner** (if not already running)
2. **Add ArgoCD secrets** to Forgejo
3. **Test workflows** on feature branch
4. **Merge to main** → Watch ArgoCD sync
5. **Celebrate:** Full GitOps pipeline working! 🎉
+269
View File
@@ -0,0 +1,269 @@
name: Cluster CI Pipeline
on:
push:
branches:
- main
- develop
paths:
- 'k8s/**'
- '.forgejo/workflows/cluster-ci.yaml'
pull_request:
paths:
- 'k8s/**'
jobs:
ci:
runs-on: docker
steps:
# === Checkout ===
- name: Checkout
run: |
REPO_URL="${{ gitea.server_url }}/${{ gitea.repository }}.git"
CLONE_URL="https://${{ secrets.CI_RUNNER }}:${{ secrets.CI_RUNNER_SECRET }}@${REPO_URL#https://}"
git clone --depth 1 "$CLONE_URL" .
git fetch origin main
git checkout main
# === Install Tools ===
- name: Install Tools
run: |
unset GITHUB_TOKEN
apt-get update && apt-get install -y \
yamllint \
python3-pip \
curl \
jq
# kubeval
curl -L https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz | tar xz
mv -f kubeval /usr/local/bin/
# kustomize
rm -f kustomize
curl -s https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh | bash
mv -f kustomize /usr/local/bin/
# argocd
curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
chmod +x /usr/local/bin/argocd
# trivy
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
# polaris
curl -L https://github.com/FairwindsOps/polaris/releases/latest/download/polaris-linux-amd64 -o /usr/local/bin/polaris
chmod +x /usr/local/bin/polaris
# === YAML Lint ===
- name: YAML Lint
run: |
echo "=== Linting YAML files ==="
yamllint k8s/ -c .yamllint.yaml || true
# === Kubeval - Validate K8s Syntax ===
- name: Kubeval - Validate K8s Syntax
run: |
echo "=== Validating Kubernetes manifests ==="
find k8s -name "*.yaml" -o -name "*.yml" | grep -v "\.archive" | while read file; do
echo "Validating $file..."
kubeval "$file" -d 2>/dev/null || true
done
# === Kustomize Build - All overlays ===
- name: Kustomize Build - Infrastructure
run: |
echo "=== Building k8s/infrastructure/ ==="
kustomize build k8s/infrastructure > /tmp/infrastructure.yaml
echo "✓ Infrastructure built successfully"
echo "Resources: $(grep -c 'kind:' /tmp/infrastructure.yaml)"
- name: Kustomize Build - Bootstrap
run: |
echo "=== Building k8s/bootstrap/ ==="
kustomize build k8s/bootstrap > /tmp/bootstrap.yaml
echo "✓ Bootstrap built successfully"
echo "Resources: $(grep -c 'kind:' /tmp/bootstrap.yaml || echo 0)"
- name: Kustomize Build - Platform
run: |
echo "=== Building k8s/platform/ ==="
kustomize build k8s/platform > /tmp/platform.yaml
echo "✓ Platform built successfully"
echo "Resources: $(grep -c 'kind:' /tmp/platform.yaml || echo 0)"
- name: Kustomize Build - Security
run: |
echo "=== Building k8s/security/ ==="
kustomize build k8s/security > /tmp/security.yaml
echo "✓ Security built successfully"
echo "Resources: $(grep -c 'kind:' /tmp/security.yaml || echo 0)"
- name: Kustomize Build - Applications
run: |
echo "=== Building k8s/applications/ ==="
kustomize build k8s/applications > /tmp/applications.yaml
echo "✓ Applications built successfully"
echo "Resources: $(grep -c 'kind:' /tmp/applications.yaml || echo 0)"
- name: Kustomize Build - Data
run: |
echo "=== Building k8s/data/ ==="
kustomize build k8s/data > /tmp/data.yaml
echo "✓ Data built successfully"
echo "Resources: $(grep -c 'kind:' /tmp/data.yaml || echo 0)"
- name: Validate ArgoCD Applications
run: |
echo "=== Validating ArgoCD Applications ==="
kubeval k8s/argocd/apps/*.yaml
# === Trivy - Scan Dockerfile ===
- name: Trivy - Scan Dockerfile
run: |
if find . -name "Dockerfile" 2>/dev/null | grep -v node_modules | head -1 | grep -q .; then
echo "=== Scanning Dockerfiles with Trivy ==="
find . -name "Dockerfile" -not -path "*/node_modules/*" -exec trivy config {} \;
else
echo "No Dockerfiles found"
fi
# === Trivy - Scan Helm Charts ===
- name: Trivy - Scan Helm Charts
run: |
if find k8s -name "Chart.yaml" 2>/dev/null | head -1 | grep -q .; then
echo "=== Scanning Helm charts with Trivy ==="
find k8s -name "Chart.yaml" -exec dirname {} \; | while read chart; do
echo "Scanning $chart..."
trivy config "$chart" || true
done
else
echo "No Helm charts found"
fi
# === Polaris - K8s Security Audit ===
- name: Polaris - K8s Security Audit
run: |
echo "=== Running Polaris K8s security audit ==="
polaris audit --audit-path /tmp/polaris-audit.json k8s/ || true
if [ -f /tmp/polaris-audit.json ]; then
echo "Security issues found:"
jq '.results[] | select(.pass == false)' /tmp/polaris-audit.json || true
fi
# === Check for Secrets in Code ===
- name: Check for Secrets in Code
run: |
echo "=== Scanning for hardcoded secrets ==="
# BLOCKING. This step used to only count findings and then exit 0, so a
# plaintext deploy key rode through it into a public remote. Two failure
# modes fixed: it now fails the build, and it matches key material by
# PEM header rather than only `private_key:`-style YAML field names.
# Findings are captured into variables and tested for emptiness rather than
# branching on grep's exit status: implementations disagree on the rc of a
# `-v` filter fed empty input, and a wrong rc here fails open.
# NOTE: --include must precede `--`; after `--` grep treats it as a filename
# and silently scans nothing.
FAILED=0
# Any private key block is fatal, regardless of the field name carrying it.
KEYS=$(grep -rIE --include="*.yaml" --include="*.yml" \
-- "-----BEGIN ([A-Z]+ )?PRIVATE KEY-----" k8s/ \
| grep -v "\.enc\.yaml" || true)
if [ -n "$KEYS" ]; then
echo "❌ Unencrypted private key material found:"
echo "$KEYS"
FAILED=1
fi
# Plaintext values in secret-ish YAML fields. SOPS output is ENC[...],
# so encrypted files never trip this.
VALS=$(grep -rInE --include="*.yaml" --include="*.yml" \
-- "^[[:space:]]*(password|token|apiKey|api_key|sshPrivateKey|client_secret):[[:space:]]*[\"']?[^\"'[:space:]{\$]{8,}" k8s/ \
| grep -v "ENC\[" | grep -v "\.enc\.yaml" || true)
if [ -n "$VALS" ]; then
echo "❌ Plaintext secret value found:"
echo "$VALS"
FAILED=1
fi
if [ "$FAILED" -ne 0 ]; then
echo "Encrypt with SOPS (see .sops.yaml) — *.enc.yaml files are exempt."
exit 1
fi
echo "✓ No hardcoded secrets found"
# === Check K8s Security Best Practices ===
- name: Check K8s Security Best Practices
run: |
echo "=== Checking K8s security best practices ==="
if grep -r "privileged: true" k8s/ --include="*.yaml" --include="*.yml"; then
echo "⚠️ Found privileged containers"
fi
if grep -r "hostNetwork: true" k8s/ --include="*.yaml" --include="*.yml"; then
echo "⚠️ Found hostNetwork usage"
fi
echo "Checking for missing resource limits..."
MISSING=0
find k8s -name "*.yaml" -o -name "*.yml" | while read file; do
if grep -q "kind: Deployment\|kind: StatefulSet\|kind: DaemonSet" "$file"; then
if ! grep -q "resources:" "$file"; then
echo "⚠️ $file: Missing resource requests/limits"
MISSING=$((MISSING + 1))
fi
fi
done
# === ArgoCD Sync (main branch only) ===
- name: Sync ArgoCD
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
env:
ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
run: |
echo "=== Syncing homelab-root ==="
argocd app sync homelab-root --force
argocd app wait homelab-root --timeout 5m
- name: Check Sync Status
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
env:
ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
run: |
echo "=== ArgoCD Applications Status ==="
argocd app list -o table
STATUS=$(argocd app get homelab-root -o jsonpath='{.status.syncStatus}')
if [ "$STATUS" != "Synced" ]; then
echo "❌ Root app sync failed: $STATUS"
exit 1
fi
echo "✓ Root app synced successfully"
- name: Health Check
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
env:
ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
run: |
echo "=== Checking Application Health ==="
argocd app get homelab-root -o wide
# === Summary ===
- name: Summary
if: always()
run: |
echo "=== CI Pipeline Summary ==="
echo "✓ YAML linted"
echo "✓ Manifests validated"
echo "✓ Kustomizations built"
echo "✓ Security scans completed"
echo "✓ Secrets check passed"
echo "✓ Best practices verified"
echo ""
echo "✓ All checks passed"
-3
View File
@@ -66,6 +66,3 @@ bootstrap-argocd.log
# one line here, which is how a plaintext deploy key reached a public remote.
k8s/**/*-secret.yaml
!k8s/**/*.enc.yaml
# IAM provisioning scripts contain credential references — never commit
scripts/iam/*.py
-1
View File
@@ -2,5 +2,4 @@ creation_rules:
# `secrets?` — singular too. A `seed-repo-secret.yaml` once slipped this regex
# and was committed in plaintext to a public remote.
- path_regex: k8s/.*secrets?.*\.ya?ml
encrypted_regex: ^(data|stringData)$
age: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-92
View File
@@ -208,95 +208,3 @@ versions without warning in your own values file.
Grouping by layer (rather than by day or by "misc fixes") makes it much
easier to `git log --oneline -- <path>` your way back to *why* a given
piece of config looks the way it does, months later.
## Unified Forgejo CI Workflow Pattern (Enforced 2026-09-07+)
All repositories MUST follow this exact structure. No variations.
```yaml
name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
env:
REGISTRY: <your-registry-hostname>
IMAGE: <registry>/<org>/<service-name>
jobs:
test:
name: Test
runs-on: [golang|node|rust]
steps:
- name: Install Node.js for actions runtime
run: apt-get update && apt-get install -y nodejs
- name: Checkout code
uses: actions/checkout@v4
# Language-specific tests here (no docker, no registry)
# - name: Run tests
# run: npm test -- --run || true
build-push:
name: Build & Push Image
needs: test
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: [golang|node|rust]
steps:
- name: Install Node.js and Docker
run: |
apt-get update
apt-get install -y nodejs docker.io
- name: Checkout code
uses: actions/checkout@v4
- name: Get short SHA
id: sha
run: |
SHORT_SHA=$(git rev-parse --short HEAD)
echo "short_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
- name: Registry login
run: |
echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
--username "${REGISTRY_USER}" --password-stdin
env:
REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
- name: Build Docker image
run: |
docker build --no-cache \
-t "${IMAGE}:${{ steps.sha.outputs.short_sha }}" \
-t "${IMAGE}:latest" \
.
- name: Push Docker image
run: |
docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}"
docker push "${IMAGE}:latest"
- name: Prune unused images
run: docker image prune -a --force 2>&1 | tail -3 || true
```
### Anti-Patterns (DO NOT USE)
-`container: image: golang:1.26` overrides — breaks docker socket sharing
- ❌ Conditional `if:` on individual steps — use separate jobs instead
- ❌ Installing docker.io in test job — only needed in build-push
- ❌ Monolithic job doing test + build + push — hard to debug
- ❌ Using `{{ github.sha }}` for image tag — use short commit SHA for readability
### How It Works
1. **PR to feature branch** → test job runs, build-push skipped, nothing pushed
2. **Push to main** → test runs, build-push runs after test passes, image pushed
3. Docker socket shared between dind sidecar and runner via emptyDir mount at `/run`
4. `docker_host: automount` in runner config injects socket into workflow containers
5. Secrets (FORGEJO_REGISTRY_USER, TOKEN) set in Forgejo repo settings, NOT in git
-80
View File
@@ -42,86 +42,6 @@ All logs + metrics centralized in Grafana for debugging
- **Secrets at rest** — Vault + encrypted etcd; credentials never in logs or ConfigMaps
- **Infrastructure-as-code** — Every service deployed via Helmfile; one `helmfile apply` recovers from total failure
## ArgoCD — GitOps Deployment Flow
**ArgoCD** pulls infrastructure changes from git and syncs the cluster automatically.
No manual `kubectl apply` — push to git, ArgoCD detects the change, and deploys within ~3 minutes.
```
Developer pushes to git
ArgoCD detects change (every 3 min or webhook)
Syncs manifests to cluster
Workloads reconcile automatically
```
Applications are deployed in waves (numbered 00, 10, 20, 30, ...) to respect dependencies —
storage deploys before databases, databases before applications.
### Tracked Git Repositories
ArgoCD monitors these repos for changes:
| Repository | Purpose |
|------------|----------|
| `https://github.com/Riotpiaole/riotpiao.homelab.com` | Main infrastructure repo (all manifests in `k8s/argocd/apps/`) |
| `https://forgejo.riotpiao.com/rock/*` | Any `rock/*` repo in in-cluster Forgejo (apps + configs) |
| `https://github.com/Riotpiaole/Poimen-*` | External Poimen services (memory, workflows) |
To deploy a new application: create a git repo, add an Application manifest to the homelab repo's
`k8s/argocd/apps/`, commit + push, and ArgoCD syncs within 3 minutes.
## Management Planes — Talos vs Kubernetes
This cluster has **two separate management planes**, each with different workflows:
| Plane | What it manages | Workflow | Tool |
|-------|-----------------|----------|------|
| **Talos (OS)** | Node configuration, kernel params, networking, CoreDNS, machine state | Edit `terraform/``terraform apply``make apply-cp` | `terraform` + `talosctl` |
| **Kubernetes (workloads)** | All pods, services, deployments, ingresses, databases | Edit `k8s/argocd/apps/``git push` → ArgoCD syncs | `git` + ArgoCD |
**Critical distinction:**
- **Kubernetes resources** (`k8s/**`) flow through **git → ArgoCD** — never use `kubectl apply`
- **Talos machine config** (`terraform/**`) uses **local `terraform apply`** (sanctioned exception — CI can't hold node credentials)
Example: To add a CoreDNS hostname rewrite, you edit `terraform/files/coredns/Corefile`, then:
```bash
cd terraform && terraform apply -var-file=terraform.tfvars.local
cd .. && make apply-cp # talosctl apply-config to all 3 control planes
```
But to add a new Kubernetes Deployment or update an Ingress, you only `git push`**never `kubectl apply`**.
### CoreDNS ConfigMap Ownership — Critical
⚠️ **Warning:** The `coredns` ConfigMap in `kube-system` namespace is **owned by Talos**, not ArgoCD or kubectl.
It is rendered from `terraform/files/coredns/Corefile` into Talos's machine config at bootstrap time.
**Do not `kubectl apply` or `kubectl edit` this ConfigMap directly.** Doing so transfers field ownership to kubectl's
client-side-apply mechanism, and Talos's inline-manifest controller will silently no-op on every future reconcile
(server-side-apply conflict, no error surfaced).
**To update CoreDNS (e.g., add a hostname rewrite):**
1. Edit `terraform/files/coredns/Corefile`
2. Commit + push
3. Run `cd terraform && terraform apply -var-file=terraform.tfvars.local`
4. Run `make apply-cp` to push config to all control planes
5. CoreDNS picks up changes via its `reload` plugin — no pod restart needed
**If you accidentally edited the ConfigMap directly and broke Talos's ownership:**
```bash
kubectl delete configmap coredns -n kube-system
# Wait ~30s for Talos's k8s.ManifestApplyController to recreate it
kubectl get configmap coredns -n kube-system -w
```
Or as a stopgap, apply the correct content yourself:
```bash
kubectl apply --server-side -f <(terraform output coredns_config)
```
## Quick Start — Deploying the Cluster
### 1. Bootstrap Talos Nodes
View File
+2 -15
View File
@@ -1,27 +1,14 @@
# Exposes agent-hub at api.riotpiao.com/console (WebSocket) and /run
# (trigger a new session) -- both are routes on the same hub.js service.
#
# Was ingressClassName: kong until Kong was retired on 2026-08-19. Pointed
# straight at nginx rather than through the replacement Go gateway because that
# gateway has no WebSocket upgrade support yet -- routing /console through it
# would break the console outright. nginx handles the upgrade natively.
#
# Path precedence: the nginx Ingress api/api catch-alls `/` on this same host
# to the gateway. nginx matches longest prefix first, so these three paths win
# over `/` and the rest of the host still reaches the gateway.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: console
namespace: agent-pod
annotations:
# A console WebSocket stays open across a whole agent session; nginx's 60s
# default read timeout would drop it mid-run.
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-buffering: "off"
konghq.com/strip-path: "false"
spec:
ingressClassName: nginx
ingressClassName: kong
rules:
- host: api.riotpiao.com
http:
+44 -164
View File
@@ -52,14 +52,6 @@ data:
const AGENT_MANAGER_BIN = process.env.AGENT_MANAGER_BIN || path.join(__dirname, "..", ".bin", "agent-manager-fork");
// agent-manager's own session-state DB -- used to detect a session that has
// actually died (process crashed/exited, status flips to "errored"/"dead")
// instead of one that's merely slow. Read-only introspection plus the one
// UPDATE in killDeadSession below, same class of operation as the tmux
// nudges already done directly against agent-manager's internals.
const AGENT_MANAGER_DB =
process.env.AGENT_MANAGER_DB || path.join(require("node:os").homedir(), ".config", "agent-manager", "state.db");
// Empty means "let pi fall back to ~/.pi/agent/settings.json's default"
// (currently anthropic/claude-sonnet-4-5, real paid usage). Set both to
// route every stage -- headless (spawnPi) and interactive (runOnPool) --
@@ -115,22 +107,6 @@ data:
"Read and follow ~/.pi/agent/skills/karpathy-guidelines/SKILL.md and " +
"~/.pi/agent/skills/caveman/SKILL.md as hard rules for this entire task, before anything else. ";
// judge (routed to homelab-reasoning) has been observed narrating an
// entire review in prose -- "I should run git diff, then check X..." --
// and then writing a verdict based on that narration without ever calling
// a real tool. Live example: a phase-judge call produced a page of
// "I would check..." reasoning, declared VERDICT: PASS, and showed the
// touch command as a fenced code block IN ITS OWN TEXT rather than
// executing it. Coordinator just timed out waiting on a sentinel that was
// never going to appear, since nothing was ever actually run. Spelled out
// explicitly since "use the judge skill" alone apparently isn't enough to
// rule this out.
const REQUIRE_REAL_TOOL_CALLS =
"Do not narrate what you would check -- actually run the commands via a real tool call and read their real " +
"output before writing anything. A verdict based on describing checks instead of executing them is invalid. " +
"Writing the verdict file and touching the sentinel are themselves tool calls you must execute, not text to " +
"display in your response. ";
function parseVerdictLine(text, label) {
if (!text) return null;
const re = new RegExp(`${label}:\\s*(\\w+)`, "i");
@@ -169,17 +145,6 @@ data:
// session: resolver's crash/stall diagnosis, and the initial clone. Kept
// exactly as before -- only the interactive per-role stages (runOnPool,
// below) go through agent-manager.
//
// Bounded by SPAWN_PI_TIMEOUT_MS -- unlike runOnPool's pooled sessions
// (which now have status polling to catch a dead session fast, see
// waitForSentinel/killDeadSession), this is a raw child_process with no
// equivalent escape hatch. Observed live: a resolver call shared the
// default backend with a concurrently-busy repo's implementer and sat for
// 6+ minutes producing nothing -- with no timeout here, that blocks the
// entire calling repo's pipeline forever, since askResolver is always
// awaited before the next stage can run.
const SPAWN_PI_TIMEOUT_MS = 5 * 60 * 1000;
function spawnPi({ agent, prompt, cwd }) {
const finalPrompt = ROLE_SKILLS.has(agent) ? `/skill:${agent} ${HARD_RULES}${prompt}` : prompt;
const args = ["-p", "--mode", "json"];
@@ -216,19 +181,7 @@ data:
stderrTail = (stderrTail + chunk.toString()).slice(-4000);
});
return new Promise((resolve) => {
let settled = false;
const timer = setTimeout(() => {
if (settled) return;
settled = true;
child.kill("SIGKILL");
resolve({ code: null, lastText, stderrTail, timedOut: true });
}, SPAWN_PI_TIMEOUT_MS);
child.on("close", (code) => {
if (settled) return;
settled = true;
clearTimeout(timer);
resolve({ code, lastText, stderrTail });
});
child.on("close", (code) => resolve({ code, lastText, stderrTail }));
});
}
@@ -240,27 +193,19 @@ data:
const STAGE_TIMEOUT_MS = 10 * 60 * 1000;
const NUDGE_TIMEOUT_MS = 5 * 60 * 1000;
// Resolves as soon as filePath appears (fs.watch on its directory, same as
// before), as soon as target's agent-manager status flips to "errored" or
// "dead" (polled -- state.db has no watch mechanism), or after limitMs with
// neither. A session that has actually crashed will never touch the
// sentinel, so without the status poll this just burns the full STAGE_
// TIMEOUT_MS waiting on a file that was never coming, same as a genuine
// stall -- polling status catches that in ~pollMs instead.
function waitForSentinel(filePath, target, limitMs, pollMs = 5000) {
// Resolves as soon as filePath appears (fs.watch on its directory), or
// after limitMs with no sign of it -- event-driven completion instead of
// hub.js's old 10s poll.
function waitForSentinel(filePath, limitMs) {
return new Promise((resolve) => {
if (fs.existsSync(filePath)) return resolve({ ok: true });
if (fs.existsSync(filePath)) return resolve(true);
const dir = path.dirname(filePath);
const id = target.replace(/^am_/, "");
let settled = false;
let watcher;
let poller;
let timer;
const finish = (result) => {
if (settled) return;
settled = true;
clearTimeout(timer);
clearInterval(poller);
if (watcher) {
try {
watcher.close();
@@ -272,36 +217,20 @@ data:
};
try {
watcher = fs.watch(dir, () => {
if (fs.existsSync(filePath)) finish({ ok: true });
if (fs.existsSync(filePath)) finish(true);
});
} catch {
// dir missing at watch time is a real bug elsewhere (cwd should
// already exist); surface it as a timeout rather than hang forever.
return finish({ timedOut: true });
return finish(false);
}
// Closes the race between the existsSync check above and the watcher
// actually being attached.
if (fs.existsSync(filePath)) return finish({ ok: true });
poller = setInterval(async () => {
const { out } = await runCmd("sqlite3", [AGENT_MANAGER_DB, `SELECT status FROM sessions WHERE id='${id}'`]);
const status = out.trim();
if (status === "errored" || status === "dead") finish({ dead: true, status });
}, pollMs);
timer = setTimeout(() => finish({ timedOut: true }), limitMs);
if (fs.existsSync(filePath)) return finish(true);
const timer = setTimeout(() => finish(false), limitMs);
});
}
// Kills a session that's actually crashed (not just slow) and archives it
// in agent-manager's own DB so it stops showing up as a live, unattended
// pane -- otherwise every crash leaves an orphaned tmux session + state.db
// row behind permanently, identical to the manually-cleaned-up poiman-
// planner ghost session found earlier this same run.
async function killDeadSession(target) {
await runAmTmux(["kill-session", "-t", target]);
const id = target.replace(/^am_/, "");
await runCmd("sqlite3", [AGENT_MANAGER_DB, `UPDATE sessions SET archived=1 WHERE id='${id}'`]);
}
// Runs one task's worth of work on a persistent per-role agent: spawns the
// role's session the first time it's ever needed for this repo, sends every
// later prompt into that same tmux pane via send-keys -- prefixed with a
@@ -312,14 +241,10 @@ data:
// shared across every task in a repo's pipeline (see runRepoPipeline) -- it
// IS the 4-agent pool, one entry per role, filled in lazily as each role
// gets its first task.
// A dead/errored session gets one respawn-and-retry (same prompt, fresh
// session) before this stage is abandoned -- matches resolver-SKILL.md's
// own documented contract of retrying a failed stage at most once.
const DEAD_SESSION_RETRIES = 1;
async function runOnPool(pool, cwd, repoId, role, prompt, sentinelFile) {
fs.rmSync(sentinelFile, { force: true });
const label = `${repoId}-${role}`;
let target = pool[role];
// A pooled session's shell cwd drifts as it explores the repo (e.g. cd
// into a Rust workspace subdirectory to read source) and nothing resets
@@ -334,19 +259,14 @@ data:
// wrong by reasoning about a relative "current directory."
const cwdReminder = `Your working directory for this task is ${cwd} -- if your shell isn't already there, run: cd ${cwd}\n\n`;
const spawnFresh = async () => {
if (!target) {
const spawnArgs = ["spawn", "--tool", "pi", "--cwd", cwd, "--name", label, "--group", repoId, "--prompt", cwdReminder + HARD_RULES + prompt];
const { provider, model } = providerModelFor(role);
if (provider) spawnArgs.push("--provider", provider);
if (model) spawnArgs.push("--model", model);
const spawned = await runCmd(AGENT_MANAGER_BIN, spawnArgs);
return spawned.code === 0 ? amSessionName(spawned.out) : null;
};
let target = pool[role];
if (!target) {
target = await spawnFresh();
if (!target) return { ok: false, crashed: true, error: "spawn failed", sessionName: label };
if (spawned.code !== 0) return { ok: false, crashed: true, error: spawned.out, sessionName: label };
target = amSessionName(spawned.out);
pool[role] = target;
} else {
await runAmTmux(["send-keys", "-t", target, "/new", "Enter"]);
@@ -354,25 +274,8 @@ data:
await runAmTmux(["send-keys", "-t", target, cwdReminder + HARD_RULES + prompt, "Enter"]);
}
for (let deadRetries = 0; ; deadRetries++) {
const outcome = await waitForSentinel(sentinelFile, target, STAGE_TIMEOUT_MS);
if (outcome.ok) return { ok: true, sessionName: label };
if (outcome.dead) {
await killDeadSession(target);
if (pool[role] === target) delete pool[role];
if (deadRetries >= DEAD_SESSION_RETRIES) {
return { ok: false, crashed: true, error: `session died (status: ${outcome.status})`, sessionName: label };
}
target = await spawnFresh();
if (!target) return { ok: false, crashed: true, error: "respawn after death failed", sessionName: label };
pool[role] = target;
continue;
}
// Plain stall -- session still alive, just slow. Ask resolver once,
// nudge if it says worth it, and stop here either way (this is not
// the death path, so no respawn/retry loop).
let ok = await waitForSentinel(sentinelFile, STAGE_TIMEOUT_MS);
if (!ok) {
const pane = await runAmTmux(["capture-pane", "-t", target, "-p", "-S", "-200"]);
const resolution = await askResolver(
cwd,
@@ -380,18 +283,12 @@ data:
`Repo ${repoId}'s "${role}" agent hasn't finished its current task after 10 minutes. Its pane tail:\n${pane.out.slice(-3000)}\n\n` +
`Decide: is it still making real progress and worth nudging to wrap up, or stuck and worth abandoning?`
);
let ok = false;
if (resolution === "RETRY") {
await runAmTmux(["send-keys", "-t", target, `Please wrap up now and run: touch ${sentinelFile}`, "Enter"]);
const nudged = await waitForSentinel(sentinelFile, target, NUDGE_TIMEOUT_MS);
ok = nudged.ok === true;
if (nudged.dead) {
await killDeadSession(target);
if (pool[role] === target) delete pool[role];
}
ok = await waitForSentinel(sentinelFile, NUDGE_TIMEOUT_MS);
}
return { ok, sessionName: label };
}
return { ok, sessionName: label };
}
function plannerPrompt(task, specHint, judgeOnly, cwd) {
@@ -434,9 +331,9 @@ data:
function judgePrompt(task, baseBranch, attempt, cwd) {
return (
`${REQUIRE_REAL_TOOL_CALLS}Use the judge skill to review the diff against ${baseBranch}...HEAD for task ${task}. ` +
`Write your verdict to ${path.join(cwd, `.task-result-${task}`)} as a single "VERDICT: PASS" or "VERDICT: FAIL" ` +
`line plus one line of rationale, then run: touch ${path.join(cwd, `.stage-done-${task}-judge-${attempt}`)}`
`Use the judge skill to review the diff against ${baseBranch}...HEAD for task ${task}. Write your verdict to ` +
`${path.join(cwd, `.task-result-${task}`)} as a single "VERDICT: PASS" or "VERDICT: FAIL" line plus one line of rationale, then ` +
`run: touch ${path.join(cwd, `.stage-done-${task}-judge-${attempt}`)}`
);
}
@@ -752,47 +649,31 @@ data:
return finish("branch-crashed");
}
// Idempotent and run every phase, NOT gated on a fresh (non-resumed)
// start -- every run this session was a resume, so the old i===0 &&
// !resuming gate meant this setup permanently never ran on poiman's
// branch, and portfolio's PLAN.md stayed tracked from before this rule
// ever existed (gitignore has no effect on an already-tracked file --
// observed live: it kept getting swept back in by every `git add -A`
// regardless of the ignore rule). Check-and-fix on every phase instead
// of once-at-genesis so a repo that's missing either self-heals on its
// very next run rather than carrying the gap forever.
const gitignorePath = path.join(cwd, ".gitignore");
const currentGitignore = fs.existsSync(gitignorePath) ? fs.readFileSync(gitignorePath, "utf8").split("\n") : [];
const requiredGitignoreLines = [
"*.tar.gz",
"*.tgz",
"*.crate",
"*.zip",
"*.bin",
"*.whl",
"vendor/",
"node_modules/",
".task-result-*",
".phase-result-*",
".stage-done-*",
"PLAN.md",
];
const missingGitignoreLines = requiredGitignoreLines.filter((line) => !currentGitignore.includes(line));
if (missingGitignoreLines.length > 0) {
fs.appendFileSync(
gitignorePath,
"\n# agent-harness: build artifacts, vendored archives, and harness bookkeeping never belong in source control\n" +
missingGitignoreLines.join("\n") +
"\n"
);
if (i === 0 && !resuming) {
const gitignoreAdditions = [
"",
"# agent-harness: build artifacts and vendored archives never belong in source control",
"*.tar.gz",
"*.tgz",
"*.crate",
"*.zip",
"*.bin",
"*.whl",
"vendor/",
"node_modules/",
"",
"# agent-harness: task/phase completion sentinel files, harness bookkeeping only",
".task-result-*",
".phase-result-*",
".stage-done-*",
"",
"# agent-harness: PLAN.md is per-task planner scratch state, never a deliverable",
"PLAN.md",
].join("\n");
fs.appendFileSync(path.join(cwd, ".gitignore"), gitignoreAdditions + "\n");
await runGit(cwd, ["add", ".gitignore"]);
await runGit(cwd, ["commit", "-m", "chore: broaden .gitignore for agent-run artifacts"]);
}
const trackedFiles = await runGit(cwd, ["ls-tree", "-r", "HEAD", "--name-only"]);
if (trackedFiles.out.split("\n").includes("PLAN.md")) {
await runGit(cwd, ["rm", "--cached", "PLAN.md"]);
await runGit(cwd, ["commit", "-m", "chore: untrack PLAN.md (already gitignored, was committed pre-rule)"]);
}
await runPhase(cwd, baseBranch, phaseBranch, phaseTasks, pool, repoId, pipelineSession);
@@ -813,7 +694,7 @@ data:
cwd,
repoId,
"judge",
`${REQUIRE_REAL_TOOL_CALLS}Use the judge skill to review the full phase diff for phase ${phaseLabel} against ` +
`Use the judge skill to review the full phase diff for phase ${phaseLabel} against ` +
`${baseBranch}...HEAD (covers every task in this phase: ${[...phaseTaskIds].join(", ")}). Every ` +
`individual task already passed its own judge review -- your job here is different: confirm the ` +
`tasks integrate correctly as one coherent narrative, and that real integration tests (not just ` +
@@ -979,4 +860,3 @@ data:
}
module.exports = { runCoordinator, runRepoPipeline, runOnPool, parseTaskBoard };
+3 -14
View File
@@ -56,7 +56,7 @@ spec:
- -c
- |
set -e
apt-get update && apt-get install -y git curl jq openssh-client tmux python3 sqlite3 gcc build-essential
apt-get update && apt-get install -y git curl jq openssh-client tmux python3 sqlite3
ssh-keygen -y -f /root/.ssh/id_forgejo > /root/.ssh/id_forgejo.pub
eval "$(ssh-agent -s)"
ssh-add /root/.ssh/id_forgejo
@@ -69,19 +69,6 @@ spec:
https://github.com/Riotpiaole/agent-manager.git /root/agent-manager-src
(cd /root/agent-manager-src && go build -o /usr/local/bin/agent-manager .)
# Language toolchains for whatever repos the implementer/investigator/
# judge roles actually build and test -- go was already fetched above
# only for building agent-manager itself, and its PATH export above is
# local to this script, invisible to `kubectl exec` sessions into the
# already-running container. Symlinking both into /usr/local/bin (on
# PATH for every exec session, interactive or not) instead of relying
# on shell rc sourcing, which pi's non-interactive tool calls don't do.
ln -sf /usr/local/go/bin/go /usr/local/bin/go
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable
ln -sf /root/.cargo/bin/cargo /usr/local/bin/cargo
ln -sf /root/.cargo/bin/rustc /usr/local/bin/rustc
ln -sf /root/.cargo/bin/rustup /usr/local/bin/rustup
node /root/hub.js
env:
- name: PI_BIN
@@ -153,6 +140,8 @@ spec:
path: judge/SKILL.md
- key: resolver-SKILL.md
path: resolver/SKILL.md
- key: brave-search-SKILL.md
path: brave-search/SKILL.md
- name: hub-src
configMap:
name: hub-src
+41 -2
View File
@@ -1,5 +1,44 @@
apiVersion: v1
data:
brave-search-SKILL.md: |
---
name: brave-search
description: Web search via the Brave Search API, called directly with curl. Use for searching documentation, facts, or any current web content.
allowed-tools: Bash
---
# Brave Search
Direct HTTP call to the Brave Search API — no separate script or package, just `curl` (`BRAVE_API_KEY` is already set in the environment).
## Search
```bash
curl -s -H "Accept: application/json" -H "X-Subscription-Token: $BRAVE_API_KEY" \
--get --data-urlencode "q=<query>" --data-urlencode "count=5" \
"https://api.search.brave.com/res/v1/web/search"
```
Options (add as extra `--data-urlencode` pairs):
- `count=<n>` — number of results (max 20, default 5)
- `country=<code>` — two-letter country code (default US)
- `freshness=pd|pw|pm|py` — past day/week/month/year, or `freshness=YYYY-MM-DDtoYYYY-MM-DD`
Response is JSON; the results live at `.web.results[]`, each with `title`, `url`, `description`, `age`. Pipe through `jq` if you want a shorter view, e.g.:
```bash
curl -s -H "Accept: application/json" -H "X-Subscription-Token: $BRAVE_API_KEY" \
--get --data-urlencode "q=<query>" "https://api.search.brave.com/res/v1/web/search" \
| jq -r '.web.results[] | "- \(.title)\n \(.url)\n \(.description)\n"'
```
There's no page-content-extraction helper here — if a result needs reading in full, `curl` the URL directly and read the raw HTML/text; don't expect readability-cleaned markdown.
## When to Use
- Searching for documentation or API references
- Looking up facts or current information
- Confirming a claim or approach against real sources
implementer-SKILL.md: |
---
name: implementer
@@ -35,7 +74,7 @@ data:
**Modes:**
- **Collect mode** (default) — search the web (`curl` against `https://api.search.brave.com/res/v1/web/search`, header `X-Subscription-Token: $BRAVE_API_KEY`, `--data-urlencode "q=<query>"`) with varied queries to cover the topic from multiple angles, then produce a structured summary: topic areas found, key facts, and links to sources for each. Do not editorialize about which approach is "right" — that's out of scope for this skill.
- **Collect mode** (default) — use the `brave-search` skill (`curl` against the Brave Search API) with varied queries to cover the topic from multiple angles, then produce a structured summary: topic areas found, key facts, and links to sources for each. Do not editorialize about which approach is "right" — that's out of scope for this skill.
- If asked to write the summary to a file, write it and report the path; otherwise return it directly in your response.
investigator-SKILL.md: |
---
@@ -46,7 +85,7 @@ data:
Check whether the plan's claims about the real world are actually true right now. Cite sources; don't assert without one.
- Read `PLAN.md` and the spec docs on disk. For each claim depending on external facts (a library's current API, a service's behavior), search the web (`curl` against `https://api.search.brave.com/res/v1/web/search`, header `X-Subscription-Token: $BRAVE_API_KEY`) to confirm or refute it. Append a `## Investigation` section to `PLAN.md`: each claim, its source(s), PASS/FLAG. Commit: `git add PLAN.md && git commit -m "investigate: confirm plan against sources"`.
- Read `PLAN.md` and the spec docs on disk. For each claim depending on external facts (a library's current API, a service's behavior), use `brave-search` to confirm or refute it. Append a `## Investigation` section to `PLAN.md`: each claim, its source(s), PASS/FLAG. Commit: `git add PLAN.md && git commit -m "investigate: confirm plan against sources"`.
- No `PLAN.md`? Just answer the question asked, citing sources.
Flag unconfirmed/contradicted claims rather than silently fixing them — that decision belongs to whoever reads the flag next.
+83
View File
@@ -0,0 +1,83 @@
# API Auth Layer — Authentik service account + Kong JWT (model invoke)
Protect the model API (`api.riotpiao.com/*`, Kong OSS 3.9) so only an Authentik
service account holding a valid **client_credentials** JWT can invoke the KServe
models. "Invoke role" = **possession of a JWT from the dedicated model-invoke
OAuth2 provider** (only the service account can obtain one).
## Flow
```
service account ── client_credentials ──▶ Authentik token endpoint
(client_id + secret) https://authentik.riotpiao.com/application/o/token/
▼ RS256 JWT (iss = https://authentik.riotpiao.com/application/o/model-invoke/)
client ── Authorization: Bearer <jwt> ──▶ Kong (api.riotpiao.com/*)
jwt plugin: verify RS256 sig via Authentik JWKS,
check iss/exp → map to KongConsumer → allow
KServe model (reasoning / ornith / ...)
```
Kong OSS has no enterprise `openid-connect` plugin, so we use the built-in
**`jwt`** plugin: it validates an RS256 signature against a public key we pin on
a KongConsumer, keyed by the token's `iss`.
## Changes
### 1. Authentik (k8s/infra/iam/scripts/authentik-provision.py)
- New **service account** user `model-invoker` (type `service_account`, no
password; Authentik issues an app-password/token for M2M).
- New **OAuth2 provider + application** `model-invoke`:
- `client_type: confidential`, `grant_types: ["client_credentials"]`
- signing key = existing RS256 keypair (same as other providers)
- mappings: `openid` (+ optionally a static `invoke` scope) — no user scopes
needed for M2M.
- Client secret written to k8s Secret `api/model-invoke-oidc`
(keys `client-id`, `client-secret`), labelled for whoever consumes it.
- Bind the service account so it (and only it) can use the provider.
### 2. Kong (k8s/apps/api/, new file `model-auth.yaml`)
- **KongConsumer** `model-invoker` (ns api).
- **`jwt` credential** on that consumer (a Secret of type
`konghq.com/v1/credential`):
- `algorithm: RS256`
- `key` = the token `iss``https://authentik.riotpiao.com/application/o/model-invoke/`
- `rsa_public_key` = the PEM public key of Authentik's `model-invoke` signing
cert (fetched from Authentik JWKS / cert, stored in git or ksops).
- **KongPlugin** `jwt-auth` (`plugin: jwt`, `config.claims_to_verify: [exp]`).
### 3. Wire onto model routes (k8s/apps/api/llm-routes.yaml)
- Add `jwt-auth` to each model Ingress's `konghq.com/plugins` annotation
(currently e.g. `llm-rewrite-reasoning`) → becomes
`llm-rewrite-reasoning,jwt-auth`.
- Leave `/models` list route open OR protect too (decision).
## Client usage (after build)
```bash
TOKEN=$(curl -s https://authentik.riotpiao.com/application/o/token/ \
-d grant_type=client_credentials \
-d client_id=model-invoke \
-d client_secret=<secret> \
-d scope=openid | jq -r .access_token)
curl https://api.riotpiao.com/v1/chat/completions \
-H "Authorization: Bearer $TOKEN" -d '{...}'
```
## Test plan
1. No token → Kong returns 401.
2. Valid client_credentials token → 200, model responds.
3. Expired/garbage token → 401.
4. Confirm the `/models` route behaviour matches the decision.
## Open items / risks
- Authentik `client_credentials` for a *service account* may require an
**app-password / JWT-assertion** flow rather than plain client_secret POST —
verify Authentik 2026.x M2M exactly (client_credentials with client_secret vs
the SA token). Adjust step 1 accordingly before wiring Kong.
- Pinning `rsa_public_key`: Authentik key rotation would break it — document a
rotation runbook, or have the provision script re-export the cert PEM into the
Kong credential on each run (keeps them in sync, same idea as ksops secrets).
- Kong `jwt` maps token→consumer by the `iss`=`key` match; ensure the provider's
issuer is stable.
+8 -11
View File
@@ -5,17 +5,14 @@
#
# nginx terminates TLS with the wildcard *.riotpiao.com cert (served as its
# default-ssl-certificate, so no per-rule `tls:` block is needed) and forwards
# plain HTTP to the gateway.
# plain HTTP to kong-proxy. Kong then does the real routing, from Ingresses
# carrying `ingressClassName: kong`.
#
# Backend was kong-proxy:80 until Kong was retired on 2026-08-19; it is now the
# Go gateway's Service, api-gateway:8080, deployed from rock/homelab-frontend.
# Reverting the cutover is a change to these two lines and nothing else.
# Catch-all `/` on purpose: everything under this host belongs to Kong. Listing
# per-API paths here would duplicate Kong's routing table inside nginx, and the
# two copies would drift.
#
# Catch-all `/` on purpose: everything under this host belongs to the gateway.
# Listing per-API paths here would duplicate the gateway's routing table inside
# nginx, and the two copies would drift.
#
# In-cluster callers should prefer http://api-gateway.api.svc.cluster.local:8080
# In-cluster callers should prefer http://kong-proxy.api.svc.cluster.local
# directly. Resolving api.riotpiao.com sends them out to nginx and back in,
# which is a pointless hairpin unless they need TLS or the public hostname.
apiVersion: networking.k8s.io/v1
@@ -41,6 +38,6 @@ spec:
pathType: Prefix
backend:
service:
name: api-gateway
name: kong-proxy
port:
number: 8080
number: 80
+19
View File
@@ -0,0 +1,19 @@
# Cluster-wide Kong Prometheus plugin -- `global: "true"` label makes the
# ingress controller apply it to every route on this Kong instance, so all
# five LLM routes (ornith/reasoning/qwen/embeddings/rerank) get RED metrics
# without touching llm-routes.yaml. Scraped via kong-values.yaml's
# serviceMonitor (status listener, already on by chart default at :8100).
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
name: prometheus
annotations:
kubernetes.io/ingress.class: kong
labels:
global: "true"
plugin: prometheus
config:
status_code_metrics: true
latency_metrics: true
bandwidth_metrics: true
upstream_health_metrics: true
+140
View File
@@ -0,0 +1,140 @@
# Kong Gateway — cluster-internal API gateway (namespace `api`).
#
# Chart: kong/kong 3.4.1 (appVersion 3.9). Only overrides are listed; every key
# here was checked against `helm show values kong/kong --version 3.4.1`, because
# Helm silently ignores unknown keys — a typo is a no-op, not an error.
#
# ── Topology ────────────────────────────────────────────────────────────────
# external: client -> nginx (TLS, wildcard *.riotpiao.com) -> kong-proxy:80
# internal: pod -> kong-proxy.api.svc.cluster.local:80
#
# nginx stays the single edge and the only LoadBalancer (192.168.1.160). Kong is
# the policy/routing layer behind it, so it needs no LB IP and no TLS of its own
# — hence ClusterIP and proxy.tls disabled. Giving Kong its own IP from
# homelab-pool would mean duplicating cert-manager wiring and diverging from the
# CoreDNS convention that sends every *.riotpiao.com host to nginx.
#
# ── Routing model ───────────────────────────────────────────────────────────
# Consumers publish an Ingress with `ingressClassName: kong`; the controller
# turns it into a Kong route. `nginx` remains the default IngressClass, so this
# is strictly opt-in and no existing Ingress changes behaviour.
# Without this the release name is prefixed onto everything (`kong-kong-proxy`).
# Pinning it keeps the Service name stable and independent of the release name,
# which matters because the nginx Ingress in k8s/bootstrap/ingress/ingress.yaml
# references it by name.
fullnameOverride: kong
# Two replicas so a node drain or rollout doesn't take the gateway down. Kong is
# stateless in DB-less mode, so replicas are pure redundancy.
replicaCount: 2
# Opt in to the `llm-serving-default-deny` NetworkPolicy, which admits port 8080
# only from pods carrying this label. That policy is a compensating control, not
# hygiene: vLLM v0.11.0 is frozen on Volta and will never receive patches for
# several remote/unauthenticated advisories, so it must not be broadly reachable.
#
# Without this label Cilium DROPS the packets rather than refusing them, so the
# symptom is a request that hangs until the client's timeout — not a connection
# error. /v1/models still worked while this was missing, because
# request-termination answers inside Kong and never touches an upstream.
podLabels:
llm-client: "true"
env:
# DB-less. Config comes from Kubernetes objects via the ingress controller, so
# git stays the source of truth. A Postgres-backed Kong would put live routing
# config in a database mutated through the Admin API — state outside git, plus
# migration Jobs on every upgrade.
database: "off"
# `nginx_proxy_<directive>` injects a directive into the proxy location block;
# this renders `proxy_buffering off;`.
#
# Required for LLM streaming. With buffering on (the default) nginx accumulates
# the upstream response before forwarding, so an SSE stream from
# `"stream": true` arrives in lumps or stalls until the generation finishes —
# which defeats the point of streaming. The matching setting is already on the
# nginx Ingress in ingress.yaml; both hops have to be unbuffered or the
# buffered one dominates.
nginx_proxy_proxy_buffering: "off"
# Any plugin that rewrites the request body — request-transformer on the
# llm-chat-* routes — reads it through `kong.request.get_body()`, and that
# returns nothing once nginx has spilled the body past
# client_body_buffer_size into a temp file. The plugin then re-serializes a
# body with no `messages`, and the upstream answers
# HTTP 400 {"error":{"message":"[] is too short - 'messages'"}}
# Measured on /v1/ornith/chat/completions: 10588 B -> 200, 11088 B -> 400.
# An agent request carrying tool schemas clears that in one turn, so the
# buffer has to hold a whole conversation, not a chat message.
nginx_http_client_body_buffer_size: "16m"
nginx_http_client_max_body_size: "16m"
ingressController:
enabled: true
ingressClass: kong
# The chart's ingress-class template is gated on
# `.Capabilities.APIVersions.Has "networking.k8s.io/v1/IngressClass"`, so a
# bare `helm template` renders nothing. ArgoCD passes --api-versions from the
# live cluster, so it does render there — verify `kubectl get ingressclass
# kong` after the first sync rather than assuming it.
createIngressClass: true
# Deliberately empty: setting is-default-class here would hijack every Ingress
# in the cluster that omits ingressClassName. nginx keeps that role.
ingressClassAnnotations: {}
proxy:
enabled: true
# Chart default is LoadBalancer, which would claim an IP from homelab-pool.
type: ClusterIP
http:
enabled: true
servicePort: 80
containerPort: 8000
# nginx already terminated TLS; a second handshake to the same cluster buys
# nothing and would need Kong to hold its own certificate.
tls:
enabled: false
# No Service for the Admin API. The controller reaches it over localhost inside
# the pod, so exposing it would only create an unauthenticated write path to the
# gateway's entire configuration.
admin:
enabled: false
# Kong Manager UI — chart default is `enabled: true` with type NodePort, which
# would open a port on every node. Not wanted.
manager:
enabled: false
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: "2"
memory: 1Gi
podDisruptionBudget:
enabled: true
minAvailable: 1
# Status listener (metrics/health) is on by default at :8100 (chart default,
# verified via `helm show values`). This just wires the ServiceMonitor the
# chart already knows how to generate for it, so kong_http_requests_total /
# kong_latency_* / kong_bandwidth_bytes land in Prometheus. Paired with the
# cluster-wide `prometheus` KongClusterPlugin in kong-metrics.yaml.
serviceMonitor:
enabled: true
labels:
release: kube-prometheus-stack
# Spread the two replicas across nodes; `ScheduleAnyway` so a single-node
# situation degrades to co-location instead of leaving a pod Pending.
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/name: kong
app.kubernetes.io/instance: kong
+7 -7
View File
@@ -1,14 +1,14 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Explicit allowlist. Anything added to this directory and not listed here is
# silently dropped — no error, no drift shown.
#
# Down to a single Ingress since Kong was retired (2026-08-19). The Kong Helm
# values, the KongClusterPlugin for Prometheus, the six KongPlugin CRs behind
# the path-per-model LLM surface, the KongConsumer and the key-auth plan all
# went with it.
# Explicit allowlist so kong-values.yaml in this directory is NOT treated as a
# manifest — it is Helm input consumed by the chart source of the `kong`
# Application, not a Kubernetes object. Anything new added here must be listed
# or it is silently dropped with no error and no drift shown.
resources:
- ingress.yaml
- kong-metrics.yaml
- llm-routes.yaml
- model-auth.yaml
# No top-level `namespace:` transformer on purpose: ingress.yaml sets its own
# namespace, and the transformer rewrites metadata.namespace on every resource
# it builds, which is a trap for anything cross-namespace added later.
+288
View File
@@ -0,0 +1,288 @@
# LLM API surface on the Kong gateway — DeepSeek/OpenAI-shaped.
#
# These live in namespace `llm-serving`, not `api`, because a Kubernetes Ingress
# can only reference a Service in its own namespace and the predictor Services
# are there. The Kong ingress controller watches all namespaces, so the routes
# still land on the gateway. They are synced by the `kong` Application (which
# has a `path: k8s/apps/api` source) so all gateway config stays in one place.
#
# ── Model -> upstream map (verified live) ───────────────────────────────────
# reasoning -> reasoning-predictor vLLM, DeepSeek-R1-Distill-32B, 2 replicas
# ornith:35b -> ornith-predictor Ollama, 2 replicas (retired verifier-
# qwen2.5:3b-instruct -> ornith-predictor Ollama predictor's vLLM PRM slot to get
# the 2nd GPU) -- k8s Service load-balances
# across both, each replica loads both
# models, so 2 concurrent implementer-style
# calls each land on an independent instance
# nomic-embed-text-v2 -> embeddings-predictor TEI
# bge-reranker-base -> reranker-predictor TEI
#
# ── Why path-per-model, and why the body is rewritten ───────────────────────
# Kong matches routes on host, path, method and headers — never on the request
# body. So a single /v1/chat/completions endpoint that dispatches on the body's
# `model` field is not expressible in Kong OSS (`ai-proxy-advanced`, which does
# multi-target model routing, is Enterprise-only).
#
# Hence the model is in the path, and each chat route force-overwrites `model`
# in the body regardless, so a client calling /v1/qwen/... with some other
# `model` value in the body can't silently get routed to the wrong weights.
# Callers may omit `model` entirely.
#
# ── Timeouts ───────────────────────────────────────────────────────────────
# Kong's upstream timeouts default to 60000ms. A 32B model generating a long
# answer on a Volta GPU routinely exceeds that, and the client would see a
# 504 mid-generation. Raised to 1h on every LLM route. Values are milliseconds.
# ── GET /v1/models ──────────────────────────────────────────────────────────
# Served entirely by Kong via request-termination: the plugin short-circuits in
# the access phase, so the backend below is never contacted. It only exists
# because an Ingress rule requires a backend.
#
# The list is static, which means it can drift from what the engines actually
# serve — notably if the Ollama pull list in the ornith InferenceService
# changes. Verify with:
# curl -s $SVC/v1/models (against each *-predictor)
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: llm-models-list
namespace: llm-serving
plugin: request-termination
config:
status_code: 200
content_type: application/json
body: |
{"object":"list","data":[
{"id":"reasoning","object":"model","owned_by":"homelab","created":0},
{"id":"ornith:35b","object":"model","owned_by":"homelab","created":0},
{"id":"qwen2.5:3b-instruct","object":"model","owned_by":"homelab","created":0},
{"id":"nomic-ai/nomic-embed-text-v2-moe","object":"model","owned_by":"homelab","created":0},
{"id":"BAAI/bge-reranker-base","object":"model","owned_by":"homelab","created":0}
]}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: llm-models
namespace: llm-serving
annotations:
konghq.com/plugins: llm-models-list # model-key-auth stripped -- see model-auth.yaml
konghq.com/strip-path: "false"
konghq.com/methods: "GET"
spec:
ingressClassName: kong
rules:
- host: api.riotpiao.com
http:
paths:
- path: /v1/models
pathType: Exact
backend:
# Never actually called — request-termination answers first.
service:
name: reasoning-predictor
port:
number: 80
---
# ── POST /v1/reasoning/chat/completions ─────────────────────────────────────
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: llm-rewrite-reasoning
namespace: llm-serving
plugin: request-transformer
config:
# `add` only applies when the field is absent, `replace` only when present.
# Both are needed to force the value in either case.
add:
body:
- "model:reasoning"
replace:
body:
- "model:reasoning"
# The model lives in the path for routing; the upstream still expects the
# canonical OpenAI path.
uri: /v1/chat/completions
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: llm-chat-reasoning
namespace: llm-serving
annotations:
konghq.com/plugins: llm-rewrite-reasoning # model-key-auth stripped -- see model-auth.yaml
konghq.com/strip-path: "false"
konghq.com/methods: "POST"
konghq.com/connect-timeout: "10000"
konghq.com/read-timeout: "3600000"
konghq.com/write-timeout: "3600000"
spec:
ingressClassName: kong
rules:
- host: api.riotpiao.com
http:
paths:
- path: /v1/reasoning/chat/completions
pathType: Prefix
backend:
service:
name: reasoning-predictor
port:
number: 80
---
# ── POST /v1/ornith/chat/completions ────────────────────────────────────────
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: llm-rewrite-ornith
namespace: llm-serving
plugin: request-transformer
config:
add:
body:
- "model:ornith:35b"
replace:
body:
- "model:ornith:35b"
uri: /v1/chat/completions
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: llm-chat-ornith
namespace: llm-serving
annotations:
konghq.com/plugins: llm-rewrite-ornith # model-key-auth stripped -- see model-auth.yaml
konghq.com/strip-path: "false"
konghq.com/methods: "POST"
konghq.com/connect-timeout: "10000"
konghq.com/read-timeout: "3600000"
konghq.com/write-timeout: "3600000"
spec:
ingressClassName: kong
rules:
- host: api.riotpiao.com
http:
paths:
- path: /v1/ornith/chat/completions
pathType: Prefix
backend:
service:
name: ornith-predictor
port:
number: 80
---
# ── POST /v1/qwen/chat/completions ──────────────────────────────────────────
# Same upstream pod as ornith — only the forced body `model` differs. Both stay
# resident because the engine runs with OLLAMA_MAX_LOADED_MODELS=2 and
# OLLAMA_KEEP_ALIVE=-1, so this does not trigger a model swap per request.
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: llm-rewrite-qwen
namespace: llm-serving
plugin: request-transformer
config:
add:
body:
- "model:qwen2.5:3b-instruct"
replace:
body:
- "model:qwen2.5:3b-instruct"
uri: /v1/chat/completions
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: llm-chat-qwen
namespace: llm-serving
annotations:
konghq.com/plugins: llm-rewrite-qwen # model-key-auth stripped -- see model-auth.yaml
konghq.com/strip-path: "false"
konghq.com/methods: "POST"
konghq.com/connect-timeout: "10000"
konghq.com/read-timeout: "3600000"
konghq.com/write-timeout: "3600000"
spec:
ingressClassName: kong
rules:
- host: api.riotpiao.com
http:
paths:
- path: /v1/qwen/chat/completions
pathType: Prefix
backend:
service:
name: ornith-predictor
port:
number: 80
---
# ── POST /v1/embeddings ─────────────────────────────────────────────────────
# No path-per-model and no rewrite: there is exactly one embeddings backend, so
# there is nothing to disambiguate, and TEI already serves the canonical
# OpenAI path (verified: /v1/embeddings returns 405 to GET, i.e. it exists).
# That makes an OpenAI SDK a drop-in here.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: llm-embeddings
namespace: llm-serving
annotations:
konghq.com/strip-path: "false"
konghq.com/methods: "POST"
konghq.com/connect-timeout: "10000"
konghq.com/read-timeout: "600000"
konghq.com/write-timeout: "600000"
spec:
ingressClassName: kong
rules:
- host: api.riotpiao.com
http:
paths:
- path: /v1/embeddings
pathType: Prefix
backend:
service:
name: embeddings-predictor
port:
number: 80
---
# ── POST /v1/rerank ─────────────────────────────────────────────────────────
# Rerank is not part of the OpenAI spec, and TEI serves it at /rerank — probing
# /v1/rerank returned 404 while /rerank returned 405, so this one genuinely
# needs the rewrite that embeddings does not.
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: llm-rewrite-rerank
namespace: llm-serving
plugin: request-transformer
config:
replace:
uri: /rerank
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: llm-rerank
namespace: llm-serving
annotations:
konghq.com/plugins: llm-rewrite-rerank # model-key-auth stripped -- see model-auth.yaml
konghq.com/strip-path: "false"
konghq.com/methods: "POST"
konghq.com/connect-timeout: "10000"
konghq.com/read-timeout: "600000"
konghq.com/write-timeout: "600000"
spec:
ingressClassName: kong
rules:
- host: api.riotpiao.com
http:
paths:
- path: /v1/rerank
pathType: Prefix
backend:
service:
name: reranker-predictor
port:
number: 80
+45
View File
@@ -0,0 +1,45 @@
# API auth layer — Kong key-auth on the model routes.
#
# TEMPORARILY RETIRED: verified live that Kong's key-auth here does not accept
# `Authorization: Bearer <key>` the way the comment below used to claim — a
# raw `apikey: <key>` header succeeds (200), the same request with only
# `Authorization: Bearer <key>` fails (401). No OpenAI-SDK-compatible client
# (pi included) sends a raw apikey header or lets you customize the header
# name, so every such client was hard-blocked. The KongPlugin below is
# commented out and every route's `konghq.com/plugins` annotation in
# llm-routes.yaml has `model-key-auth` stripped, so the model routes are
# unauthenticated for now. Re-enable once there's a Bearer-compatible fix
# (e.g. a request-transformer that copies the Bearer token into an `apikey`
# header before key-auth runs) — do not just uncomment this as-is, that
# reintroduces the exact block every real client hits.
#
# The key itself lives in the ksops-managed Secret model-invoke-apikey
# (labelled konghq.com/credential: key-auth) and is bound to the KongConsumer
# below, which stays defined (harmless without the plugin) so re-enabling
# later is a two-line uncomment instead of a rebuild.
---
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
name: model-invoker
namespace: api
annotations:
kubernetes.io/ingress.class: kong
username: model-invoker
credentials:
- model-invoke-apikey
# ---
# apiVersion: configuration.konghq.com/v1
# kind: KongPlugin
# metadata:
# name: model-key-auth
# namespace: llm-serving
# plugin: key-auth
# config:
# key_names:
# - apikey
# - authorization
# key_in_header: true
# key_in_query: false
# key_in_body: false
# hide_credentials: true
-82
View File
@@ -1,82 +0,0 @@
# ComfyUI — GPU-accelerated image generation on worker-1.
# Uses 1x V100 32GB (sm70). Freed by scaling ornith 2→1.
apiVersion: apps/v1
kind: Deployment
metadata:
name: comfyui
namespace: comfyui
labels:
app: comfyui
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: comfyui
template:
metadata:
labels:
app: comfyui
spec:
nodeSelector:
kubernetes.io/hostname: worker-1
runtimeClassName: nvidia
# k8s Service named 'comfyui' injects COMFYUI_PORT=tcp://... into pod env,
# which clobbers ai-dock's own COMFYUI_PORT variable (expects a port number).
# Disable service link injection to avoid the collision.
enableServiceLinks: false
containers:
- name: comfyui
image: ghcr.io/ai-dock/comfyui:v2-cuda-12.1.1-base-22.04
ports:
- containerPort: 8188
protocol: TCP
env:
- name: NVIDIA_VISIBLE_DEVICES
value: "all"
resources:
requests:
cpu: "4"
memory: 8Gi
nvidia.com/gpu: "1"
limits:
cpu: "8"
memory: 16Gi
nvidia.com/gpu: "1"
volumeMounts:
- mountPath: /workspace/ComfyUI/models
name: models
- mountPath: /workspace/ComfyUI/output
name: output
readinessProbe:
httpGet:
path: /
port: 8188
periodSeconds: 10
initialDelaySeconds: 30
startupProbe:
httpGet:
path: /
port: 8188
failureThreshold: 120
periodSeconds: 10
volumes:
- name: models
persistentVolumeClaim:
claimName: comfyui-models
- name: output
emptyDir: {}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: comfyui-models
namespace: comfyui
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 50Gi
-33
View File
@@ -1,33 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: comfyui
namespace: comfyui
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
# WebSocket support for ComfyUI's live preview
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
spec:
ingressClassName: nginx
tls:
- secretName: comfyui-tls
hosts:
- comfyui.riotpiao.com
rules:
- host: comfyui.riotpiao.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: comfyui
port:
number: 80
-7
View File
@@ -1,7 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- ingress.yaml
-14
View File
@@ -1,14 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: comfyui
namespace: comfyui
labels:
app: comfyui
spec:
selector:
app: comfyui
ports:
- port: 80
targetPort: 8188
protocol: TCP
-107
View File
@@ -1,107 +0,0 @@
# Gotify — Push Notifications + Email Relay
Self-hosted notification server with SMTP email forwarding sidecar.
## Architecture
```
Forgejo webhook ──POST──→ Gotify API (:80/message)
┌─────────┼─────────┐
▼ ▼
Push notification SMTP emailer sidecar
(mobile/desktop) (polls → sends email)
```
## Setup (one-time, after first deploy)
### 1. Encrypt secrets before committing
```bash
# Edit secrets.yaml with real values first, then:
sops -e -i k8s/apps/gotify/secrets.yaml
```
### 2. Create Gotify app + client tokens
1. Login to `https://gotify.riotpiao.com` with admin creds
2. **Applications** → Create `forgejo` → copy **app token**
3. **Clients** → Create `smtp-emailer` → copy **client token**
4. Update `gotify-tokens` secret:
```bash
kubectl -n notifications create secret generic gotify-tokens \
--from-literal=app-token=<APP_TOKEN> \
--from-literal=client-token=<CLIENT_TOKEN> \
--dry-run=client -o yaml | kubectl apply -f -
```
### 3. Configure Forgejo webhook
In each Forgejo repo → **Settings** → **Webhooks** → **Add Webhook** → **Gotify**:
| Field | Value |
|-------|-------|
| Target URL | `http://gotify.notifications.svc.cluster.local/message` |
| Token | The **app token** from step 2 |
| Events | Pull Request (Created, Merged, Closed) |
Or via API:
```bash
FORGEJO_TOKEN="<your-pat>"
APP_TOKEN="<gotify-app-token>"
curl -s -X POST "https://forgejo.riotpiao.com/api/v1/repos/rock/homelab/hooks" \
-H "Authorization: token $FORGEJO_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"type": "gotify",
"active": true,
"config": {
"content_type": "json",
"url": "http://gotify.notifications.svc.cluster.local/message?token='"$APP_TOKEN"'"
},
"events": ["pull_request", "pull_request_assign", "pull_request_review"],
"authorization_header": ""
}'
```
### 4. Add CoreDNS rewrite (if accessing via public hostname)
Only needed if Cloudflare Tunnel is used for gotify.riotpiao.com:
```
# terraform/files/coredns/Corefile — add rewrite:
rewrite name gotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
```
Then: `cd terraform && terraform apply && cd .. && make apply-cp`
### 5. SMTP providers
| Provider | Host | Port | Notes |
|----------|------|------|-------|
| Gmail | smtp.gmail.com | 587 | Use App Password (2FA required) |
| Resend | smtp.resend.com | 587 | Free 100 emails/day |
| Sendgrid | smtp.sendgrid.net | 587 | Free 100 emails/day |
| Mailgun | smtp.mailgun.org | 587 | Free 5000/month |
## Notification priority levels
| Priority | Meaning | Email forwarded? |
|----------|---------|-----------------|
| 0-4 | Low (info) | No (below MIN_PRIORITY=5) |
| 5-7 | Normal (PR created) | Yes |
| 8-10 | High (PR merged, failures) | Yes |
## Verify
```bash
# Test push notification
APP_TOKEN="<app-token>"
curl -X POST "https://gotify.riotpiao.com/message?token=$APP_TOKEN" \
-H "Content-Type: application/json" \
-d '{"title":"Test","message":"Hello from homelab","priority":5}'
# Check email sidecar logs
kubectl -n notifications logs deployment/gotify -c smtp-emailer --tail=20
```
-35
View File
@@ -1,35 +0,0 @@
# CNPG Postgres for Gotify. Lightweight — 2 instances, 2Gi storage.
# CNPG generates secret `gotify-db-app` + service `gotify-db-rw` in ns notifications.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: gotify-db
namespace: notifications
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:16.2
bootstrap:
initdb:
database: gotify
owner: app
encoding: UTF8
localeCollate: C
localeCType: C
enableSuperuserAccess: false
resources:
requests: { memory: "256Mi", cpu: "100m" }
limits: { memory: "512Mi", cpu: "500m" }
storage:
size: 2Gi
storageClass: longhorn-cnpg
monitoring:
enablePodMonitor: true
affinity:
podAntiAffinityType: preferred
topologyKey: kubernetes.io/hostname
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
-204
View File
@@ -1,204 +0,0 @@
# Gotify — self-hosted push notification server + SMTP email relay.
# Forgejo webhooks → Gotify → push notifications + email forwarding.
# Runs on control plane (no GPU needed), lightweight.
apiVersion: apps/v1
kind: Deployment
metadata:
name: gotify
namespace: notifications
labels:
app: gotify
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: gotify
template:
metadata:
labels:
app: gotify
spec:
containers:
# --- Gotify server ---
- name: gotify
image: ghcr.io/gotify/server:2.6.1
command: ["/bin/sh", "-c"]
args:
- |
export GOTIFY_DATABASE_DIALECT=postgres
export GOTIFY_DATABASE_CONNECTION="host=gotify-db-rw.notifications port=5432 user=${DB_USER} password=${DB_PASS} dbname=gotify sslmode=disable"
exec /app/gotify-app
ports:
- containerPort: 80
protocol: TCP
env:
- name: GOTIFY_DEFAULTUSER_NAME
valueFrom:
secretKeyRef:
name: gotify-admin
key: username
- name: GOTIFY_DEFAULTUSER_PASS
valueFrom:
secretKeyRef:
name: gotify-admin
key: password
- name: DB_USER
valueFrom:
secretKeyRef:
name: gotify-db-app
key: username
- name: DB_PASS
valueFrom:
secretKeyRef:
name: gotify-db-app
key: password
- name: GOTIFY_SERVER_PORT
value: "80"
- name: GOTIFY_SERVER_KEEPALIVEPERIODSECONDS
value: "0"
- name: TZ
value: Asia/Tokyo
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi
livenessProbe:
httpGet:
path: /health
port: 80
periodSeconds: 30
initialDelaySeconds: 10
readinessProbe:
httpGet:
path: /health
port: 80
periodSeconds: 10
initialDelaySeconds: 5
# --- SMTP emailer sidecar ---
# Watches Gotify WebSocket stream, forwards messages as email.
# https://github.com/eternal-flame-AD/gotify-broadcast
- name: smtp-emailer
image: ghcr.io/gotify/server:2.6.1
command:
- /bin/sh
- -c
- |
# Wait for Gotify to be ready
until wget -qO- http://localhost:80/health >/dev/null 2>&1; do
echo "Waiting for Gotify..."
sleep 2
done
echo "Gotify is ready, starting email relay..."
# Poll Gotify messages and forward via SMTP using msmtp
# Install msmtp for lightweight SMTP sending
apk add --no-cache msmtp curl jq
# Configure msmtp
cat > /tmp/msmtprc <<MSMTP
defaults
auth on
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile /tmp/msmtp.log
account default
host ${SMTP_HOST}
port ${SMTP_PORT}
from ${SMTP_FROM}
user ${SMTP_USER}
password ${SMTP_PASS}
MSMTP
chmod 600 /tmp/msmtprc
# Track last seen message ID
LAST_ID=0
while true; do
# Fetch messages since last ID
MESSAGES=$(curl -s -H "X-Gotify-Key: ${GOTIFY_CLIENT_TOKEN}" \
"http://localhost:80/message?since=${LAST_ID}&limit=10" 2>/dev/null)
if [ -n "$MESSAGES" ]; then
echo "$MESSAGES" | jq -r '.messages[]? | @base64' | while read -r MSG; do
DECODED=$(echo "$MSG" | base64 -d)
ID=$(echo "$DECODED" | jq -r '.id')
TITLE=$(echo "$DECODED" | jq -r '.title // "Notification"')
BODY=$(echo "$DECODED" | jq -r '.message // ""')
PRIORITY=$(echo "$DECODED" | jq -r '.priority // 5')
APP=$(echo "$DECODED" | jq -r '.appid // 0')
DATE=$(echo "$DECODED" | jq -r '.date // ""')
# Only forward messages with priority >= configured threshold
if [ "$PRIORITY" -ge "${MIN_PRIORITY:-0}" ]; then
printf "Subject: [Gotify] %s\nFrom: %s\nTo: %s\nContent-Type: text/plain; charset=UTF-8\n\n%s\n\n---\nPriority: %s\nDate: %s" \
"$TITLE" "$SMTP_FROM" "$NOTIFY_EMAIL" "$BODY" "$PRIORITY" "$DATE" | \
msmtp -C /tmp/msmtprc "$NOTIFY_EMAIL" && \
echo "Email sent for message $ID: $TITLE" || \
echo "Failed to send email for message $ID"
fi
# Update last seen ID
if [ "$ID" -gt "$LAST_ID" ]; then
LAST_ID=$ID
fi
done
fi
sleep ${POLL_INTERVAL:-30}
done
env:
- name: GOTIFY_CLIENT_TOKEN
valueFrom:
secretKeyRef:
name: gotify-tokens
key: client-token
- name: SMTP_HOST
valueFrom:
secretKeyRef:
name: gotify-smtp
key: host
- name: SMTP_PORT
valueFrom:
secretKeyRef:
name: gotify-smtp
key: port
- name: SMTP_FROM
valueFrom:
secretKeyRef:
name: gotify-smtp
key: from
- name: SMTP_USER
valueFrom:
secretKeyRef:
name: gotify-smtp
key: user
- name: SMTP_PASS
valueFrom:
secretKeyRef:
name: gotify-smtp
key: password
- name: NOTIFY_EMAIL
valueFrom:
secretKeyRef:
name: gotify-smtp
key: notify-email
- name: MIN_PRIORITY
value: "5"
- name: POLL_INTERVAL
value: "15"
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 100m
memory: 64Mi
# No volumes — Postgres handles persistence
-26
View File
@@ -1,26 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gotify
namespace: notifications
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
# WebSocket support for Gotify client connections
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
spec:
ingressClassName: nginx
rules:
- host: gotify.riotpiao.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: gotify
port:
number: 80
-8
View File
@@ -1,8 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- db.yaml
- deployment.yaml
- service.yaml
- ingress.yaml
-6
View File
@@ -1,6 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: notifications
labels:
kubernetes.io/metadata.name: notifications
-14
View File
@@ -1,14 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: gotify
namespace: notifications
labels:
app: gotify
spec:
selector:
app: gotify
ports:
- port: 80
targetPort: 80
protocol: TCP
-14
View File
@@ -1,14 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: immich-config
data:
DB_HOSTNAME: "immich-db-rw"
DB_DATABASE_NAME: "immich"
# Only pgvector is installed (see db.yaml) - no vectorchord extension image
# exists for pg18 in CNPG's catalog yet. Explicit instead of relying on
# auto-detect's vectorchord-first preference order.
DB_VECTOR_EXTENSION: "pgvector"
REDIS_HOSTNAME: "immich-redis"
IMMICH_MACHINE_LEARNING_URL: "http://immich-machine-learning:3003"
TZ: "America/Los_Angeles"
-58
View File
@@ -1,58 +0,0 @@
# Dedicated CNPG Postgres for Immich. Same recipe as paperless-db/authentik-db
# (2 instances, default longhorn storage class) except the operand is
# PostgreSQL 18, not 16.2 - the official CNPG pgvector extension image
# (ghcr.io/cloudnative-pg/pgvector) is only published for pg18, no pg16 tags
# exist in that registry. Immich itself supports pg18 fine (immich-app's own
# postgres image already ships 18-vectorchord builds).
#
# pgvector loaded via CNPG's ImageVolume extension mechanism (CNPG 1.27+,
# k8s ImageVolume feature - both present here: operator is 1.30.0, cluster is
# v1.36.1). No shared_preload_libraries needed - pgvector doesn't require
# preload, just CREATE EXTENSION, which immich-server issues itself at
# startup. Distro/pg-major must match between the operand image and the
# extension image (both "18"+"trixie" here) - CNPG's own compatibility rule.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: immich-db
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:18-minimal-trixie
postgresql:
extensions:
- name: pgvector
image:
reference: ghcr.io/cloudnative-pg/pgvector:0.8.1-18-trixie
bootstrap:
initdb:
database: immich
owner: app
encoding: UTF8
localeCollate: C
localeCType: C
# CREATE EXTENSION vector requires superuser (pgvector's control file
# isn't marked trusted) and the "app" owner role isn't one
# (enableSuperuserAccess: false, repo convention) - postInitApplicationSQL
# runs as superuser during initdb, before the app ever connects. Only
# fires on a fresh bootstrap; the live cluster already had this run
# manually once (kubectl exec ... psql -U postgres -c 'CREATE EXTENSION').
postInitApplicationSQL:
- "CREATE EXTENSION IF NOT EXISTS vector;"
- "CREATE EXTENSION IF NOT EXISTS cube;"
- "CREATE EXTENSION IF NOT EXISTS earthdistance;"
enableSuperuserAccess: false
resources:
requests: { memory: "512Mi", cpu: "250m" }
limits: { memory: "2Gi", cpu: "1" }
storage:
size: 20Gi
storageClass: longhorn
affinity:
podAntiAffinityType: preferred
topologyKey: kubernetes.io/hostname
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
-100
View File
@@ -1,100 +0,0 @@
# immich-server: pinned to talos-cp-3, same reasoning as paperless
# (deployment.yaml comment there) - immich-media is a ReadWriteOnce Longhorn
# volume with a single replica physically on that node's disk (shared with
# paperless-media on the same 4TB HDD). Recreate strategy for the same
# reason: two pods can't both attach an RWO volume.
apiVersion: apps/v1
kind: Deployment
metadata:
name: immich-server
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: immich-server
template:
metadata:
labels:
app: immich-server
spec:
serviceAccountName: immich
nodeSelector:
kubernetes.io/hostname: talos-cp-3
containers:
- name: immich-server
image: ghcr.io/immich-app/immich-server:release
ports:
- containerPort: 2283
envFrom:
- configMapRef:
name: immich-config
env:
- name: DB_USERNAME
valueFrom:
secretKeyRef:
name: immich-db-app
key: username
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: immich-db-app
key: password
# Composed by k8s/infra/iam's provisioning script (system-config
# JSON, oauth section) - see immich-oidc Secret.
- name: IMMICH_CONFIG_FILE
value: /config/immich.json
resources:
requests: { cpu: "500m", memory: "1Gi" }
limits: { cpu: "2", memory: "4Gi" }
volumeMounts:
- name: media
mountPath: /usr/src/app/upload
- name: oidc-config
mountPath: /config
readOnly: true
volumes:
- name: media
persistentVolumeClaim:
claimName: immich-media
- name: oidc-config
secret:
secretName: immich-oidc
items:
- key: config.json
path: immich.json
---
# CPU-only for now - the cluster's one GPU node (worker-1) is already
# dedicated to llm-serving predictors. Not node-pinned: its cache PVC is on
# the default 3-replica pool, not the single-disk cp-3 HDD.
apiVersion: apps/v1
kind: Deployment
metadata:
name: immich-machine-learning
spec:
replicas: 1
selector:
matchLabels:
app: immich-machine-learning
template:
metadata:
labels:
app: immich-machine-learning
spec:
serviceAccountName: immich
containers:
- name: immich-machine-learning
image: ghcr.io/immich-app/immich-machine-learning:release
ports:
- containerPort: 3003
resources:
requests: { cpu: "500m", memory: "1Gi" }
limits: { cpu: "2", memory: "4Gi" }
volumeMounts:
- name: ml-cache
mountPath: /cache
volumes:
- name: ml-cache
persistentVolumeClaim:
claimName: immich-ml-cache
-24
View File
@@ -1,24 +0,0 @@
# Direct nginx ingress, same reasoning as paperless: large uploads (photos/
# videos) and long-lived operations (video transcode, big batch uploads) need
# proxy-body-size/timeouts raised past nginx's defaults.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: immich
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
spec:
ingressClassName: nginx
rules:
- host: img.riotpiao.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: immich-server
port:
number: 2283
-14
View File
@@ -1,14 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: immich
resources:
- db.yaml
- pvc.yaml
- configmap.yaml
- redis.yaml
- deployment.yaml
- service.yaml
- ingress.yaml
- rbac.yaml
# immich-oidc Secret written by the PostSync provisioning Job in
# k8s/infra/iam (same as paperless-oidc) - not duplicated here.
-38
View File
@@ -1,38 +0,0 @@
# Two volumes:
#
# - media: original photos/videos + generated thumbnails/encoded videos.
# Shares the cp-3 USB HDD with paperless-media, same StorageClass/disk tag,
# single replica (single disk, no redundancy possible - same tradeoff
# paperless already accepts). Sized 1400Gi, not 2000Gi: the disk's real
# usable capacity (~3724GiB, formatting overhead) minus paperless-media's
# 2000Gi and ~231GiB of other apps' default-class replicas that Longhorn
# placed here anyway (disk tags only pull matching volumes in, they don't
# exclude non-matching ones when the untagged pool elsewhere is full) only
# leaves ~1493Gi of real scheduling headroom right now.
# - ml-cache: downloaded ML model weights for immich-machine-learning
# (face detection / CLIP embeddings). Small, disposable (re-downloads on
# loss), but persisted so a pod restart doesn't re-pull multi-GB models -
# default 3-replica pool, not node-pinned.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: immich-media
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn-paperless-media
resources:
requests:
storage: 1400Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: immich-ml-cache
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 5Gi
-43
View File
@@ -1,43 +0,0 @@
# Scoped operator access for immich-admins: restart/config-edit rights on
# just this service's own resources, nothing CNPG-managed (immich-db-*) or
# provisioning-managed (immich-oidc). Same pattern as
# k8s/apps/paperless/rbac.yaml. Inert until kube-apiserver's OIDC wiring
# lands (--oidc-groups-claim=groups, --oidc-groups-prefix=oidc:).
apiVersion: v1
kind: ServiceAccount
metadata:
name: immich
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: immich-operator
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
resourceNames: ["immich-server", "immich-machine-learning"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["immich-config"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["immich-oidc"]
verbs: ["get", "list", "watch", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: immich-admins-binding
subjects:
- kind: Group
name: "oidc:immich-admins"
apiGroup: rbac.authorization.k8s.io
- kind: ServiceAccount
name: immich
namespace: immich
roleRef:
kind: Role
name: immich-operator
apiGroup: rbac.authorization.k8s.io
-37
View File
@@ -1,37 +0,0 @@
# Job queue broker for immich-server. No PVC: queue state is disposable - a
# lost queue on restart just re-triggers the affected background jobs
# (thumbnail generation, ML jobs, etc.), no photo data loss since originals
# live on immich-media.
apiVersion: apps/v1
kind: Deployment
metadata:
name: immich-redis
spec:
replicas: 1
selector:
matchLabels:
app: immich-redis
template:
metadata:
labels:
app: immich-redis
spec:
containers:
- name: redis
image: redis:7-alpine
ports:
- containerPort: 6379
resources:
requests: { cpu: "50m", memory: "64Mi" }
limits: { cpu: "250m", memory: "256Mi" }
---
apiVersion: v1
kind: Service
metadata:
name: immich-redis
spec:
selector:
app: immich-redis
ports:
- port: 6379
targetPort: 6379
-21
View File
@@ -1,21 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: immich-server
spec:
selector:
app: immich-server
ports:
- port: 2283
targetPort: 2283
---
apiVersion: v1
kind: Service
metadata:
name: immich-machine-learning
spec:
selector:
app: immich-machine-learning
ports:
- port: 3003
targetPort: 3003
-8
View File
@@ -45,14 +45,6 @@ spec:
volumeMounts:
- mountPath: /mnt/models
name: models
podMetadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/metrics"
labels:
app.kubernetes.io/name: llm-embeddings
app.kubernetes.io/part-of: llm-serving
maxReplicas: 1
minReplicas: 1
nodeSelector:
-2
View File
@@ -14,7 +14,5 @@ resources:
- ornith.yaml
- reasoning.yaml
- reranker.yaml
- qwen-cpu.yaml
- networkpolicy.yaml
# No namespace transformer: every file sets its own, and the transformer would
# rewrite metadata.namespace on anything cross-namespace added later.
-76
View File
@@ -1,76 +0,0 @@
# NetworkPolicy for LLM inference engines (llm-serving namespace).
#
# These pods have NO auth — vLLM, Ollama, and TEI accept any request.
# All access MUST go through the api-gateway, which validates JWTs and
# injects identity headers (X-Forwarded-User, X-Auth-Verified).
#
# Replaces the hand-applied llm-serving-default-deny policy that used
# `llm-client: "true"` pod label as a selector — any pod in any namespace
# could self-grant access by adding that label, which defeats the purpose.
#
# This policy restricts ingress to:
# 1. api namespace (gateway) — the sole entry point for inference
# 2. monitoring namespace — Prometheus scraping vLLM/TEI /metrics
# 3. intra-namespace — pod-to-pod (future: multi-replica comms)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: llm-serving-ingress
namespace: llm-serving
labels:
app.kubernetes.io/part-of: llm-serving
spec:
podSelector:
matchLabels:
app.kubernetes.io/part-of: llm-serving
policyTypes:
- Ingress
ingress:
# Allow from api-gateway (namespace: api)
# Gateway proxies /v1/chat/completions, /v1/embeddings, /v1/rerank
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: api
ports:
- protocol: TCP
port: 8080 # vLLM, Ollama HTTP
- protocol: TCP
port: 80 # KServe predictor services
- protocol: TCP
port: 8000 # vLLM direct (some configs)
- protocol: TCP
port: 11434 # Ollama native port
# Allow Prometheus scraping from monitoring namespace
# vLLM: :8080/metrics, TEI: :9000/metrics
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
ports:
- protocol: TCP
port: 8080
- protocol: TCP
port: 9000
# Allow from paperless namespace (paperless-ai auto-tagging)
# Bypasses gateway until service-account JWT token exchange is implemented.
# paperless-ai-agent has llm:inference role in Authentik.
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: paperless
ports:
- protocol: TCP
port: 8080
- protocol: TCP
port: 80
- protocol: TCP
port: 8000
# Allow intra-namespace (pod-to-pod within llm-serving)
- from:
- podSelector:
matchLabels:
app.kubernetes.io/part-of: llm-serving
ports:
- protocol: TCP
port: 8080
+29 -23
View File
@@ -3,15 +3,20 @@ kind: InferenceService
metadata:
annotations:
serving.kserve.io/deploymentMode: RawDeployment
# The konghq.com/{connect,read,write}-timeout annotations that used to live
# here went with Kong (retired 2026-08-19). They existed because Kong read
# its upstream timeouts off the Kubernetes Service, and its 60s default cut
# off the first request after any pod restart — a restart flushes VRAM and
# reloading ornith:35b takes longer than that. OLLAMA_KEEP_ALIVE=-1 hid the
# problem in steady state.
# Kong reads its timeouts from the Kubernetes Service, not the Ingress —
# Ingress annotations configure Route entities (strip-path, methods,
# plugins), these configure the Service entity. They were on
# llm-chat-ornith's Ingress and therefore ignored, leaving Kong's 60s
# default in force. KServe propagates InferenceService annotations to the
# Service it generates, which is how they reach Kong from here.
#
# The equivalent budget now belongs to the Go gateway's per-route timeout
# config in rock/homelab-frontend, not to an annotation on this object.
# This was invisible while OLLAMA_KEEP_ALIVE=-1 kept the model resident: no
# request ever waited on a cold load. A pod restart flushes VRAM, and
# loading ornith:35b takes longer than 60s, so the first request after any
# restart returned 504.
konghq.com/connect-timeout: "10000"
konghq.com/read-timeout: "3600000"
konghq.com/write-timeout: "3600000"
labels:
app.kubernetes.io/name: llm-ornith
app.kubernetes.io/part-of: llm-serving
@@ -33,8 +38,12 @@ spec:
ollama pull ornith:35b
ollama pull qwen2.5:3b-instruct
ollama run ornith:35b "ok" >/dev/null 2>&1 || true
ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
wait $SERVE_PID
'
@@ -50,7 +59,7 @@ spec:
- name: OLLAMA_NUM_PARALLEL
value: '1'
- name: OLLAMA_MAX_LOADED_MODELS
value: '1'
value: '2'
image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
name: kserve-container
ports:
@@ -61,7 +70,8 @@ spec:
command:
- /bin/sh
- -c
- ollama ps 2>/dev/null | grep -q ornith
- ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
grep -q qwen2.5
periodSeconds: 10
resources:
limits:
@@ -77,26 +87,22 @@ spec:
command:
- /bin/sh
- -c
- ollama ps 2>/dev/null | grep -q ornith
- ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
grep -q qwen2.5
failureThreshold: 120
periodSeconds: 15
volumeMounts:
- mountPath: /mnt/models
name: models
podMetadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/metrics"
labels:
app.kubernetes.io/name: llm-ornith
app.kubernetes.io/part-of: llm-serving
deploymentStrategy:
type: Recreate
# 1 replica -- ornith:35b only. qwen2.5:3b moved to CPU on cp-2.
# Frees 1 GPU for ComfyUI.
maxReplicas: 1
minReplicas: 1
# 2 replicas -- each its own GPU, each loading both ornith:35b and
# qwen2.5:3b-instruct -- so 2 concurrent implementer-style calls each
# get an independent instance instead of contending on one, at the
# cost of judge/qwen traffic still sharing whichever replica an
# implementer call also lands on.
maxReplicas: 2
minReplicas: 2
nodeSelector:
kubernetes.io/hostname: worker-1
runtimeClassName: nvidia
-115
View File
@@ -1,115 +0,0 @@
# qwen2.5:3b-instruct on CPU (talos-cp-2, 144GB RAM, 24 cores).
# Moved off GPU to free a V100 for ComfyUI. Latency ~10x slower
# than GPU but sufficient for lightweight tasks (summarization,
# classification, quick answers).
apiVersion: apps/v1
kind: Deployment
metadata:
name: qwen-cpu
namespace: llm-serving
labels:
app: qwen-cpu
app.kubernetes.io/name: qwen-cpu
app.kubernetes.io/part-of: llm-serving
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: qwen-cpu
template:
metadata:
labels:
app: qwen-cpu
app.kubernetes.io/name: qwen-cpu
app.kubernetes.io/part-of: llm-serving
spec:
nodeSelector:
kubernetes.io/hostname: talos-cp-2
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
- name: ollama
image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
command: ["/bin/sh", "-c"]
args:
- |
ollama serve &
SERVE_PID=$!
until ollama list >/dev/null 2>&1; do sleep 2; done
ollama pull qwen2.5:3b-instruct
ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
wait $SERVE_PID
env:
- name: OLLAMA_HOST
value: "0.0.0.0:8080"
- name: OLLAMA_MODELS
value: /root/.ollama/models
- name: OLLAMA_CONTEXT_LENGTH
value: "32768"
- name: OLLAMA_KEEP_ALIVE
value: "-1"
- name: OLLAMA_MAX_LOADED_MODELS
value: "1"
- name: OLLAMA_NUM_PARALLEL
value: "2"
ports:
- containerPort: 8080
protocol: TCP
readinessProbe:
exec:
command: ["/bin/sh", "-c", "ollama ps 2>/dev/null | grep -q qwen2.5"]
periodSeconds: 10
startupProbe:
exec:
command: ["/bin/sh", "-c", "ollama ps 2>/dev/null | grep -q qwen2.5"]
failureThreshold: 60
periodSeconds: 10
resources:
requests:
cpu: "4"
memory: 4Gi
limits:
cpu: "8"
memory: 8Gi
volumeMounts:
- mountPath: /root/.ollama
name: ollama-data
volumes:
- name: ollama-data
persistentVolumeClaim:
claimName: qwen-cpu-data
---
apiVersion: v1
kind: Service
metadata:
name: qwen-cpu
namespace: llm-serving
labels:
app: qwen-cpu
app.kubernetes.io/part-of: llm-serving
spec:
selector:
app: qwen-cpu
ports:
- port: 80
targetPort: 8080
protocol: TCP
---
# Small PVC for qwen2.5:3b model weights (~1.9GB).
# Separate from llm-models PVC which is pinned to worker-1.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: qwen-cpu-data
namespace: llm-serving
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 5Gi
+35 -56
View File
@@ -12,59 +12,46 @@ spec:
predictor:
containers:
- args:
# bnb-4bit retired: no int4 tensor cores on sm70/V100, dequant-then-
# matmul is two slow kernel launches instead of one fused int4 GEMM,
# decode crawled at 2.5-10 tok/s regardless of TP/PP. Switched to
# JunHowie/Qwen3-32B-GPTQ-Int4 -- same dense Qwen3-32B weights, same
# hermes/qwen3 parser stack (no narration-bug risk, same as before),
# only the quant format changes. Plain (non-Marlin) GPTQ kernel is
# confirmed Volta-compatible; Marlin needs sm80+ and vLLM would try
# to auto-upgrade to it, so --quantization is pinned explicitly to
# `gptq` to force the plain kernel. Verified checkpoint size: 19.34GB
# (summed from the real safetensors index, not bits-per-param math).
# max-model-len=131072 is Qwen3-32B's real ceiling (config.json YaRN:
# factor=4.0, original_max_position_embeddings=32768) -- 200k was
# asked for but exceeds this architecturally regardless of VRAM.
# KV cache math: 256KB/token total (64 layers, 8 KV heads, 128
# head_dim, fp16), PP=2 splits both weights and KV load ~evenly, so
# each GPU carries ~9.67GB weights + ~128KB/token KV. At
# gpu-memory-utilization=0.90 (28.8GB/GPU usable), that leaves
# ~19.1GB/GPU for KV cache -> ~156k tokens/GPU capacity, comfortably
# above the 131072 target with room to spare -- the old
# OffloadingConnector CPU-DRAM spillover (tuned for the previous
# model's much smaller 16384 context) is no longer needed and is
# dropped. Staying on PP=2 and vLLM 0.11.0 (no version bump needed,
# this checkpoint only requires vllm>=0.9.2) -- plain GPTQ has no
# TP>1 restriction unlike bnb, so tensor-parallel-size=2 is worth
# trying later, but not risking a parallelism-strategy change in the
# same rollout as the quant+context-length change.
# This GPTQ requant's own config.json ships max_position_embeddings=
# 40960 and rope_scaling=None -- confirmed directly (curl'd the raw
# config.json), the base Qwen3-32B repo's YaRN block did NOT carry
# over during quantization. Re-applying it explicitly here restores
# the same math the base model documents (32768 * 4.0 = 131072);
# without this, --max-model-len=131072 fails ModelConfig validation
# against the checkpoint's own (unscaled) 40960 ceiling.
- --model=JunHowie/Qwen3-32B-GPTQ-Int4
- --model=unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit
- --served-model-name=reasoning
- --quantization=gptq
- --quantization=bitsandbytes
- --dtype=float16
- --kv-cache-dtype=auto
- --rope-scaling={"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}
- --tensor-parallel-size=1
- --pipeline-parallel-size=2
- --max-model-len=131072
- --max-model-len=16384
- --gpu-memory-utilization=0.90
- --max-num-seqs=4
- --enable-chunked-prefill
- --enable-prefix-caching
# qwen3 is vLLM's dedicated reasoning parser for this family's <think>
# blocks.
- --reasoning-parser=qwen3
# hermes is the documented tool-call parser for general (non-Coder)
# Qwen3 models -- native chat template support, not narrated text.
- --reasoning-parser=deepseek_r1
# Without these, any client sending tool_choice="auto" (pi does, for
# Read/Bash/etc.) gets a 400: "auto" tool choice requires
# --enable-auto-tool-choice and --tool-call-parser to be set.
# deepseek_v3 (matching --reasoning-parser above) 400s here --
# "DeepSeek-V3 Tool parser could not locate tool call start/end
# tokens in the tokenizer" -- this checkpoint is a Qwen2.5-32B base
# distilled on R1 traces, so its tokenizer never got DeepSeek-V3's
# own special tool-call tokens registered even though it kept R1's
# <think> reasoning format. hermes parses tool calls from plain text
# patterns instead of special tokens, so it works against the
# underlying Qwen tokenizer regardless. Verified live: deepseek_v3
# 400s, hermes returns a real tool_calls response.
- --enable-auto-tool-choice
- --tool-call-parser=hermes
# vLLM 0.11.0's native OffloadingConnector -- spills KV cache blocks
# to CPU DRAM instead of discarding them on preemption (max-num-seqs=4
# + max-model-len=16384 means concurrent long sequences compete for
# the same GPU KV space). No extra dependency, built into vLLM core.
# num_cpu_blocks is a starting point sized against the +4Gi/replica
# headroom added below (worker-1 has ~18Gi of request headroom across
# both replicas as of 2026-08-19) -- watch actual host memory usage
# and adjust; block_size=128 tokens matches vLLM's own example.
# Note: 0.11.0 ships the original (fragmented, small-transfer-block)
# version of this connector -- 0.12.0 consolidates KV data into one
# contiguous block per request and is reported an order of magnitude
# faster for this specific feature, so this is a real but not yet
# optimal implementation until the image gets bumped.
- --kv-transfer-config={"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"num_cpu_blocks":2000,"block_size":128}}
- --host=0.0.0.0
- --port=8080
env:
@@ -87,12 +74,12 @@ spec:
resources:
limits:
cpu: '16'
memory: 36Gi
nvidia.com/gpu: '2'
memory: 24Gi
nvidia.com/gpu: '1'
requests:
cpu: '8'
memory: 12Gi
nvidia.com/gpu: '2'
nvidia.com/gpu: '1'
startupProbe:
failureThreshold: 80
httpGet:
@@ -104,18 +91,10 @@ spec:
name: models
- mountPath: /dev/shm
name: shm
podMetadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/metrics"
labels:
app.kubernetes.io/name: llm-reasoning
app.kubernetes.io/part-of: llm-serving
deploymentStrategy:
type: Recreate
maxReplicas: 1
minReplicas: 1
maxReplicas: 2
minReplicas: 2
nodeSelector:
kubernetes.io/hostname: worker-1
runtimeClassName: nvidia
-8
View File
@@ -45,14 +45,6 @@ spec:
volumeMounts:
- mountPath: /mnt/models
name: models
podMetadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/metrics"
labels:
app.kubernetes.io/name: llm-reranker
app.kubernetes.io/part-of: llm-serving
maxReplicas: 1
minReplicas: 1
nodeSelector:
@@ -13,7 +13,6 @@ spec:
labels:
app: management-service
spec:
serviceAccountName: kmsvc
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
@@ -2,7 +2,7 @@ namespace: sqs
replicaCount: 3
image:
repository: forgejo.riotpiao.com/rock/kmsvc-manage
repository: ghcr.io/riotpiaole/kmsvc-management-service
tag: latest
pullPolicy: Always
@@ -1,6 +0,0 @@
apiVersion: v2
name: memory-queues
description: Kafka queues (DLQ) for Poimen Memory service (Phase 6.6)
type: application
version: 0.1.0
appVersion: "1.0"
@@ -1,20 +0,0 @@
{{- range .Values.queues }}
---
apiVersion: kmsvc.io/v1alpha1
kind: Queue
metadata:
name: {{ .name }}
namespace: {{ $.Values.namespace }}
labels:
app: memory-service
queue: dlq
spec:
name: {{ .name }}
description: {{ .description }}
partitions: {{ .partitions }}
replicationFactor: {{ .replicationFactor }}
config:
retention.ms: "{{ .config.retention.ms }}"
message.retention.seconds: "{{ .config.message.retention.seconds }}"
visibility.timeout.seconds: "{{ .config.visibility.timeout.seconds }}"
{{- end }}
@@ -1,25 +0,0 @@
# Poimen Memory Service Kafka Queues (kmsvc)
# Phase 6.6: DLQ topics for webhook + metrics failures
queues:
# DLQ for extraction, webhook, and agent failures
- name: poimen-memory-dlq
description: "DLQ for extraction, webhook, and agent failures"
partitions: 3
replicationFactor: 1
config:
retention.ms: "1209600000" # 14 days
message.retention.seconds: "1209600"
visibility.timeout.seconds: "300"
# DLQ for metrics persistence failures
- name: poimen-memory-metric-dlq
description: "DLQ for metrics persistence failures"
partitions: 3
replicationFactor: 1
config:
retention.ms: "1209600000" # 14 days
message.retention.seconds: "1209600"
visibility.timeout.seconds: "300"
namespace: sqs
@@ -18,6 +18,15 @@ rules:
- apiGroups: ["kmsvc.io"]
resources: ["queues/finalizers"]
verbs: ["update"]
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers/status"]
verbs: ["get", "update", "patch"]
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers/finalizers"]
verbs: ["update"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
@@ -27,6 +36,9 @@ rules:
- apiGroups: [""]
resources: ["pods", "nodes"]
verbs: ["get"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
@@ -0,0 +1,62 @@
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: temporalworkers.kmsvc.io
spec:
group: kmsvc.io
names:
kind: TemporalWorker
plural: temporalworkers
singular: temporalworker
scope: Namespaced
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
required:
- spec
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
type: object
description: Temporal worker specification
properties:
namespace:
type: string
description: Temporal namespace
taskQueue:
type: string
description: Task queue name
workflowTypes:
type: array
items:
type: string
description: List of workflow types to execute
activityTypes:
type: array
items:
type: string
description: List of activity types to execute
concurrency:
type: integer
minimum: 1
description: Worker concurrency level
status:
type: object
description: Temporal worker status
properties:
ready:
type: boolean
lastHeartbeat:
type: string
format: date-time
error:
type: string
+1 -1
View File
@@ -1,7 +1,7 @@
namespace: sqs
image:
repository: forgejo.riotpiao.com/rock/kmsvc-manage
repository: ghcr.io/riotpiaole/kmsvc-management-service
tag: latest
pullPolicy: Always
-95
View File
@@ -1,95 +0,0 @@
# Overrides paperless-ngx's own paperless/adapter.py at the same import path
# (mounted via subPath in deployment.yaml) - settings.py hardcodes
# SOCIALACCOUNT_ADAPTER = "paperless.adapter.CustomSocialAccountAdapter", so
# no Django setting needs to change, just the file content underneath it.
#
# Stock CustomSocialAccountAdapter.populate_user() is a stub ("kept in case
# global default permissions are implemented in the future" - they aren't),
# so every OIDC signup lands with zero permissions and 403s on every API
# endpoint. This adds the actual mapping: Authentik's "permissions" claim
# (via the permissions scope, requested in PAPERLESS_SOCIALACCOUNT_PROVIDERS,
# computed server-side from group membership by authentik-provision.py) ->
# "paperless:write" or "*" (homelab-admins) grants is_staff+is_superuser,
# same convention already used for MinIO's policy claim and Grafana's
# role_attribute_path. Checking the permission string rather than a literal
# group name decouples "what grants access" from which group happens to
# hold it - same pattern applies to every other service's Role/RoleBinding
# in k8s/infra/rbac/.
apiVersion: v1
kind: ConfigMap
metadata:
name: paperless-adapter
data:
adapter.py: |
from urllib.parse import quote
from allauth.account.adapter import DefaultAccountAdapter
from allauth.core import context
from allauth.socialaccount.adapter import DefaultSocialAccountAdapter
from django.conf import settings
from django.forms import ValidationError
from django.urls import reverse
REQUIRED_PERMISSIONS = {"paperless:write", "*"}
class CustomAccountAdapter(DefaultAccountAdapter):
def is_open_for_signup(self, request):
allow_signups = super().is_open_for_signup(request)
return getattr(settings, "ACCOUNT_ALLOW_SIGNUPS", allow_signups)
def pre_authenticate(self, request, **credentials):
if settings.DISABLE_REGULAR_LOGIN:
raise ValidationError("Regular login is disabled")
return super().pre_authenticate(request, **credentials)
def is_safe_url(self, url):
from django.utils.http import url_has_allowed_host_and_scheme
allowed_hosts = {context.request.get_host()} | set(settings.ALLOWED_HOSTS)
if "*" in allowed_hosts:
allowed_hosts.remove("*")
allowed_hosts.add(context.request.get_host())
return url_has_allowed_host_and_scheme(url, allowed_hosts=allowed_hosts)
return url_has_allowed_host_and_scheme(url, allowed_hosts=allowed_hosts)
def get_reset_password_from_key_url(self, key):
if settings.PAPERLESS_URL is None:
return super().get_reset_password_from_key_url(key)
path = reverse(
"account_reset_password_from_key",
kwargs={"uidb36": "UID", "key": "KEY"},
)
path = path.replace("UID-KEY", quote(key))
return settings.PAPERLESS_URL + path
class CustomSocialAccountAdapter(DefaultSocialAccountAdapter):
def is_open_for_signup(self, request, sociallogin):
allow_signups = super().is_open_for_signup(request, sociallogin)
return getattr(settings, "SOCIALACCOUNT_ALLOW_SIGNUPS", allow_signups)
def get_connect_redirect_url(self, request, socialaccount):
return reverse("base")
def populate_user(self, request, sociallogin, data):
user = super().populate_user(request, sociallogin, data)
perms = set(sociallogin.account.extra_data.get("permissions") or [])
if perms & REQUIRED_PERMISSIONS:
user.is_staff = True
user.is_superuser = True
return user
def save_user(self, request, sociallogin, form=None):
# populate_user() sets the flags on the in-memory user, but
# allauth's default save_user() re-derives is_staff from
# ACCOUNT_DEFAULT_HTTP_PROTOCOL-independent defaults and can
# overwrite them on save - re-apply after super().save_user()
# persists the row, matching the permissions check above exactly.
user = super().save_user(request, sociallogin, form)
perms = set(sociallogin.account.extra_data.get("permissions") or [])
if perms & REQUIRED_PERMISSIONS and not (user.is_staff and user.is_superuser):
user.is_staff = True
user.is_superuser = True
user.save(update_fields=["is_staff", "is_superuser"])
return user
-94
View File
@@ -1,94 +0,0 @@
# Nightly: pg_dump the paperless DB + mirror the media PVC into the scoped
# `paperless` MinIO bucket (see minio-provision-paperless-job.yaml). This is a
# BACKUP target, not live storage - paperless-ngx has no native S3 backend, it
# only ever reads/writes the local media PVC directly.
#
# Pinned to talos-cp-3, same as deployment.yaml: media is a ReadWriteOnce
# Longhorn volume with a single replica physically on that node's disk -
# mounting it read-only here from a different node would conflict with the
# live webserver's attachment.
apiVersion: batch/v1
kind: CronJob
metadata:
name: paperless-backup
spec:
schedule: "0 3 * * *" # 03:00 daily, low-traffic window
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
nodeSelector:
kubernetes.io/hostname: talos-cp-3
initContainers:
- name: pg-dump
image: postgres:16-alpine
env:
- name: PGHOST
value: paperless-db-rw
- name: PGDATABASE
value: paperless
- name: PGUSER
valueFrom:
secretKeyRef:
name: paperless-db-app
key: username
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: paperless-db-app
key: password
command:
- sh
- -c
- pg_dump --format=custom --file=/backup/paperless-db.dump
volumeMounts:
- name: backup
mountPath: /backup
containers:
- name: mc-mirror
image: quay.io/minio/mc:latest
env:
- name: ACCESS_KEY
valueFrom:
secretKeyRef:
name: paperless-minio-creds
key: ACCESS_KEY
- name: SECRET_KEY
valueFrom:
secretKeyRef:
name: paperless-minio-creds
key: SECRET_KEY
- name: BUCKET
valueFrom:
secretKeyRef:
name: paperless-minio-creds
key: BUCKET
- name: ENDPOINT
valueFrom:
secretKeyRef:
name: paperless-minio-creds
key: ENDPOINT
command:
- /bin/sh
- -c
- |
set -e
mc alias set b "$ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY"
mc cp /backup/paperless-db.dump "b/$BUCKET/db/paperless-db-$(date +%Y%m%d).dump"
mc mirror --overwrite /media "b/$BUCKET/media"
echo "Backup done."
volumeMounts:
- name: backup
mountPath: /backup
- name: media
mountPath: /media
readOnly: true
volumes:
- name: backup
emptyDir: {}
- name: media
persistentVolumeClaim:
claimName: paperless-media
readOnly: true
-22
View File
@@ -1,22 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: paperless-config
data:
PAPERLESS_URL: "https://paperless.riotpiao.com"
PAPERLESS_TIME_ZONE: "America/Los_Angeles"
PAPERLESS_OCR_LANGUAGE: "eng"
PAPERLESS_DBHOST: "paperless-db-rw"
PAPERLESS_DBNAME: "paperless"
PAPERLESS_REDIS: "redis://paperless-redis:6379"
# django-allauth generic OIDC provider. The client_id/secret/server_url
# bundle itself lives in the paperless-oidc Secret
# (SOCIALACCOUNT_PROVIDERS_JSON key, composed by authentik-provision.py) -
# env vars can't be split across a ConfigMap + Secret for the same key, so
# this whole value is sourced from the Secret in deployment.yaml instead.
PAPERLESS_APPS: "allauth.socialaccount.providers.openid_connect"
# Authentik already verifies identity via OIDC - a second email-confirmation
# step has no SMTP configured to send it anyway, and paperless-ngx doesn't
# wire up allauth's confirm-email view, so signup 500s with NoReverseMatch
# on 'account_confirm_email' without this.
PAPERLESS_ACCOUNT_EMAIL_VERIFICATION: "none"
-103
View File
@@ -1,103 +0,0 @@
# Single container runs webserver + consumer + scheduler (paperless-ngx's
# stock entrypoint does this internally) - no need to split into separate
# Deployments. replicas: 1 only: paperless-media is ReadWriteOnce, and the
# consumer polling the media dir doesn't benefit from horizontal scaling here.
#
# Pinned to talos-cp-3: paperless-media's disk physically lives there. Longhorn
# RWO volumes can only be attached from one node at a time, and the nightly
# backup-cronjob.yaml also mounts this same PVC (read-only) to mirror it into
# MinIO - pinning both to the same node avoids a cross-node attach conflict,
# and keeps the 3.5Ti read/write path off the network entirely.
apiVersion: apps/v1
kind: Deployment
metadata:
name: paperless
spec:
replicas: 1
strategy:
type: Recreate # ReadWriteOnce media PVC - avoid two pods fighting over it
selector:
matchLabels:
app: paperless
template:
metadata:
labels:
app: paperless
spec:
# Kubernetes injects legacy Docker-links env vars for every Service in
# this namespace (<SVC>_SERVICE_HOST, <SVC>_PORT, ...). The Service here
# is named "paperless", so that becomes PAPERLESS_PORT=tcp://<ip>:8000 -
# paperless-ngx's own entrypoint reads PAPERLESS_PORT for gunicorn's
# bind address, collides, and gunicorn crash-loops on "not a valid port
# number". Disable the injection instead of renaming the Service.
enableServiceLinks: false
nodeSelector:
kubernetes.io/hostname: talos-cp-3
containers:
- name: paperless
image: ghcr.io/paperless-ngx/paperless-ngx:2.20.15
ports:
- containerPort: 8000
envFrom:
- configMapRef:
name: paperless-config
env:
- name: PAPERLESS_DBUSER
valueFrom:
secretKeyRef:
name: paperless-db-app
key: username
- name: PAPERLESS_DBPASS
valueFrom:
secretKeyRef:
name: paperless-db-app
key: password
- name: PAPERLESS_SECRET_KEY
valueFrom:
secretKeyRef:
name: paperless-secrets
key: PAPERLESS_SECRET_KEY
- name: PAPERLESS_ADMIN_USER
valueFrom:
secretKeyRef:
name: paperless-secrets
key: PAPERLESS_ADMIN_USER
- name: PAPERLESS_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: paperless-secrets
key: PAPERLESS_ADMIN_PASSWORD
- name: PAPERLESS_SOCIALACCOUNT_PROVIDERS
valueFrom:
secretKeyRef:
name: paperless-oidc
key: SOCIALACCOUNT_PROVIDERS_JSON
resources:
requests: { cpu: "500m", memory: "1Gi" }
limits: { cpu: "2", memory: "4Gi" }
volumeMounts:
- name: media
mountPath: /usr/src/paperless/media
- name: data
mountPath: /usr/src/paperless/data
- name: consume
mountPath: /usr/src/paperless/consume
# Overrides paperless-ngx's own adapter.py in place - settings.py
# hardcodes the import path, so no Django setting changes, just
# the file content underneath it (see adapter-configmap.yaml).
- name: adapter
mountPath: /usr/src/paperless/src/paperless/adapter.py
subPath: adapter.py
readOnly: true
volumes:
- name: media
persistentVolumeClaim:
claimName: paperless-media
- name: data
persistentVolumeClaim:
claimName: paperless-data
- name: consume
emptyDir: {}
- name: adapter
configMap:
name: paperless-adapter
-24
View File
@@ -1,24 +0,0 @@
# Direct nginx ingress to the paperless Service - not routed via the Go
# api-gateway (api.riotpiao.com), which has no WebSocket upgrade support and
# paperless-ngx keeps a long-lived /ws/ connection open for live task status.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: paperless
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "0" # large scanned PDF uploads
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
spec:
ingressClassName: nginx
rules:
- host: paperless.riotpiao.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: paperless
port:
number: 8000
-18
View File
@@ -1,18 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: paperless
resources:
- pvc.yaml
- configmap.yaml
- redis.yaml
- deployment.yaml
- service.yaml
- ingress.yaml
- backup-cronjob.yaml
- adapter-configmap.yaml
- rbac.yaml
- paperless-ai.yaml
# postgres: paperless-db CNPG Cluster, deployed by k8s/infra/databases (wave 2,
# before this app at wave 8) - not duplicated here. Same for the paperless-oidc
# and paperless-minio-creds Secrets, written by PostSync provisioning Jobs in
# k8s/infra/iam and k8s/infra/minio respectively.
-64
View File
@@ -1,64 +0,0 @@
# Secret paperless-ai-config managed via SOPS (argocd/secrets/paperless-ai-secrets.enc.yaml)
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: paperless-ai
namespace: paperless
labels:
app.kubernetes.io/name: paperless-ai
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: paperless-ai
template:
metadata:
labels:
app.kubernetes.io/name: paperless-ai
spec:
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
- name: paperless-ai
image: clusterzx/paperless-ai:latest
env:
# Paperless-ngx connection
- name: PAPERLESS_API_URL
value: "http://paperless.paperless.svc.cluster.local:8000"
- name: PAPERLESS_API_TOKEN
valueFrom:
secretKeyRef:
name: paperless-ai-config
key: PAPERLESS_API_TOKEN
- name: PAPERLESS_USERNAME
value: "admin"
# LLM API — local gateway, no auth required (phase 3 not built yet)
- name: AI_PROVIDER
value: "custom"
- name: CUSTOM_BASE_URL
value: "http://reasoning-predictor.llm-serving.svc.cluster.local:80/v1"
- name: CUSTOM_API_KEY
value: "not-required"
- name: CUSTOM_MODEL
value: "reasoning"
# Behavior
- name: SCAN_INTERVAL
value: "300"
- name: PROCESS_PREDEFINED_DOCUMENTS
value: "no"
- name: ADD_AI_TAG
value: "yes"
- name: AI_TAG_NAME
value: "ai-processed"
- name: USE_PROMPT_TAGS
value: "yes"
resources:
requests:
cpu: 100m
memory: 512Mi
limits:
cpu: "1"
memory: 2Gi
-35
View File
@@ -1,35 +0,0 @@
# Two volumes, deliberately separate storage classes:
#
# - media: the actual documents (originals + OCR'd archive PDFs + thumbnails).
# Lives on the cp-3 USB HDD, single replica (see
# k8s/infra/longhorn/longhorn-paperless-storageclass.yaml). Shares the disk
# with Immich's immich-media PVC (k8s/apps/immich/pvc.yaml, 2000Gi) - photo
# libraries grow much faster than scanned documents, so paperless gets the
# smaller 500Gi share.
# - data: the SQLite classification model + search index. Small (low GB),
# frequently rewritten, and disposable (rebuilds from the DB + media on
# next consume) - stays on the default 3-replica pool instead of the
# single-disk HDD.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: paperless-media
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn-paperless-media
resources:
requests:
storage: 500Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: paperless-data
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 5Gi
-35
View File
@@ -1,35 +0,0 @@
# Scoped operator access for paperless-admins: restart/config-edit rights on
# just this service's own resources, nothing CNPG-managed (paperless-db-*)
# or provisioning-managed (paperless-oidc, paperless-minio-creds). Inert
# until kube-apiserver's OIDC wiring lands (--oidc-groups-claim=groups,
# --oidc-groups-prefix=oidc:) - subject name below assumes that prefix.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: paperless-operator
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
resourceNames: ["paperless"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["paperless-config"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["paperless-secrets"]
verbs: ["get", "list", "watch", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: paperless-admins-binding
subjects:
- kind: Group
name: "oidc:paperless-admins"
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: paperless-operator
apiGroup: rbac.authorization.k8s.io
-37
View File
@@ -1,37 +0,0 @@
# Task queue broker + websocket channel layer for paperless-ngx. No PVC:
# queued/scheduled task state is disposable - a lost queue on restart just
# means re-triggering consumption, not data loss (documents themselves live
# on paperless-media).
apiVersion: apps/v1
kind: Deployment
metadata:
name: paperless-redis
spec:
replicas: 1
selector:
matchLabels:
app: paperless-redis
template:
metadata:
labels:
app: paperless-redis
spec:
containers:
- name: redis
image: redis:7-alpine
ports:
- containerPort: 6379
resources:
requests: { cpu: "50m", memory: "64Mi" }
limits: { cpu: "250m", memory: "256Mi" }
---
apiVersion: v1
kind: Service
metadata:
name: paperless-redis
spec:
selector:
app: paperless-redis
ports:
- port: 6379
targetPort: 6379
-10
View File
@@ -1,10 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: paperless
spec:
selector:
app: paperless
ports:
- port: 8000
targetPort: 8000
@@ -1,144 +0,0 @@
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: secretrotations.homelab.riotpiao.com
spec:
group: homelab.riotpiao.com
names:
kind: SecretRotation
plural: secretrotations
scope: Namespaced
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
metadata:
type: object
spec:
type: object
required:
- provider
- rotationInterval
properties:
# External system: authentik | forgejo | minio | vault
provider:
type: string
enum: [authentik, forgejo, minio, vault]
# How often to rotate (hours)
rotationInterval:
type: integer
minimum: 24
# Application ID in external system
appId:
type: string
# k8s Secret to update (name, namespace, key)
secretRef:
type: object
required: [name, namespace]
properties:
name:
type: string
namespace:
type: string
key:
type: string
description: "Secret key to update (e.g., MINIO_IDENTITY_OPENID_CLIENT_SECRET)"
# Path to git file that holds the secret (for .enc.yaml files)
gitPath:
type: string
description: "Path in homelab repo to .enc.yaml file"
# Ansible template values to substitute
templateValues:
type: object
additionalProperties:
type: string
status:
type: object
properties:
lastRotationTime:
type: string
format: date-time
nextRotationTime:
type: string
format: date-time
lastRotationStatus:
type: string
enum: [Success, Failed, Pending]
lastRotationError:
type: string
lastCommitHash:
type: string
---
# Example usage:
apiVersion: homelab.riotpiao.com/v1
kind: SecretRotation
metadata:
name: minio-oidc
namespace: secret-rotation
spec:
provider: authentik
rotationInterval: 2160 # 90 days in hours
appId: minio
secretRef:
name: minio-oidc
namespace: storage
key: MINIO_IDENTITY_OPENID_CLIENT_SECRET
gitPath: k8s/argocd/secrets/minio-oidc.enc.yaml
---
apiVersion: homelab.riotpiao.com/v1
kind: SecretRotation
metadata:
name: portfolio-agent-oidc
namespace: secret-rotation
spec:
provider: authentik
rotationInterval: 2160
appId: portfolio-agent
secretRef:
name: portfolio-agent-oidc
namespace: portfolio
key: CLIENT_SECRET
gitPath: k8s/argocd/secrets/portfolio-agent-oidc.enc.yaml
---
apiVersion: homelab.riotpiao.com/v1
kind: SecretRotation
metadata:
name: forgejo-registry-token
namespace: secret-rotation
spec:
provider: forgejo
rotationInterval: 2160
appId: rock/riotpiao.com
secretRef:
name: forgejo-registry-secret
namespace: kube-system
key: REGISTRY_TOKEN
gitPath: k8s/argocd/secrets/forgejo-registry-secret.enc.yaml
---
apiVersion: homelab.riotpiao.com/v1
kind: SecretRotation
metadata:
name: minio-root-credentials
namespace: secret-rotation
spec:
provider: minio
rotationInterval: 4320 # 180 days in hours
appId: root
secretRef:
name: minio-creds
namespace: storage
gitPath: k8s/argocd/secrets/minio-secrets.enc.yaml
@@ -1,92 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: secret-rotation-controller
namespace: secret-rotation
spec:
replicas: 1
selector:
matchLabels:
app: secret-rotation-controller
template:
metadata:
labels:
app: secret-rotation-controller
spec:
serviceAccountName: secret-rotation-controller
containers:
- name: controller
image: secret-rotation-controller:latest
imagePullPolicy: IfNotPresent
env:
# SOPS reads age key from this file
- name: SOPS_AGE_KEY_FILE
value: /etc/sops/age/private-key.txt
# Vault auth (token in projected volume)
- name: VAULT_ADDR
value: http://vault.vault.svc.cluster.local:8200
- name: VAULT_TOKEN_FILE
value: /var/run/secrets/vault/token
# Authentik
- name: AUTHENTIK_URL
value: http://authentik-server.iam.svc.cluster.local
- name: AUTHENTIK_BOOTSTRAP_TOKEN
valueFrom:
secretKeyRef:
name: authentik-bootstrap
key: token
# Git
- name: GIT_REPO
value: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- name: GIT_AUTHOR_EMAIL
value: [email protected]
- name: GIT_AUTHOR_NAME
value: Secret Rotation Controller
- name: FORGEJO_TOKEN
valueFrom:
secretKeyRef:
name: forgejo-registry-secret
key: REGISTRY_TOKEN
volumeMounts:
# Age key from ExternalSecret (synced from Vault)
- name: age-key
mountPath: /etc/sops/age
readOnly: true
# Vault auth token (projected)
- name: vault-token
mountPath: /var/run/secrets/vault
readOnly: true
# Temp working dir
- name: tmp
mountPath: /tmp
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
volumes:
- name: age-key
secret:
secretName: sops-age-key
defaultMode: 0400
- name: vault-token
projected:
sources:
- serviceAccountToken:
path: token
audience: vault
expirationSeconds: 3600
- name: tmp
emptyDir: {}
@@ -1,15 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: secret-rotation
resources:
- rbac.yaml
- crd.yaml
- external-secret.yaml
- deployment.yaml
commonLabels:
app.kubernetes.io/name: secret-rotation-controller
app.kubernetes.io/component: automation
managed-by: argocd
@@ -1,53 +0,0 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: secret-rotation-controller
namespace: secret-rotation
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: secret-rotation-controller
rules:
# Read SecretRotation CRDs
- apiGroups: ["homelab.riotpiao.com"]
resources: ["secretrotations"]
verbs: ["get", "list", "watch"]
# Update status
- apiGroups: ["homelab.riotpiao.com"]
resources: ["secretrotations/status"]
verbs: ["get", "patch", "update"]
# Read k8s secrets that will be rotated
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"]
# For recording events
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: secret-rotation-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: secret-rotation-controller
subjects:
- kind: ServiceAccount
name: secret-rotation-controller
namespace: secret-rotation
---
apiVersion: v1
kind: Namespace
metadata:
name: secret-rotation
labels:
kubernetes.io/metadata.name: secret-rotation
-18
View File
@@ -1,18 +0,0 @@
apiVersion: kmsvc.io/v1
kind: TemporalWorker
metadata:
name: worker-production
namespace: temporal
spec:
namespace: production
taskQueue: worker-production
concurrency: 10
workflowTypes:
- HelloWorldWorkflow
- GreeterWorkflow
- ProcessOrderWorkflow
activityTypes:
- GreetActivity
- ValidateOrderActivity
- ProcessPaymentActivity
- NotifyCustomerActivity
-25
View File
@@ -1,25 +0,0 @@
# Wave -1 — AppProject definitions (must sync before any Application that references them).
# Syncs k8s/argocd/projects/ which was previously applied by hand.
# Enabled by Stage 1 (A2).
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: argocd-projects
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
annotations:
argocd.argoproj.io/sync-wave: "-1"
spec:
project: homelab
revisionHistoryLimit: 3
syncPolicy:
automated:
prune: true
selfHeal: true
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/argocd/projects
destination:
server: https://kubernetes.default.svc
+1 -1
View File
@@ -17,7 +17,7 @@ spec:
syncOptions:
- CreateNamespace=true
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
# ksops decrypts every *.enc.yaml here at kustomize-build time (repo-server
# runs `kustomize build --enable-alpha-plugins --enable-exec`). Replaces the
+3 -75
View File
@@ -21,7 +21,7 @@ spec:
helm:
valueFiles:
- $values/k8s/bootstrap/cert-manager/cert-manager-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
ref: values
destination:
@@ -93,7 +93,7 @@ spec:
project: homelab
revisionHistoryLimit: 3
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
# A real kustomization.yaml (resources: the 3 issuer/CA files) renders these
# deterministically. The previous directory.include with bare filenames
@@ -127,7 +127,7 @@ spec:
project: homelab
revisionHistoryLimit: 3
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/bootstrap/ingress
destination:
@@ -138,75 +138,3 @@ spec:
selfHeal: true
syncOptions:
- CreateNamespace=true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cluster-maintenance
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/infra/cluster-maintenance
destination:
server: https://kubernetes.default.svc
namespace: kube-system
syncPolicy:
automated:
prune: true
selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: kyverno
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
project: homelab
source:
repoURL: https://kyverno.github.io/kyverno/
chart: kyverno
targetRevision: "1.14.0"
helm:
valueFiles:
- $values/k8s/bootstrap/kyverno/kyverno-values.yaml
sources:
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: kyverno
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: kyverno-policies
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/bootstrap/kyverno
destination:
server: https://kubernetes.default.svc
namespace: kyverno
syncPolicy:
automated:
prune: true
selfHeal: true
-33
View File
@@ -1,33 +0,0 @@
# ArgoCD Image Updater - auto-updates Application images from registry
# Watches forgejo.riotpiao.com for new image tags and updates Applications
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: argocd-image-updater
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
project: homelab
revisionHistoryLimit: 3
sources:
- repoURL: https://argoproj.github.io/argo-helm
chart: argocd-image-updater
targetRevision: "0.11.2"
helm:
valueFiles:
- $values/k8s/infra/argocd-image-updater/values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=false
-55
View File
@@ -1,55 +0,0 @@
# Tekton Pipelines v0.68.0
#
# Install method: vendored release.yaml in k8s/infra/tekton/
# downloaded from https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.68.0/release.yaml
#
# To upgrade:
# 1. Download new release.yaml from https://github.com/tektoncd/pipeline/releases
# 2. Replace k8s/infra/tekton/release.yaml
# 3. Commit and push — ArgoCD syncs automatically
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: tekton-pipelines
namespace: argocd
labels:
app.kubernetes.io/name: tekton-pipelines
app.kubernetes.io/part-of: homelab-infra
wave: "06"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/infra/tekton
destination:
server: https://kubernetes.default.svc
namespace: tekton-pipelines
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
ignoreDifferences:
- group: admissionregistration.k8s.io
kind: ValidatingWebhookConfiguration
jsonPointers:
- /webhooks/0/clientConfig/caBundle
- /webhooks
- group: admissionregistration.k8s.io
kind: MutatingWebhookConfiguration
jsonPointers:
- /webhooks/0/clientConfig/caBundle
- /webhooks
-32
View File
@@ -1,32 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: secret-rotation
namespace: argocd
labels:
app.kubernetes.io/name: secret-rotation
spec:
project: homelab
sources:
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
path: k8s/apps/secret-rotation-controller
targetRevision: main
destination:
server: https://kubernetes.default.svc
namespace: secret-rotation
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- RespectIgnoreDifferences=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
+7 -33
View File
@@ -17,7 +17,7 @@ spec:
helm:
valueFiles:
- $values/k8s/infra/minio/minio-operator-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
ref: values
destination:
@@ -41,7 +41,7 @@ metadata:
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/infra/minio
destination:
@@ -66,7 +66,7 @@ metadata:
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/infra/longhorn
destination:
@@ -102,7 +102,7 @@ spec:
skipCrds: true
valueFiles:
- $values/k8s/infra/monitoring/prometheus-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
ref: values
destination:
@@ -152,7 +152,7 @@ metadata:
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/infra/monitoring/crds
destination:
@@ -183,7 +183,7 @@ metadata:
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/infra/monitoring
destination:
@@ -213,7 +213,7 @@ spec:
helm:
valueFiles:
- $values/k8s/infra/monitoring/blackbox-exporter-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
ref: values
destination:
@@ -223,29 +223,3 @@ spec:
automated:
prune: true
selfHeal: true
---
# Distributed tracing: Tempo + OpenTelemetry Collector.
# Receives traces from instrumented services, stores in local volume (72h retention).
# Grafana datasource auto-configured, service graph + latency dashboards included.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: tracing
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/infra/tracing
destination:
server: https://kubernetes.default.svc
namespace: tracing
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
+3 -3
View File
@@ -19,7 +19,7 @@ spec:
helm:
valueFiles:
- $values/k8s/infra/logging/loki-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
ref: values
destination:
@@ -53,7 +53,7 @@ spec:
helm:
valueFiles:
- $values/k8s/infra/logging/grafana-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
ref: values
destination:
@@ -87,7 +87,7 @@ spec:
helm:
valueFiles:
- $values/k8s/infra/logging/promtail-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
ref: values
destination:
+6 -140
View File
@@ -17,7 +17,7 @@ spec:
helm:
valueFiles:
- $values/k8s/infra/iam/vault-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
ref: values
destination:
@@ -46,7 +46,7 @@ spec:
helm:
valueFiles:
- $values/k8s/infra/iam/authentik-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
ref: values
destination:
@@ -68,7 +68,7 @@ metadata:
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/infra/iam
destination:
@@ -79,90 +79,18 @@ spec:
prune: true
selfHeal: true
---
# Forgejo itself. Was a bootstrap Helm release (phase 3) until it was brought
# under Argo, because values changes there were inert — a proxy-body-size fix
# sat committed while the live Ingress kept nginx's 1m default and rejected
# every OCI push with 413.
#
# Wave 3: after databases (wave 2) — Forgejo needs CNPG and Redis up first.
#
# Retiring the Helm release: Argo adopts the existing objects on first sync.
# Delete the release secrets afterwards so helm stops claiming ownership:
# kubectl -n cicd delete secret -l owner=helm,name=forgejo
#
# automated sync is deliberately absent. This chart owns the Forgejo PVC and
# the git forge itself; the first sync is manual so its diff can be read before
# anything is applied. Turn on automated+selfHeal once that diff is clean.
# Forgejo runner (local chart). Forgejo itself is Phase 0 (bootstrap).
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: forgejo
name: forgejo-runner
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "3"
spec:
project: homelab
sources:
- repoURL: https://dl.gitea.com/charts/
chart: gitea
targetRevision: 12.7.0
helm:
valueFiles:
- $values/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: cicd
# Reloader injects a STAKATER_* env var carrying a hash of the config Secret,
# so the pod rolls when that Secret changes. The chart does not render it, so
# Argo would strip it on every sync — and with selfHeal on, Argo and Reloader
# would fight over the field and Recreate the forge each round.
ignoreDifferences:
- group: apps
kind: Deployment
name: forgejo-gitea
jqPathExpressions:
- '.spec.template.spec.containers[].env[] | select(.name | startswith("STAKATER_"))'
syncPolicy:
syncOptions:
# Adopt the objects the bootstrap Helm release already created rather
# than failing on "already exists".
- ServerSideApply=true
---
# Forgejo runners (local chart, one instance per language), replacing the
# single generic "docker"-labeled runner. Each instance is a full standalone
# Deployment with its own dind sidecar, own PVCs (registration + layer
# cache) and own registered label -- there is no shared generic runner
# anymore, so each instance also builds and pushes images for the repos it
# serves (the chart's ConfigMap/NetworkPolicy fixes for that -- valid_volumes,
# network: host, egress to ingress-nginx -- apply identically to all three).
#
# `values.yaml` is the chart's default and doubles as the golang instance's
# config; node and rust layer a small values-<lang>.yaml override on top for
# just runner.name/runner.labels. All three share one runner-token Secret
# (Forgejo registration tokens are reusable across multiple runners, unlike
# GitHub's one-time tokens) -- if that assumption is ever wrong, registration
# will fail loudly in the register initContainer's logs, not silently.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: forgejo-runner-golang
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "3"
argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-golang
argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
argocd-image-updater.argoproj.io/write-back-method: git
argocd-image-updater.argoproj.io/git-branch: main
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/infra/forgejo-runner
destination:
@@ -172,65 +100,3 @@ spec:
automated:
prune: true
selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: forgejo-runner-node
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "3"
argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-node
argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
argocd-image-updater.argoproj.io/write-back-method: git
argocd-image-updater.argoproj.io/git-branch: main
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/infra/forgejo-runner
helm:
valueFiles:
- values-node.yaml
destination:
server: https://kubernetes.default.svc
namespace: cicd
syncPolicy:
automated:
prune: true
selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: forgejo-runner-rust
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "3"
argocd-image-updater.argoproj.io/image-list: runner=forgejo.riotpiao.com/rock/forgejo-runner-rust
argocd-image-updater.argoproj.io/runner.update-strategy: newest-build
argocd-image-updater.argoproj.io/runner.allow-tags: regexp:^[0-9a-f]{7}$|^latest$|^v[0-9]+$
argocd-image-updater.argoproj.io/runner.helm.image-name: runner.image.repository
argocd-image-updater.argoproj.io/runner.helm.image-tag: runner.image.tag
argocd-image-updater.argoproj.io/write-back-method: git
argocd-image-updater.argoproj.io/git-branch: main
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/infra/forgejo-runner
helm:
valueFiles:
- values-rust.yaml
destination:
server: https://kubernetes.default.svc
namespace: cicd
syncPolicy:
automated:
prune: true
selfHeal: true
+1 -1
View File
@@ -14,7 +14,7 @@ metadata:
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/infra/databases
destination:
-20
View File
@@ -1,20 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: memory-queues
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "7"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/apps/messaging/memory-queues
destination:
server: https://kubernetes.default.svc
namespace: sqs
syncPolicy:
automated:
prune: true
selfHeal: true
+46 -6
View File
@@ -1,6 +1,6 @@
# Wave 5 — Kafka (Strimzi operator + cluster CR), Redis infrastructure.
# Strimzi/Redis are public Helm charts; kafka-cluster is a local chart.
# queue-crd and management-service are managed by kmsvc-root (kmsvc-manage.git).
# Wave 5 — Kafka (Strimzi operator + cluster CR), Redis, and the SQS-like
# queue services. Strimzi/Redis are public Helm charts; kafka-cluster/queue-crd/
# management-service are local charts (rendered from their own Chart.yaml).
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
@@ -68,7 +68,7 @@ metadata:
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/apps/messaging/kafka-cluster
destination:
@@ -78,5 +78,45 @@ spec:
automated:
prune: true
selfHeal: true
# queue-crd and management-service moved to kmsvc-manage.git repo
# Managed by kmsvc-root Application
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: queue-crd
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "6"
spec:
project: homelab
source:
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/apps/messaging/queue-crd
destination:
server: https://kubernetes.default.svc
namespace: sqs
syncPolicy:
automated:
prune: true
selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: management-service
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "7"
spec:
project: homelab
source:
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/apps/messaging/management-service
destination:
server: https://kubernetes.default.svc
namespace: sqs
syncPolicy:
automated:
prune: true
selfHeal: true
+29 -39
View File
@@ -1,54 +1,40 @@
# Wave 7 — api-gw, the cluster's API gateway (namespace `api`).
# Wave 7 — Kong, the cluster's internal API gateway (namespace `api`).
#
# Replaces Kong OSS 3.4.1, removed 2026-08-19. Kong existed to route
# `api.riotpiao.com`, but Kong OSS cannot dispatch on a request body, so the
# LLM surface had to be expressed as one path per model
# (`/v1/reasoning/chat/completions`, `/v1/ornith/...`, `/v1/qwen/...`) with a
# `request-transformer` plugin forcing the body's `model` field on each. The Go
# gateway reads the body and picks the upstream, so a single canonical
# `POST /v1/chat/completions` covers every model. See
# docs/adr/ADR-0001-retire-kong-for-go-gateway.md in the frontend repo.
# Sits between nginx and the backend services: nginx owns the edge and TLS,
# Kong owns routing policy, auth and rate limiting. Wave 7 puts it after the
# data/messaging tiers it fronts and before the wave-8 applications that
# publish routes into it.
#
# UPDATED 2026-08-22: Tracks main branch of homelab-frontend (auto-syncs on each push).
# Image built on every main commit with tag <commit-sha>.
# ArgoCD auto-pulls the latest image (live reconciliation ~3min).
# DB-less: routing config comes from Kubernetes objects (Ingress with
# `ingressClassName: kong`, plus KongPlugin/KongConsumer CRDs), so git remains
# the source of truth and there are no migration Jobs on upgrade.
#
# Two sources:
# 1. rock/homelab-frontend on the in-cluster Forgejo (prod branch) — the gateway's own
# kustomization (Deployment, Service, ConfigMap, RBAC, NetworkPolicy). It
# sets `namespace: api` itself, so no transformer is needed here. The
# Forgejo host must stay listed in the `homelab` AppProject sourceRepos or
# this Application is rejected with "is not permitted in project".
# 2. k8s/apps/api in this repo — the nginx edge Ingress for
# api.riotpiao.com, inherited from the retired `kong` Application. It
# cannot move to k8s/bootstrap/ingress/ingress.yaml because that syncs in
# wave 1, before namespace `api` exists.
#
# No resources-finalizer: deleting this Application leaves the workload running
# rather than cascading the delete.
# CRDs ship in the chart's crds/ directory; ArgoCD applies those by default
# (helm.skipCrds is left false).
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: api-gw
name: kong
namespace: argocd
labels:
app.kubernetes.io/name: api-gateway
app.kubernetes.io/component: gateway
annotations:
argocd.argoproj.io/sync-wave: "7"
# ArgoCD Image Updater - auto-update on new image push
argocd-image-updater.argoproj.io/image-list: gw=forgejo.riotpiao.com/rock/api-gateway
argocd-image-updater.argoproj.io/gw.update-strategy: digest
argocd-image-updater.argoproj.io/gw.allow-tags: regexp:^latest$
argocd-image-updater.argoproj.io/write-back-method: argocd
spec:
project: homelab
revisionHistoryLimit: 3
sources:
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
- repoURL: https://charts.konghq.com
chart: kong
targetRevision: "3.4.1"
helm:
valueFiles:
- $values/k8s/apps/api/kong-values.yaml
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
ref: values
# The nginx Ingress for api.riotpiao.com. Kept in this Application rather
# than the central k8s/bootstrap/ingress/ingress.yaml because that one syncs
# in wave 1, before namespace `api` exists.
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/apps/api
destination:
@@ -60,9 +46,13 @@ spec:
selfHeal: true
syncOptions:
- CreateNamespace=true
# The chart's CRDs exceed the annotation size limit that client-side
# apply relies on; server-side apply avoids the
# "metadata.annotations: Too long" failure CRDs commonly hit.
- ServerSideApply=true
retry:
limit: 5
limit: 3
backoff:
duration: 5s
duration: 10s
factor: 2
maxDuration: 3m
+3 -3
View File
@@ -1,7 +1,7 @@
# Wave 6 — the model servers behind api.riotpiao.com (namespace `llm-serving`).
#
# Syncs before wave 7 (api-gw), so the predictor Services exist before the
# gateway that routes to them. KServe itself is part of the substrate; this Application
# Syncs before wave 7 (Kong), so the predictor Services exist before the routes
# that point at them. KServe itself is part of the substrate; this Application
# owns only the InferenceServices.
#
# Adopted from live state on 2026-08-15. These five had been `kubectl apply`-ed
@@ -20,7 +20,7 @@ spec:
project: homelab
revisionHistoryLimit: 3
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/apps/llm-serving
destination:
-32
View File
@@ -1,32 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: comfyui
namespace: argocd
labels:
app.kubernetes.io/name: comfyui
app.kubernetes.io/component: image-generation
annotations:
argocd.argoproj.io/sync-wave: "8"
spec:
project: homelab
revisionHistoryLimit: 3
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/apps/comfyui
destination:
server: https://kubernetes.default.svc
namespace: comfyui
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
-32
View File
@@ -1,32 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: gotify
namespace: argocd
labels:
app.kubernetes.io/name: gotify
app.kubernetes.io/component: notifications
annotations:
argocd.argoproj.io/sync-wave: "8"
spec:
project: homelab
revisionHistoryLimit: 3
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/apps/gotify
destination:
server: https://kubernetes.default.svc
namespace: notifications
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
+7 -128
View File
@@ -19,12 +19,9 @@ spec:
helm:
valueFiles:
- $values/k8s/apps/temporal/temporal-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
ref: values
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/apps/temporal
destination:
server: https://kubernetes.default.svc
namespace: temporal
@@ -51,7 +48,7 @@ spec:
helm:
valueFiles:
- $values/k8s/apps/portainer/portainer-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
ref: values
destination:
@@ -74,7 +71,7 @@ metadata:
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/apps/cloudflared
destination:
@@ -97,7 +94,7 @@ metadata:
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/apps/agent-pod
destination:
@@ -130,7 +127,7 @@ metadata:
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/apps/sms
destination:
@@ -141,67 +138,6 @@ spec:
prune: true
selfHeal: true
---
# Document management. Raw manifests (no Helm): postgres is the dedicated
# paperless-db CNPG cluster in k8s/infra/databases (wave 2), redis is
# in-cluster only (no PVC), media lives on the cp-3 USB HDD (see
# k8s/infra/longhorn/longhorn-paperless-storageclass.yaml). OIDC via
# Authentik provisioned by k8s/infra/iam's PostSync job; MinIO backup bucket
# creds provisioned by k8s/infra/minio's PostSync job.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: paperless
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "8"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/apps/paperless
destination:
server: https://kubernetes.default.svc
namespace: paperless
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
---
# Photo/video backup. Self-contained (unlike paperless, its CNPG Postgres
# lives here too, not in k8s/infra/databases) - CreateNamespace=true creates
# the namespace before any manifest in this Application applies, including
# the Cluster CR, so no separate wave-2 pre-creation step is needed. Postgres
# is pg18 (not this repo's usual 16.2) because CNPG's official pgvector
# extension image only publishes pg18 builds - see k8s/apps/immich/db.yaml.
# media PVC shares the cp-3 HDD 2TB/2TB with paperless-media. OIDC via
# Authentik provisioned by k8s/infra/iam's PostSync job (immich entry in
# SERVICES + immich_role scope mapping for admin-via-claim).
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: immich
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "8"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/apps/immich
destination:
server: https://kubernetes.default.svc
namespace: immich
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
---
# Consolidated: homarr + homarr-patches → homarr
# Helm chart + values + PostSync hook patch (fix-probes-job.yaml)
apiVersion: argoproj.io/v1alpha1
@@ -220,10 +156,10 @@ spec:
helm:
valueFiles:
- $values/k8s/apps/homarr/homarr-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
ref: values
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/apps/homarr # PostSync hook: fix-probes-job.yaml
destination:
@@ -235,60 +171,3 @@ spec:
selfHeal: true
syncOptions:
- CreateNamespace=true
---
# Portfolio site at riotpiao.com - static Next.js site from rock/riotpiao.com repo.
# Points directly to infra/portfolio/base (bypassing repo's own argocd-apps.yaml
# which has wrong URLs). Image built by Forgejo Actions on rock/portfolio repo.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: portfolio
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "8"
# ArgoCD Image Updater - auto-update on new image push
argocd-image-updater.argoproj.io/image-list: app=forgejo.riotpiao.com/rock/portfolio
argocd-image-updater.argoproj.io/app.update-strategy: digest
argocd-image-updater.argoproj.io/app.allow-tags: regexp:^latest$
argocd-image-updater.argoproj.io/write-back-method: argocd
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/rock/riotpiao.com.git
targetRevision: main
path: infra/portfolio/base
destination:
server: https://kubernetes.default.svc
namespace: portfolio
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
---
# Wave 9 - per-service scoped RBAC (Role/RoleBinding), deliberately last so
# every target namespace above already exists. Inert until kube-apiserver
# gets --oidc-groups-claim=groups wired up (separate, not-yet-applied
# terraform/talosctl change) - these grant nothing until then.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: rbac
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "9"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/infra/rbac
destination:
server: https://kubernetes.default.svc
# No namespace: cluster-scoped resources (ClusterRoleBinding, etc.)
# Namespace is set per-resource in kustomization
syncPolicy:
automated:
prune: true
selfHeal: true
-28
View File
@@ -1,28 +0,0 @@
# kmsvc-manage bootstrap — manages itself and its supporting services
# (Strimzi/Kafka, Redis, queue-operator, message-plane server) from
# the kmsvc-manage repo's own k8s/argocd/ structure on the main branch.
# Image built on every main commit, auto-deployed to sqs namespace.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: kmsvc-root
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "6"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/kmsvc-manage.git
targetRevision: main
path: k8s/argocd/apps
directory:
recurse: false
destination:
server: https://kubernetes.default.svc
namespace: sqs
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
-40
View File
@@ -1,40 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: poimen
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "7"
# Image Updater: auto-update on new image push (SHA tag filter)
argocd-image-updater.argoproj.io/image-list: |
memory=forgejo.riotpiao.com/rock/poimen-memory
workflows=forgejo.riotpiao.com/rock/poimen-workflows
frontend=forgejo.riotpiao.com/rock/poimen-frontend
argocd-image-updater.argoproj.io/memory.update-strategy: digest
argocd-image-updater.argoproj.io/memory.allow-tags: regexp:^latest$
argocd-image-updater.argoproj.io/workflows.update-strategy: digest
argocd-image-updater.argoproj.io/workflows.allow-tags: regexp:^latest$
argocd-image-updater.argoproj.io/frontend.update-strategy: digest
argocd-image-updater.argoproj.io/frontend.allow-tags: regexp:^latest$
argocd-image-updater.argoproj.io/write-back-method: argocd
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-workflows.git
targetRevision: main
path: k8s
destination:
server: https://kubernetes.default.svc
namespace: poimen
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
-15
View File
@@ -12,19 +12,6 @@ spec:
description: Homelab GitOps — single-repo, in-cluster destinations only
sourceRepos:
- https://github.com/Riotpiaole/riotpiao.homelab.com.git
# Poimen services (GitHub)
- https://github.com/Riotpiaole/Poimen-memory.git
- https://github.com/Riotpiaole/Poimen-workflows.git
- https://github.com/Riotpiaole/poimen*.git
# In-cluster Forgejo repos — explicit allowlist (no wildcard)
- https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
- https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
- https://forgejo.riotpiao.com/riotpiao-poimen/kmsvc-manage.git
- https://forgejo.riotpiao.com/riotpiao-poimen/poimen.git
- https://forgejo.riotpiao.com/riotpiao-poimen/poimen-memory.git
- https://forgejo.riotpiao.com/riotpiao-poimen/poimen-workflows.git
- https://forgejo.riotpiao.com/riotpiao-poimen/poimen-frontend.git
- https://forgejo.riotpiao.com/rock/riotpiao.com.git
# Public Helm chart repos referenced by k8s/argocd/apps/* and bootstrap/*
- https://cloudnative-pg.github.io/charts
- https://dl.gitea.com/charts/
@@ -43,8 +30,6 @@ spec:
- https://charts.jetstack.io
- https://kubernetes.github.io/ingress-nginx
- https://stakater.github.io/stakater-charts
# ArgoCD ecosystem charts
- https://argoproj.github.io/argo-helm
destinations:
- server: https://kubernetes.default.svc
namespace: "*"
-10
View File
@@ -1,10 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
name: argocd-projects
# AppProject definitions for ArgoCD. Synced by wave -1 Application
# (k8s/argocd/apps/-1-projects.yaml) so they exist before any Application
# references them. Enabled by Stage 1 (A2).
resources:
- homelab-project.yaml
+1 -2
View File
@@ -12,7 +12,7 @@ metadata:
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
repoURL: https://github.com/Riotpiaole/riotpiao.homelab.com.git
targetRevision: main
path: k8s/argocd/apps
directory:
@@ -26,4 +26,3 @@ spec:
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
+13 -13
View File
@@ -1,23 +1,23 @@
apiVersion: ENC[AES256_GCM,data:C3U=,iv:J6yvL9HYwzrR4AidMrxmTQZAA1AqtAO/nn9AQnS40JY=,tag:WkqaErH6Xfrpd68+4QrfrQ==,type:str]
kind: ENC[AES256_GCM,data:BJsIWY38,iv:/8AGCpKvtaKoi+iuQNkJbKCSo/jSKi6WRs0f1tJj6d0=,tag:6/Rja8mrZu25kSqg0IzjYQ==,type:str]
apiVersion: ENC[AES256_GCM,data:zdc=,iv:VvjvrS5PVNAMIaOE0LaWU+tHcUIYVQDnCANQz6myktY=,tag:xGyWDhRCwwiNny7hPllf5g==,type:str]
kind: ENC[AES256_GCM,data:2bf5Zfy8,iv:5Oz423GzUWmgdaaZHbrtedwRHIAPIuLh4iDMieLL05s=,tag:GNW1ROjlzGvO/4tIOSuH3Q==,type:str]
metadata:
name: ENC[AES256_GCM,data:lTfngGBtsNA+,iv:shwcjVXeWhFRE+IMYlW6ffPyY4JqVzw5yySqcfyU+4I=,tag:qMmCemQcBjLwubaTnnKqTg==,type:str]
namespace: ENC[AES256_GCM,data:/VJI5GUu+jSX,iv:XCgFTLvytlYl8K09JyhGSsmVaTVCyztY5Xvw4TSkfEg=,tag:3eWHcCiedFHLvcKC8WVnUg==,type:str]
type: ENC[AES256_GCM,data:aibi62c0,iv:MJQRYJ27pTgtaRUKoJI2nb1qKZP47c4Ma+PvjIrCiE0=,tag:jB6mYfAPSnWnZUnY+rC+zQ==,type:str]
name: ENC[AES256_GCM,data:7RO0Qxkc+/sA,iv:wKe9A8d7QJSx/6rlEY5H6lU8V24TJqr5IpXQCBc8QgM=,tag:iF6z1MNEXG3pCz2cQ18gLg==,type:str]
namespace: ENC[AES256_GCM,data:RcduxHHLtjH9,iv:opOVx1lL2ltDqQgsleN7NdMAq0TyFr/YQO3FFsHh5AA=,tag:fIwO31apqIatRRzBvamw6g==,type:str]
type: ENC[AES256_GCM,data:C+JjyZt5,iv:xs49Lz6zRzcf3spiPzdUKTm2HZ+VgFahN6wjIe81JI4=,tag:VllJ54MqU+klA8xAIebjrA==,type:str]
stringData:
models.json: ENC[AES256_GCM,data:SZNtmDYpM+ivOATbvUcOGylj5i7RIu6sps3tp63jQcPwrEjM9bNVCcIEdfC8owq3JU2yT3mUMdC5c/ViNuCMhIilLSGf35Da/uTnOTe4e3URNQ2r1LXTBYr5R4ESt6uZrI7llzgc68C7j+k1JEldbPeoyAef0sHHSIGdbQHaVzd/+j6RSnmDcpwNw5fopyeBkZivdnKgY1vW0bz/IEgMUHRpsIeZkENSJvp63pUwsmv7NZ0qaFWk2rgzZiZszIBFYP8K/AcDAaTid8T7F6Ro9A3ClRdHW9K8Zary2TSXMrZJ+Mo9Rcqzcrw5LBiCRCn+lSmILovZfohPHUN6atKbB5YAewo7XBVlJJa5Wl+V7sq3FHZH7+rbLSFQEPovxt8z8SW2k1W/ifz+0HYWLRrtJLADZ3iV6eDKCrkSnDiCvi9pAczwD4jrbQYDdnVb9Djgp+8qhUyjEZT3C7ONs/ZaTAawX434KqsUN4O5LSEKgNlITnvHgmoxMd6eR1Xc48Gb3CBmI9ChZNpXvtgzZX0t9EuKCR0HBoBg9IPz6vADdbEVRGNLVVT1ucDZU7Sp+vMXnaM9ZIw3jDgWnmX+L4sBBpe2H2mC71muYVpF0swHt7+H0o7Nc7vxNsNIqnqWJjTQt/w4joENvbKB4dBs5lNRCeXuWuwyh6/f+MZbz4ZfqtPfvoEsWqiXglGs0eQQttDlXS36R3lsF1brFKbHdO4j7rS7YEeZdKgJIALqp3I+w/v9hAyLh+bu+R6tqXz3QoKWQkDl10Jjip0I/GIkkw9W8ZDBgsy9YcVf9J0ZOa+wm0GZXhBYWyfrE55FiMkCL1IYP2GABb5HdAbMyccIB1dkx9/TAGWeYxsMPmjCvBghf3LawZNa3nDNMnfH7+MxQFfvlHlGJBWbgO7M4V/rOLQu0RhoMIplb1ZyucHSVMWDNt71kteKR7Fme3VdFES+nIoDi8usfUYSQraW99XMB9IsIkPMCz9nFAUoNrXhd/ADkyRmaXe+gb8UohY++7zBy6YllTICHzZFVvucU1YniZLR9i5lVBv16Gcpe0PxnLuCGFwQi+RdgIPglyXTEFh5Woo7ahrr0H/9knrG7p2cqpfJs777ZSWexQL7ncN40kk73fZyAQZFbYw6vhkRGdTHb5Dq2hrLC4FSPev33lWN9v1V4uMo/wa0CGXkxbjTcHeb2NkzfXWFM0+OlqSYTev8fnbZoBGvh/N/UcmdadAG20wPoD4O7kRaZMqZe3vf5s5AsirZtKGMB+FlpBw8f3BQ8GGV9X9bBcFgq42hhcXzGrelqdCwPKFqPpz5ItEasDAdU48dHBFnAn3OWYiKrDn4/uc0XURcMKg5JR0cwJEaAWd6ZPvPy58qYQwWoFA+MVqk1/fql2GjUxW10hTPNMePLq70FYEl5pRqcWAzS2v/9LSM53E/UD/tlyjm61llqPhirDt7QhYmnEawJeQxFzzrgizAdx4jp4SGCpa3envp1oPySJdEyTJkedf/zYxXHLKZ/sc9uuhM/Bl103SeXeuAQrXrFE/kJazdUBgEstOPiDiLSWq9FjMxgMIr9SMcChYCFm7U4irsjWCdboVWKb6iMZH96AFnm2AwP+QvXUSN6dNoRAGY9E7yeiebuAnvCMGQIZ4rxYJ+xsQvRDpIwZZhs1oDAKsvMaozp5o8zZXwU+UCoRfiHVQ+LmAKhDWW0PfLhMVRSsBA9tHDnigwwRNth+A+117IBwnOgpMf/vS2xOTPszAyso24+4np4etTRMWBZLai/sScnkWflGlEwTD2lb38IeCVfTH4Vp1wPfViI3bRGvpj8je6aBg5buRaUO/M2NPXdK9eroM3OQewEtv1QTtuGy7ihsXukeoll8GvOor+1m8qhLeK4/pMH2kEMvE7fFjiG7cVxUYczN1RPl3353BIh65BdSqi8ZbxezHbPuPnoST3teoBdiwVoHcrz80by76n5kIiLKrunuzDVugq3H7NgE8xpnIpttCADiYHL2jtx7Tg7ft3rSMEjep6OSThUWyKyJUUCgyoENmrR7zyWhk50Z1YCkFW/1hOma+chdPY012aGz6nhDBxOOYrI54YlD61jKsheE51IYWKdQXAKfsp0UCGR3Q+t69QuCm+lclZgatuoBWO+tt/WKekO3T4Y1HMx1aoDk/kAKYZwfZXPyCPQ/cIK0GUpS3H8Fa8uDIpaf5KqTj3DTV9qCcrtvrmrf4Qj8YiyN+TBp/wtLiMLtofjI+apTxBqQSGss7oB/JsuH1wacA5EGoKwO8tZAuMWLM6wuPjaSgVRlBezbGsYqKUAiDZkcXLOHco6AbEptMbK91t5Mgw0V8ActHRg+3LbuHo5P/tWwzdF+1Ht7J7EvJIzvz4lfuVGyUA4fh6uVHHeeJG2OrNtjrqr2zq5iSEECJ4tc7hNDnGR7wqHc3TPEi0YIicUNU5701v3wcKgxqiPrrfRS1ONR7SLqiTHijpdFBOZyFrh/bd0216w/6vUk8uZoq/3u0PjUkVG10ofb3farN4GUfOwvQBgSRkRjIe5K4DH1ZKnE0SQtV9UonzWPqU6jWGepPsOlTMiBkdoTeUhaeomAygbJjRuBsKzGGTDDaZwBpVvB0rBVIV5xge1NKVCSTnOyGYi2gYkmEcWgSdgWk+OOI=,iv:9PDXlSUYz+vl2EzVcwMHZTgyamXLNZU7C+XdcAEi9j4=,tag:bRkH77im+qHjDewMW6PsbA==,type:str]
models.json: ENC[AES256_GCM,data:FvlbdMkcngJchi0GEjIEDjxXpQjRwh+3xnlZNpApgd3v1SpMd0qD2d3TbQr39+feSfsvpxhdmmyU/PmRHZYVdr/QpIKlZBWBD6qTCaOSq62NQbR4ck336muIaWD4AmahWCvRouYEJdEsnG7nedjAivw4cls18vMfnL5pNm2t96foScSyHosWoh3RdT6CWLmJE39+kS6/fwZ1D3Z/GEm/E2zHkVtEbEw/fk1PiqsnmZWt8QcSRAeX8aIDkphV7wOKlkri11r1NuFQKMVIau07VPce1YEi5rsJqsircvDxQelQiMGQt2y2M4GGlXx2NeJgKEtP4Hf9mVGhTeN3CYLcXi8Qyg7k1GIawGVW/Em0kWWfy0GMjvOMfYozNmIpi0YQdYBbr3j6pkixtVM2dWezvld5QIYLnHoGMxw4M6L4IqVIonC3j6tk9pV1DaL5IFskEXyb6ScTrmVY4mQ+VEXvfLYA38jRauWxx/4qf4I7B0RzfGzOx67UJh2jKj0bPWNFHKt5Dz4CK+ul39d1KUPHHMoyaH5gCpOAfLSkjMm44X1Cn+O6uY+QFCugh9MI7bJg3EbovTCZxz95GCupWAvDshv59f7gpGL0T8AuRZHlNp6yZMLdviR9yf79d2rW26PIM1QuNOUWELrOcTs5IAmyuFYr0PI5W5XSf4klOdz3UdhPfMDcyRqtSzVDsSo9oC/WeWOo1+yurtdqqW0wLiFpU/kTaZ4JN5Kl080twqFEqgyPnJrbmlxwWlt4XN2SV9D7pcfE61FzUTwoi5PD/8xt94Fp3XerSgJtwhQ6X4Neo47wCFoMR4Y69mlHrFgSJA4kLFezDc2ISnyAix0W5to0cjwJqQ2EgonIhCXD4xWTvYeApHJMOC2o4B74K5efOZ7klb6PwDMvzM3LyeGzjoJh47aJGvhWN/MpQZC2lSHT8PdMxdGDc+OkEVmsvMncddWMGjQhkTo+69sUexARxLY4H22TCt58azqQGcWa5e7s9NUHb6bRVKHzXh4HMiIhQ0jBev++Jz5gIOyYRnw5xoGbo3ROv8ndczVAqXyAla23EM/VBWT0hLMjr6xxonuLQacP9dPJE9QUofjRMeYblkVvQdIw8WuKGbAlbTR/bokQwGdVp2bywSFQZj7BLlPjzCr0/LJmxYMdq6CKB049PdTlS9/UKBHApddMQ/QRXfOYUCX+lm4k4J19jd7dWk34WNUrTEPSR2d1U/2wTnMvzlrnQHx/zk4/lmI1newfjIbybIbXypX8xVEqzFWXpxUe+iFLP19KmmmHG7gGsV32k1VSGvYFI0HOokZga98QsAbJGbi1gQq8gYpMR/mBdhjJVcnAjOYpg+YaR0vCyrhBN55aSCDoUbT1JOqxvFqmlBHDxTWorD/3KGFxEH/wvSsxussFmUCWdEgYmjGO1ktnQBw1YpOB7mhDUlzvYCSxfpexsYufDYqTbrb/R5gmATC0pEN6EgTth5YqFOL7AiIsogy14RGjRym1wvjVl5SUU9bMQlWGJbQuvNRRwL/TzR/QP210jMsnAh9t8sncDnFPIbhFgeBLsU/ctYoBs0nFVWGWzcuoP9HTnvdWTT2xKmWAeFz8I3DZ5sjm2B787Q4bCrAszTvVLeLiRw/pOE32BeNfG4nNiZDvMQhUomAKpaQypWTu2wLsv3ISm00gud5sRKB6ASJANuSu+EhG9k4OYcBEzAT9PKUxclupFbmsFnXx0u9CXDj3tErrPCUs7DuFSr+fz0ehCLMH+GrcqfytjNDVoTSBRZg+lJYKjAqVCGvq8uRf5V/rdulbt3ggx/uhGSiPzYrBQ0g9iMdjQqpNGBwjlxGkhRE9D2n/PoerAAnHTfwc9Hezffncostjp09xw2CB/xOx9HZFGalQ1R4F5+0lm0iR4lMzc+se+o38foNIiMeMIwAVBJu7IO++UMuKCxpxmdAGqEYj/Ddpzfi3ZcRwe/1HhanVvKRMJsPOhnv0SB2xBrdb7QHnEzLbRTriVRBBdOI77HXWgDt9uQru0tEXCEQ0n+f0BWqHVa/RLN4Yj0dQVWLXU75ddyHnRaCiVdLQnObMvG/QXqY1k/bJTzGNgVF8/KGY51NgdfqskOXC9Fv5IGiyWq+/DL7tQcrmThkzUhZlwDBmwUzPFmzuUC7Z5PkXv2EV+eUrYF16HYqp5KgqTkBHr/JXR8ljb/5gMN929Jk2fBVEuzaEG6W9Yek+Qbx1cnXYrMBI9GWJUJsMsxsAj+1bAjtQeQwt0nXgZesTaV90dfQYq+nTGXMNjqaMhsrJhR6lU1+CDpwY+IObHSttS8AKB+Czsl0nUB44o1a/zKRDNTCGGmPFuv1Q5Ub/1myxorLDDF1mYDpre5TedygtZ/1YSz/YOpKwmWdW32FsnY+m3qjEAcfV5f923N/z8ENVxVpeiWL6mqf7h6F3qTiaSsFq2O8m8tVm70X+oRdig3hPnKqlK/KoxVmzZErhEJ+KmP5I2uvIlb4PViE6imu8kw4VqMClb7jJ7Yk+2Zy84C8RGuDCN8GCDG7FvAahA9xVGxB6rmEpzvqFBxos2EeLOlyfewf7rCw1JAL3C9hZrciA9Jqb/t8B7KqCmCOsKK7xVt1yqyG1dsa364JG/1SKSMKebTheuymZ1+VnGjRZ+pgSAkQGjCtp7B3fTlhjt+WwxW4Ml27yfjokQQ==,iv:8Z39eWukGSMePh/3Dj35e6Zahejil+eeMSqwMYf3snI=,tag:FyBg1y0IQEg/m0mgpf1ESg==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4aENRQllpaWh3bnRJYjEw
eTFkM250eXBSVUFrQWJXSzJUTGgrNmJqL1NVCklRT0tRWTlRU0duYWNzVFExQllS
bWo0TndLVWl2VGllb00zR1c5ZERpWmsKLS0tIDh6Uk5hUFV4bmRkK0lHWWN6L2Jk
dlU3cXJlVFZYYi8yMm5kVUJveU91OGcKyin8Tr7OkCocRxf1dzWl/QsC4l2XW4dn
g/it6hJQx1P+23STw9pDZVPqEj4fOdqjnNoRqVCkM8wH4SXJfwrnlA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBZzB5REh6amhCNzRmY0ta
MkN3Y0ZlR29lQ3h2SWo3cW5CUThkL2RnU1NVCmt2ZkhIZTlHN1RQRkFrTjVvbjVw
RGRrTXRoYmdQcnlMSEo3ZWsrZUQ5cHMKLS0tIGU2TGJqZDRxUGJpZzRveEtZankx
MzhrT1R2akxxby9QVzd1RXB0RDY1LzQKOF+/e5z5lPX6Y1sMTAHuDj3YqW1m+sBd
u/0R0YnBonYM3wS5nJE3NZMkImaAdQlUjOzQepfBldG+lz++rlnAww==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
lastmodified: "2026-08-20T07:26:38Z"
mac: ENC[AES256_GCM,data:YIN51aoCGuFrJwxJIGbCf7vY/+S4uHR4HwQ6un084hMKnInh1uK96FxWrQBlcheqBDfoaXHXXmHAd14LVhrVEsj3R1cFPpxiQpqjwt+d+mON4YBeOrC/VcStAU8joKcLsc8H0PF41PGMcBmflQVDX30D/+am60hZ7FMnGavZqgA=,iv:N6tv/3gh9BJvZdWXAQqTwaceR5nLbiA4YwOz01uwbtg=,tag:6VG9ZkEVpE9xLCNy0HLYcw==,type:str]
lastmodified: "2026-08-18T20:06:08Z"
mac: ENC[AES256_GCM,data:IM9HkpdwtQE2wCkjwDWOmHH4uP7TlIsrK4TVytiecvYz4SiLk6IRUSIu7I3a+F+dtltC2WtokoATaB69DTXPoI54amzzptirxiFD5FbaU+u2gLjo7KI7V0smYGuKqMYwnod2L/4GdlvP6xjVxFWuA01rQRaYBkFSumS4NABl/I4=,iv:al1MyBmFwni8gap7PPZxWaCwqFKCicfPq6nVmrSn9Xc=,tag:KJGTDQ8xRGF8qMOjoQ+dng==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
@@ -1,23 +0,0 @@
apiVersion: ENC[AES256_GCM,data:ECE=,iv:bISz4HovH++X7DW1Qj8Cw0L6+EvPB0+68hh7tfyW5C0=,tag:w+SjD7MsfeIuSf62n+Zl7Q==,type:str]
kind: ENC[AES256_GCM,data:AxXaV3Mb,iv:xfJb354Rrjz3zLctW2i6hl40yh9EsfIqKxDvmn8jqnU=,tag:1xQ/RR3KhPLKCTWCYjebJg==,type:str]
metadata:
name: ENC[AES256_GCM,data:gdQ4OoMYKUPEbsUeA5OI4i1xnh8=,iv:IA2kaCyqQmkYpLolvcTF4aleh+yd/ImXJMhRMvpGCgo=,tag:8qyYYw4EhBKKPzEmpepAeQ==,type:str]
namespace: ENC[AES256_GCM,data:/mOSyXWmRg==,iv:cpeHUTlMqlJzJttGtuR3DoiMtvVqFmDS0/5Tl7K7c2M=,tag:q4GxWgWn2wQJxJHqnq4WQA==,type:str]
type: ENC[AES256_GCM,data:92OcrsuV,iv:Z1XBy6iZ6unGrK4/SSdDa58pbPL3022gU/f1AOM4uvc=,tag:cd4wWRq/ohm6BJ/Eo6HOIw==,type:str]
stringData:
REGISTRY_PAT: ENC[AES256_GCM,data:zDKruOXhzsIcFZTTp4r4r9rwHih9uf1cCp06KZ6eIXSIZDvR6aQY4A==,iv:5Vch5Z7xn1KkRxrgOs2p/M58nd6WhtcuUhoZDCi8LSY=,tag:14SAZIWGOxJMqLBtOorjYQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIaG5QVUYyNjhvaENPcUFy
K3RSSVM2N2hTeFhNK0M5YWZmODhzZU1HaHlFCmNGZWdFcFEvanVwcXpGSmVvRHVx
aG45c2lBY1RuSTYwbDZTVk1QeHNWOTQKLS0tIDg4TnVMNjNtaU1VQk5zQjUvU2hM
R245ZVdqc0ZWWVhJb3dOZ3lpU3JjUlUKawSg09ZPq8FKx5tvOVZZ+K4yh7eTQsUp
be8mWUpS0+eEmNqh35BwU3HrETMQFA6a1kjVp30JOMtqa5rbYlzF7w==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
lastmodified: "2026-08-23T23:14:53Z"
mac: ENC[AES256_GCM,data:yzi6woUaUCi6w9pG/eKnU7k/VfZgoXg/tW8p9joG8p7iabLWMdlXlx23m/CItw/NE0zeGWA5iZPPFZXOit2vN36VzK3kQNoFW6QbhvYLZi+78C/RWIdcpZkxB/tPIRG0vq8Q1SA+rwGjG/0xeAFh+R7k+YBTMd2XWBH3P7EI5T4=,iv:kf6MDaAVrtDvPIEjHMMLxXDSDRC3I1GpsPeJFUYppiw=,tag:NukVbGLa9EoMYmRsa4nBtA==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
@@ -1,24 +0,0 @@
apiVersion: ENC[AES256_GCM,data:D9Y=,iv:EH+zD6bogxh/h/Oe+RxDCtfO96tkc56ou14V+68nK7k=,tag:xTeuhSzqZyrS/ltqvtHcgw==,type:str]
data:
.dockerconfigjson: ENC[AES256_GCM,data:P8x3bhPbJTFvFIKE8WQY1P7KqPwrxNTYTNR+4/Z7nZSr2YQab/Db0SRcvFO9lr4ImdO6waZ8EZ6CI1RzIYx1WcjGXs8FzX1jpBXSw9EG69vRTZCqRRJR8c2yg7qmNFfSvcPAUhK9tciYCzzFWRHZEDAmQ52cox/sMdxE/YR61dhYCy1E9kPwTnaDmDj5Hu72mbLJDxIYpN3T2hVWlw02lHYHgFnuKsPb0Tf1lxn174j/gMMxUT6ynVhWSEExzHvTqQ0RZzuP+VXAh3N9HasbtfcAabt3FtEjYNZqYJ+OE8oiX/A6YjgPDqUOR3AJFZE3,iv:6YsyIHQc8xp8T8XUWhN/pBeaYVI/VdIHOe/w/hb5e6U=,tag:nmGFdkv2hLgj8Dp8/0MOkw==,type:str]
kind: ENC[AES256_GCM,data:hOA36Sjr,iv:y0XHfUOUnut8z0yM2g7Beo3qiqxJhLLhffPpNlUhaec=,tag:45s3FjYCr83vYCSXDXu2/g==,type:str]
metadata:
creationTimestamp: null
name: ENC[AES256_GCM,data:yD8Fo/fncAA4qkaY7RjhRg==,iv:qwc4WId/kGwradgxFUwG5B5XIVYRlMY2HxsaGJAzrjw=,tag:O8GwNdDHMJreL609G/gykA==,type:str]
namespace: ENC[AES256_GCM,data:zg2o,iv:KDpM2L71/LDI4JLTQUtbcv5SV0IraAbKpNEBzSFn/rE=,tag:uFm+3EkgZN3NC2hUPXo9Pg==,type:str]
type: ENC[AES256_GCM,data:5/a+6lRp9Ea5rU/+gEMIgoDYs72xRWkfwefMcy+h,iv:jYirTXlb4rFwvb+nLcgB5X4x1Q/S+3LsVL+7ypS7mkQ=,tag:5Wc0DrMZxIpC0e39MzlAlw==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2ekxNWW9VQ09RTUlMUzVy
QlVrRm12Smt2akRaYmkvMHRPRmJvOTdBTzE0CmdhVFNyWDJnWDV3eTFFUm1majk0
L01ZWTNZdmcwc1MwMHgzQmdVZy9KcjgKLS0tIHZLVGg5VmNRQ2ZNY1lIRHlzaUlo
TWVNelZRcHFveGRiNTVvNzJCNWtDSGMKyV3Puscgx3RqK65KSL6SYaTauxsBY3qd
CeFU928hcB86DwAG/Atq2Qtd7S9pzuzOVQmXRZxwpCDTTyRVhU7eVA==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
lastmodified: "2026-08-20T04:43:53Z"
mac: ENC[AES256_GCM,data:Leus5j38xJwJz3Ge9WggBjQSAh2ESlIMUnX9SylE4oIcAt71f8WadtSCOmnqT3ZK+uN8f+Huq6WetGHWUdLfZjZUngHQHWLoRP1xpTVvB5HwJK4F1ASvmL+u1rC88AG3JsZc3Kc2N1G+m6QruQK/9HSmkMi/HGXy658ijzErEzk=,iv:1ga2dKDqTalj9WnjVT6AubXsL7130CuJp3SbBkTb/64=,tag:YsD9metJdYiJmdVXpPuIsQ==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
@@ -1,24 +0,0 @@
apiVersion: ENC[AES256_GCM,data:uS8=,iv:EEoo9U+C244eAJMSTOQVkf5AE6BeHrc5DWPjtWnPRdk=,tag:H+YmBSGvcON3wh0p22LXDQ==,type:str]
kind: ENC[AES256_GCM,data:j1/PFkTS,iv:ja4q8X+nzE/ZczwcY+Qe2DnnsxG64W1fkRPQvOaoqvA=,tag:WiilMGagudEKUlc2JdbGyg==,type:str]
metadata:
name: ENC[AES256_GCM,data:J9iAIHboMyHu6xn/,iv:p3z3uzlkM7VDzOT3WDCelCdZ2t+QqSPmFtqYfvXPQMs=,tag:rKRtpUexVkuZKOJ34a0Xjw==,type:str]
namespace: ENC[AES256_GCM,data:su0FvA==,iv:6SPvwxZ/4aoL0Z07zdPC9r6L7HOHuKv3RodXpVcii04=,tag:njwkj+yvSgN8sliJP8T/Kw==,type:str]
type: ENC[AES256_GCM,data:Qd6WJN1c,iv:M3fc78LvemcEbWzesuYeQ/GZyIOl7Eg/0EmUe3p0qAk=,tag:Tnzwewn0vwdowxy/EyuPdA==,type:str]
stringData:
user: ENC[AES256_GCM,data:/Z7gPrsTUZOLzfoZ8cZGD10wrZE=,iv:0ydBSeq+yHB2b1J4W7FwIv55Eh+N3qfQ6Q7PwoUAvYo=,tag:CO19F8nElaYZyFiFR6N/Tg==,type:str]
password: ENC[AES256_GCM,data:AOl4StpeQIHSLX+oEFnkfg==,iv:mYsZ+5sum6YqyUPndtPCniTraxldKc803ULlKs8Gsaw=,tag:hK3w51ifZ/omAL7XDf8seg==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaMlhKVEM2bFdFOFAwV1lp
WUd6SEN4ZTF1TEpGYXhOYmJLK0tpcWZyc2w4ClVzZmI2Qk1KUnV1OXpyTEV2WWFa
VlJ2eHRSaEhqUnA2dUJVbWJUcEgxcFkKLS0tIG9IaFVnenNpSFRzdStiWjJvakVL
aDk2bTZGR3Zya3ROUS9vd1hEQVRFaG8KbeXA6IebHEaB79N6u795336aHesHOgzO
uZvvBUzSBy3t3jfFk8bJP4aH79I33Ha2eK5rsvdsiv/orwCMXUINKg==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
lastmodified: "2026-09-12T21:00:46Z"
mac: ENC[AES256_GCM,data:DjzPtR+Ueihh166Bvd3jCLtZFQdrvrxOoF87cv6lxKGWPEptT5vbw/EfuIFJBb6lvEJFDWKRC/r5ASdGwComYsPn0DZs4BPCzeKBtLfP9K2OGCXnAC7eGqt4mzMrrW0CQd1QSGcuXw8CY7uJGkMPGPKe3/0mQWiis2OSpHxTM18=,iv:/QqSEFU9RuX9z5Z6aSaD7KlP/cgGTOYvTH+VJOnDTYM=,tag:yFbFXn+iWqQZx7wW4dKppg==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
@@ -1,24 +0,0 @@
apiVersion: ENC[AES256_GCM,data:wR4=,iv:cRBzbvu0eUYCYeKeysua1/P3Meli/rQyj/mIV6VWnPM=,tag:oyTPA/mLYNUX+QDkYLlZyQ==,type:str]
kind: ENC[AES256_GCM,data:bdKQdadW,iv:F3DQiBI9xhSx87jkS1Hyevo48uUCBjsJSjbScIBznKA=,tag:PFakzzBj5S9F65dxu0L2rg==,type:str]
metadata:
name: ENC[AES256_GCM,data:XHFYrKClPo+IwRbM,iv:ZGuL+cN840Y1bOTC62NhDDcoZpTzDWQ4GfqPJX/QWmI=,tag:hlCVjzvfcxXGOASc3vda/Q==,type:str]
namespace: ENC[AES256_GCM,data:0BbhgZBog+1qY/iRJA==,iv:88tiSpEDqIokT5VP/d6bB2+aUyh1kZ7NEHwZWoJW3XU=,tag:CBR01jh2U9h7kNEcK1e1sA==,type:str]
type: ENC[AES256_GCM,data:Mpv1V73w,iv:BW3r6RpLnwe3XDKwrZySyOjrWUnSwIG5JOoPLzP/5gM=,tag:oyJySL0haOei1m4i+1AJ1g==,type:str]
stringData:
username: ENC[AES256_GCM,data:8xmxLGE=,iv:J/vEvXGoD+ka6FDgnSwDz0fs9IxuJIZW9r7Oyu2qxC8=,tag:dN9jd6NSavMN8+uzvemYWg==,type:str]
password: ENC[AES256_GCM,data:vB9PaaUiQZ8FY8pO0cq1fHLi7Gq5t4U=,iv:C0Y1m2SGYP3oTIFoau5JavVmLblj6te/SPMLodIwiZw=,tag:3Ip4YvR4WPzR0bBpTyjytA==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRT3JoTnhVVW5SUUJXbTdz
dUpLcmtBWWhyaGk3Y1hBM1ArcVI5eHREbmp3CiswUGVmd0NhejZwQ2UvNEdxS3ow
L1RweS9Kb2paeStLZ0tLSFdWbVZqQW8KLS0tIHJHT3hiMmxtM0Q1Ym5KOVpLVFpl
enR3NmdNdmdwVitTQVJlRHFWcjR0N2sKQw9ZZs+Ji/Zq/feO3qy4DwaCfWgDOQ/z
FVVhcCXweN58tb+9fzCJ+pNi/hSmvUkCMbb1+60qBvEehNzOoMRJ5g==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
lastmodified: "2026-09-10T13:31:00Z"
mac: ENC[AES256_GCM,data:3HWV/NG0yTbsxY28u41zTRmAPc1kokGb4nCLmAsyq8/uvxcP+evDNyZXYpS93eVdPRfRZ1OqVBzyVGJEnj7JSXQVmlzor9JcWEL2fcpogoLVfJZKTRD6hk6optnBMjj84iQEEOx3A80JrDOdoXsH0pd9bJBABwoUOJwuufbXcIU=,iv:KLZsmAcapQgV08pFhGIvPt5ylsWg3xazAAjPuJYOaXA=,tag:1vKA3fISMIurHzxZ04F3lg==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
@@ -1,28 +0,0 @@
apiVersion: ENC[AES256_GCM,data:FOg=,iv:15mfPeWXV5LQEYahaYvNf1z2bHbxk4j5i8DXT6mdG/0=,tag:1f9/jnnu+wGeeiXGNFgKzA==,type:str]
kind: ENC[AES256_GCM,data:dOFfNQiM,iv:HVLosbRpcCgu4iiYKV8MDLiQ0uhj8DXBfVpRBW2NFNo=,tag:mCbRIzD1OHRhBLc+7xcVug==,type:str]
metadata:
name: ENC[AES256_GCM,data:kHoKhGyiIQvCfng=,iv:dAhxjeLlXK3yueMKfrHxmh9YmNA7AYKFBquLikCNzcE=,tag:3Xoaw7YNBCU/GINlaQOpBw==,type:str]
namespace: ENC[AES256_GCM,data:fTJMZhbqSQWD3/cnWg==,iv:D7zWa91+Fgerfyrywf8VA5YNzGrThhLz7CvYXucfiq0=,tag:RcIegCl2UR1JsbfIQm928w==,type:str]
type: ENC[AES256_GCM,data:Qh51bTsM,iv:2Htv0Xze7Hd1m6zkP6rtFXAlg8qm0AKylSjwJxRjpBk=,tag:NCu5iVUTNVfYgErZSvwHIg==,type:str]
stringData:
host: ENC[AES256_GCM,data:aZXSvQ8B7h78q1kHmh0=,iv:uUyL9X3ehIHqztGAtXtAQWgduvbSsSwZBZlTFMdOlBo=,tag:OM/vdO19VKDSa4W5WQAKNQ==,type:str]
port: ENC[AES256_GCM,data:5ZBB,iv:1/XAfq+PJKdBsufx+EPvvB/cm9nbEwYigc8eACXjR8g=,tag:WlNyz7gTPh0KMe5oKVd0BA==,type:str]
from: ENC[AES256_GCM,data:2/akr8cXgmgQGnqJaFcLJMDcWw==,iv:0rT/6aqyXxn1jIJ5umYCDZR4S8Pbh4vUgaXNrxd3JF8=,tag:k0pQrPOrqWTyE1Xu9oSzVA==,type:str]
user: ENC[AES256_GCM,data:d9BLaFYOYm++HZMzlf1gVauKkUQ=,iv:Wd48T5UPVn6z3bS0t02X9GbrnWal3QoIQv2EgM9z9Wg=,tag:cmg2KnLY4Atwco+j2wVdeg==,type:str]
password: ENC[AES256_GCM,data:pmqEj9623A6bThKrkCCjuA==,iv:M/QwJ0s//UvuOjOljl67EDhvZt/9Wp8JicvCcows51A=,tag:bfRyIdj1cgIZxU3WPbKteg==,type:str]
notify-email: ENC[AES256_GCM,data:+Xo22g8U2wDKsnPnFq/+GKxgYOI=,iv:PE+C1etlp6046ragiv1iMDQbhfo5IULd/5akD0+Sc6A=,tag:cCZmXByA85fuZL9yozVLjw==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpVVdxQTZFTHcwMXFFZEFk
ckNsYk4rdWtkeXFMVnlyVmdaRDBuRWsvZkQ4CitSSTBXOTZhWVpUZFFlR0JITVV4
Uy9lSlNHNjFNaFhHNTN0WnRaRlNNVGcKLS0tIHRlMnZpQVlScGRYYm5nT3Z5TWd6
ak1UZ1RobkpQZHBXR3MyenBDcU53Mk0KvH9O6bgwrjay0+1/A6TGX8GhITiDjWoO
RNX4fDtqNwhzmCfXbVjK30vlBjOFe+Bb7Z2n+hWMmHHDgFdS0xIAzA==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
lastmodified: "2026-09-12T20:57:18Z"
mac: ENC[AES256_GCM,data:V/wgjnpUBvaV39BCTlecCwQy2/1h+0vixEXleJc/I9y+AOvuwVZNH7M86hdjoAdsKiIoNYySj4Flz+MJ9C8jQoAxBTDPbolP9UwDFWQ5KMJTKPPDoJGLNjy7bUq51WoyePUM6WWAYIiWHIB3OvAxHaSzECzL+ZoAhQy92cPKa1o=,iv:vLtIoD/C5yeXPEr+2Lim6XnWzAj42vr9qQgsxKpuhA8=,tag:PBq8qPHrx4RRgAKCWrxG3Q==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
@@ -1,27 +0,0 @@
apiVersion: ENC[AES256_GCM,data:YE0=,iv:s3yNqcBn7/DKUQNgbGGwVmlM1HzxYGLBKyB6Y2ii2WE=,tag:77KxAjOFU3G0fSfrgudKkg==,type:str]
kind: ENC[AES256_GCM,data:PFW4VV9T,iv:n8gzMjE5C2TAfUTpp/8ex64B+hWUGG1K+2oQ4deN03Q=,tag:18Xqb6Di5izMHmzR5EU+QA==,type:str]
metadata:
name: ENC[AES256_GCM,data:Xhg1fQrD4itrbvDW0g==,iv:/THWSXAThb9fj+mm3wcxqzSdzdt7nE06+xOpkCxyImE=,tag:UChokr06VkbDZcFZDcUY2A==,type:str]
namespace: ENC[AES256_GCM,data:r25U5MuuzDd8JJ2YjQ==,iv:uDUcS4ZTpZe8HmZMArzkdu7LV5GUoLSP2wIs5HB1gv0=,tag:fci7j30hoxnVKJwPFNfd1Q==,type:str]
type: ENC[AES256_GCM,data:agEG8Fu8,iv:ckYX0buX8md1CWHXfxbAXQJLzoTxTJI16nlQ9LSzYi0=,tag:4tZWf3REbGOmmBiT14+dew==,type:str]
stringData:
#ENC[AES256_GCM,data:ehjcsmz1R0i/n31f8M0IIUT4KDBwzz6f5ds=,iv:j5swFAKpC67ZvGioiCCC1D651LxUl9gME8GqK5Uc+ys=,tag:BUJ8k9LHs8NAPPZAwPuifA==,type:comment]
#ENC[AES256_GCM,data:ndAB00yun636VHDMonqTghO/jCWodNd/hmdbm1QmCvOAi4FvTLl8bY3wYR+ZtlkSQYvFNqH798MJixv5OZwxBj5H,iv:pS+7B60jaS5jhqDRw2LbBFv7mD1HO6+hjjv+iNpenXc=,tag:NU6+gxCHTGxqeDMfvkwJWw==,type:comment]
#ENC[AES256_GCM,data:T3mhSm/5q7JTSXNLrjhpP5GhqA7nXz8uAhJcEV1RDctAX0Dyfq8Qn4bNFO51ibwTETx4SnunPuFZVs++AvRnsA==,iv:oRGM8N4iEcwBrMzoEXQAeKqCKzUq+jXTMVq9W2l6oh4=,tag:GrrbC/tkWpA5kp5UYCyRBA==,type:comment]
app-token: ENC[AES256_GCM,data:jw+AqbsxyoYRvNrUDrq+GNq/TNfweFCs,iv:67vSSgMZCMV0GG37ZxUxHAWZqOOGMYCmo3J43WgLyNk=,tag:xlrnip4XomXBbMmooW9FKg==,type:str]
client-token: ENC[AES256_GCM,data:gsxeCqKh4e+DFjpn9jM5GfemAdBf8dSv,iv:l2bBMdxQUil8jU9XDjXGn3EtvFVrK5ibXDCeGaPbYTY=,tag:ELf5S6cdNQJ84Es5upu1IQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAza3YrRW1VTVNSOGlMK21a
bDhzQlEyZDNnZTZrSVNnTUEzU1hSdnM1ZFVzCjdBSUlmWG5EcnlzbSs5bXdDaGkx
ME1nMnRqQzF1Vkc3b3FXSUVHT1g0ZDgKLS0tIHpMUytEMjdLQ2E0Unp2di9KS3JQ
eHBraDk1clJyanhLY2dGM0tESmJIUFkKHTl3y9uQiEofOFD8j2vH3YK/CVzlq11w
GfShIji1yCvvowKGzYYhsQK0UM0FzhzBv0GFMYWQCBq8pGdoPVmO5g==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
lastmodified: "2026-09-10T13:31:00Z"
mac: ENC[AES256_GCM,data:JhnO0V6cd2QVY/VhwHAJ5J75GeVlOVHBfVBTZstkj/IvpkAcwS/JE2b6jXiCwEC4n/vdA8DJ074noysUBRxh4Y7O4NKuvWcTniQKgqULNL1Hzgj3NdUkcQlWT+Z0HQ7FlvFH97VVXVfasU+CLHG48ShT2fDSj8JusEllb9CwSvg=,iv:PZo2krxvJc1TLJbvx+S9ClthoBe4w7WigUkq7dg0gNk=,tag:2IF5g/WTRP4XdnCZzDbiIA==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2

Some files were not shown because too many files have changed in this diff Show More