Commit Graph
78 Commits
Author SHA1 Message Date
rock 6743f7c25f Phase 6.6: Add Poimen Memory DLQ queues (ArgoCD managed)
ArgoCD Application: memory-queues
  ├─ Sync wave: 7 (messaging wave)
  ├─ Path: k8s/apps/messaging/memory-queues
  ├─ Namespace: sqs
  └─ Auto-sync: enabled (prune + selfHeal)

Helm Chart: memory-queues
  ├─ Chart.yaml: v0.1.0
  ├─ values.yaml: Queue config
  └─ templates/queues.yaml: Queue CRD resources

Queues Created:

1. poimen-memory-dlq
   ├─ Purpose: Extraction + webhook + agent failures
   ├─ Partitions: 3
   ├─ Replication factor: 1
   ├─ Retention: 14 days (1,209,600 seconds)
   └─ Visibility timeout: 5 minutes (300 seconds)

2. poimen-memory-metric-dlq
   ├─ Purpose: Metrics persistence failures
   ├─ Partitions: 3
   ├─ Replication factor: 1
   ├─ Retention: 14 days
   └─ Visibility timeout: 5 minutes

Resource: Queue CRD (kmsvc.io/v1alpha1)
  └─ Managed by: queue-operator (already running in sqs ns)

Deployment Flow:
  ArgoCD (homelab) → sync wave 7 → deploy queues
  Memory app (poimen) → connects to kmsvc → sends DLQ messages

Files:
  ├─ k8s/apps/messaging/memory-queues/Chart.yaml (new)
  ├─ k8s/apps/messaging/memory-queues/values.yaml (new)
  ├─ k8s/apps/messaging/memory-queues/templates/queues.yaml (new)
  └─ k8s/argocd/apps/50-memory-queues.yaml (new)
2026-09-05 01:10:51 -07:00
rock 266f0637a4 Revert "feat: add memory service queues (processing, indexing, dlq)"
This reverts commit af6fc84a18.
2026-09-05 01:09:30 -07:00
rock af6fc84a18 feat: add memory service queues (processing, indexing, dlq)
- processing-queue: high-throughput, auto-scaling (1-4 shards)
- indexing-queue: FIFO with deduplication (1-2 shards)
- memory-dlq: dead letter queue for redelivery failures
- ArgoCD Application (wave 7) to auto-sync queue lifecycle
2026-09-05 01:05:01 -07:00
rock 449c2a9109 gitops: add secret-rotation controller ArgoCD Application
- Syncs k8s/apps/secret-rotation-controller/ kustomization
- Auto-prune and self-heal enabled
- Creates secret-rotation namespace
- ArgoCD will deploy CRD, RBAC, ExternalSecret, Deployment
2026-09-04 13:51:26 -07:00
rock 87786d8733 messaging: remove queue-crd/management-service (moved to kmsvc-manage)
- Applications now managed by kmsvc-root from kmsvc-manage.git
- Added ServerSideApply to homelab-root for proper annotation sync
- Avoids duplicate Application conflicts with Image Updater
2026-09-03 08:28:38 -07:00
rock 17a98afa3f image-updater: filter to SHA tags only (skip :latest)
allow-tags: regexp:^[0-9a-f]{7}$ ensures newest-build strategy
compares commit SHA tags, not the stale :latest tag
2026-09-03 08:07:19 -07:00
rock c6032cc354 argocd: add Image Updater for auto-deploy on image push
- Install argocd-image-updater via Helm (wave 1)
- Configure Forgejo registry (anonymous pulls)
- Annotate apps for auto-update: api-gw, portfolio, management-service, queue-crd
- Uses newest-build strategy for commit SHA tags
2026-09-02 20:31:09 -07:00
rock 1af9232613 feat: add tempo and otel-collector for distributed tracing 2026-08-31 15:02:15 -07:00
rock be55d68571 feat(argocd): add portfolio app, explicit sourceRepos, coredns rewrite for riotpiao.com
- Add Application for rock/riotpiao.com repo (portfolio site)
- Replace wildcard sourceRepos with explicit repo list
- Add CoreDNS rewrite for root domain riotpiao.com
2026-08-31 14:18:57 -07:00
rock 2038253d48 feat(security): add Kyverno for image scanning and Pod security policies
- Install Kyverno policy engine for admission control
- Add ClusterPolicies:
  * Disallow 'latest' tags (require explicit versions)
  * Restrict to trusted registries (docker.io, ghcr.io, quay.io, etc.)
  * Require non-root containers
  * Drop all Linux capabilities by default
  * Require securityContext on all containers
  * Require read-only root filesystem (audit only)
  * Require resource requests/limits (prevent starvation)
- All policies in audit mode initially (failurePolicy: ignore)
- Ready to graduate to enforce after testing
- Fixes: missing image scanning from security audit
2026-08-31 11:43:04 -07:00
rock 136bfaf0f2 feat(ops): add cluster-wide stale job/pod cleanup CronJob
Daily 04:00 UTC sweeper in kube-system:
- Delete failed Jobs older than 24h (any namespace)
- Delete completed standalone Jobs older than 72h (no CronJob owner)
- Delete orphan Error/Evicted pods older than 1h
- Self-cleans via ttlSecondsAfterFinished
2026-08-30 07:27:50 -07:00
Story Crater Bot 61a1975669 feat: deploy Immich with Authentik OIDC, rock as admin
Self-hosted photo backup (Google Photos replacement) - raw manifests,
no Helm chart, self-contained under k8s/apps/immich including its own
CNPG Postgres. Media PVC shares the cp-3 HDD 2TB/2TB with
paperless-media.

Postgres is pg18, not this repo's usual 16.2: CNPG's official pgvector
extension image (ghcr.io/cloudnative-pg/pgvector) is only published
for pg18, loaded via CNPG's ImageVolume extension mechanism (operator
1.30.0 / k8s 1.36.1 both support it). Immich auto-manages CREATE
EXTENSION itself at startup.

OIDC via a new "immich_role" Authentik scope mapping (homelab-admins/
immich-admins -> "admin" claim, else "user"), consumed by Immich's
OAuth roleClaim setting which re-syncs isAdmin on every login - more
reliable than Immich's racy first-user-is-admin fallback. Config
composed into an immich-oidc Secret and mounted as IMMICH_CONFIG_FILE,
matching the paperless-oidc pattern. k8s RBAC (immich-operator Role +
oidc:immich-admins binding) mirrors paperless/rbac.yaml.

immich namespace pre-created in k8s/infra/databases/namespaces.yaml
(not just immich's own CreateNamespace=true) since the iam PostSync
job's RoleBinding needs it to exist before wave 8.
2026-08-25 18:21:28 -07:00
Story Crater Bot 89f01b6f2e feat: add homelab-wide Authentik RBAC model and k8s OIDC auth wiring
Adds permissions claim + per-service admin groups in Authentik, scoped
Role/RoleBinding per service, public PKCE kubernetes OAuth2 client, and
kube-apiserver OIDC extraArgs. Also fixes paperless OIDC signup permissions
via adapter override and adds CoreDNS rewrite for authentik.riotpiao.com.
2026-08-25 15:03:44 -07:00
Story Crater Bot bea76eeba6 fix: repoint ArgoCD Applications from GitHub to in-cluster Forgejo
homelab-root and every child Application still tracked github.com/Riotpiaole/riotpiao.homelab.com, which had diverged from origin (Forgejo) for a while - pushes to Forgejo were never picked up by ArgoCD. Repointed to forgejo.riotpiao.com/rock/homelab.git, already covered by the AppProject's rock/* wildcard.
2026-08-25 11:20:26 -07:00
Story Crater Bot e8e5acfb13 feat: add paperless-ngx with OIDC, CNPG db, cp-3 HDD media, MinIO backup
Fixes controlplane.tftpl's install.wipe:true (should be false, live CPs already run false) and syncs coredns Corefile back to what's actually deployed (drops an unrolled-out, stale Kong-era rewrite).
2026-08-25 11:11:37 -07:00
Story Crater Bot 9eb5c8ea1a Remove separate memory app, bundle into wave 2 databases 2026-08-22 23:16:40 -07:00
Story Crater Bot aa6b2ae9c3 Add Poimen Memory to ArgoCD wave 2 deployment (namespace: poimen) 2026-08-22 23:09:02 -07:00
Story Crater Bot 06855b3768 fix(argocd): update poimen repoURL after rock/poiman rename to rock/poimen
The poiman repo was renamed to poimen on Forgejo; the stale repoURL made
poimen-root fail with a 301 redirect ComparisonError (ArgoCD's git
client doesn't follow redirects on smart-HTTP fetch), blocking sync for
poimen-root and everything under it.
2026-08-21 21:57:36 -07:00
Story Crater Bot 2f5793899b fix(argocd): poimen-root point to k8s/argocd directory 2026-08-21 21:28:18 -07:00
Story Crater Bot 2a1cb77443 fix(argocd): poimen-root point to k8s/argocd/apps like kmsvc-root 2026-08-21 21:27:38 -07:00
Story Crater Bot 9415a30309 fix(argocd): poimen-root use single source from poiman, remove workflows 2026-08-21 21:25:55 -07:00
Story Crater Bot a63838bae3 feat(argocd): enable poimen-root Application for poiman orchestration 2026-08-21 21:24:12 -07:00
Story Crater Bot 0eb0dd89f6 chore(argocd): track main branch instead of prod for auto-sync on every commit 2026-08-21 20:55:20 -07:00
Story Crater Bot 7a7e0fe813 chore(argocd): update api-gateway to track homelab-frontend prod branch 2026-08-21 20:47:38 -07:00
Story Crater Bot 9e5e51733d chore(argocd): add poimen application placeholder 2026-08-21 20:44:49 -07:00
Story Crater Bot 544fa371f6 chore(argocd): add kmsvc-manage bootstrap application 2026-08-21 20:44:35 -07:00
Story Crater Bot dbc4a55b02 feat(forgejo-runner): split into golang/node/rust runners, retire generic docker runner 2026-08-21 16:49:26 -07:00
Story Crater Bot 130746e6a1 Add Temporal worker for production task queue 2026-08-21 16:44:59 -07:00
Story Crater Bot bedf062906 chore(forgejo-runner): arm for cascading delete ahead of 3-runner migration 2026-08-21 16:35:01 -07:00
Story Crater Bot c938a58544 stage1: A1-A2 AppProject and projects Application
A1: Replace per-repo Forgejo entries with https://forgejo.riotpiao.com/rock/*
    wildcard so onboarding never requires touching AppProject.

A2: Add wave -1 Application for k8s/argocd/projects/ so it syncs before
    any Application references the AppProject.

Also add kustomization.yaml to k8s/argocd/projects/ to make it renderable.

Enabled by Stage 1 (A1, A2).
2026-08-20 21:31:05 -07:00
Story Crater Bot 7e40a4baf6 feat: cut api.riotpiao.com over to the Go gateway and retire Kong
Ingress api/api now backs onto api-gateway:8080; the kong Application, its
Helm values, plugins and llm-routes are removed. Gateway image v0.0.0 is in
the Forgejo registry and the pull secret is in the api namespace.
2026-08-19 22:51:53 -07:00
Story Crater Bot f8136f4e1e fix: ignore Reloader's injected env var on the Forgejo Deployment
Argo would otherwise strip STAKATER_* on each sync and fight Reloader for it,
recreating the forge pod every reconcile.
2026-08-19 22:43:29 -07:00
Story Crater Bot df5e623347 feat: manage Forgejo with Argo instead of the bootstrap Helm release
Values changes were inert as a bootstrap release, so the proxy-body-size fix
never reached the live Ingress. First sync is manual — the chart owns the
Forgejo PVC.
2026-08-19 22:37:56 -07:00
Story Crater Bot afb9b35292 fix(agent-pod): remote tui session for multi-agent 2026-08-18 15:08:04 -07:00
Story Crater Bot 5f16f16f0f fix(argocd): clone the public GitHub seed anonymously over HTTPS and delete the SSH deploy-key Secret — its private half had been committed in plaintext to a public remote, and a public repo needs no credential at all 2026-08-18 15:08:04 -07:00
Story Crater Bot c3ffc611f4 fix(homarr): add AUTH_OIDC_URI + email account linking — homarr hides the Authentik sign-in button unless AUTH_OIDC_URI (authorize endpoint) is set alongside AUTH_OIDC_ISSUER (per authentik/homarr SSO docs); was the missing var 2026-08-18 15:08:04 -07:00
Story Crater Bot fc10a9871a refactor(ingress): drop redundant ArgoCD ingress-nginx app — chart 4.15.1 was double-managed by both the helm-bootstrap release and this ArgoCD app (same chart), fighting over the controller/LB service (ingress-config drift). ingress-nginx is bootstrap-critical (ArgoCD's own reachability path), so helm-bootstrap is the single owner 2026-08-18 15:08:04 -07:00
Story Crater Bot 71fb7e9826 feat(sms): add BlueBubbles iMessage delivery (Docker-OSX macOS VM pinned to worker-2) + ArgoCD app + dedicated longhorn-imessage-local SC — default longhorn SC can't schedule a 3-replica 200Gi volume (only worker-1 has 200Gi free at 100% over-provisioning) and Immediate binding would pin the qcow2 to the wrong node
- namespace: PodSecurity privileged, needed for /dev/kvm + privileged QEMU
- storageclass: 1 replica, strict-local, WaitForFirstConsumer
- deployment: nodeSelector workload=imessage + matching NoSchedule toleration,
  Recreate strategy (two QEMU procs on one qcow2 corrupts it), no readiness
  probe (guest install is interactive and takes many minutes)
- services: ClusterIP only; VNC is an unauthenticated console, reach it with
  port-forward, never an Ingress
- networkpolicy: default-deny, opt-in via sms-client=true on port 1234
2026-08-18 15:08:04 -07:00
Story Crater Bot 61b906cce1 feat(monitoring): enable Alertmanager (null receiver, longhorn PVC, az-a) + fix forgejo-rules ns forgejo->cicd — alerting delivery was disabled; forgejo PrometheusRule targeted a nonexistent namespace 2026-08-18 15:08:04 -07:00
Story Crater Bot 650fcf1b61 fix(coredns): own Corefile+hostname rewrites via Talos inlineManifest (single-source terraform/files/coredns/Corefile), drop ArgoCD coredns-config app — in-cluster *.riotpiao.com now resolves to nginx ingress so MinIO/OIDC discovery works; update cp-2 IP .213->.214 2026-08-18 15:08:04 -07:00
Story Crater Bot 78c9946cf6 feat(reloader): enable autoReloadAll + reloadOnCreate — watch all workloads without per-Deployment annotations (charts like homarr don't expose them); auto-restart pods when ksops secrets are created/rotated 2026-08-18 15:08:04 -07:00
Story Crater Bot 51d938d13a chore(duckdns): remove duckdns updater entirely — superseded by cloudflared tunnel; drop app-def, manifests, kube-system Deployment 2026-08-18 15:08:04 -07:00
Story Crater Bot 13ebfe158a fix(cert-manager): cert-manager-issuers directory.include renders empty — switch to explicit resources list, restore automated sync 2026-08-18 15:08:04 -07:00
Story Crater Bot 8ff3060ff0 refactor(argocd): replace SOPS CMP with ksops kustomize generator, rotate age key — CMP discover glob silently shadowed kustomize rendering of any app whose path held a .enc.yaml (MinIO Tenant/cloudflared/authentik jobs never applied); centralize 8 Secret manifests under k8s/argocd/secrets, defer 4 helm-values fragments 2026-08-18 15:08:04 -07:00
Story Crater Bot b99eeac0e8 fix(argocd): use comma-separated include list, not brace expansion
ArgoCD directory.include uses Go filepath.Match glob syntax, not shell
brace expansion - {a,b,c} silently matched nothing, only the original 2
files stayed tracked.
2026-08-18 15:08:04 -07:00
Story Crater Bot 9257525b38 feat(cert-manager): add self-signed homelab-ca ClusterIssuer + trust bundle, fix grafana-oidc secret
homelab-ca was referenced by 6 manifests (authentik, forgejo-runner,
blackbox-exporter, management-service) as a CA trust ConfigMap but never
existed anywhere - not in git, not live in cluster. Generated a new
10-year self-signed root CA, wired it as a ClusterIssuer (cert-manager
namespace) and distributed the public cert as a ConfigMap to every
consuming namespace (iam, cicd, monitoring, sqs). Private key lives only
in the encrypted Secret. Widened cert-manager-issuers' directory include
glob rather than creating a new Application - destination.namespace is
just a fallback default on a plain directory source, not a transformer,
so it doesn't fight with each ConfigMap's own explicit namespace.

Also adds grafana-oidc secret (GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET),
same pre-existing gap as grafana-admin - was meant to come from a deleted
manual script, value already available in .env.
2026-08-18 15:08:04 -07:00
Story Crater Bot efb9389093 feat(argocd): migrate all applications from Forgejo to GitHub
- Replace all forgejo.riotpiao.com repo URLs with [email protected] SSH URLs
- Enables immediate GitOps sync without waiting for Forgejo mirror setup
- Includes ingress-nginx now fully ArgoCD-managed (wave 0)
- SOPS secrets can now sync and decrypt TLS certificates
2026-08-18 15:08:04 -07:00
Story Crater Bot 820702c748 feat(argocd): migrate ingress-nginx to full GitOps management
- Create ArgoCD Application for ingress-nginx controller (wave 0)
- Source: GitHub repo + Helm chart with local values file
- Adopts existing bootstrap Helm release (no downtime)
- Enables automated sync and self-heal for nginx configuration
2026-08-18 15:08:03 -07:00
Story Crater Bot 5b3307ffee fix(bootstrap): correct cluster config + complete Phase4 ArgoCD bootstrap permanent fixes 2026-08-18 15:08:03 -07:00
Story Crater Bot 41f5b05395 refactor(k8s): consolidate to infra/+apps/ single-source tree, dedicated per-app CNPG (authentik-db/temporal-db), wire monitoring-config, forgejo→cicd ns, drop orphan/stale (data-schemas, ollama, story-crater, sqs/argocd, key-rotation) 2026-08-18 15:08:03 -07:00