Merge pull request 'fix(monitoring): cap Prometheus retention below the PVC size' (#68 ) from fix/prometheus-retention-size into main

fix(monitoring): cap Prometheus retention below the PVC size
retentionSize was 95GB against a 25Gi volume, so the limit never fired. The volume filled, and Prometheus crashlooped with: write /prometheus/queries.active: no space left on device failed to initialize active query tracker 18GB keeps roughly 7GiB of headroom for the WAL and block overhead.
2026-10-25 04:43:21 +00:00 · 2026-10-24 21:43:01 -07:00 · 2026-10-24 20:05:20 -07:00 · 2026-10-25 01:26:38 +00:00 · 2026-10-24 18:26:21 -07:00 · 2026-10-25 01:24:43 +00:00
29 changed files with 587 additions and 528 deletions
@@ -0,0 +1,37 @@
+{
+  "schema_version": 1,
+  "diagram_type": "architecture",
+  "meta": {
+    "title": "Poimen Memory — Schema Relations",
+    "quality_profile": "showcase",
+    "viewBox": [900, 480]
+  },
+  "components": [
+    { "id": "node", "type": "database", "label": "memory_node", "sublabel": "Episodes (raw messages)", "pos": [40, 60], "size": [150, 55], "tag": "BIGSERIAL" },
+    { "id": "entity", "type": "database", "label": "memory_entity", "sublabel": "Extracted entities", "pos": [260, 60], "size": [150, 55], "tag": "UUID" },
+    { "id": "edge", "type": "database", "label": "memory_edge", "sublabel": "Facts / relationships", "pos": [480, 60], "size": [150, 55], "tag": "UUID" },
+    { "id": "community", "type": "database", "label": "memory_community", "sublabel": "Entity clusters", "pos": [700, 60], "size": [150, 55], "tag": "UUID" },
+    { "id": "pgvector", "type": "cloud", "label": "pgvector HNSW", "sublabel": "768-dim cosine search", "pos": [140, 260], "size": [160, 55], "tag": "nomic-embed-v2" },
+    { "id": "tsvector", "type": "cloud", "label": "pg ts_rank", "sublabel": "Full-text search", "pos": [400, 260], "size": [160, 55], "tag": "GIN index" },
+    { "id": "tenant", "type": "security", "label": "tenant_id", "sublabel": "Multi-tenant isolation", "pos": [660, 260], "size": [140, 55], "tag": "W7 / mig 014" }
+  ],
+  "boundaries": [
+    { "kind": "region", "label": "Temporal Knowledge Graph (Zep §2)", "wraps": ["node", "entity", "edge", "community"] },
+    { "kind": "security-group", "label": "Search & Isolation", "wraps": ["pgvector", "tsvector", "tenant"] }
+  ],
+  "connections": [
+    { "id": "node-entity", "from": "node", "to": "entity", "variant": "emphasis" },
+    { "id": "entity-edge", "from": "entity", "to": "edge", "variant": "emphasis" },
+    { "id": "entity-community", "from": "community", "to": "entity", "variant": "dashed" },
+    { "id": "edge-node", "from": "edge", "to": "node", "variant": "dashed" },
+    { "id": "entity-vec", "from": "entity", "to": "pgvector", "variant": "default" },
+    { "id": "entity-fts", "from": "entity", "to": "tsvector", "variant": "dashed" },
+    { "id": "edge-fts", "from": "edge", "to": "tsvector", "variant": "dashed" },
+    { "id": "tenant-all", "from": "tenant", "to": "entity", "variant": "security" }
+  ],
+  "cards": [
+    { "dot": "emerald", "title": "Three-Tier Graph", "items": ["memory_node → episode subgraph", "memory_entity → entity subgraph", "memory_edge → fact relationships", "memory_community → clusters (Phase 4)"] },
+    { "dot": "rose", "title": "Bi-Temporal", "items": ["t_valid / t_invalid: event time", "t_created / t_expired: transaction time", "Contradiction: active → candidate → confirmed"] },
+    { "dot": "cyan", "title": "Search", "items": ["HNSW: name, summary, fact embeddings", "ts_rank: BM25 fallback (always available)", "tenant_id: per-table isolation (mig 014)"] }
+  ]
+}
@@ -0,0 +1,158 @@
+{
+  "schema_version": 1,
+  "diagram_type": "dataflow",
+  "meta": {
+    "title": "Poimen Memory \u2014 Schema Relations",
+    "viewBox": [
+      960,
+      480
+    ],
+    "quality_profile": "showcase"
+  },
+  "stages": [
+    {
+      "label": "Episodes"
+    },
+    {
+      "label": "Entities"
+    },
+    {
+      "label": "Relationships"
+    },
+    {
+      "label": "Indexes"
+    }
+  ],
+  "nodes": [
+    {
+      "id": "node",
+      "type": "database",
+      "label": "memory_node",
+      "sublabel": "Raw messages",
+      "stage": 0,
+      "row": 0,
+      "tag": "BIGSERIAL"
+    },
+    {
+      "id": "entity",
+      "type": "database",
+      "label": "memory_entity",
+      "sublabel": "Named entities",
+      "stage": 1,
+      "row": 0,
+      "tag": "UUID + tenant_id"
+    },
+    {
+      "id": "community",
+      "type": "database",
+      "label": "memory_community",
+      "sublabel": "Clusters (Phase 4)",
+      "stage": 1,
+      "row": 1,
+      "tag": "UUID"
+    },
+    {
+      "id": "edge",
+      "type": "database",
+      "label": "memory_edge",
+      "sublabel": "Facts + bi-temporal",
+      "stage": 2,
+      "row": 0,
+      "tag": "UUID + tenant_id"
+    },
+    {
+      "id": "hnsw",
+      "type": "cloud",
+      "label": "HNSW Index",
+      "sublabel": "768-dim cosine",
+      "stage": 3,
+      "row": 0,
+      "tag": "pgvector"
+    },
+    {
+      "id": "fts",
+      "type": "cloud",
+      "label": "GIN Index",
+      "sublabel": "ts_rank BM25",
+      "stage": 3,
+      "row": 1,
+      "tag": "full-text"
+    }
+  ],
+  "flows": [
+    {
+      "from": "node",
+      "to": "entity",
+      "label": "LLM extraction",
+      "classification": "ingest",
+      "variant": "emphasis",
+      "route": "straight"
+    },
+    {
+      "from": "entity",
+      "to": "edge",
+      "label": "source + target FK",
+      "classification": "foreign key",
+      "variant": "emphasis",
+      "route": "straight"
+    },
+    {
+      "from": "entity",
+      "to": "community",
+      "label": "community_id",
+      "classification": "membership",
+      "variant": "dashed",
+      "route": "straight",
+      "labelDy": 30
+    },
+    {
+      "from": "edge",
+      "to": "hnsw",
+      "classification": "vector",
+      "variant": "default",
+      "route": "straight",
+      "label": "embedding",
+      "labelDy": -30
+    },
+    {
+      "from": "edge",
+      "to": "fts",
+      "label": "fact text",
+      "classification": "text",
+      "variant": "dashed",
+      "route": "vertical-channel",
+      "labelDy": 30
+    }
+  ],
+  "cards": [
+    {
+      "dot": "emerald",
+      "title": "Three-Tier Graph",
+      "items": [
+        "memory_node \u2192 episodes (raw messages)",
+        "memory_entity \u2192 extracted entities",
+        "memory_edge \u2192 facts between entities",
+        "memory_community \u2192 clusters (Phase 4)"
+      ]
+    },
+    {
+      "dot": "rose",
+      "title": "Bi-Temporal + Tenant",
+      "items": [
+        "t_valid/t_invalid: when true in reality",
+        "t_created/t_expired: system transaction time",
+        "tenant_id: row-level multi-tenant isolation",
+        "Contradiction: active \u2192 candidate \u2192 confirmed"
+      ]
+    },
+    {
+      "dot": "cyan",
+      "title": "Search Indexes",
+      "items": [
+        "HNSW: name, summary, fact embeddings",
+        "GIN: ts_rank on name+summary and fact",
+        "RRF fusion: sem_weight + lex_weight"
+      ]
+    }
+  ]
+}
@@ -0,0 +1,63 @@
+{
+  "schema_version": 1,
+  "diagram_type": "sequence",
+  "meta": {
+    "title": "Service Acting on Behalf of User",
+    "viewBox": [1000, 600],
+    "column_fit": "spread",
+    "quality_profile": "showcase"
+  },
+  "participants": [
+    { "id": "user", "type": "external", "label": "User", "sublabel": "browser" },
+    { "id": "app", "type": "frontend", "label": "Web App", "sublabel": "poimen UI" },
+    { "id": "authentik", "type": "security", "label": "Authentik", "sublabel": "OIDC" },
+    { "id": "worker", "type": "backend", "label": "Backend Worker", "sublabel": "temporal/queue" },
+    { "id": "memory", "type": "backend", "label": "Memory API", "sublabel": "poimen-memory" },
+    { "id": "jwks", "type": "security", "label": "JWKS", "sublabel": "RS256" }
+  ],
+  "segments": [
+    { "from": 150, "to": 260, "label": "User Auth" },
+    { "from": 270, "to": 370, "label": "Delegation" },
+    { "from": 380, "to": 510, "label": "Downstream" }
+  ],
+  "messages": [
+    { "id": "login", "from": "user", "to": "app", "y": 165, "label": "login (OIDC)", "variant": "default" },
+    { "id": "oidc-flow", "from": "app", "to": "authentik", "y": 200, "label": "authorization_code flow", "variant": "emphasis" },
+    { "id": "user-jwt", "from": "authentik", "to": "app", "y": 235, "label": "user JWT {groups,roles}", "variant": "return" },
+    { "id": "trigger-job", "from": "user", "to": "app", "y": 285, "label": "trigger async job", "variant": "default" },
+    { "id": "queue-job", "from": "app", "to": "worker", "y": 320, "label": "enqueue job + user JWT", "variant": "emphasis", "note": "JWT passed as job metadata" },
+    { "id": "job-ack", "from": "worker", "to": "app", "y": 355, "label": "job queued", "variant": "return" },
+    { "id": "call-memory", "from": "worker", "to": "memory", "y": 400, "label": "POST /memory/ingest", "variant": "emphasis", "note": "Bearer <user JWT>" },
+    { "id": "fetch-jwks", "from": "memory", "to": "jwks", "y": 435, "label": "GET /jwks", "variant": "dashed" },
+    { "id": "jwks-keys", "from": "jwks", "to": "memory", "y": 470, "label": "RSA pubkey", "variant": "return" },
+    { "id": "ingest-ok", "from": "memory", "to": "worker", "y": 505, "label": "202 Accepted", "variant": "return", "note": "uses USER's groups/roles" }
+  ],
+  "activations": [
+    { "participant": "user", "from": 160, "to": 291, "type": "external" },
+    { "participant": "app", "from": 165, "to": 361, "type": "frontend" },
+    { "participant": "authentik", "from": 195, "to": 241, "type": "security" },
+    { "participant": "worker", "from": 315, "to": 511, "type": "backend" },
+    { "participant": "memory", "from": 395, "to": 511, "type": "backend" },
+    { "participant": "jwks", "from": 430, "to": 476, "type": "security" }
+  ],
+  "cards": [
+    {
+      "dot": "emerald",
+      "title": "Delegation Pattern",
+      "items": [
+        "User JWT travels with the job payload",
+        "Worker uses user's token for downstream",
+        "Memory sees USER as the principal"
+      ]
+    },
+    {
+      "dot": "rose",
+      "title": "Security Model",
+      "items": [
+        "No service-account escalation",
+        "User's permissions bound the action",
+        "Audit trail traces to user.sub"
+      ]
+    }
+  ]
+}
@@ -0,0 +1,42 @@
+{
+  "schema_version": 1,
+  "diagram_type": "architecture",
+  "meta": {
+    "title": "Memory Service Fine-Grained RBAC",
+    "quality_profile": "showcase",
+    "viewBox": [900, 520]
+  },
+  "components": [
+    { "id": "jwt", "type": "security", "label": "JWT Token", "sublabel": "groups[] + roles[]", "pos": [40, 60], "size": [140, 60], "tag": "Authentik" },
+    { "id": "cap-check", "type": "backend", "label": "Capability Check", "sublabel": "has_capability()", "pos": [260, 60], "size": [160, 60], "tag": "Phase 1" },
+    { "id": "scope-checker", "type": "backend", "label": "Scope Checker", "sublabel": "filter_resources()", "pos": [500, 60], "size": [160, 60], "tag": "Phase 2" },
+    { "id": "decision", "type": "backend", "label": "AccessDecision", "sublabel": "Allow / Deny", "pos": [740, 60], "size": [130, 60] },
+    { "id": "project-scope", "type": "database", "label": "ProjectScope", "sublabel": "dev/staging/prod/*", "pos": [40, 200], "size": [140, 55] },
+    { "id": "visibility-scope", "type": "database", "label": "VisibilityScope", "sublabel": "public/private", "pos": [220, 200], "size": [140, 55] },
+    { "id": "owner-scope", "type": "database", "label": "OwnerScope", "sublabel": "self/any/user", "pos": [400, 200], "size": [140, 55] },
+    { "id": "group-scope", "type": "database", "label": "GroupScope", "sublabel": "in_any_group()", "pos": [580, 200], "size": [140, 55] },
+    { "id": "entities", "type": "database", "label": "Entities", "sublabel": "memory_entity", "pos": [40, 340], "size": [120, 50] },
+    { "id": "edges", "type": "database", "label": "Edges", "sublabel": "memory_edge", "pos": [200, 340], "size": [120, 50] },
+    { "id": "projects", "type": "database", "label": "Projects", "sublabel": "project_id", "pos": [360, 340], "size": [120, 50] },
+    { "id": "convos", "type": "database", "label": "Conversations", "sublabel": "owner_id", "pos": [520, 340], "size": [120, 50] }
+  ],
+  "boundaries": [
+    { "kind": "security-group", "label": "Access Control Pipeline", "wraps": ["cap-check", "scope-checker", "decision"] },
+    { "kind": "region", "label": "Scope Checkers", "wraps": ["project-scope", "visibility-scope", "owner-scope", "group-scope"] },
+    { "kind": "region", "label": "Protected Resources", "wraps": ["entities", "edges", "projects", "convos"] }
+  ],
+  "connections": [
+    { "id": "jwt-to-cap", "from": "jwt", "to": "cap-check", "label": "claims", "variant": "emphasis" },
+    { "id": "cap-to-scope", "from": "cap-check", "to": "scope-checker", "variant": "emphasis" },
+    { "id": "scope-to-decision", "from": "scope-checker", "to": "decision", "variant": "emphasis" },
+    { "id": "scope-to-proj", "from": "scope-checker", "to": "project-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" },
+    { "id": "scope-to-vis", "from": "scope-checker", "to": "visibility-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" },
+    { "id": "scope-to-owner", "from": "scope-checker", "to": "owner-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" },
+    { "id": "scope-to-group", "from": "scope-checker", "to": "group-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" }
+  ],
+  "cards": [
+    { "dot": "emerald", "title": "Phase 1: Capability", "items": ["has_capability(memory:read)", "Checks roles[] claim", "Binary allow/deny"] },
+    { "dot": "rose", "title": "Phase 2: Scope", "items": ["Project: dev/staging/prod", "Visibility: public/private", "Owner: self-owned only", "Group: poimen-admins/devs"] },
+    { "dot": "cyan", "title": "JWT Claims", "items": ["groups: user.groups", "roles: attributes.roles", "permissions: group names"] }
+  ]
+}
@@ -0,0 +1,54 @@
+{
+  "schema_version": 1,
+  "diagram_type": "sequence",
+  "meta": {
+    "title": "Service-to-Service (M2M) Authentication",
+    "viewBox": [1080, 520],
+    "column_fit": "spread",
+    "quality_profile": "showcase"
+  },
+  "participants": [
+    { "id": "worker", "type": "backend", "label": "Ingest Worker", "sublabel": "temporal/cron" },
+    { "id": "authentik", "type": "security", "label": "Authentik", "sublabel": "OIDC provider" },
+    { "id": "memory", "type": "backend", "label": "Memory Service", "sublabel": "poimen-memory" },
+    { "id": "jwks", "type": "security", "label": "JWKS", "sublabel": "RS256 keys" }
+  ],
+  "segments": [
+    { "from": 150, "to": 230, "label": "Token Mint" },
+    { "from": 240, "to": 380, "label": "API + RBAC" }
+  ],
+  "messages": [
+    { "id": "token-req", "from": "worker", "to": "authentik", "y": 165, "label": "POST /token grant_type=password", "variant": "emphasis", "note": "client_id + username + app_password + scope" },
+    { "id": "jwt-resp", "from": "authentik", "to": "worker", "y": 205, "label": "JWT {groups,roles,aud,iss}", "variant": "return", "note": "groups=[poimen-admins] roles=[memory:write]" },
+    { "id": "api-call", "from": "worker", "to": "memory", "y": 260, "label": "POST /memory/ingest", "variant": "emphasis", "note": "Authorization: Bearer <JWT>" },
+    { "id": "fetch-jwks", "from": "memory", "to": "jwks", "y": 295, "label": "GET /jwks (cached)", "variant": "dashed" },
+    { "id": "jwks-resp", "from": "jwks", "to": "memory", "y": 330, "label": "RSA pubkey (kid)", "variant": "return" },
+    { "id": "verify-ok", "from": "memory", "to": "worker", "y": 365, "label": "202 Accepted", "variant": "return", "note": "sig ✓ iss ✓ aud ✓ has_capability ✓" }
+  ],
+  "activations": [
+    { "participant": "worker", "from": 160, "to": 371, "type": "backend" },
+    { "participant": "authentik", "from": 165, "to": 211, "type": "security" },
+    { "participant": "memory", "from": 255, "to": 371, "type": "backend" },
+    { "participant": "jwks", "from": 290, "to": 336, "type": "security" }
+  ],
+  "cards": [
+    {
+      "dot": "emerald",
+      "title": "M2M Pattern",
+      "items": [
+        "grant_type=password (Authentik aliases client_credentials)",
+        "App-password token authenticates the service account",
+        "NOT client_secret — that creates nameless SA"
+      ]
+    },
+    {
+      "dot": "rose",
+      "title": "JWT Claims",
+      "items": [
+        "groups: from user.groups (poimen-admins)",
+        "roles: from user.attributes.roles",
+        "iss/aud: per-provider issuer"
+      ]
+    }
+  ]
+}
@@ -0,0 +1,61 @@
+{
+  "schema_version": 1,
+  "diagram_type": "sequence",
+  "meta": {
+    "title": "User Direct Access with JWT",
+    "viewBox": [1000, 640],
+    "column_fit": "spread",
+    "quality_profile": "showcase"
+  },
+  "participants": [
+    { "id": "user", "type": "external", "label": "User", "sublabel": "browser" },
+    { "id": "app", "type": "frontend", "label": "Web App", "sublabel": "poimen UI" },
+    { "id": "authentik", "type": "security", "label": "Authentik", "sublabel": "OIDC provider" },
+    { "id": "memory", "type": "backend", "label": "Memory API", "sublabel": "poimen-memory" },
+    { "id": "jwks", "type": "security", "label": "JWKS", "sublabel": "RS256 keys" }
+  ],
+  "segments": [
+    { "from": 150, "to": 370, "label": "OIDC Login" },
+    { "from": 380, "to": 540, "label": "API + RBAC" }
+  ],
+  "messages": [
+    { "id": "click-login", "from": "user", "to": "app", "y": 165, "label": "click Login", "variant": "default" },
+    { "id": "redirect-auth", "from": "app", "to": "authentik", "y": 200, "label": "redirect /authorize", "variant": "emphasis", "note": "scope=openid groups roles" },
+    { "id": "show-login", "from": "authentik", "to": "user", "y": 235, "label": "login form", "variant": "default" },
+    { "id": "submit-creds", "from": "user", "to": "authentik", "y": 270, "label": "username + password", "variant": "security" },
+    { "id": "auth-code", "from": "authentik", "to": "app", "y": 305, "label": "redirect ?code=xxx", "variant": "return" },
+    { "id": "exchange-token", "from": "app", "to": "authentik", "y": 340, "label": "POST /token code=xxx", "variant": "emphasis" },
+    { "id": "jwt-resp", "from": "authentik", "to": "app", "y": 375, "label": "JWT {groups,roles,sub}", "variant": "return" },
+    { "id": "query-api", "from": "app", "to": "memory", "y": 420, "label": "GET /memory/query", "variant": "emphasis", "note": "Authorization: Bearer <JWT>" },
+    { "id": "fetch-jwks", "from": "memory", "to": "jwks", "y": 455, "label": "GET /jwks (cached)", "variant": "dashed" },
+    { "id": "jwks-keys", "from": "jwks", "to": "memory", "y": 490, "label": "RSA pubkey", "variant": "return" },
+    { "id": "query-resp", "from": "memory", "to": "app", "y": 525, "label": "200 JSON results", "variant": "return", "note": "has_capability(memory:read) ✓" }
+  ],
+  "activations": [
+    { "participant": "user", "from": 160, "to": 276, "type": "external" },
+    { "participant": "app", "from": 165, "to": 531, "type": "frontend" },
+    { "participant": "authentik", "from": 195, "to": 381, "type": "security" },
+    { "participant": "memory", "from": 415, "to": 531, "type": "backend" },
+    { "participant": "jwks", "from": 450, "to": 496, "type": "security" }
+  ],
+  "cards": [
+    {
+      "dot": "emerald",
+      "title": "Authorization Code Flow",
+      "items": [
+        "User authenticates directly with Authentik",
+        "App exchanges code for JWT",
+        "Scopes: openid groups roles permissions"
+      ]
+    },
+    {
+      "dot": "rose",
+      "title": "JWT Contents",
+      "items": [
+        "iss: .../application/o/poimen/",
+        "aud: poimen (provider client_id)",
+        "groups: user's group memberships"
+      ]
+    }
+  ]
+}
@@ -12,6 +12,19 @@ rules:
  - apiGroups: ["kmsvc.io"]
    resources: ["queues"]
    verbs: ["get", "list", "watch", "update", "patch"]
+  # The operator watches TemporalWorker objects and creates Deployments for
+  # them. Without these two rules the controller logs
+  # "cannot list resource temporalworkers/deployments ... at the cluster scope"
+  # and never reconciles.
+  - apiGroups: ["kmsvc.io"]
+    resources: ["temporalworkers"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+  - apiGroups: ["kmsvc.io"]
+    resources: ["temporalworkers/status"]
+    verbs: ["get", "update", "patch"]
+  - apiGroups: ["apps"]
+    resources: ["deployments"]
+    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: ["kmsvc.io"]
    resources: ["queues/status"]
    verbs: ["get", "update", "patch"]
@@ -1,115 +0,0 @@
-# macOS VM (Docker-OSX) hosting the BlueBubbles server.
-#
-# ── Why a VM and not a container ────────────────────────────────────────────
-# Containers share the host kernel. macOS binaries are Mach-O and need XNU plus
-# Cocoa/IOKit, which a Linux kernel cannot provide, so no macOS container exists
-# or can exist. Docker-OSX is QEMU running a macOS guest, packaged in a
-# container — a VM in a box, not a macOS container.
-#
-# ── Why this works on worker-2 ──────────────────────────────────────────────
-# Verified on the existing hardware: amd64, `vmx` (Intel VT-x) present, and
-# /dev/kvm exists on Talos nodes (KVM is compiled into Talos' kernel, not a
-# module). Bare metal, so no nested virtualisation needed.
-#
-# ── Read this before relying on it ──────────────────────────────────────────
-# 1. Setup is INTERACTIVE. First boot runs the macOS installer: connect over
-#    VNC (:5999), erase the disk in Disk Utility, install, create a user, sign
-#    into iMessage, THEN install BlueBubbles inside the guest. This manifest
-#    only provides the machine; it does not provision macOS.
-# 2. iMessage activation on non-Apple hardware is a coin flip. BlueBubbles'
-#    own guidance: "test sending an iMessage to yourself. If it does not
-#    succeed, it's likely best to restart from the beginning."
-# 3. Apple's macOS licence permits virtualisation only on Apple hardware. This
-#    is a Hackintosh. Use a throwaway Apple ID, not a primary one.
-# 4. BlueBubbles labels this path "not for beginners", "no guarantees or
-#    warranty".
-#
-# Private API (reactions, typing indicators, edit/unsend) needs SIP disabled
-# inside the guest and is NOT required for plain send/receive. Skip it.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: macos-bluebubbles
-  labels:
-    app.kubernetes.io/name: macos-bluebubbles
-    app.kubernetes.io/part-of: sms
-spec:
-  replicas: 1
-  # Recreate: the qcow2 disk is RWO and a second pod must never attach it
-  # concurrently — two QEMU processes on one image corrupts it.
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: macos-bluebubbles
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: macos-bluebubbles
-        app.kubernetes.io/part-of: sms
-    spec:
-      # Dedicated node. The taint keeps everything else off worker-2; this
-      # toleration is what lets the VM on. Both halves are required.
-      nodeSelector:
-        workload: imessage
-      tolerations:
-        - key: workload
-          operator: Equal
-          value: imessage
-          effect: NoSchedule
-      containers:
-        - name: macos
-          image: sickcodes/docker-osx:latest@sha256:3a3c82c79bc4e73531f819ccdfa4053b3084efd7c1f645678dbf8b4b3a24369c
-          # QEMU needs /dev/kvm; Talos enforces `baseline` cluster-wide, so this
-          # only schedules because the sms namespace is labelled privileged.
-          securityContext:
-            privileged: true
-          env:
-            # Generates a unique serial / board-serial / UUID / MAC and persists
-            # them to bootdisk.qcow2. This synthetic identity is what iMessage
-            # activates against — it must stay stable across restarts, which is
-            # why the PVC matters.
-            - name: GENERATE_UNIQUE
-              value: "true"
-            # Identity is only plausible if it matches a real product line.
-            - name: DEVICE_MODEL
-              value: "iMacPro1,1"
-            - name: RAM
-              value: "12"
-            - name: CORES
-              value: "6"
-            - name: EXTRA
-              # Expose the BlueBubbles server port from the guest to the pod.
-              # Guest :1234 (BlueBubbles default) -> pod :1234.
-              value: "-device virtio-net-pci,netdev=net0 -netdev user,id=net0,hostfwd=tcp::1234-:1234"
-          ports:
-            - name: vnc
-              containerPort: 5999
-              protocol: TCP
-            - name: bluebubbles
-              containerPort: 1234
-              protocol: TCP
-          resources:
-            requests:
-              cpu: "6"
-              memory: 14Gi
-            limits:
-              cpu: "12"
-              memory: 20Gi
-          volumeMounts:
-            - name: macos-disk
-              mountPath: /home/arch/OSX-KVM/disk
-            - name: kvm
-              mountPath: /dev/kvm
-          # No readiness probe on purpose. The guest takes many minutes to boot,
-          # and until macOS + BlueBubbles are installed BY HAND there is nothing
-          # listening on 1234. A probe here would crash-loop the pod through the
-          # entire interactive install.
-      volumes:
-        - name: macos-disk
-          persistentVolumeClaim:
-            claimName: macos-disk
-        - name: kvm
-          hostPath:
-            path: /dev/kvm
-            type: CharDevice
@@ -1,10 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-namespace: sms
-resources:
-  - namespace.yaml
-  - storageclass.yaml
-  - pvc-macos.yaml
-  - deployment-macos.yaml
-  - service.yaml
-  - networkpolicy.yaml
@@ -1,19 +0,0 @@
-# iMessage delivery for the cluster.
-#
-# BlueBubbles' server is a macOS Electron app paired with an Objective-C helper
-# that hooks Messages.app private APIs — it cannot be containerised on Linux,
-# because containers share the host kernel and macOS needs XNU + Cocoa. The only
-# way to run it on Talos is a full macOS VM under QEMU/KVM (Docker-OSX), which
-# needs a privileged pod with /dev/kvm.
-#
-# Hence privileged PodSecurity: the cluster default from the Talos controlplane
-# is `enforce: baseline`, which forbids privileged containers and host devices.
-# Scope is limited to this namespace.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: sms
-  labels:
-    pod-security.kubernetes.io/enforce: privileged
-    pod-security.kubernetes.io/audit: privileged
-    pod-security.kubernetes.io/warn: privileged
@@ -1,22 +0,0 @@
-# Default-deny. This namespace runs a privileged QEMU VM signed into an Apple
-# ID and exposes an unauthenticated VNC console; nothing should reach it except
-# opted-in clients.
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: sms-default-deny
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/part-of: sms
-  policyTypes:
-    - Ingress
-  ingress:
-    - from:
-        - namespaceSelector: {}
-          podSelector:
-            matchLabels:
-              sms-client: "true"
-      ports:
-        - protocol: TCP
-          port: 1234
@@ -1,25 +0,0 @@
-# Persistent macOS disk image + generated hardware identity (bootdisk.qcow2).
-#
-# This volume is NOT disposable: it holds the VM's serial number, board serial,
-# UUID and MAC, which together form the identity iMessage was activated against.
-# Losing it means re-running activation, which is the least reliable step of the
-# whole setup.
-#
-# Docker-OSX documents 128GB minimum for the guest image; 200Gi leaves room for
-# the installer, the base system, and qcow2 growth.
-#
-# ⚠️ Single replica (see storageclass.yaml — capacity and IO both rule out 3).
-# Losing worker-2's disk therefore means losing the activated identity and
-# redoing iMessage activation. Once the guest is installed and activated, take
-# a Longhorn snapshot/backup of this volume; that is the only redundancy here.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: macos-disk
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: longhorn-imessage-local
-  resources:
-    requests:
-      storage: 200Gi
@@ -1,31 +0,0 @@
-# VNC is how you drive the interactive macOS install. Deliberately ClusterIP —
-# it is an unauthenticated console onto a machine holding a live Apple ID
-# session. Reach it with `kubectl port-forward`, never an Ingress.
-apiVersion: v1
-kind: Service
-metadata:
-  name: macos-vnc
-spec:
-  type: ClusterIP
-  selector:
-    app.kubernetes.io/name: macos-bluebubbles
-  ports:
-    - name: vnc
-      port: 5999
-      targetPort: vnc
---
-# The BlueBubbles REST API, once installed inside the guest. This is the stable
-# name cluster services use, so callers never depend on the pod IP or on whether
-# the backend is this VM or a real Mac mini later.
-apiVersion: v1
-kind: Service
-metadata:
-  name: bluebubbles
-spec:
-  type: ClusterIP
-  selector:
-    app.kubernetes.io/name: macos-bluebubbles
-  ports:
-    - name: http
-      port: 1234
-      targetPort: bluebubbles
@@ -1,32 +0,0 @@
-# Dedicated StorageClass for the macOS VM disk.
-#
-# The default `longhorn` class does not work here, for two independent reasons:
-#
-# 1. Replica count. Default is 3, and Longhorn schedules against
-#    storageMaximum - storageScheduled with over-provisioning at 100%. Free
-#    space is cp-1 146Gi / cp-2 8Gi / cp-3 146Gi / worker-1 292Gi, so a 200Gi
-#    volume has only one node that can hold even a single replica — a 3-replica
-#    volume fails outright with ReplicaSchedulingFailure.
-# 2. Binding mode. `Immediate` provisions the volume the moment the PVC is
-#    created, before any pod is scheduled. Combined with strict-local that
-#    pins the data to an arbitrary node, not the one the VM runs on.
-#
-# So: one replica, kept local to the VM, bound only once the pod has a node.
-apiVersion: storage.k8s.io/v1
-kind: StorageClass
-metadata:
-  name: longhorn-imessage-local
-provisioner: driver.longhorn.io
-allowVolumeExpansion: true
-reclaimPolicy: Delete
-# The pod is pinned to worker-2 by nodeSelector; wait for it to be scheduled so
-# the replica is placed on that node and not somewhere else.
-volumeBindingMode: WaitForFirstConsumer
-parameters:
-  # A qcow2 backing a live VM is latency-sensitive and rewritten constantly.
-  # Serving it over the network from another node's disk would be the single
-  # worst thing for guest responsiveness, so force it local.
-  numberOfReplicas: "1"
-  dataLocality: "strict-local"
-  staleReplicaTimeout: "30"
-  fsType: "ext4"
@@ -110,37 +110,6 @@ spec:
    syncOptions:
      - CreateNamespace=true
 ---
-# iMessage/SMS delivery. Raw manifests: a privileged macOS VM (Docker-OSX)
-# running the BlueBubbles server, plus its dedicated local StorageClass.
-#
-# Pinned to worker-2 via nodeSelector `workload: imessage` + a matching
-# toleration for that node's taint. Until worker-2 is provisioned this app
-# syncs everything except the pod, which stays Pending — that is expected.
-#
-# No CreateNamespace: namespace.yaml carries `pod-security: privileged`, which
-# the VM needs (/dev/kvm, privileged), and an ArgoCD-created namespace would
-# not have those labels.
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: sms
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: "8"
-spec:
-  project: homelab
-  source:
-    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
-    targetRevision: main
-    path: k8s/apps/sms
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: sms
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
---
 # Document management. Raw manifests (no Helm): postgres is the dedicated
 # paperless-db CNPG cluster in k8s/infra/databases (wave 2), redis is
 # in-cluster only (no PVC), media lives on the cp-3 USB HDD (see
@@ -1,36 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: azhi-riotpiao
-  namespace: argocd
-  annotations:
-    argocd.argoproj.io/sync-wave: '8'
-spec:
-  project: homelab
-  source:
-    # Private GitHub. The read-only credential is Secret azhi-infra-github in
-    # namespace argocd, created out-of-band.
-    repoURL: https://github.com/ToyoGlobal/azhi-infra.git
-    # The repository's default branch. It is not `main`: this repo's work
-    # happens on the staging slice branch, and `main` is behind it.
-    targetRevision: feat/azi-riotpiao-cnpg
-    # One directory, because ArgoCD reads a single path per source. That
-    # directory is a root kustomization over foundation, postgres and
-    # azhi-server.
-    path: k8s/overlays/azi-riotpiao
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: azi-riotpiao
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-    - CreateNamespace=true
-    # The two Secrets this namespace needs are credentials, so they are created
-    # by hand and NOT by this Application:
-    #   acr-pull                 — the private Alibaba ACR pull credential
-    #   azhi-riotpiao-app-env    — DB credentials, JWT secrets, AES key
-    # A missing imagePullSecret fails the pod outright and a missing app-env
-    # Secret leaves it in CreateContainerConfigError, which is the intended
-    # loud failure rather than a pod that runs against unknown values.
@@ -0,0 +1,20 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: talos-repair
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "6"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/infra/talos-repair
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: cicd
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
@@ -16,12 +16,6 @@ spec:
    - https://github.com/Riotpiaole/Poimen-memory.git
    - https://github.com/Riotpiaole/Poimen-workflows.git
    - https://github.com/Riotpiaole/poimen*.git
-    # ToyoGlobal (GitHub, private). The azhi-infra Application reads its
-    # manifests from here, so the URL must be listed or ArgoCD rejects the
-    # Application with "repository not permitted". A read-only credential for
-    # this repo lives in Secret azhi-infra-github in namespace argocd; it is
-    # created out-of-band and is not committed, because it is a credential.
-    - https://github.com/ToyoGlobal/azhi-infra.git
    # In-cluster Forgejo repos — explicit allowlist (no wildcard)
    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
@@ -41,6 +35,7 @@ spec:
    - https://grafana.github.io/helm-charts
    - https://helm.releases.hashicorp.com
    - https://charts.goauthentik.io
+    - https://charts.zitadel.com
    - https://strimzi.io/charts/
    - https://charts.bitnami.com/bitnami
    - https://homarr-labs.github.io/charts
@@ -1,23 +0,0 @@
-apiVersion: ENC[AES256_GCM,data:ECE=,iv:bISz4HovH++X7DW1Qj8Cw0L6+EvPB0+68hh7tfyW5C0=,tag:w+SjD7MsfeIuSf62n+Zl7Q==,type:str]
-kind: ENC[AES256_GCM,data:AxXaV3Mb,iv:xfJb354Rrjz3zLctW2i6hl40yh9EsfIqKxDvmn8jqnU=,tag:1xQ/RR3KhPLKCTWCYjebJg==,type:str]
-metadata:
-    name: ENC[AES256_GCM,data:gdQ4OoMYKUPEbsUeA5OI4i1xnh8=,iv:IA2kaCyqQmkYpLolvcTF4aleh+yd/ImXJMhRMvpGCgo=,tag:8qyYYw4EhBKKPzEmpepAeQ==,type:str]
-    namespace: ENC[AES256_GCM,data:/mOSyXWmRg==,iv:cpeHUTlMqlJzJttGtuR3DoiMtvVqFmDS0/5Tl7K7c2M=,tag:q4GxWgWn2wQJxJHqnq4WQA==,type:str]
-type: ENC[AES256_GCM,data:92OcrsuV,iv:Z1XBy6iZ6unGrK4/SSdDa58pbPL3022gU/f1AOM4uvc=,tag:cd4wWRq/ohm6BJ/Eo6HOIw==,type:str]
-stringData:
-    REGISTRY_PAT: ENC[AES256_GCM,data:zDKruOXhzsIcFZTTp4r4r9rwHih9uf1cCp06KZ6eIXSIZDvR6aQY4A==,iv:5Vch5Z7xn1KkRxrgOs2p/M58nd6WhtcuUhoZDCi8LSY=,tag:14SAZIWGOxJMqLBtOorjYQ==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIaG5QVUYyNjhvaENPcUFy
-            K3RSSVM2N2hTeFhNK0M5YWZmODhzZU1HaHlFCmNGZWdFcFEvanVwcXpGSmVvRHVx
-            aG45c2lBY1RuSTYwbDZTVk1QeHNWOTQKLS0tIDg4TnVMNjNtaU1VQk5zQjUvU2hM
-            R245ZVdqc0ZWWVhJb3dOZ3lpU3JjUlUKawSg09ZPq8FKx5tvOVZZ+K4yh7eTQsUp
-            be8mWUpS0+eEmNqh35BwU3HrETMQFA6a1kjVp30JOMtqa5rbYlzF7w==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-08-23T23:14:53Z"
-    mac: ENC[AES256_GCM,data:yzi6woUaUCi6w9pG/eKnU7k/VfZgoXg/tW8p9joG8p7iabLWMdlXlx23m/CItw/NE0zeGWA5iZPPFZXOit2vN36VzK3kQNoFW6QbhvYLZi+78C/RWIdcpZkxB/tPIRG0vq8Q1SA+rwGjG/0xeAFh+R7k+YBTMd2XWBH3P7EI5T4=,iv:kf6MDaAVrtDvPIEjHMMLxXDSDRC3I1GpsPeJFUYppiw=,tag:NukVbGLa9EoMYmRsa4nBtA==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.2
@@ -11,7 +11,6 @@ files:
    - agent-pod-ssh-key.enc.yaml
    - authentik-secrets.enc.yaml
    - cloudflare-secrets.enc.yaml
-    - forgejo-registry-pat.enc.yaml
    - forgejo-registry-pat-api.enc.yaml
    - forgejo-registry-pull.enc.yaml
    - forgejo-runner-token.enc.yaml
@@ -38,4 +37,4 @@ files:
    - forgejo-registry-token.enc.yaml
    - pi-sandbox-secrets.enc.yaml
    - etcd-defrag-creds.enc.yaml
-  - zitadel-masterkey.enc.yaml
+    - zitadel-masterkey.enc.yaml
@@ -0,0 +1,23 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: talos-repair-config
+    namespace: cicd
+type: Opaque
+stringData:
+    config: ENC[AES256_GCM,data:WO+jinh/2Rq5VeGLoDDJOQL4zfYJINSTYZCwGWndbzfIHw4XRwxDB7BixjCdpPuLOkNIBAXHQpKPBhlQpvRTso70L26njgVdVENUGeNhxvb6/cwl3VYRsfQQxKoQW9Qu6DyCD7ItuyazXMdIbENKBoih7GA2IjKdGeEtpB+a6PO+6rpI0AQRIjM9h65yJ6gvWeLZmb6U0qCFW61SelHmVSsfehxH/R9EKQROKwD0Z/TgYnoS28OuDtixAQVfnrI3gtgUAeXgnPQTJf1Vb9iwbRBvaSWkPSIqa7hOqWEIM2qyrvhTsEb8XDRUHTmp4BCxYktIxoiSWZEbeh0y0HKRzekZIj4rHA+c/JYSPqS0dvn1Ga+8PTc1LNdXDWIcKI58YQ2jzA0oE2vm0YzmOjcDTzTGLDBk49gIql+AMfwoxCop7t1PB7BrJw6M4bj9M5aHIq8ZlGXDvtWZcSEBj0ChZ0q20cB4v4bvMkbuJR/PS25uNkeG+UBCPidBQqCUw37aNf8SDwOaAC+6PeHdLmcJ/dkKntGFlU5MQzWQXfqs/fp95ze3mr7f3et7SqVl7fAlYSdEt6dA2xxnFnqgNwAm3UacV/We6SRnjrFlBAHZ3Q5hIuKkxmszsS4Bh/VuNtZI5YJ4ldnoIZC3aWKDt+rVB6P+Qxp1T1XqZj8GY715DZRP1q+Z7iQ9RcXiJBTD5qWVpnkESOvbTEYv/W3oXTufVRrJ8Ps0on/xWRF5NU04+tqvToC8+MIQDz5rb+eXYCEWjZG9ZhInY+M2HRTLg+c49dRlyqVnnqfNWBejSS/hoQbf5HmkUbtzsCftvu2NwS2nOMhm55xcCsAunB+0uY+Bj4jqyqRvWUjA99yHIdQH5JTJXE4FQPxdBTmBFp8xVdYWvlnCojK4qfv5EurLUunBugdSdYidi2TKYEoW2mX19RdVnkQgBjW9caaZUX9BIi+zNhObKenoWxRORCb0dHw72mwXMaSQM4IDygcuwi/79OtkYik3juECVmI58bu/GFRo2NtmgkGdMD29SMV6rerDtYjGjetgFvWkDy6hrB/Vz4dI37eGt/EH/7G0z/IT9Q2sM8nzJDtCYdicepLQ1WaM0P4BEn2rrqhv81VvDH7M9e7wtz4xly5RYQUr5wkzvoSEIglFekPvPmhIxEsNdaBj6MlVmLTbEbv85oBLwm0ZLRJcgUsrBbZghGKQaI5udAk1dgwOng9v0Ai+ilJ2RSt2VI6WG1rwRnDG+Hv+H1cKT/NwA8ocmM/kxQlUE3BxESrxhTnKq6W0rzPInIj+xeS8v1X4sloGJIiweCCI8YrqUnmYKKQ6Rtd5B4jS5fAa8jp274/R4xvLaR2r70wKp890Iwfb948PwqjNpH/XlOUcfIxPu3jHO65OEyX3OG/+qm+g91AmxlrwZRDYeSioHztz55Du+JKX2Twnb5ttEwVm6I7eQWGbzw50Qgg6I0cHM1YRrXv/h35C8pkXxV7vMlyR0g5ELAoRWYkuveniruKNIIPBw6SY81RtjIIoPvKzNAFssNdiNxfYEsHiDlfK9kUvTLs8qREmLt1rzgD/vHtanmKh7WMtHN42tSUp+NhoHPlEyaaJLUMkFS7suh2S9xPmhisw61npROXrc2PBxLR52i8t0cJYFa1XuZ40KMCZ7kw0UmKiUxeFDTlZkaOTocVqw3qLY906argYnVdkpaoJMUXsnE3JmXch9HG1qpP+hY8LJSOl5ejAiQADHYDm8P/+9VVlNL6eo2O6qFUs9lC+I8hVeSOvJd8+aLeWisFeboKjarhw/52eXwZdTEpPHa7yMDsIQgciqN82yagJYrEL6BevvZ/kGHP5TszYYfcKcN28k9y/R2JYND2TvYeIKb2Nmj9rhwuXBOFnlep4Exw1QS0If80gJG1BCub+u6GKk9FaFHUSH/x3MIRHwVzCHRnZxqbwc1uGZ2I3rRxVq5MHo5p7htS/KYi7+cQjSdRvfdlCofo6YKP38qQwh9y9yAUqMEOABtjUaEJSeWlGq2yh0onm+Uqm54fcCSr7S5MvBTZ1oblLqVUrZBDTIkY5cdbuiyyoj7GPLHrxUo25yiGI+FXHKy136E0Rv7p5N8o3U5O8ycPdk5vOVYgvM3pdLS1zHHsVYIYY/Y/svtDUnMhJONl5AwQ2sDZ2GiWx1nWJ4VumLuwb/Yow/BHtnW6SOeWFSwRkgJdNL5mzrNoexytbrDaSWR0IpuLUbvK/Gi4MJaEkm1acQB1CMOjIymVu1pcih0H1Zl4O+fhQM+bVke6NKrBSUKErHWca9iNH16Du7iXBbaK0Dz73nIhALJKCivr1rWviXXxeQuzTFMRXdK/SalJySHM7ov6qOikQTG/Q97zQ+UYiUnLo3UxlWngXrnzGy8fAb58v9bHhEtnvCrAhiJTer0x4GYQJGwYjQDbN9ucpu00PDnQU0GqYJHqCf4Vm9Y3SYTs92T6Lftvu3nw/ZH6PblcznC6sueEPON5QzzQppepezdolRNyx4g8XYsI+XJlCNrExWxDxErLBTOm3f7aB19RYCS1tG0G/aKEJTXh9RE5VyWSyBzPNMuLZF9HIIHp3yGpqGM7xsYKsMtngBiyXC+i/C/geIgUFJym/sqWn4iPDWtd0YeELg46B5krcolTg87KEeiqDAebieqL5ZiftnVi10kV2L2XPs4ofJjpVI1Q53RsgT89zWXYv97DTZeDwSE8FY2vbADnUpGk/UZ2WMQlX8iwFOTo1woE6AHyqrq6kzGamUGsyoAarqmJk77o2ktzO2M4OqsVqJErSWTcCLZOlxsDWUn/OsQX5+9vaAhHwDHMmrqWo6cJuAtvaLYOHVB/BUOiI/MJLSYeDIrKUML3Ew0GbX89jkR45SPjRdsXQ/oVCJDxm7WjF9jd6bly2wPTmWCzoqeTJU48jPk47tDHWA7QjLEah6PajxbAs/xSN4c2uK9ltR3K/urNpOTYwsFMLgRfzontPe8RMB9A6aZcwQnIhLTpgX8fw/j+aqwqJwVcIk57/KvcOksBjN4rwPQ78kCmpLJ/KEjYh1z8EwIdVX3UDvq8c8YSZnzwsTAVfhWxigvLGzQDTXrr8aNMDXwB2iUVwhNdGw/E1UCSvsKbvw+p9YRaCAzPnnKa8AVAJySAc1Ys/DAYskdKYaTPUwdCXyZfLzfOsFvEzBxqrRgxxK77ij5UQXKINjQpAIP6pkDze+JTwWQsPfcMZ5xtb2tpiUF9FvCVqR1dfblppq3/f3ZHSl7f4s/sOM2/ZRkiI4m8yxhWDMtauxyrnkgiLsOBUcK4MclD5E3H6KtaJODFiCyH+NPtAMmaZ1iduRM51qxmrHlcI5B4ErJum/iZg+zIokXqIi/U6B+tbjj0EKugN+5V03JaoTvdaGQKNvJHnyP0odFN05ZgZnYCKGGh41EGyCZ8/lmrNbAfZRPjRtHw5ypBUiW8Yz+SkumgWA637ASf1AklCNs9MBBuOPmGckDWy8zTvUCgWCwYehIjKSQ+1hYoWpraLNOZy7tHlTQZ7G5zIFR3cJljO4bfmpAGDsSeDQtBsAI90gD4P3ipKFXVQTfl7c3CD1kB8TiOpG0xVGlYQ4dP15z1wTUfHRz82eFxfwn2fc/R/AKmINq9P5PEL31jhLM+gTxctb2BiGn8cCRuorLCg9FtkwC4jy2AluVj6uZZOGvDCYyGqFsdObgwb+FdIoCYDdeZXo+1dxlxhL1258L6h5+yJBgSy6QDN26N6MMNnK2iD7QTQwtB9CfaX3Bk6T0nZ0TT6zSxu7ix7UG+E5AKwoMVcD0EuLlmJ0IhLI+4ZZ8z/QUIzeTZ9ve+r8uyHKmWr2oR8ec5m95PGG5JzWmw30sDelFFl6S3KxsHwzlJsXeXbE5I/Z0RWZQBa8xPbmY54qYXPzp3O+9WYys6A697UzL6fcCORYokB/TR5exBS2saW4OhkwX5ss9aUJGZiMFSgfF1paxqXkGIx3IuDH5xP3QxhGxNMB+T+Yn+meApf2A00dEUng1HRtgBU5Mk/TM19g8l3nVtwTrmAZDm50jIn+G51GpeoBeYPONhGfb1Ngao383TPkvShaNujE3zsJLfJCrNda76rIechIhAC25wWauUMGUU8daL8cdtJnAEeidnolR+lVf5WFKUxvO36X84FCSaR4PqzvlNk7hOvd607h7IGOEVx7/XRLpWDDrrT6hpY653lNf3RCsxdQ7kS9e5hi/RpU1mf9mdTOsf+AtHkkog336jKdyl1/plDV/AlRS8/QjlPfeyvGmA6gEC8cZmBkvyZPbs81tYmif8q2Amd9La3wldh3evYOOkC/b1QQ3fLryoeUl5/9oVprW/aEfmcTxG9CytaFfsx6cAejAb7g6fz/iaWE57fXfth7K7ATi827ZwAs0XX9lLkyGn60ePtqg771isK5/v1jCsbDQDwHwNLehH+BX9zZWWjaQxK85/o09Oaiy0oGY1nivv5kHelmGwj8qqd,iv:a4NHhErZolhOuzNCUU51GQiCfCT8GzZ3GQ24s7fkuyU=,tag:w+GrXmaYWADTrBxMko3xtg==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjQmk5aE1GN2dpVlREZU1s
+            N3V5ZUQweWxMaVo3Y1VjdzZTYW96eFBYZGxvCnNJcjVXKzI4VnliTWFXQy9iSy9k
+            eEZmWlVPQzhZK09NZGowZUVXc1BoZTQKLS0tIHIrMEpTM0lMU0w0eG02aUxJY2VZ
+            aUQ0cmkvK0UwdzhqcDJwM1RDcnpEU0UKQRiTa9/yNrqlkIdlNeaIDzIKa/xvx5nW
+            7wTNfjq2M3go0DMnaVjJA3e15JZBOpOs0QP9z9ydKF/FS4PeCIEFyQ==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-07T04:27:33Z"
+    mac: ENC[AES256_GCM,data:CxZdlGJdIFFTwaeeS9LW2XDDgvUmSsBeBKtFYyDM1JliS8BHtgEv48yuCWxP5sR3SaYtcgNSIQ55zUa89jM6dQHqvjK+1jSYu1xzFWPOlxUu8gVSlU0QLS4GU1RChXuPpTUOjxzn4ymZukGgclZk9iuMvI0/U+nGD8nvs8k0zpM=,iv:MGZ7YsOsjelJPjLYWtALOrM6r8Vd1HgbU/PKL4c3r3k=,tag:dIE38TCobESY3j56NVEvIw==,type:str]
+    version: 3.13.2
@@ -5,19 +5,19 @@ metadata:
    namespace: iam
 type: Opaque
 stringData:
-    masterkey: ENC[AES256_GCM,data:dGp0wSCUJORP6W5PHLKYaQEqrZsgiAwU1GqcuvuZV+g=,iv:Wwt6E/XXwsoVEQxe4gG1m5oJLOcNUXngftADQvZQepw=,tag:1Vget5gs+t455U9zLavdwg==,type:str]
+    masterkey: ENC[AES256_GCM,data:Da543Jt48o6p3f6/AbZRsjChz3NexPJoi/mC/LB6jUI=,iv:irS0n76tZ6QzoQMm0dWrvbw+SPfXz4h13jQb/Bjfgwg=,tag:tfbgaAdxmPIjleB1OXLn0Q==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBld0tUWHNWem13cDlxRG14
-            QWRlTjA5SzZnS2pBd2M0WWkzQWxKZW9HZWdJCk00QWJJcWZFM1ZTSUlReUlGZG4z
-            RFBSUWRlVUx2NGN2OFAwazBYRTVVNkEKLS0tIHpSSW9IWHhNa0d3SmEydXU4akVj
-            MGJkN2U4WmtLbVFMZm55U1hEZElZMlUKKV9twnC9GC7MSmNLxEBHbPl4RN5Hsd+z
-            NViCKxUoQAB8apn2XVHZcDnyfKNfr1P3aSa3IIHyVY3prgo1OFDpdw==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOZld2VG9NaFhIVWpBcGpO
+            SmVwNm9FT2wrQmFKb0RGQmY4TCtITjFEZjBjCmNuNXNJYTdsZE44cDNzR2hiNTdy
+            QWlVdzBtdjd0cSsxUjQxMzNyblhWTkkKLS0tIHBLT2x1MDBGdjE1eHFOdHVGa1NO
+            ZGM1K2lDTksraFdPeFdSZ1JyU1VtcjAKgTjjhr7u4YRuQVZQ7kAa1clJ0Rzy10hQ
+            Umcd4/YStyXHZjsJJpFSpOFdP5cZscGo3xBgpXtxAscKTS1Nw4derg==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    encrypted_regex: ^(data|stringData)$
-    lastmodified: "2026-10-24T08:57:27Z"
-    mac: ENC[AES256_GCM,data:hXDAddVmnlf/yinY2c2Q8yfsjqi6n9cRbEbwykqYScd6mDPfdF7BriTiu9K+CV/IvIT4zcMTZbhBeQ/vTCAh0TZDp1lOVglwtNH/imgagyzPmcCvbCIkZKwF5vwWQIGHcDI6pk4/zVQkqt7zKTf8+Qj0pNh5UrAjVeEFE7cuJkQ=,iv:oIQvROFGbiESmMTthHOJiNM+Gy5BCfM5XtDWbszOPmc=,tag:AaYrJrwXuKD7w0p+TTa7FA==,type:str]
+    lastmodified: "2026-10-25T01:24:21Z"
+    mac: ENC[AES256_GCM,data:QFVk7KLsMLCAMH/PHVGCRyfU3k2JiO8AE3jJXmhnlUB4C8MYCTyOWQbnTFXozCMOr0kp+hAt3dYn/e23SWhyvJ7BgVLRb5QF1py704Dr46atVoLUvwHpFI7dyGK/LDTxqiM40RXAnDtWUJlTkVCzKdSFiV8CokIoByKWwxFPayM=,iv:t7HWMcpQl/DJ8Rp51Du7b5gURZO+C+FodsZSJuy5jBc=,tag:qEXSS4S0Y5cvn7T6EjakhA==,type:str]
    version: 3.13.2
@@ -55,7 +55,10 @@ alertmanager:
 prometheus:
  prometheusSpec:
    retention: 30d
-    retentionSize: "95GB"
+    # Must stay under the PVC size below (25Gi). At 95GB the limit never
+    # triggered, the volume filled, and Prometheus crashlooped with
+    # "no space left on device" writing /prometheus/queries.active.
+    retentionSize: "18GB"

    # scrapeTimeout MUST be <= scrapeInterval or the operator refuses to generate
    # the Prometheus config ("scrapeTimeout greater than scrapeInterval") and no
@@ -31,29 +31,3 @@ roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: oidc-bootstrap
---
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: oidc-bootstrap-restart-apiserver-v2
-  namespace: kube-system
-spec:
-  backoffLimit: 4
-  ttlSecondsAfterFinished: 7200
-  template:
-    spec:
-      serviceAccountName: oidc-bootstrap
-      restartPolicy: Never
-      containers:
-        - name: restart
-          image: docker.io/curlimages/curl:8.22.0
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -u
-              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
-              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
-              for pod in kube-apiserver-talos-cp-1 kube-apiserver-talos-cp-2 kube-apiserver-talos-cp-3; do
-                curl -sS --max-time 15 --cacert "$CA" -H "Authorization: Bearer $TOKEN"                   -X DELETE "https://kubernetes.default.svc/api/v1/namespaces/kube-system/pods/$pod?gracePeriodSeconds=0"                   -o /dev/null -w "$pod -> HTTP %{http_code}\n" || echo "$pod -> request failed"
-              done
-              echo "deletes requested; kubelet recreates each static pod in ~20-40s"
@@ -1,141 +0,0 @@
-# One-shot: restart the control-plane API servers so their OIDC authenticator initialises.
-#
-# Two earlier mechanisms failed and are documented so nobody repeats them:
-#   1. deleting the mirror pod does NOT restart a static pod (deletes returned 200, no restart);
-#   2. matching /proc/*/cmdline for 'kube-apiserver' also matched this script's own cmdline,
-#      so every pod SIGTERMed itself (exit 143) before reaching the real process.
-# This version matches /proc/<pid>/comm (the process name) and skips \$\$, so self-kill is
-# impossible. hostPID + privileged are permitted in the 'cicd' namespace. Delays stagger the
-# three control planes by 2 minutes each, so only one API server is down at a time.
---
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: oidc-restart-apiserver-cp1-v2
-  namespace: cicd
-spec:
-  backoffLimit: 0
-  ttlSecondsAfterFinished: 3600
-  template:
-    spec:
-      hostPID: true
-      restartPolicy: Never
-      nodeSelector:
-        kubernetes.io/hostname: talos-cp-1
-      tolerations:
-        - key: node-role.kubernetes.io/control-plane
-          operator: Exists
-          effect: NoSchedule
-        - key: node-role.kubernetes.io/master
-          operator: Exists
-          effect: NoSchedule
-      containers:
-        - name: kill
-          image: docker.io/library/busybox:1.36
-          securityContext:
-            privileged: true
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              echo "sleeping 10s before restarting the API server on talos-cp-1"
-              sleep 10
-              self=$$
-              killed=0
-              for d in /proc/[0-9]*; do
-                pid=${d#/proc/}
-                [ "$pid" = "$self" ] && continue
-                comm=$(cat "$d/comm" 2>/dev/null)
-                [ "$comm" = "kube-apiserver" ] || continue
-                echo "SIGTERM -> pid $pid"
-                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
-              done
-              echo "signalled $killed API server process(es); kubelet recreates the static pod"
-
---
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: oidc-restart-apiserver-cp2-v2
-  namespace: cicd
-spec:
-  backoffLimit: 0
-  ttlSecondsAfterFinished: 3600
-  template:
-    spec:
-      hostPID: true
-      restartPolicy: Never
-      nodeSelector:
-        kubernetes.io/hostname: talos-cp-2
-      tolerations:
-        - key: node-role.kubernetes.io/control-plane
-          operator: Exists
-          effect: NoSchedule
-        - key: node-role.kubernetes.io/master
-          operator: Exists
-          effect: NoSchedule
-      containers:
-        - name: kill
-          image: docker.io/library/busybox:1.36
-          securityContext:
-            privileged: true
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              echo "sleeping 120s before restarting the API server on talos-cp-2"
-              sleep 120
-              self=$$
-              killed=0
-              for d in /proc/[0-9]*; do
-                pid=${d#/proc/}
-                [ "$pid" = "$self" ] && continue
-                comm=$(cat "$d/comm" 2>/dev/null)
-                [ "$comm" = "kube-apiserver" ] || continue
-                echo "SIGTERM -> pid $pid"
-                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
-              done
-              echo "signalled $killed API server process(es); kubelet recreates the static pod"
-
---
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: oidc-restart-apiserver-cp3-v2
-  namespace: cicd
-spec:
-  backoffLimit: 0
-  ttlSecondsAfterFinished: 3600
-  template:
-    spec:
-      hostPID: true
-      restartPolicy: Never
-      nodeSelector:
-        kubernetes.io/hostname: talos-cp-3
-      tolerations:
-        - key: node-role.kubernetes.io/control-plane
-          operator: Exists
-          effect: NoSchedule
-        - key: node-role.kubernetes.io/master
-          operator: Exists
-          effect: NoSchedule
-      containers:
-        - name: kill
-          image: docker.io/library/busybox:1.36
-          securityContext:
-            privileged: true
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              echo "sleeping 240s before restarting the API server on talos-cp-3"
-              sleep 240
-              self=$$
-              killed=0
-              for d in /proc/[0-9]*; do
-                pid=${d#/proc/}
-                [ "$pid" = "$self" ] && continue
-                comm=$(cat "$d/comm" 2>/dev/null)
-                [ "$comm" = "kube-apiserver" ] || continue
-                echo "SIGTERM -> pid $pid"
-                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
-              done
-              echo "signalled $killed API server process(es); kubelet recreates the static pod"
-
@@ -0,0 +1,12 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+# Stable secret name: the job mounts talos-repair-config by name, so the generated
+# Secret must not get a hash suffix (same convention as k8s/argocd/secrets).
+generatorOptions:
+  disableNameSuffixHash: true
+# ksops decrypts talosconfig-secret.enc.yaml into the Secret the job mounts.
+# Kept local to this app on purpose: no edits to the shared secrets generator.
+generators:
+  - secret-generator.yaml
+resources:
+  - restart-kubelet.yaml
@@ -0,0 +1,57 @@
+# One-shot repair: the kubelets on cp-1, cp-2 and worker-1 stopped posting node status at
+# 04:11-04:14 UTC and never reconnected, so those nodes are NotReady/unreachable and every
+# pod on them (Cilium included) shows NotReady. This job runs on the one healthy node (cp-3),
+# talks to the Talos API on the LAN, and uses Talos' endpoint->node proxying to reach the
+# other three: it reports the kubelet service state, then restarts it.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: talos-restart-stalled-kubelets
+  namespace: cicd
+spec:
+  backoffLimit: 1
+  ttlSecondsAfterFinished: 86400
+  template:
+    spec:
+      restartPolicy: Never
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-3
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: talosctl
+          image: docker.io/curlimages/curl:8.22.0
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              set -u
+              echo "fetching talosctl v1.13.3 (matches the node version)"
+              curl -sSL -o /tmp/talosctl \
+                https://github.com/siderolabs/talos/releases/download/v1.13.3/talosctl-linux-amd64 \
+                || { echo "DOWNLOAD FAILED"; exit 1; }
+              chmod +x /tmp/talosctl
+              T="/tmp/talosctl --talosconfig /talos/config -e 192.168.1.162"
+              NODES="192.168.1.166 192.168.1.214 192.168.1.223"
+              echo "=== BEFORE: kubelet service state ==="
+              for n in $NODES; do echo "--- $n"; $T -n $n service kubelet 2>&1 || true; done
+              echo "=== kernel log tail (last 15 lines per node) ==="
+              for n in $NODES; do echo "--- $n"; $T -n $n dmesg --tail 15 2>&1 | tail -15 || true; done
+              echo "=== ACTION: restart kubelet ==="
+              for n in $NODES; do echo "--- restarting $n"; $T -n $n restart kubelet 2>&1 || true; sleep 20; done
+              sleep 60
+              echo "=== AFTER ==="
+              for n in $NODES; do echo "--- $n"; $T -n $n service kubelet 2>&1 || true; done
+          volumeMounts:
+            - name: talos
+              mountPath: /talos
+              readOnly: true
+      volumes:
+        - name: talos
+          secret:
+            secretName: talos-repair-config
+            defaultMode: 0444
+            items:
+              - key: config
+                path: config
@@ -0,0 +1,10 @@
+apiVersion: viaduct.ai/v1
+kind: ksops
+metadata:
+  name: talos-repair-secret-generator
+  annotations:
+    config.kubernetes.io/function: |
+      exec:
+        path: ksops
+files:
+  - talosconfig-secret.enc.yaml
@@ -0,0 +1,23 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: talos-repair-config
+    namespace: cicd
+type: Opaque
+stringData:
+    config: ENC[AES256_GCM,data:WO+jinh/2Rq5VeGLoDDJOQL4zfYJINSTYZCwGWndbzfIHw4XRwxDB7BixjCdpPuLOkNIBAXHQpKPBhlQpvRTso70L26njgVdVENUGeNhxvb6/cwl3VYRsfQQxKoQW9Qu6DyCD7ItuyazXMdIbENKBoih7GA2IjKdGeEtpB+a6PO+6rpI0AQRIjM9h65yJ6gvWeLZmb6U0qCFW61SelHmVSsfehxH/R9EKQROKwD0Z/TgYnoS28OuDtixAQVfnrI3gtgUAeXgnPQTJf1Vb9iwbRBvaSWkPSIqa7hOqWEIM2qyrvhTsEb8XDRUHTmp4BCxYktIxoiSWZEbeh0y0HKRzekZIj4rHA+c/JYSPqS0dvn1Ga+8PTc1LNdXDWIcKI58YQ2jzA0oE2vm0YzmOjcDTzTGLDBk49gIql+AMfwoxCop7t1PB7BrJw6M4bj9M5aHIq8ZlGXDvtWZcSEBj0ChZ0q20cB4v4bvMkbuJR/PS25uNkeG+UBCPidBQqCUw37aNf8SDwOaAC+6PeHdLmcJ/dkKntGFlU5MQzWQXfqs/fp95ze3mr7f3et7SqVl7fAlYSdEt6dA2xxnFnqgNwAm3UacV/We6SRnjrFlBAHZ3Q5hIuKkxmszsS4Bh/VuNtZI5YJ4ldnoIZC3aWKDt+rVB6P+Qxp1T1XqZj8GY715DZRP1q+Z7iQ9RcXiJBTD5qWVpnkESOvbTEYv/W3oXTufVRrJ8Ps0on/xWRF5NU04+tqvToC8+MIQDz5rb+eXYCEWjZG9ZhInY+M2HRTLg+c49dRlyqVnnqfNWBejSS/hoQbf5HmkUbtzsCftvu2NwS2nOMhm55xcCsAunB+0uY+Bj4jqyqRvWUjA99yHIdQH5JTJXE4FQPxdBTmBFp8xVdYWvlnCojK4qfv5EurLUunBugdSdYidi2TKYEoW2mX19RdVnkQgBjW9caaZUX9BIi+zNhObKenoWxRORCb0dHw72mwXMaSQM4IDygcuwi/79OtkYik3juECVmI58bu/GFRo2NtmgkGdMD29SMV6rerDtYjGjetgFvWkDy6hrB/Vz4dI37eGt/EH/7G0z/IT9Q2sM8nzJDtCYdicepLQ1WaM0P4BEn2rrqhv81VvDH7M9e7wtz4xly5RYQUr5wkzvoSEIglFekPvPmhIxEsNdaBj6MlVmLTbEbv85oBLwm0ZLRJcgUsrBbZghGKQaI5udAk1dgwOng9v0Ai+ilJ2RSt2VI6WG1rwRnDG+Hv+H1cKT/NwA8ocmM/kxQlUE3BxESrxhTnKq6W0rzPInIj+xeS8v1X4sloGJIiweCCI8YrqUnmYKKQ6Rtd5B4jS5fAa8jp274/R4xvLaR2r70wKp890Iwfb948PwqjNpH/XlOUcfIxPu3jHO65OEyX3OG/+qm+g91AmxlrwZRDYeSioHztz55Du+JKX2Twnb5ttEwVm6I7eQWGbzw50Qgg6I0cHM1YRrXv/h35C8pkXxV7vMlyR0g5ELAoRWYkuveniruKNIIPBw6SY81RtjIIoPvKzNAFssNdiNxfYEsHiDlfK9kUvTLs8qREmLt1rzgD/vHtanmKh7WMtHN42tSUp+NhoHPlEyaaJLUMkFS7suh2S9xPmhisw61npROXrc2PBxLR52i8t0cJYFa1XuZ40KMCZ7kw0UmKiUxeFDTlZkaOTocVqw3qLY906argYnVdkpaoJMUXsnE3JmXch9HG1qpP+hY8LJSOl5ejAiQADHYDm8P/+9VVlNL6eo2O6qFUs9lC+I8hVeSOvJd8+aLeWisFeboKjarhw/52eXwZdTEpPHa7yMDsIQgciqN82yagJYrEL6BevvZ/kGHP5TszYYfcKcN28k9y/R2JYND2TvYeIKb2Nmj9rhwuXBOFnlep4Exw1QS0If80gJG1BCub+u6GKk9FaFHUSH/x3MIRHwVzCHRnZxqbwc1uGZ2I3rRxVq5MHo5p7htS/KYi7+cQjSdRvfdlCofo6YKP38qQwh9y9yAUqMEOABtjUaEJSeWlGq2yh0onm+Uqm54fcCSr7S5MvBTZ1oblLqVUrZBDTIkY5cdbuiyyoj7GPLHrxUo25yiGI+FXHKy136E0Rv7p5N8o3U5O8ycPdk5vOVYgvM3pdLS1zHHsVYIYY/Y/svtDUnMhJONl5AwQ2sDZ2GiWx1nWJ4VumLuwb/Yow/BHtnW6SOeWFSwRkgJdNL5mzrNoexytbrDaSWR0IpuLUbvK/Gi4MJaEkm1acQB1CMOjIymVu1pcih0H1Zl4O+fhQM+bVke6NKrBSUKErHWca9iNH16Du7iXBbaK0Dz73nIhALJKCivr1rWviXXxeQuzTFMRXdK/SalJySHM7ov6qOikQTG/Q97zQ+UYiUnLo3UxlWngXrnzGy8fAb58v9bHhEtnvCrAhiJTer0x4GYQJGwYjQDbN9ucpu00PDnQU0GqYJHqCf4Vm9Y3SYTs92T6Lftvu3nw/ZH6PblcznC6sueEPON5QzzQppepezdolRNyx4g8XYsI+XJlCNrExWxDxErLBTOm3f7aB19RYCS1tG0G/aKEJTXh9RE5VyWSyBzPNMuLZF9HIIHp3yGpqGM7xsYKsMtngBiyXC+i/C/geIgUFJym/sqWn4iPDWtd0YeELg46B5krcolTg87KEeiqDAebieqL5ZiftnVi10kV2L2XPs4ofJjpVI1Q53RsgT89zWXYv97DTZeDwSE8FY2vbADnUpGk/UZ2WMQlX8iwFOTo1woE6AHyqrq6kzGamUGsyoAarqmJk77o2ktzO2M4OqsVqJErSWTcCLZOlxsDWUn/OsQX5+9vaAhHwDHMmrqWo6cJuAtvaLYOHVB/BUOiI/MJLSYeDIrKUML3Ew0GbX89jkR45SPjRdsXQ/oVCJDxm7WjF9jd6bly2wPTmWCzoqeTJU48jPk47tDHWA7QjLEah6PajxbAs/xSN4c2uK9ltR3K/urNpOTYwsFMLgRfzontPe8RMB9A6aZcwQnIhLTpgX8fw/j+aqwqJwVcIk57/KvcOksBjN4rwPQ78kCmpLJ/KEjYh1z8EwIdVX3UDvq8c8YSZnzwsTAVfhWxigvLGzQDTXrr8aNMDXwB2iUVwhNdGw/E1UCSvsKbvw+p9YRaCAzPnnKa8AVAJySAc1Ys/DAYskdKYaTPUwdCXyZfLzfOsFvEzBxqrRgxxK77ij5UQXKINjQpAIP6pkDze+JTwWQsPfcMZ5xtb2tpiUF9FvCVqR1dfblppq3/f3ZHSl7f4s/sOM2/ZRkiI4m8yxhWDMtauxyrnkgiLsOBUcK4MclD5E3H6KtaJODFiCyH+NPtAMmaZ1iduRM51qxmrHlcI5B4ErJum/iZg+zIokXqIi/U6B+tbjj0EKugN+5V03JaoTvdaGQKNvJHnyP0odFN05ZgZnYCKGGh41EGyCZ8/lmrNbAfZRPjRtHw5ypBUiW8Yz+SkumgWA637ASf1AklCNs9MBBuOPmGckDWy8zTvUCgWCwYehIjKSQ+1hYoWpraLNOZy7tHlTQZ7G5zIFR3cJljO4bfmpAGDsSeDQtBsAI90gD4P3ipKFXVQTfl7c3CD1kB8TiOpG0xVGlYQ4dP15z1wTUfHRz82eFxfwn2fc/R/AKmINq9P5PEL31jhLM+gTxctb2BiGn8cCRuorLCg9FtkwC4jy2AluVj6uZZOGvDCYyGqFsdObgwb+FdIoCYDdeZXo+1dxlxhL1258L6h5+yJBgSy6QDN26N6MMNnK2iD7QTQwtB9CfaX3Bk6T0nZ0TT6zSxu7ix7UG+E5AKwoMVcD0EuLlmJ0IhLI+4ZZ8z/QUIzeTZ9ve+r8uyHKmWr2oR8ec5m95PGG5JzWmw30sDelFFl6S3KxsHwzlJsXeXbE5I/Z0RWZQBa8xPbmY54qYXPzp3O+9WYys6A697UzL6fcCORYokB/TR5exBS2saW4OhkwX5ss9aUJGZiMFSgfF1paxqXkGIx3IuDH5xP3QxhGxNMB+T+Yn+meApf2A00dEUng1HRtgBU5Mk/TM19g8l3nVtwTrmAZDm50jIn+G51GpeoBeYPONhGfb1Ngao383TPkvShaNujE3zsJLfJCrNda76rIechIhAC25wWauUMGUU8daL8cdtJnAEeidnolR+lVf5WFKUxvO36X84FCSaR4PqzvlNk7hOvd607h7IGOEVx7/XRLpWDDrrT6hpY653lNf3RCsxdQ7kS9e5hi/RpU1mf9mdTOsf+AtHkkog336jKdyl1/plDV/AlRS8/QjlPfeyvGmA6gEC8cZmBkvyZPbs81tYmif8q2Amd9La3wldh3evYOOkC/b1QQ3fLryoeUl5/9oVprW/aEfmcTxG9CytaFfsx6cAejAb7g6fz/iaWE57fXfth7K7ATi827ZwAs0XX9lLkyGn60ePtqg771isK5/v1jCsbDQDwHwNLehH+BX9zZWWjaQxK85/o09Oaiy0oGY1nivv5kHelmGwj8qqd,iv:a4NHhErZolhOuzNCUU51GQiCfCT8GzZ3GQ24s7fkuyU=,tag:w+GrXmaYWADTrBxMko3xtg==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjQmk5aE1GN2dpVlREZU1s
+            N3V5ZUQweWxMaVo3Y1VjdzZTYW96eFBYZGxvCnNJcjVXKzI4VnliTWFXQy9iSy9k
+            eEZmWlVPQzhZK09NZGowZUVXc1BoZTQKLS0tIHIrMEpTM0lMU0w0eG02aUxJY2VZ
+            aUQ0cmkvK0UwdzhqcDJwM1RDcnpEU0UKQRiTa9/yNrqlkIdlNeaIDzIKa/xvx5nW
+            7wTNfjq2M3go0DMnaVjJA3e15JZBOpOs0QP9z9ydKF/FS4PeCIEFyQ==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-07T04:27:33Z"
+    mac: ENC[AES256_GCM,data:CxZdlGJdIFFTwaeeS9LW2XDDgvUmSsBeBKtFYyDM1JliS8BHtgEv48yuCWxP5sR3SaYtcgNSIQ55zUa89jM6dQHqvjK+1jSYu1xzFWPOlxUu8gVSlU0QLS4GU1RChXuPpTUOjxzn4ymZukGgclZk9iuMvI0/U+nGD8nvs8k0zpM=,iv:MGZ7YsOsjelJPjLYWtALOrM6r8Vd1HgbU/PKL4c3r3k=,tag:dIE38TCobESY3j56NVEvIw==,type:str]
+    version: 3.13.2
Author	SHA1	Message	Date
rock	7251032081	Merge pull request 'fix(monitoring): cap Prometheus retention below the PVC size' (#68 ) from fix/prometheus-retention-size into main	2026-10-25 04:43:21 +00:00
rock	b7fc6ed60b	fix(monitoring): cap Prometheus retention below the PVC size retentionSize was 95GB against a 25Gi volume, so the limit never fired. The volume filled, and Prometheus crashlooped with: write /prometheus/queries.active: no space left on device failed to initialize active query tracker 18GB keeps roughly 7GiB of headroom for the WAL and block overhead.	2026-10-24 21:43:01 -07:00
rock	6312674058	chore(sms): retire the stuck macOS namespace The macOS workload is scaled to 0 and its PVC macos-disk has been Pending for 72 days against storage class longhorn-imessage-local. No node carries the workload=imessage label it pins to. The namespace only holds that dead PVC plus two unused Services. Remove the Application, the manifests and the namespace so ArgoCD prunes them.	2026-10-24 20:05:20 -07:00
rock	2fe6d50ffd	Merge pull request 'fix(argocd): allow the Zitadel chart repo in the homelab AppProject' (#67 ) from fix/allow-zitadel-chart-repo into main	2026-10-25 01:26:38 +00:00
rock	e2d9717ce1	fix(argocd): allow the Zitadel chart repo in the homelab AppProject The zitadel Application failed its first sync with InvalidSpecError: application repo https://charts.zitadel.com is not permitted in project 'homelab'. AppProject sourceRepos is an allowlist, so the chart repo must be listed before any chart from it can be reconciled.	2026-10-24 18:26:21 -07:00
rock	111b19dac2	Merge pull request 'fix(sops): repair the ksops generator list and rotate the Zitadel masterkey' (#66 ) from fix/sops-generator-indent into main	2026-10-25 01:24:43 +00:00
rock	e335694cbb	fix(sops): repair the generator list and rotate the Zitadel masterkey Two defects in the ksops generator: 1. The `zitadel-masterkey.enc.yaml` entry was added with two-space indentation while every other entry uses four. YAML rejected the list, and kustomize reported it as `loading generator plugins ... MalformedYAMLError: yaml: line 39`. That failed the whole sops-secrets application, so no Secret in that app could be created. Verified by running the same kustomize build inside argocd-repo-server: exit 1 before the fix, exit 0 with 33 manifests after it. 2. The masterkey value was generated during the first verification pass and appeared in that session's output. Rotate it now, before the instance exists: a fresh 32-byte value, SOPS-encrypted, with the offline copy updated. No ZITADEL data exists yet, so rotation costs nothing. After the first sync it would mean re-encrypting system.encryption_keys. The offline copy lives outside the repo at ~/zitadel-masterkey.offline (0600).	2026-10-24 18:24:28 -07:00
rock	d1a7d3be6e	fix(oidc): remove the API-server restart Jobs from GitOps The oidc-bootstrap ArgoCD Application runs with selfHeal and prune. Three Jobs in restart-apiserver.yaml kill the kube-apiserver process on each control plane, and bootstrap.yaml carried a fourth Job that deletes the mirror pods. ArgoCD recreated a deleted Job within two minutes, so the kill repeated and every API server reached CrashLoopBackOff. Remove all four. The OIDC authenticator already loaded, so the one-shot restart is not needed again.	2026-10-24 15:06:32 -07:00
rock	2f664c32d9	fix(cluster): grant queue-operator watches and retire the macOS workload queue-operator could not list temporalworkers.kmsvc.io or deployments.apps at cluster scope, so its informers failed on every retry and it reconciled nothing. Add those rules to the ClusterRole. macos-bluebubbles pins itself to a node labelled workload=imessage, and no such node exists. Its PVC macos-disk uses StorageClass longhorn-imessage-local and stayed unbound for 71 days. Scale the Deployment to 0 and record why.	2026-10-24 04:03:02 -07:00
rock	6c98ca7436	Merge pull request 'fix(sops): drop the orphan forgejo-registry-pat file' (#65 ) from fix/sops-drop-orphan-forgejo-pat into main	2026-10-24 10:47:14 +00:00
rock	6ee22401fd	fix(sops): drop the orphan forgejo-registry-pat file ksops failed the whole sops-secrets application on this file: it declares Secret forgejo-registry-pat in namespace forgejo, and no such namespace exists. The failure blocks every other file in the generator list, including zitadel-masterkey. The same Secret name is already generated in namespace api by forgejo-registry-pat-api.enc.yaml, and that is the copy the gateway reads. The forgejo namespace copy is stale: Forgejo runs in namespace cicd, and namespace forgejo was never created. Remove the file and its generator entry. ArgoCD never created the Secret, so pruning has nothing to remove.	2026-10-24 03:46:29 -07:00
rock	e9817283e6	Merge pull request 'feat(iam): wire Zitadel as the Authentik replacement' (#63 ) from harness/zitadel-iam-wiring into main	2026-10-24 10:41:17 +00:00
rock	6e9586aa15	Merge pull request 'chore(argocd): delete the azhi-riotpiao Application' (#64 ) from chore/remove-azhi-riotpiao-app into main	2026-10-24 10:41:09 +00:00
rock	8e9ac37583	chore(argocd): delete the azhi-riotpiao Application The Application pointed at github.com/ToyoGlobal/azhi-infra, branch feat/azi-riotpiao-cnpg, path k8s/overlays/azi-riotpiao. It never reconciled: status stayed Unknown because the source repository could not be read, and the destination namespace azi-riotpiao was never created. ArgoCD reported it as an OutOfSync managed resource of homelab-root with nothing behind it. Also drops the ToyoGlobal URL from the AppProject allowlist. No manifest from that repository is referenced any more, so the allowlist entry is dead weight. Not touched here, because it is not in git: Secret azhi-infra-github in namespace argocd holds the read-only GitHub credential for that repository. Delete it separately once no other Application needs the repo. The live Application object was deleted by hand. Until this commit reaches main, the next homelab-root sync recreates it, because main still declares it.	2026-10-24 03:35:52 -07:00