Compare commits

..
Author SHA1 Message Date
rock 7251032081 Merge pull request 'fix(monitoring): cap Prometheus retention below the PVC size' (#68) from fix/prometheus-retention-size into main 2026-10-25 04:43:21 +00:00
rock b7fc6ed60b fix(monitoring): cap Prometheus retention below the PVC size
retentionSize was 95GB against a 25Gi volume, so the limit never fired. The
volume filled, and Prometheus crashlooped with:

  write /prometheus/queries.active: no space left on device
  failed to initialize active query tracker

18GB keeps roughly 7GiB of headroom for the WAL and block overhead.
2026-10-24 21:43:01 -07:00
rock 6312674058 chore(sms): retire the stuck macOS namespace
The macOS workload is scaled to 0 and its PVC macos-disk has been Pending
for 72 days against storage class longhorn-imessage-local. No node carries
the workload=imessage label it pins to. The namespace only holds that dead
PVC plus two unused Services. Remove the Application, the manifests and the
namespace so ArgoCD prunes them.
2026-10-24 20:05:20 -07:00
rock 2fe6d50ffd Merge pull request 'fix(argocd): allow the Zitadel chart repo in the homelab AppProject' (#67) from fix/allow-zitadel-chart-repo into main 2026-10-25 01:26:38 +00:00
rock e2d9717ce1 fix(argocd): allow the Zitadel chart repo in the homelab AppProject
The zitadel Application failed its first sync with InvalidSpecError:
application repo https://charts.zitadel.com is not permitted in project
'homelab'. AppProject sourceRepos is an allowlist, so the chart repo must be
listed before any chart from it can be reconciled.
2026-10-24 18:26:21 -07:00
rock 111b19dac2 Merge pull request 'fix(sops): repair the ksops generator list and rotate the Zitadel masterkey' (#66) from fix/sops-generator-indent into main 2026-10-25 01:24:43 +00:00
rock e335694cbb fix(sops): repair the generator list and rotate the Zitadel masterkey
Two defects in the ksops generator:

1. The `zitadel-masterkey.enc.yaml` entry was added with two-space indentation
   while every other entry uses four. YAML rejected the list, and kustomize
   reported it as `loading generator plugins ... MalformedYAMLError: yaml: line
   39`. That failed the whole sops-secrets application, so no Secret in that app
   could be created. Verified by running the same kustomize build inside
   argocd-repo-server: exit 1 before the fix, exit 0 with 33 manifests after it.

2. The masterkey value was generated during the first verification pass and
   appeared in that session's output. Rotate it now, before the instance exists:
   a fresh 32-byte value, SOPS-encrypted, with the offline copy updated. No
   ZITADEL data exists yet, so rotation costs nothing. After the first sync it
   would mean re-encrypting system.encryption_keys.

The offline copy lives outside the repo at ~/zitadel-masterkey.offline (0600).
2026-10-24 18:24:28 -07:00
rock d1a7d3be6e fix(oidc): remove the API-server restart Jobs from GitOps
The oidc-bootstrap ArgoCD Application runs with selfHeal and prune. Three
Jobs in restart-apiserver.yaml kill the kube-apiserver process on each
control plane, and bootstrap.yaml carried a fourth Job that deletes the
mirror pods. ArgoCD recreated a deleted Job within two minutes, so the
kill repeated and every API server reached CrashLoopBackOff.

Remove all four. The OIDC authenticator already loaded, so the one-shot
restart is not needed again.
2026-10-24 15:06:32 -07:00
rock 2f664c32d9 fix(cluster): grant queue-operator watches and retire the macOS workload
queue-operator could not list temporalworkers.kmsvc.io or deployments.apps
at cluster scope, so its informers failed on every retry and it reconciled
nothing. Add those rules to the ClusterRole.

macos-bluebubbles pins itself to a node labelled workload=imessage, and no
such node exists. Its PVC macos-disk uses StorageClass longhorn-imessage-local
and stayed unbound for 71 days. Scale the Deployment to 0 and record why.
2026-10-24 04:03:02 -07:00
rock 6c98ca7436 Merge pull request 'fix(sops): drop the orphan forgejo-registry-pat file' (#65) from fix/sops-drop-orphan-forgejo-pat into main 2026-10-24 10:47:14 +00:00
27 changed files with 587 additions and 462 deletions
@@ -0,0 +1,37 @@
{
"schema_version": 1,
"diagram_type": "architecture",
"meta": {
"title": "Poimen Memory — Schema Relations",
"quality_profile": "showcase",
"viewBox": [900, 480]
},
"components": [
{ "id": "node", "type": "database", "label": "memory_node", "sublabel": "Episodes (raw messages)", "pos": [40, 60], "size": [150, 55], "tag": "BIGSERIAL" },
{ "id": "entity", "type": "database", "label": "memory_entity", "sublabel": "Extracted entities", "pos": [260, 60], "size": [150, 55], "tag": "UUID" },
{ "id": "edge", "type": "database", "label": "memory_edge", "sublabel": "Facts / relationships", "pos": [480, 60], "size": [150, 55], "tag": "UUID" },
{ "id": "community", "type": "database", "label": "memory_community", "sublabel": "Entity clusters", "pos": [700, 60], "size": [150, 55], "tag": "UUID" },
{ "id": "pgvector", "type": "cloud", "label": "pgvector HNSW", "sublabel": "768-dim cosine search", "pos": [140, 260], "size": [160, 55], "tag": "nomic-embed-v2" },
{ "id": "tsvector", "type": "cloud", "label": "pg ts_rank", "sublabel": "Full-text search", "pos": [400, 260], "size": [160, 55], "tag": "GIN index" },
{ "id": "tenant", "type": "security", "label": "tenant_id", "sublabel": "Multi-tenant isolation", "pos": [660, 260], "size": [140, 55], "tag": "W7 / mig 014" }
],
"boundaries": [
{ "kind": "region", "label": "Temporal Knowledge Graph (Zep §2)", "wraps": ["node", "entity", "edge", "community"] },
{ "kind": "security-group", "label": "Search & Isolation", "wraps": ["pgvector", "tsvector", "tenant"] }
],
"connections": [
{ "id": "node-entity", "from": "node", "to": "entity", "variant": "emphasis" },
{ "id": "entity-edge", "from": "entity", "to": "edge", "variant": "emphasis" },
{ "id": "entity-community", "from": "community", "to": "entity", "variant": "dashed" },
{ "id": "edge-node", "from": "edge", "to": "node", "variant": "dashed" },
{ "id": "entity-vec", "from": "entity", "to": "pgvector", "variant": "default" },
{ "id": "entity-fts", "from": "entity", "to": "tsvector", "variant": "dashed" },
{ "id": "edge-fts", "from": "edge", "to": "tsvector", "variant": "dashed" },
{ "id": "tenant-all", "from": "tenant", "to": "entity", "variant": "security" }
],
"cards": [
{ "dot": "emerald", "title": "Three-Tier Graph", "items": ["memory_node → episode subgraph", "memory_entity → entity subgraph", "memory_edge → fact relationships", "memory_community → clusters (Phase 4)"] },
{ "dot": "rose", "title": "Bi-Temporal", "items": ["t_valid / t_invalid: event time", "t_created / t_expired: transaction time", "Contradiction: active → candidate → confirmed"] },
{ "dot": "cyan", "title": "Search", "items": ["HNSW: name, summary, fact embeddings", "ts_rank: BM25 fallback (always available)", "tenant_id: per-table isolation (mig 014)"] }
]
}
@@ -0,0 +1,158 @@
{
"schema_version": 1,
"diagram_type": "dataflow",
"meta": {
"title": "Poimen Memory \u2014 Schema Relations",
"viewBox": [
960,
480
],
"quality_profile": "showcase"
},
"stages": [
{
"label": "Episodes"
},
{
"label": "Entities"
},
{
"label": "Relationships"
},
{
"label": "Indexes"
}
],
"nodes": [
{
"id": "node",
"type": "database",
"label": "memory_node",
"sublabel": "Raw messages",
"stage": 0,
"row": 0,
"tag": "BIGSERIAL"
},
{
"id": "entity",
"type": "database",
"label": "memory_entity",
"sublabel": "Named entities",
"stage": 1,
"row": 0,
"tag": "UUID + tenant_id"
},
{
"id": "community",
"type": "database",
"label": "memory_community",
"sublabel": "Clusters (Phase 4)",
"stage": 1,
"row": 1,
"tag": "UUID"
},
{
"id": "edge",
"type": "database",
"label": "memory_edge",
"sublabel": "Facts + bi-temporal",
"stage": 2,
"row": 0,
"tag": "UUID + tenant_id"
},
{
"id": "hnsw",
"type": "cloud",
"label": "HNSW Index",
"sublabel": "768-dim cosine",
"stage": 3,
"row": 0,
"tag": "pgvector"
},
{
"id": "fts",
"type": "cloud",
"label": "GIN Index",
"sublabel": "ts_rank BM25",
"stage": 3,
"row": 1,
"tag": "full-text"
}
],
"flows": [
{
"from": "node",
"to": "entity",
"label": "LLM extraction",
"classification": "ingest",
"variant": "emphasis",
"route": "straight"
},
{
"from": "entity",
"to": "edge",
"label": "source + target FK",
"classification": "foreign key",
"variant": "emphasis",
"route": "straight"
},
{
"from": "entity",
"to": "community",
"label": "community_id",
"classification": "membership",
"variant": "dashed",
"route": "straight",
"labelDy": 30
},
{
"from": "edge",
"to": "hnsw",
"classification": "vector",
"variant": "default",
"route": "straight",
"label": "embedding",
"labelDy": -30
},
{
"from": "edge",
"to": "fts",
"label": "fact text",
"classification": "text",
"variant": "dashed",
"route": "vertical-channel",
"labelDy": 30
}
],
"cards": [
{
"dot": "emerald",
"title": "Three-Tier Graph",
"items": [
"memory_node \u2192 episodes (raw messages)",
"memory_entity \u2192 extracted entities",
"memory_edge \u2192 facts between entities",
"memory_community \u2192 clusters (Phase 4)"
]
},
{
"dot": "rose",
"title": "Bi-Temporal + Tenant",
"items": [
"t_valid/t_invalid: when true in reality",
"t_created/t_expired: system transaction time",
"tenant_id: row-level multi-tenant isolation",
"Contradiction: active \u2192 candidate \u2192 confirmed"
]
},
{
"dot": "cyan",
"title": "Search Indexes",
"items": [
"HNSW: name, summary, fact embeddings",
"GIN: ts_rank on name+summary and fact",
"RRF fusion: sem_weight + lex_weight"
]
}
]
}
@@ -0,0 +1,63 @@
{
"schema_version": 1,
"diagram_type": "sequence",
"meta": {
"title": "Service Acting on Behalf of User",
"viewBox": [1000, 600],
"column_fit": "spread",
"quality_profile": "showcase"
},
"participants": [
{ "id": "user", "type": "external", "label": "User", "sublabel": "browser" },
{ "id": "app", "type": "frontend", "label": "Web App", "sublabel": "poimen UI" },
{ "id": "authentik", "type": "security", "label": "Authentik", "sublabel": "OIDC" },
{ "id": "worker", "type": "backend", "label": "Backend Worker", "sublabel": "temporal/queue" },
{ "id": "memory", "type": "backend", "label": "Memory API", "sublabel": "poimen-memory" },
{ "id": "jwks", "type": "security", "label": "JWKS", "sublabel": "RS256" }
],
"segments": [
{ "from": 150, "to": 260, "label": "User Auth" },
{ "from": 270, "to": 370, "label": "Delegation" },
{ "from": 380, "to": 510, "label": "Downstream" }
],
"messages": [
{ "id": "login", "from": "user", "to": "app", "y": 165, "label": "login (OIDC)", "variant": "default" },
{ "id": "oidc-flow", "from": "app", "to": "authentik", "y": 200, "label": "authorization_code flow", "variant": "emphasis" },
{ "id": "user-jwt", "from": "authentik", "to": "app", "y": 235, "label": "user JWT {groups,roles}", "variant": "return" },
{ "id": "trigger-job", "from": "user", "to": "app", "y": 285, "label": "trigger async job", "variant": "default" },
{ "id": "queue-job", "from": "app", "to": "worker", "y": 320, "label": "enqueue job + user JWT", "variant": "emphasis", "note": "JWT passed as job metadata" },
{ "id": "job-ack", "from": "worker", "to": "app", "y": 355, "label": "job queued", "variant": "return" },
{ "id": "call-memory", "from": "worker", "to": "memory", "y": 400, "label": "POST /memory/ingest", "variant": "emphasis", "note": "Bearer <user JWT>" },
{ "id": "fetch-jwks", "from": "memory", "to": "jwks", "y": 435, "label": "GET /jwks", "variant": "dashed" },
{ "id": "jwks-keys", "from": "jwks", "to": "memory", "y": 470, "label": "RSA pubkey", "variant": "return" },
{ "id": "ingest-ok", "from": "memory", "to": "worker", "y": 505, "label": "202 Accepted", "variant": "return", "note": "uses USER's groups/roles" }
],
"activations": [
{ "participant": "user", "from": 160, "to": 291, "type": "external" },
{ "participant": "app", "from": 165, "to": 361, "type": "frontend" },
{ "participant": "authentik", "from": 195, "to": 241, "type": "security" },
{ "participant": "worker", "from": 315, "to": 511, "type": "backend" },
{ "participant": "memory", "from": 395, "to": 511, "type": "backend" },
{ "participant": "jwks", "from": 430, "to": 476, "type": "security" }
],
"cards": [
{
"dot": "emerald",
"title": "Delegation Pattern",
"items": [
"User JWT travels with the job payload",
"Worker uses user's token for downstream",
"Memory sees USER as the principal"
]
},
{
"dot": "rose",
"title": "Security Model",
"items": [
"No service-account escalation",
"User's permissions bound the action",
"Audit trail traces to user.sub"
]
}
]
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"diagram_type": "architecture",
"meta": {
"title": "Memory Service Fine-Grained RBAC",
"quality_profile": "showcase",
"viewBox": [900, 520]
},
"components": [
{ "id": "jwt", "type": "security", "label": "JWT Token", "sublabel": "groups[] + roles[]", "pos": [40, 60], "size": [140, 60], "tag": "Authentik" },
{ "id": "cap-check", "type": "backend", "label": "Capability Check", "sublabel": "has_capability()", "pos": [260, 60], "size": [160, 60], "tag": "Phase 1" },
{ "id": "scope-checker", "type": "backend", "label": "Scope Checker", "sublabel": "filter_resources()", "pos": [500, 60], "size": [160, 60], "tag": "Phase 2" },
{ "id": "decision", "type": "backend", "label": "AccessDecision", "sublabel": "Allow / Deny", "pos": [740, 60], "size": [130, 60] },
{ "id": "project-scope", "type": "database", "label": "ProjectScope", "sublabel": "dev/staging/prod/*", "pos": [40, 200], "size": [140, 55] },
{ "id": "visibility-scope", "type": "database", "label": "VisibilityScope", "sublabel": "public/private", "pos": [220, 200], "size": [140, 55] },
{ "id": "owner-scope", "type": "database", "label": "OwnerScope", "sublabel": "self/any/user", "pos": [400, 200], "size": [140, 55] },
{ "id": "group-scope", "type": "database", "label": "GroupScope", "sublabel": "in_any_group()", "pos": [580, 200], "size": [140, 55] },
{ "id": "entities", "type": "database", "label": "Entities", "sublabel": "memory_entity", "pos": [40, 340], "size": [120, 50] },
{ "id": "edges", "type": "database", "label": "Edges", "sublabel": "memory_edge", "pos": [200, 340], "size": [120, 50] },
{ "id": "projects", "type": "database", "label": "Projects", "sublabel": "project_id", "pos": [360, 340], "size": [120, 50] },
{ "id": "convos", "type": "database", "label": "Conversations", "sublabel": "owner_id", "pos": [520, 340], "size": [120, 50] }
],
"boundaries": [
{ "kind": "security-group", "label": "Access Control Pipeline", "wraps": ["cap-check", "scope-checker", "decision"] },
{ "kind": "region", "label": "Scope Checkers", "wraps": ["project-scope", "visibility-scope", "owner-scope", "group-scope"] },
{ "kind": "region", "label": "Protected Resources", "wraps": ["entities", "edges", "projects", "convos"] }
],
"connections": [
{ "id": "jwt-to-cap", "from": "jwt", "to": "cap-check", "label": "claims", "variant": "emphasis" },
{ "id": "cap-to-scope", "from": "cap-check", "to": "scope-checker", "variant": "emphasis" },
{ "id": "scope-to-decision", "from": "scope-checker", "to": "decision", "variant": "emphasis" },
{ "id": "scope-to-proj", "from": "scope-checker", "to": "project-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" },
{ "id": "scope-to-vis", "from": "scope-checker", "to": "visibility-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" },
{ "id": "scope-to-owner", "from": "scope-checker", "to": "owner-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" },
{ "id": "scope-to-group", "from": "scope-checker", "to": "group-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" }
],
"cards": [
{ "dot": "emerald", "title": "Phase 1: Capability", "items": ["has_capability(memory:read)", "Checks roles[] claim", "Binary allow/deny"] },
{ "dot": "rose", "title": "Phase 2: Scope", "items": ["Project: dev/staging/prod", "Visibility: public/private", "Owner: self-owned only", "Group: poimen-admins/devs"] },
{ "dot": "cyan", "title": "JWT Claims", "items": ["groups: user.groups", "roles: attributes.roles", "permissions: group names"] }
]
}
+54
View File
@@ -0,0 +1,54 @@
{
"schema_version": 1,
"diagram_type": "sequence",
"meta": {
"title": "Service-to-Service (M2M) Authentication",
"viewBox": [1080, 520],
"column_fit": "spread",
"quality_profile": "showcase"
},
"participants": [
{ "id": "worker", "type": "backend", "label": "Ingest Worker", "sublabel": "temporal/cron" },
{ "id": "authentik", "type": "security", "label": "Authentik", "sublabel": "OIDC provider" },
{ "id": "memory", "type": "backend", "label": "Memory Service", "sublabel": "poimen-memory" },
{ "id": "jwks", "type": "security", "label": "JWKS", "sublabel": "RS256 keys" }
],
"segments": [
{ "from": 150, "to": 230, "label": "Token Mint" },
{ "from": 240, "to": 380, "label": "API + RBAC" }
],
"messages": [
{ "id": "token-req", "from": "worker", "to": "authentik", "y": 165, "label": "POST /token grant_type=password", "variant": "emphasis", "note": "client_id + username + app_password + scope" },
{ "id": "jwt-resp", "from": "authentik", "to": "worker", "y": 205, "label": "JWT {groups,roles,aud,iss}", "variant": "return", "note": "groups=[poimen-admins] roles=[memory:write]" },
{ "id": "api-call", "from": "worker", "to": "memory", "y": 260, "label": "POST /memory/ingest", "variant": "emphasis", "note": "Authorization: Bearer <JWT>" },
{ "id": "fetch-jwks", "from": "memory", "to": "jwks", "y": 295, "label": "GET /jwks (cached)", "variant": "dashed" },
{ "id": "jwks-resp", "from": "jwks", "to": "memory", "y": 330, "label": "RSA pubkey (kid)", "variant": "return" },
{ "id": "verify-ok", "from": "memory", "to": "worker", "y": 365, "label": "202 Accepted", "variant": "return", "note": "sig ✓ iss ✓ aud ✓ has_capability ✓" }
],
"activations": [
{ "participant": "worker", "from": 160, "to": 371, "type": "backend" },
{ "participant": "authentik", "from": 165, "to": 211, "type": "security" },
{ "participant": "memory", "from": 255, "to": 371, "type": "backend" },
{ "participant": "jwks", "from": 290, "to": 336, "type": "security" }
],
"cards": [
{
"dot": "emerald",
"title": "M2M Pattern",
"items": [
"grant_type=password (Authentik aliases client_credentials)",
"App-password token authenticates the service account",
"NOT client_secret — that creates nameless SA"
]
},
{
"dot": "rose",
"title": "JWT Claims",
"items": [
"groups: from user.groups (poimen-admins)",
"roles: from user.attributes.roles",
"iss/aud: per-provider issuer"
]
}
]
}
+61
View File
@@ -0,0 +1,61 @@
{
"schema_version": 1,
"diagram_type": "sequence",
"meta": {
"title": "User Direct Access with JWT",
"viewBox": [1000, 640],
"column_fit": "spread",
"quality_profile": "showcase"
},
"participants": [
{ "id": "user", "type": "external", "label": "User", "sublabel": "browser" },
{ "id": "app", "type": "frontend", "label": "Web App", "sublabel": "poimen UI" },
{ "id": "authentik", "type": "security", "label": "Authentik", "sublabel": "OIDC provider" },
{ "id": "memory", "type": "backend", "label": "Memory API", "sublabel": "poimen-memory" },
{ "id": "jwks", "type": "security", "label": "JWKS", "sublabel": "RS256 keys" }
],
"segments": [
{ "from": 150, "to": 370, "label": "OIDC Login" },
{ "from": 380, "to": 540, "label": "API + RBAC" }
],
"messages": [
{ "id": "click-login", "from": "user", "to": "app", "y": 165, "label": "click Login", "variant": "default" },
{ "id": "redirect-auth", "from": "app", "to": "authentik", "y": 200, "label": "redirect /authorize", "variant": "emphasis", "note": "scope=openid groups roles" },
{ "id": "show-login", "from": "authentik", "to": "user", "y": 235, "label": "login form", "variant": "default" },
{ "id": "submit-creds", "from": "user", "to": "authentik", "y": 270, "label": "username + password", "variant": "security" },
{ "id": "auth-code", "from": "authentik", "to": "app", "y": 305, "label": "redirect ?code=xxx", "variant": "return" },
{ "id": "exchange-token", "from": "app", "to": "authentik", "y": 340, "label": "POST /token code=xxx", "variant": "emphasis" },
{ "id": "jwt-resp", "from": "authentik", "to": "app", "y": 375, "label": "JWT {groups,roles,sub}", "variant": "return" },
{ "id": "query-api", "from": "app", "to": "memory", "y": 420, "label": "GET /memory/query", "variant": "emphasis", "note": "Authorization: Bearer <JWT>" },
{ "id": "fetch-jwks", "from": "memory", "to": "jwks", "y": 455, "label": "GET /jwks (cached)", "variant": "dashed" },
{ "id": "jwks-keys", "from": "jwks", "to": "memory", "y": 490, "label": "RSA pubkey", "variant": "return" },
{ "id": "query-resp", "from": "memory", "to": "app", "y": 525, "label": "200 JSON results", "variant": "return", "note": "has_capability(memory:read) ✓" }
],
"activations": [
{ "participant": "user", "from": 160, "to": 276, "type": "external" },
{ "participant": "app", "from": 165, "to": 531, "type": "frontend" },
{ "participant": "authentik", "from": 195, "to": 381, "type": "security" },
{ "participant": "memory", "from": 415, "to": 531, "type": "backend" },
{ "participant": "jwks", "from": 450, "to": 496, "type": "security" }
],
"cards": [
{
"dot": "emerald",
"title": "Authorization Code Flow",
"items": [
"User authenticates directly with Authentik",
"App exchanges code for JWT",
"Scopes: openid groups roles permissions"
]
},
{
"dot": "rose",
"title": "JWT Contents",
"items": [
"iss: .../application/o/poimen/",
"aud: poimen (provider client_id)",
"groups: user's group memberships"
]
}
]
}
@@ -12,6 +12,19 @@ rules:
- apiGroups: ["kmsvc.io"] - apiGroups: ["kmsvc.io"]
resources: ["queues"] resources: ["queues"]
verbs: ["get", "list", "watch", "update", "patch"] verbs: ["get", "list", "watch", "update", "patch"]
# The operator watches TemporalWorker objects and creates Deployments for
# them. Without these two rules the controller logs
# "cannot list resource temporalworkers/deployments ... at the cluster scope"
# and never reconciles.
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers/status"]
verbs: ["get", "update", "patch"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: ["kmsvc.io"] - apiGroups: ["kmsvc.io"]
resources: ["queues/status"] resources: ["queues/status"]
verbs: ["get", "update", "patch"] verbs: ["get", "update", "patch"]
-115
View File
@@ -1,115 +0,0 @@
# macOS VM (Docker-OSX) hosting the BlueBubbles server.
#
# ── Why a VM and not a container ────────────────────────────────────────────
# Containers share the host kernel. macOS binaries are Mach-O and need XNU plus
# Cocoa/IOKit, which a Linux kernel cannot provide, so no macOS container exists
# or can exist. Docker-OSX is QEMU running a macOS guest, packaged in a
# container — a VM in a box, not a macOS container.
#
# ── Why this works on worker-2 ──────────────────────────────────────────────
# Verified on the existing hardware: amd64, `vmx` (Intel VT-x) present, and
# /dev/kvm exists on Talos nodes (KVM is compiled into Talos' kernel, not a
# module). Bare metal, so no nested virtualisation needed.
#
# ── Read this before relying on it ──────────────────────────────────────────
# 1. Setup is INTERACTIVE. First boot runs the macOS installer: connect over
# VNC (:5999), erase the disk in Disk Utility, install, create a user, sign
# into iMessage, THEN install BlueBubbles inside the guest. This manifest
# only provides the machine; it does not provision macOS.
# 2. iMessage activation on non-Apple hardware is a coin flip. BlueBubbles'
# own guidance: "test sending an iMessage to yourself. If it does not
# succeed, it's likely best to restart from the beginning."
# 3. Apple's macOS licence permits virtualisation only on Apple hardware. This
# is a Hackintosh. Use a throwaway Apple ID, not a primary one.
# 4. BlueBubbles labels this path "not for beginners", "no guarantees or
# warranty".
#
# Private API (reactions, typing indicators, edit/unsend) needs SIP disabled
# inside the guest and is NOT required for plain send/receive. Skip it.
apiVersion: apps/v1
kind: Deployment
metadata:
name: macos-bluebubbles
labels:
app.kubernetes.io/name: macos-bluebubbles
app.kubernetes.io/part-of: sms
spec:
replicas: 1
# Recreate: the qcow2 disk is RWO and a second pod must never attach it
# concurrently — two QEMU processes on one image corrupts it.
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: macos-bluebubbles
template:
metadata:
labels:
app.kubernetes.io/name: macos-bluebubbles
app.kubernetes.io/part-of: sms
spec:
# Dedicated node. The taint keeps everything else off worker-2; this
# toleration is what lets the VM on. Both halves are required.
nodeSelector:
workload: imessage
tolerations:
- key: workload
operator: Equal
value: imessage
effect: NoSchedule
containers:
- name: macos
image: sickcodes/docker-osx:latest@sha256:3a3c82c79bc4e73531f819ccdfa4053b3084efd7c1f645678dbf8b4b3a24369c
# QEMU needs /dev/kvm; Talos enforces `baseline` cluster-wide, so this
# only schedules because the sms namespace is labelled privileged.
securityContext:
privileged: true
env:
# Generates a unique serial / board-serial / UUID / MAC and persists
# them to bootdisk.qcow2. This synthetic identity is what iMessage
# activates against — it must stay stable across restarts, which is
# why the PVC matters.
- name: GENERATE_UNIQUE
value: "true"
# Identity is only plausible if it matches a real product line.
- name: DEVICE_MODEL
value: "iMacPro1,1"
- name: RAM
value: "12"
- name: CORES
value: "6"
- name: EXTRA
# Expose the BlueBubbles server port from the guest to the pod.
# Guest :1234 (BlueBubbles default) -> pod :1234.
value: "-device virtio-net-pci,netdev=net0 -netdev user,id=net0,hostfwd=tcp::1234-:1234"
ports:
- name: vnc
containerPort: 5999
protocol: TCP
- name: bluebubbles
containerPort: 1234
protocol: TCP
resources:
requests:
cpu: "6"
memory: 14Gi
limits:
cpu: "12"
memory: 20Gi
volumeMounts:
- name: macos-disk
mountPath: /home/arch/OSX-KVM/disk
- name: kvm
mountPath: /dev/kvm
# No readiness probe on purpose. The guest takes many minutes to boot,
# and until macOS + BlueBubbles are installed BY HAND there is nothing
# listening on 1234. A probe here would crash-loop the pod through the
# entire interactive install.
volumes:
- name: macos-disk
persistentVolumeClaim:
claimName: macos-disk
- name: kvm
hostPath:
path: /dev/kvm
type: CharDevice
-10
View File
@@ -1,10 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: sms
resources:
- namespace.yaml
- storageclass.yaml
- pvc-macos.yaml
- deployment-macos.yaml
- service.yaml
- networkpolicy.yaml
-19
View File
@@ -1,19 +0,0 @@
# iMessage delivery for the cluster.
#
# BlueBubbles' server is a macOS Electron app paired with an Objective-C helper
# that hooks Messages.app private APIs — it cannot be containerised on Linux,
# because containers share the host kernel and macOS needs XNU + Cocoa. The only
# way to run it on Talos is a full macOS VM under QEMU/KVM (Docker-OSX), which
# needs a privileged pod with /dev/kvm.
#
# Hence privileged PodSecurity: the cluster default from the Talos controlplane
# is `enforce: baseline`, which forbids privileged containers and host devices.
# Scope is limited to this namespace.
apiVersion: v1
kind: Namespace
metadata:
name: sms
labels:
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/audit: privileged
pod-security.kubernetes.io/warn: privileged
-22
View File
@@ -1,22 +0,0 @@
# Default-deny. This namespace runs a privileged QEMU VM signed into an Apple
# ID and exposes an unauthenticated VNC console; nothing should reach it except
# opted-in clients.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: sms-default-deny
spec:
podSelector:
matchLabels:
app.kubernetes.io/part-of: sms
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector: {}
podSelector:
matchLabels:
sms-client: "true"
ports:
- protocol: TCP
port: 1234
-25
View File
@@ -1,25 +0,0 @@
# Persistent macOS disk image + generated hardware identity (bootdisk.qcow2).
#
# This volume is NOT disposable: it holds the VM's serial number, board serial,
# UUID and MAC, which together form the identity iMessage was activated against.
# Losing it means re-running activation, which is the least reliable step of the
# whole setup.
#
# Docker-OSX documents 128GB minimum for the guest image; 200Gi leaves room for
# the installer, the base system, and qcow2 growth.
#
# ⚠️ Single replica (see storageclass.yaml — capacity and IO both rule out 3).
# Losing worker-2's disk therefore means losing the activated identity and
# redoing iMessage activation. Once the guest is installed and activated, take
# a Longhorn snapshot/backup of this volume; that is the only redundancy here.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: macos-disk
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn-imessage-local
resources:
requests:
storage: 200Gi
-31
View File
@@ -1,31 +0,0 @@
# VNC is how you drive the interactive macOS install. Deliberately ClusterIP —
# it is an unauthenticated console onto a machine holding a live Apple ID
# session. Reach it with `kubectl port-forward`, never an Ingress.
apiVersion: v1
kind: Service
metadata:
name: macos-vnc
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: macos-bluebubbles
ports:
- name: vnc
port: 5999
targetPort: vnc
---
# The BlueBubbles REST API, once installed inside the guest. This is the stable
# name cluster services use, so callers never depend on the pod IP or on whether
# the backend is this VM or a real Mac mini later.
apiVersion: v1
kind: Service
metadata:
name: bluebubbles
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: macos-bluebubbles
ports:
- name: http
port: 1234
targetPort: bluebubbles
-32
View File
@@ -1,32 +0,0 @@
# Dedicated StorageClass for the macOS VM disk.
#
# The default `longhorn` class does not work here, for two independent reasons:
#
# 1. Replica count. Default is 3, and Longhorn schedules against
# storageMaximum - storageScheduled with over-provisioning at 100%. Free
# space is cp-1 146Gi / cp-2 8Gi / cp-3 146Gi / worker-1 292Gi, so a 200Gi
# volume has only one node that can hold even a single replica — a 3-replica
# volume fails outright with ReplicaSchedulingFailure.
# 2. Binding mode. `Immediate` provisions the volume the moment the PVC is
# created, before any pod is scheduled. Combined with strict-local that
# pins the data to an arbitrary node, not the one the VM runs on.
#
# So: one replica, kept local to the VM, bound only once the pod has a node.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: longhorn-imessage-local
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
# The pod is pinned to worker-2 by nodeSelector; wait for it to be scheduled so
# the replica is placed on that node and not somewhere else.
volumeBindingMode: WaitForFirstConsumer
parameters:
# A qcow2 backing a live VM is latency-sensitive and rewritten constantly.
# Serving it over the network from another node's disk would be the single
# worst thing for guest responsiveness, so force it local.
numberOfReplicas: "1"
dataLocality: "strict-local"
staleReplicaTimeout: "30"
fsType: "ext4"
-31
View File
@@ -110,37 +110,6 @@ spec:
syncOptions: syncOptions:
- CreateNamespace=true - CreateNamespace=true
--- ---
# iMessage/SMS delivery. Raw manifests: a privileged macOS VM (Docker-OSX)
# running the BlueBubbles server, plus its dedicated local StorageClass.
#
# Pinned to worker-2 via nodeSelector `workload: imessage` + a matching
# toleration for that node's taint. Until worker-2 is provisioned this app
# syncs everything except the pod, which stays Pending — that is expected.
#
# No CreateNamespace: namespace.yaml carries `pod-security: privileged`, which
# the VM needs (/dev/kvm, privileged), and an ArgoCD-created namespace would
# not have those labels.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: sms
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "8"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/apps/sms
destination:
server: https://kubernetes.default.svc
namespace: sms
syncPolicy:
automated:
prune: true
selfHeal: true
---
# Document management. Raw manifests (no Helm): postgres is the dedicated # Document management. Raw manifests (no Helm): postgres is the dedicated
# paperless-db CNPG cluster in k8s/infra/databases (wave 2), redis is # paperless-db CNPG cluster in k8s/infra/databases (wave 2), redis is
# in-cluster only (no PVC), media lives on the cp-3 USB HDD (see # in-cluster only (no PVC), media lives on the cp-3 USB HDD (see
+20
View File
@@ -0,0 +1,20 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: talos-repair
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "6"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/infra/talos-repair
destination:
server: https://kubernetes.default.svc
namespace: cicd
syncPolicy:
automated:
prune: true
selfHeal: true
+1
View File
@@ -35,6 +35,7 @@ spec:
- https://grafana.github.io/helm-charts - https://grafana.github.io/helm-charts
- https://helm.releases.hashicorp.com - https://helm.releases.hashicorp.com
- https://charts.goauthentik.io - https://charts.goauthentik.io
- https://charts.zitadel.com
- https://strimzi.io/charts/ - https://strimzi.io/charts/
- https://charts.bitnami.com/bitnami - https://charts.bitnami.com/bitnami
- https://homarr-labs.github.io/charts - https://homarr-labs.github.io/charts
@@ -0,0 +1,23 @@
apiVersion: v1
kind: Secret
metadata:
name: talos-repair-config
namespace: cicd
type: Opaque
stringData:
config: ENC[AES256_GCM,data:WO+jinh/2Rq5VeGLoDDJOQL4zfYJINSTYZCwGWndbzfIHw4XRwxDB7BixjCdpPuLOkNIBAXHQpKPBhlQpvRTso70L26njgVdVENUGeNhxvb6/cwl3VYRsfQQxKoQW9Qu6DyCD7ItuyazXMdIbENKBoih7GA2IjKdGeEtpB+a6PO+6rpI0AQRIjM9h65yJ6gvWeLZmb6U0qCFW61SelHmVSsfehxH/R9EKQROKwD0Z/TgYnoS28OuDtixAQVfnrI3gtgUAeXgnPQTJf1Vb9iwbRBvaSWkPSIqa7hOqWEIM2qyrvhTsEb8XDRUHTmp4BCxYktIxoiSWZEbeh0y0HKRzekZIj4rHA+c/JYSPqS0dvn1Ga+8PTc1LNdXDWIcKI58YQ2jzA0oE2vm0YzmOjcDTzTGLDBk49gIql+AMfwoxCop7t1PB7BrJw6M4bj9M5aHIq8ZlGXDvtWZcSEBj0ChZ0q20cB4v4bvMkbuJR/PS25uNkeG+UBCPidBQqCUw37aNf8SDwOaAC+6PeHdLmcJ/dkKntGFlU5MQzWQXfqs/fp95ze3mr7f3et7SqVl7fAlYSdEt6dA2xxnFnqgNwAm3UacV/We6SRnjrFlBAHZ3Q5hIuKkxmszsS4Bh/VuNtZI5YJ4ldnoIZC3aWKDt+rVB6P+Qxp1T1XqZj8GY715DZRP1q+Z7iQ9RcXiJBTD5qWVpnkESOvbTEYv/W3oXTufVRrJ8Ps0on/xWRF5NU04+tqvToC8+MIQDz5rb+eXYCEWjZG9ZhInY+M2HRTLg+c49dRlyqVnnqfNWBejSS/hoQbf5HmkUbtzsCftvu2NwS2nOMhm55xcCsAunB+0uY+Bj4jqyqRvWUjA99yHIdQH5JTJXE4FQPxdBTmBFp8xVdYWvlnCojK4qfv5EurLUunBugdSdYidi2TKYEoW2mX19RdVnkQgBjW9caaZUX9BIi+zNhObKenoWxRORCb0dHw72mwXMaSQM4IDygcuwi/79OtkYik3juECVmI58bu/GFRo2NtmgkGdMD29SMV6rerDtYjGjetgFvWkDy6hrB/Vz4dI37eGt/EH/7G0z/IT9Q2sM8nzJDtCYdicepLQ1WaM0P4BEn2rrqhv81VvDH7M9e7wtz4xly5RYQUr5wkzvoSEIglFekPvPmhIxEsNdaBj6MlVmLTbEbv85oBLwm0ZLRJcgUsrBbZghGKQaI5udAk1dgwOng9v0Ai+ilJ2RSt2VI6WG1rwRnDG+Hv+H1cKT/NwA8ocmM/kxQlUE3BxESrxhTnKq6W0rzPInIj+xeS8v1X4sloGJIiweCCI8YrqUnmYKKQ6Rtd5B4jS5fAa8jp274/R4xvLaR2r70wKp890Iwfb948PwqjNpH/XlOUcfIxPu3jHO65OEyX3OG/+qm+g91AmxlrwZRDYeSioHztz55Du+JKX2Twnb5ttEwVm6I7eQWGbzw50Qgg6I0cHM1YRrXv/h35C8pkXxV7vMlyR0g5ELAoRWYkuveniruKNIIPBw6SY81RtjIIoPvKzNAFssNdiNxfYEsHiDlfK9kUvTLs8qREmLt1rzgD/vHtanmKh7WMtHN42tSUp+NhoHPlEyaaJLUMkFS7suh2S9xPmhisw61npROXrc2PBxLR52i8t0cJYFa1XuZ40KMCZ7kw0UmKiUxeFDTlZkaOTocVqw3qLY906argYnVdkpaoJMUXsnE3JmXch9HG1qpP+hY8LJSOl5ejAiQADHYDm8P/+9VVlNL6eo2O6qFUs9lC+I8hVeSOvJd8+aLeWisFeboKjarhw/52eXwZdTEpPHa7yMDsIQgciqN82yagJYrEL6BevvZ/kGHP5TszYYfcKcN28k9y/R2JYND2TvYeIKb2Nmj9rhwuXBOFnlep4Exw1QS0If80gJG1BCub+u6GKk9FaFHUSH/x3MIRHwVzCHRnZxqbwc1uGZ2I3rRxVq5MHo5p7htS/KYi7+cQjSdRvfdlCofo6YKP38qQwh9y9yAUqMEOABtjUaEJSeWlGq2yh0onm+Uqm54fcCSr7S5MvBTZ1oblLqVUrZBDTIkY5cdbuiyyoj7GPLHrxUo25yiGI+FXHKy136E0Rv7p5N8o3U5O8ycPdk5vOVYgvM3pdLS1zHHsVYIYY/Y/svtDUnMhJONl5AwQ2sDZ2GiWx1nWJ4VumLuwb/Yow/BHtnW6SOeWFSwRkgJdNL5mzrNoexytbrDaSWR0IpuLUbvK/Gi4MJaEkm1acQB1CMOjIymVu1pcih0H1Zl4O+fhQM+bVke6NKrBSUKErHWca9iNH16Du7iXBbaK0Dz73nIhALJKCivr1rWviXXxeQuzTFMRXdK/SalJySHM7ov6qOikQTG/Q97zQ+UYiUnLo3UxlWngXrnzGy8fAb58v9bHhEtnvCrAhiJTer0x4GYQJGwYjQDbN9ucpu00PDnQU0GqYJHqCf4Vm9Y3SYTs92T6Lftvu3nw/ZH6PblcznC6sueEPON5QzzQppepezdolRNyx4g8XYsI+XJlCNrExWxDxErLBTOm3f7aB19RYCS1tG0G/aKEJTXh9RE5VyWSyBzPNMuLZF9HIIHp3yGpqGM7xsYKsMtngBiyXC+i/C/geIgUFJym/sqWn4iPDWtd0YeELg46B5krcolTg87KEeiqDAebieqL5ZiftnVi10kV2L2XPs4ofJjpVI1Q53RsgT89zWXYv97DTZeDwSE8FY2vbADnUpGk/UZ2WMQlX8iwFOTo1woE6AHyqrq6kzGamUGsyoAarqmJk77o2ktzO2M4OqsVqJErSWTcCLZOlxsDWUn/OsQX5+9vaAhHwDHMmrqWo6cJuAtvaLYOHVB/BUOiI/MJLSYeDIrKUML3Ew0GbX89jkR45SPjRdsXQ/oVCJDxm7WjF9jd6bly2wPTmWCzoqeTJU48jPk47tDHWA7QjLEah6PajxbAs/xSN4c2uK9ltR3K/urNpOTYwsFMLgRfzontPe8RMB9A6aZcwQnIhLTpgX8fw/j+aqwqJwVcIk57/KvcOksBjN4rwPQ78kCmpLJ/KEjYh1z8EwIdVX3UDvq8c8YSZnzwsTAVfhWxigvLGzQDTXrr8aNMDXwB2iUVwhNdGw/E1UCSvsKbvw+p9YRaCAzPnnKa8AVAJySAc1Ys/DAYskdKYaTPUwdCXyZfLzfOsFvEzBxqrRgxxK77ij5UQXKINjQpAIP6pkDze+JTwWQsPfcMZ5xtb2tpiUF9FvCVqR1dfblppq3/f3ZHSl7f4s/sOM2/ZRkiI4m8yxhWDMtauxyrnkgiLsOBUcK4MclD5E3H6KtaJODFiCyH+NPtAMmaZ1iduRM51qxmrHlcI5B4ErJum/iZg+zIokXqIi/U6B+tbjj0EKugN+5V03JaoTvdaGQKNvJHnyP0odFN05ZgZnYCKGGh41EGyCZ8/lmrNbAfZRPjRtHw5ypBUiW8Yz+SkumgWA637ASf1AklCNs9MBBuOPmGckDWy8zTvUCgWCwYehIjKSQ+1hYoWpraLNOZy7tHlTQZ7G5zIFR3cJljO4bfmpAGDsSeDQtBsAI90gD4P3ipKFXVQTfl7c3CD1kB8TiOpG0xVGlYQ4dP15z1wTUfHRz82eFxfwn2fc/R/AKmINq9P5PEL31jhLM+gTxctb2BiGn8cCRuorLCg9FtkwC4jy2AluVj6uZZOGvDCYyGqFsdObgwb+FdIoCYDdeZXo+1dxlxhL1258L6h5+yJBgSy6QDN26N6MMNnK2iD7QTQwtB9CfaX3Bk6T0nZ0TT6zSxu7ix7UG+E5AKwoMVcD0EuLlmJ0IhLI+4ZZ8z/QUIzeTZ9ve+r8uyHKmWr2oR8ec5m95PGG5JzWmw30sDelFFl6S3KxsHwzlJsXeXbE5I/Z0RWZQBa8xPbmY54qYXPzp3O+9WYys6A697UzL6fcCORYokB/TR5exBS2saW4OhkwX5ss9aUJGZiMFSgfF1paxqXkGIx3IuDH5xP3QxhGxNMB+T+Yn+meApf2A00dEUng1HRtgBU5Mk/TM19g8l3nVtwTrmAZDm50jIn+G51GpeoBeYPONhGfb1Ngao383TPkvShaNujE3zsJLfJCrNda76rIechIhAC25wWauUMGUU8daL8cdtJnAEeidnolR+lVf5WFKUxvO36X84FCSaR4PqzvlNk7hOvd607h7IGOEVx7/XRLpWDDrrT6hpY653lNf3RCsxdQ7kS9e5hi/RpU1mf9mdTOsf+AtHkkog336jKdyl1/plDV/AlRS8/QjlPfeyvGmA6gEC8cZmBkvyZPbs81tYmif8q2Amd9La3wldh3evYOOkC/b1QQ3fLryoeUl5/9oVprW/aEfmcTxG9CytaFfsx6cAejAb7g6fz/iaWE57fXfth7K7ATi827ZwAs0XX9lLkyGn60ePtqg771isK5/v1jCsbDQDwHwNLehH+BX9zZWWjaQxK85/o09Oaiy0oGY1nivv5kHelmGwj8qqd,iv:a4NHhErZolhOuzNCUU51GQiCfCT8GzZ3GQ24s7fkuyU=,tag:w+GrXmaYWADTrBxMko3xtg==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjQmk5aE1GN2dpVlREZU1s
N3V5ZUQweWxMaVo3Y1VjdzZTYW96eFBYZGxvCnNJcjVXKzI4VnliTWFXQy9iSy9k
eEZmWlVPQzhZK09NZGowZUVXc1BoZTQKLS0tIHIrMEpTM0lMU0w0eG02aUxJY2VZ
aUQ0cmkvK0UwdzhqcDJwM1RDcnpEU0UKQRiTa9/yNrqlkIdlNeaIDzIKa/xvx5nW
7wTNfjq2M3go0DMnaVjJA3e15JZBOpOs0QP9z9ydKF/FS4PeCIEFyQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
encrypted_regex: ^(data|stringData)$
lastmodified: "2026-10-07T04:27:33Z"
mac: ENC[AES256_GCM,data:CxZdlGJdIFFTwaeeS9LW2XDDgvUmSsBeBKtFYyDM1JliS8BHtgEv48yuCWxP5sR3SaYtcgNSIQ55zUa89jM6dQHqvjK+1jSYu1xzFWPOlxUu8gVSlU0QLS4GU1RChXuPpTUOjxzn4ymZukGgclZk9iuMvI0/U+nGD8nvs8k0zpM=,iv:MGZ7YsOsjelJPjLYWtALOrM6r8Vd1HgbU/PKL4c3r3k=,tag:dIE38TCobESY3j56NVEvIw==,type:str]
version: 3.13.2
@@ -5,19 +5,19 @@ metadata:
namespace: iam namespace: iam
type: Opaque type: Opaque
stringData: stringData:
masterkey: ENC[AES256_GCM,data:dGp0wSCUJORP6W5PHLKYaQEqrZsgiAwU1GqcuvuZV+g=,iv:Wwt6E/XXwsoVEQxe4gG1m5oJLOcNUXngftADQvZQepw=,tag:1Vget5gs+t455U9zLavdwg==,type:str] masterkey: ENC[AES256_GCM,data:Da543Jt48o6p3f6/AbZRsjChz3NexPJoi/mC/LB6jUI=,iv:irS0n76tZ6QzoQMm0dWrvbw+SPfXz4h13jQb/Bjfgwg=,tag:tfbgaAdxmPIjleB1OXLn0Q==,type:str]
sops: sops:
age: age:
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBld0tUWHNWem13cDlxRG14 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOZld2VG9NaFhIVWpBcGpO
QWRlTjA5SzZnS2pBd2M0WWkzQWxKZW9HZWdJCk00QWJJcWZFM1ZTSUlReUlGZG4z SmVwNm9FT2wrQmFKb0RGQmY4TCtITjFEZjBjCmNuNXNJYTdsZE44cDNzR2hiNTdy
RFBSUWRlVUx2NGN2OFAwazBYRTVVNkEKLS0tIHpSSW9IWHhNa0d3SmEydXU4akVj QWlVdzBtdjd0cSsxUjQxMzNyblhWTkkKLS0tIHBLT2x1MDBGdjE1eHFOdHVGa1NO
MGJkN2U4WmtLbVFMZm55U1hEZElZMlUKKV9twnC9GC7MSmNLxEBHbPl4RN5Hsd+z ZGM1K2lDTksraFdPeFdSZ1JyU1VtcjAKgTjjhr7u4YRuQVZQ7kAa1clJ0Rzy10hQ
NViCKxUoQAB8apn2XVHZcDnyfKNfr1P3aSa3IIHyVY3prgo1OFDpdw== Umcd4/YStyXHZjsJJpFSpOFdP5cZscGo3xBgpXtxAscKTS1Nw4derg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
encrypted_regex: ^(data|stringData)$ encrypted_regex: ^(data|stringData)$
lastmodified: "2026-10-24T08:57:27Z" lastmodified: "2026-10-25T01:24:21Z"
mac: ENC[AES256_GCM,data:hXDAddVmnlf/yinY2c2Q8yfsjqi6n9cRbEbwykqYScd6mDPfdF7BriTiu9K+CV/IvIT4zcMTZbhBeQ/vTCAh0TZDp1lOVglwtNH/imgagyzPmcCvbCIkZKwF5vwWQIGHcDI6pk4/zVQkqt7zKTf8+Qj0pNh5UrAjVeEFE7cuJkQ=,iv:oIQvROFGbiESmMTthHOJiNM+Gy5BCfM5XtDWbszOPmc=,tag:AaYrJrwXuKD7w0p+TTa7FA==,type:str] mac: ENC[AES256_GCM,data:QFVk7KLsMLCAMH/PHVGCRyfU3k2JiO8AE3jJXmhnlUB4C8MYCTyOWQbnTFXozCMOr0kp+hAt3dYn/e23SWhyvJ7BgVLRb5QF1py704Dr46atVoLUvwHpFI7dyGK/LDTxqiM40RXAnDtWUJlTkVCzKdSFiV8CokIoByKWwxFPayM=,iv:t7HWMcpQl/DJ8Rp51Du7b5gURZO+C+FodsZSJuy5jBc=,tag:qEXSS4S0Y5cvn7T6EjakhA==,type:str]
version: 3.13.2 version: 3.13.2
+4 -1
View File
@@ -55,7 +55,10 @@ alertmanager:
prometheus: prometheus:
prometheusSpec: prometheusSpec:
retention: 30d retention: 30d
retentionSize: "95GB" # Must stay under the PVC size below (25Gi). At 95GB the limit never
# triggered, the volume filled, and Prometheus crashlooped with
# "no space left on device" writing /prometheus/queries.active.
retentionSize: "18GB"
# scrapeTimeout MUST be <= scrapeInterval or the operator refuses to generate # scrapeTimeout MUST be <= scrapeInterval or the operator refuses to generate
# the Prometheus config ("scrapeTimeout greater than scrapeInterval") and no # the Prometheus config ("scrapeTimeout greater than scrapeInterval") and no
-26
View File
@@ -31,29 +31,3 @@ roleRef:
apiGroup: rbac.authorization.k8s.io apiGroup: rbac.authorization.k8s.io
kind: Role kind: Role
name: oidc-bootstrap name: oidc-bootstrap
---
apiVersion: batch/v1
kind: Job
metadata:
name: oidc-bootstrap-restart-apiserver-v2
namespace: kube-system
spec:
backoffLimit: 4
ttlSecondsAfterFinished: 7200
template:
spec:
serviceAccountName: oidc-bootstrap
restartPolicy: Never
containers:
- name: restart
image: docker.io/curlimages/curl:8.22.0
command: ["/bin/sh", "-c"]
args:
- |
set -u
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
for pod in kube-apiserver-talos-cp-1 kube-apiserver-talos-cp-2 kube-apiserver-talos-cp-3; do
curl -sS --max-time 15 --cacert "$CA" -H "Authorization: Bearer $TOKEN" -X DELETE "https://kubernetes.default.svc/api/v1/namespaces/kube-system/pods/$pod?gracePeriodSeconds=0" -o /dev/null -w "$pod -> HTTP %{http_code}\n" || echo "$pod -> request failed"
done
echo "deletes requested; kubelet recreates each static pod in ~20-40s"
@@ -1,141 +0,0 @@
# One-shot: restart the control-plane API servers so their OIDC authenticator initialises.
#
# Two earlier mechanisms failed and are documented so nobody repeats them:
# 1. deleting the mirror pod does NOT restart a static pod (deletes returned 200, no restart);
# 2. matching /proc/*/cmdline for 'kube-apiserver' also matched this script's own cmdline,
# so every pod SIGTERMed itself (exit 143) before reaching the real process.
# This version matches /proc/<pid>/comm (the process name) and skips \$\$, so self-kill is
# impossible. hostPID + privileged are permitted in the 'cicd' namespace. Delays stagger the
# three control planes by 2 minutes each, so only one API server is down at a time.
---
apiVersion: batch/v1
kind: Job
metadata:
name: oidc-restart-apiserver-cp1-v2
namespace: cicd
spec:
backoffLimit: 0
ttlSecondsAfterFinished: 3600
template:
spec:
hostPID: true
restartPolicy: Never
nodeSelector:
kubernetes.io/hostname: talos-cp-1
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
containers:
- name: kill
image: docker.io/library/busybox:1.36
securityContext:
privileged: true
command: ["/bin/sh", "-c"]
args:
- |
echo "sleeping 10s before restarting the API server on talos-cp-1"
sleep 10
self=$$
killed=0
for d in /proc/[0-9]*; do
pid=${d#/proc/}
[ "$pid" = "$self" ] && continue
comm=$(cat "$d/comm" 2>/dev/null)
[ "$comm" = "kube-apiserver" ] || continue
echo "SIGTERM -> pid $pid"
kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
done
echo "signalled $killed API server process(es); kubelet recreates the static pod"
---
apiVersion: batch/v1
kind: Job
metadata:
name: oidc-restart-apiserver-cp2-v2
namespace: cicd
spec:
backoffLimit: 0
ttlSecondsAfterFinished: 3600
template:
spec:
hostPID: true
restartPolicy: Never
nodeSelector:
kubernetes.io/hostname: talos-cp-2
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
containers:
- name: kill
image: docker.io/library/busybox:1.36
securityContext:
privileged: true
command: ["/bin/sh", "-c"]
args:
- |
echo "sleeping 120s before restarting the API server on talos-cp-2"
sleep 120
self=$$
killed=0
for d in /proc/[0-9]*; do
pid=${d#/proc/}
[ "$pid" = "$self" ] && continue
comm=$(cat "$d/comm" 2>/dev/null)
[ "$comm" = "kube-apiserver" ] || continue
echo "SIGTERM -> pid $pid"
kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
done
echo "signalled $killed API server process(es); kubelet recreates the static pod"
---
apiVersion: batch/v1
kind: Job
metadata:
name: oidc-restart-apiserver-cp3-v2
namespace: cicd
spec:
backoffLimit: 0
ttlSecondsAfterFinished: 3600
template:
spec:
hostPID: true
restartPolicy: Never
nodeSelector:
kubernetes.io/hostname: talos-cp-3
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
containers:
- name: kill
image: docker.io/library/busybox:1.36
securityContext:
privileged: true
command: ["/bin/sh", "-c"]
args:
- |
echo "sleeping 240s before restarting the API server on talos-cp-3"
sleep 240
self=$$
killed=0
for d in /proc/[0-9]*; do
pid=${d#/proc/}
[ "$pid" = "$self" ] && continue
comm=$(cat "$d/comm" 2>/dev/null)
[ "$comm" = "kube-apiserver" ] || continue
echo "SIGTERM -> pid $pid"
kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
done
echo "signalled $killed API server process(es); kubelet recreates the static pod"
+12
View File
@@ -0,0 +1,12 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Stable secret name: the job mounts talos-repair-config by name, so the generated
# Secret must not get a hash suffix (same convention as k8s/argocd/secrets).
generatorOptions:
disableNameSuffixHash: true
# ksops decrypts talosconfig-secret.enc.yaml into the Secret the job mounts.
# Kept local to this app on purpose: no edits to the shared secrets generator.
generators:
- secret-generator.yaml
resources:
- restart-kubelet.yaml
@@ -0,0 +1,57 @@
# One-shot repair: the kubelets on cp-1, cp-2 and worker-1 stopped posting node status at
# 04:11-04:14 UTC and never reconnected, so those nodes are NotReady/unreachable and every
# pod on them (Cilium included) shows NotReady. This job runs on the one healthy node (cp-3),
# talks to the Talos API on the LAN, and uses Talos' endpoint->node proxying to reach the
# other three: it reports the kubelet service state, then restarts it.
apiVersion: batch/v1
kind: Job
metadata:
name: talos-restart-stalled-kubelets
namespace: cicd
spec:
backoffLimit: 1
ttlSecondsAfterFinished: 86400
template:
spec:
restartPolicy: Never
nodeSelector:
kubernetes.io/hostname: talos-cp-3
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
- name: talosctl
image: docker.io/curlimages/curl:8.22.0
command: ["/bin/sh", "-c"]
args:
- |
set -u
echo "fetching talosctl v1.13.3 (matches the node version)"
curl -sSL -o /tmp/talosctl \
https://github.com/siderolabs/talos/releases/download/v1.13.3/talosctl-linux-amd64 \
|| { echo "DOWNLOAD FAILED"; exit 1; }
chmod +x /tmp/talosctl
T="/tmp/talosctl --talosconfig /talos/config -e 192.168.1.162"
NODES="192.168.1.166 192.168.1.214 192.168.1.223"
echo "=== BEFORE: kubelet service state ==="
for n in $NODES; do echo "--- $n"; $T -n $n service kubelet 2>&1 || true; done
echo "=== kernel log tail (last 15 lines per node) ==="
for n in $NODES; do echo "--- $n"; $T -n $n dmesg --tail 15 2>&1 | tail -15 || true; done
echo "=== ACTION: restart kubelet ==="
for n in $NODES; do echo "--- restarting $n"; $T -n $n restart kubelet 2>&1 || true; sleep 20; done
sleep 60
echo "=== AFTER ==="
for n in $NODES; do echo "--- $n"; $T -n $n service kubelet 2>&1 || true; done
volumeMounts:
- name: talos
mountPath: /talos
readOnly: true
volumes:
- name: talos
secret:
secretName: talos-repair-config
defaultMode: 0444
items:
- key: config
path: config
@@ -0,0 +1,10 @@
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
name: talos-repair-secret-generator
annotations:
config.kubernetes.io/function: |
exec:
path: ksops
files:
- talosconfig-secret.enc.yaml
@@ -0,0 +1,23 @@
apiVersion: v1
kind: Secret
metadata:
name: talos-repair-config
namespace: cicd
type: Opaque
stringData:
config: ENC[AES256_GCM,data:WO+jinh/2Rq5VeGLoDDJOQL4zfYJINSTYZCwGWndbzfIHw4XRwxDB7BixjCdpPuLOkNIBAXHQpKPBhlQpvRTso70L26njgVdVENUGeNhxvb6/cwl3VYRsfQQxKoQW9Qu6DyCD7ItuyazXMdIbENKBoih7GA2IjKdGeEtpB+a6PO+6rpI0AQRIjM9h65yJ6gvWeLZmb6U0qCFW61SelHmVSsfehxH/R9EKQROKwD0Z/TgYnoS28OuDtixAQVfnrI3gtgUAeXgnPQTJf1Vb9iwbRBvaSWkPSIqa7hOqWEIM2qyrvhTsEb8XDRUHTmp4BCxYktIxoiSWZEbeh0y0HKRzekZIj4rHA+c/JYSPqS0dvn1Ga+8PTc1LNdXDWIcKI58YQ2jzA0oE2vm0YzmOjcDTzTGLDBk49gIql+AMfwoxCop7t1PB7BrJw6M4bj9M5aHIq8ZlGXDvtWZcSEBj0ChZ0q20cB4v4bvMkbuJR/PS25uNkeG+UBCPidBQqCUw37aNf8SDwOaAC+6PeHdLmcJ/dkKntGFlU5MQzWQXfqs/fp95ze3mr7f3et7SqVl7fAlYSdEt6dA2xxnFnqgNwAm3UacV/We6SRnjrFlBAHZ3Q5hIuKkxmszsS4Bh/VuNtZI5YJ4ldnoIZC3aWKDt+rVB6P+Qxp1T1XqZj8GY715DZRP1q+Z7iQ9RcXiJBTD5qWVpnkESOvbTEYv/W3oXTufVRrJ8Ps0on/xWRF5NU04+tqvToC8+MIQDz5rb+eXYCEWjZG9ZhInY+M2HRTLg+c49dRlyqVnnqfNWBejSS/hoQbf5HmkUbtzsCftvu2NwS2nOMhm55xcCsAunB+0uY+Bj4jqyqRvWUjA99yHIdQH5JTJXE4FQPxdBTmBFp8xVdYWvlnCojK4qfv5EurLUunBugdSdYidi2TKYEoW2mX19RdVnkQgBjW9caaZUX9BIi+zNhObKenoWxRORCb0dHw72mwXMaSQM4IDygcuwi/79OtkYik3juECVmI58bu/GFRo2NtmgkGdMD29SMV6rerDtYjGjetgFvWkDy6hrB/Vz4dI37eGt/EH/7G0z/IT9Q2sM8nzJDtCYdicepLQ1WaM0P4BEn2rrqhv81VvDH7M9e7wtz4xly5RYQUr5wkzvoSEIglFekPvPmhIxEsNdaBj6MlVmLTbEbv85oBLwm0ZLRJcgUsrBbZghGKQaI5udAk1dgwOng9v0Ai+ilJ2RSt2VI6WG1rwRnDG+Hv+H1cKT/NwA8ocmM/kxQlUE3BxESrxhTnKq6W0rzPInIj+xeS8v1X4sloGJIiweCCI8YrqUnmYKKQ6Rtd5B4jS5fAa8jp274/R4xvLaR2r70wKp890Iwfb948PwqjNpH/XlOUcfIxPu3jHO65OEyX3OG/+qm+g91AmxlrwZRDYeSioHztz55Du+JKX2Twnb5ttEwVm6I7eQWGbzw50Qgg6I0cHM1YRrXv/h35C8pkXxV7vMlyR0g5ELAoRWYkuveniruKNIIPBw6SY81RtjIIoPvKzNAFssNdiNxfYEsHiDlfK9kUvTLs8qREmLt1rzgD/vHtanmKh7WMtHN42tSUp+NhoHPlEyaaJLUMkFS7suh2S9xPmhisw61npROXrc2PBxLR52i8t0cJYFa1XuZ40KMCZ7kw0UmKiUxeFDTlZkaOTocVqw3qLY906argYnVdkpaoJMUXsnE3JmXch9HG1qpP+hY8LJSOl5ejAiQADHYDm8P/+9VVlNL6eo2O6qFUs9lC+I8hVeSOvJd8+aLeWisFeboKjarhw/52eXwZdTEpPHa7yMDsIQgciqN82yagJYrEL6BevvZ/kGHP5TszYYfcKcN28k9y/R2JYND2TvYeIKb2Nmj9rhwuXBOFnlep4Exw1QS0If80gJG1BCub+u6GKk9FaFHUSH/x3MIRHwVzCHRnZxqbwc1uGZ2I3rRxVq5MHo5p7htS/KYi7+cQjSdRvfdlCofo6YKP38qQwh9y9yAUqMEOABtjUaEJSeWlGq2yh0onm+Uqm54fcCSr7S5MvBTZ1oblLqVUrZBDTIkY5cdbuiyyoj7GPLHrxUo25yiGI+FXHKy136E0Rv7p5N8o3U5O8ycPdk5vOVYgvM3pdLS1zHHsVYIYY/Y/svtDUnMhJONl5AwQ2sDZ2GiWx1nWJ4VumLuwb/Yow/BHtnW6SOeWFSwRkgJdNL5mzrNoexytbrDaSWR0IpuLUbvK/Gi4MJaEkm1acQB1CMOjIymVu1pcih0H1Zl4O+fhQM+bVke6NKrBSUKErHWca9iNH16Du7iXBbaK0Dz73nIhALJKCivr1rWviXXxeQuzTFMRXdK/SalJySHM7ov6qOikQTG/Q97zQ+UYiUnLo3UxlWngXrnzGy8fAb58v9bHhEtnvCrAhiJTer0x4GYQJGwYjQDbN9ucpu00PDnQU0GqYJHqCf4Vm9Y3SYTs92T6Lftvu3nw/ZH6PblcznC6sueEPON5QzzQppepezdolRNyx4g8XYsI+XJlCNrExWxDxErLBTOm3f7aB19RYCS1tG0G/aKEJTXh9RE5VyWSyBzPNMuLZF9HIIHp3yGpqGM7xsYKsMtngBiyXC+i/C/geIgUFJym/sqWn4iPDWtd0YeELg46B5krcolTg87KEeiqDAebieqL5ZiftnVi10kV2L2XPs4ofJjpVI1Q53RsgT89zWXYv97DTZeDwSE8FY2vbADnUpGk/UZ2WMQlX8iwFOTo1woE6AHyqrq6kzGamUGsyoAarqmJk77o2ktzO2M4OqsVqJErSWTcCLZOlxsDWUn/OsQX5+9vaAhHwDHMmrqWo6cJuAtvaLYOHVB/BUOiI/MJLSYeDIrKUML3Ew0GbX89jkR45SPjRdsXQ/oVCJDxm7WjF9jd6bly2wPTmWCzoqeTJU48jPk47tDHWA7QjLEah6PajxbAs/xSN4c2uK9ltR3K/urNpOTYwsFMLgRfzontPe8RMB9A6aZcwQnIhLTpgX8fw/j+aqwqJwVcIk57/KvcOksBjN4rwPQ78kCmpLJ/KEjYh1z8EwIdVX3UDvq8c8YSZnzwsTAVfhWxigvLGzQDTXrr8aNMDXwB2iUVwhNdGw/E1UCSvsKbvw+p9YRaCAzPnnKa8AVAJySAc1Ys/DAYskdKYaTPUwdCXyZfLzfOsFvEzBxqrRgxxK77ij5UQXKINjQpAIP6pkDze+JTwWQsPfcMZ5xtb2tpiUF9FvCVqR1dfblppq3/f3ZHSl7f4s/sOM2/ZRkiI4m8yxhWDMtauxyrnkgiLsOBUcK4MclD5E3H6KtaJODFiCyH+NPtAMmaZ1iduRM51qxmrHlcI5B4ErJum/iZg+zIokXqIi/U6B+tbjj0EKugN+5V03JaoTvdaGQKNvJHnyP0odFN05ZgZnYCKGGh41EGyCZ8/lmrNbAfZRPjRtHw5ypBUiW8Yz+SkumgWA637ASf1AklCNs9MBBuOPmGckDWy8zTvUCgWCwYehIjKSQ+1hYoWpraLNOZy7tHlTQZ7G5zIFR3cJljO4bfmpAGDsSeDQtBsAI90gD4P3ipKFXVQTfl7c3CD1kB8TiOpG0xVGlYQ4dP15z1wTUfHRz82eFxfwn2fc/R/AKmINq9P5PEL31jhLM+gTxctb2BiGn8cCRuorLCg9FtkwC4jy2AluVj6uZZOGvDCYyGqFsdObgwb+FdIoCYDdeZXo+1dxlxhL1258L6h5+yJBgSy6QDN26N6MMNnK2iD7QTQwtB9CfaX3Bk6T0nZ0TT6zSxu7ix7UG+E5AKwoMVcD0EuLlmJ0IhLI+4ZZ8z/QUIzeTZ9ve+r8uyHKmWr2oR8ec5m95PGG5JzWmw30sDelFFl6S3KxsHwzlJsXeXbE5I/Z0RWZQBa8xPbmY54qYXPzp3O+9WYys6A697UzL6fcCORYokB/TR5exBS2saW4OhkwX5ss9aUJGZiMFSgfF1paxqXkGIx3IuDH5xP3QxhGxNMB+T+Yn+meApf2A00dEUng1HRtgBU5Mk/TM19g8l3nVtwTrmAZDm50jIn+G51GpeoBeYPONhGfb1Ngao383TPkvShaNujE3zsJLfJCrNda76rIechIhAC25wWauUMGUU8daL8cdtJnAEeidnolR+lVf5WFKUxvO36X84FCSaR4PqzvlNk7hOvd607h7IGOEVx7/XRLpWDDrrT6hpY653lNf3RCsxdQ7kS9e5hi/RpU1mf9mdTOsf+AtHkkog336jKdyl1/plDV/AlRS8/QjlPfeyvGmA6gEC8cZmBkvyZPbs81tYmif8q2Amd9La3wldh3evYOOkC/b1QQ3fLryoeUl5/9oVprW/aEfmcTxG9CytaFfsx6cAejAb7g6fz/iaWE57fXfth7K7ATi827ZwAs0XX9lLkyGn60ePtqg771isK5/v1jCsbDQDwHwNLehH+BX9zZWWjaQxK85/o09Oaiy0oGY1nivv5kHelmGwj8qqd,iv:a4NHhErZolhOuzNCUU51GQiCfCT8GzZ3GQ24s7fkuyU=,tag:w+GrXmaYWADTrBxMko3xtg==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjQmk5aE1GN2dpVlREZU1s
N3V5ZUQweWxMaVo3Y1VjdzZTYW96eFBYZGxvCnNJcjVXKzI4VnliTWFXQy9iSy9k
eEZmWlVPQzhZK09NZGowZUVXc1BoZTQKLS0tIHIrMEpTM0lMU0w0eG02aUxJY2VZ
aUQ0cmkvK0UwdzhqcDJwM1RDcnpEU0UKQRiTa9/yNrqlkIdlNeaIDzIKa/xvx5nW
7wTNfjq2M3go0DMnaVjJA3e15JZBOpOs0QP9z9ydKF/FS4PeCIEFyQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
encrypted_regex: ^(data|stringData)$
lastmodified: "2026-10-07T04:27:33Z"
mac: ENC[AES256_GCM,data:CxZdlGJdIFFTwaeeS9LW2XDDgvUmSsBeBKtFYyDM1JliS8BHtgEv48yuCWxP5sR3SaYtcgNSIQ55zUa89jM6dQHqvjK+1jSYu1xzFWPOlxUu8gVSlU0QLS4GU1RChXuPpTUOjxzn4ymZukGgclZk9iuMvI0/U+nGD8nvs8k0zpM=,iv:MGZ7YsOsjelJPjLYWtALOrM6r8Vd1HgbU/PKL4c3r3k=,tag:dIE38TCobESY3j56NVEvIw==,type:str]
version: 3.13.2