From f3015522e3b1ed669a4c5349fdef3a6471d062d8 Mon Sep 17 00:00:00 2001 From: rock Date: Tue, 6 Oct 2026 01:48:57 +0000 Subject: [PATCH 1/2] fix(storage): pin minio tenant to a published image release (#53) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Why `minio/minio:RELEASE.2025-07-23T15-54-02Z` no longer exists. The Docker Hub repository `minio/minio` returns **404 for every tag** (checked: the pinned tag and `latest`), and MinIO has moved its images behind registry authentication (`quay.io/minio/minio` → 401, `registry.min.io/minio/minio` → auth required, `dl.min.io` binaries → 410 Gone). The tenant therefore sat in `ImagePullBackOff` — 44,229 attempts over 6d23h — and the chain from there is: ``` minio tenant pod not ready (1/2) -> headless Service minio-cluster-hl has no endpoints -> CoreDNS: lookup minio-cluster-hl.storage.svc.cluster.local -> no such host -> Vault S3 storage init fails at startup -> vault-0 CrashLoopBackOff, 1943 restarts, ~7 days ``` ## Control checks (so this is not a network story) | check | result | | --- | --- | | `alpine:3.20` pulled on talos-cp-1 | Succeeded — docker.io works | | `quay.io/prometheus/node-exporter:v1.8.2` on the same node | Succeeded — public quay works | | `minio/minio` (docker.io) | HTTP 404, repo removed | | old tag `RELEASE.2024-06-13T20-48-48Z` with `IfNotPresent` | not cached on the node | ## Change Pin `spec.image` to `coollabsio/minio:RELEASE.2025-10-15T17-29-55Z` — a public mirror, and a newer upstream release than the previous pin. Verified by pulling it on talos-cp-1 and running it: the image carries `/usr/bin/docker-entrypoint.sh` (the layout the operator expects, which runs `minio server … --console-address :9090`) and reports `minio version RELEASE.2025-10-15T17-29-55Z`. ## Caveat This is a community mirror, not MinIO's official image, because the official ones now require credentials. If MinIO registry credentials are available, add an imagePullSecret and switch back to `quay.io/minio/minio`. Supply-chain decision, flagged deliberately. ## After merge (ArgoCD, not kubectl) The storage app tracks `k8s/infra/minio`, so the sync rolls the tenant; then: ``` kubectl get pods -n storage # minio-cluster-az-a-0 -> 2/2 kubectl get pods -n iam | grep vault # vault-0 -> Running (storage init succeeds) ``` Residual risk, unrelated to this fix: the tenant's 4 x 25Gi volumes are Longhorn-backed, and Longhorn currently fails to create engines for new volume attaches cluster-wide. If the restarted pod cannot attach, that is a second, separate fault.Reviewed-on: https://forgejo.riotpiao.com/riotpiao-poimen/homelab/pulls/53 Co-authored-by: rock --- k8s/infra/minio/minio-tenant.yaml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/k8s/infra/minio/minio-tenant.yaml b/k8s/infra/minio/minio-tenant.yaml index e756d30..810821e 100644 --- a/k8s/infra/minio/minio-tenant.yaml +++ b/k8s/infra/minio/minio-tenant.yaml @@ -11,7 +11,12 @@ metadata: prometheus.io/port: "9000" prometheus.io/scrape: "true" spec: - image: minio/minio:RELEASE.2025-07-23T15-54-02Z + # Was minio/minio:RELEASE.2025-07-23T15-54-02Z — that Docker Hub repository no longer + # exists (404 for every tag), so the tenant sat in ImagePullBackOff and Vault's S3 + # backend could not resolve minio-cluster-hl. Official images now need registry auth + # (quay.io/minio/minio returns 401), so this pins a public mirror of the same upstream + # release; swap it for the official image once MinIO registry credentials exist. + image: coollabsio/minio:RELEASE.2025-10-15T17-29-55Z # Disable operator auto-TLS: MinIO serves plain HTTP internally on 9000. # External TLS is terminated at nginx ingress (wildcard riotpiao-com-tls cert). From b6e317b34ce657157cc48904b78d23d33705c996 Mon Sep 17 00:00:00 2001 From: rock Date: Tue, 6 Oct 2026 01:49:12 +0000 Subject: [PATCH 2/2] feat(poimen): pi sandbox workspace volume, encrypted api keys, cluster dns rewrite (#54) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the pi coding-agent sandbox's cluster side: - `k8s/infra/poimen/pi-agent-workspace-pv.yaml` — static hostPath PV + PVC for the sandbox /workspace (Longhorn cannot attach new volumes in this cluster; this gives real persistence and survived a pod delete/recreate test: marker file + 38 KB session.jsonl intact, the session answered from its own history afterwards). - `k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml` — the sandbox's API-key Secret, SOPS-encrypted (recipient from .sops.yaml, payload only: `encrypted_regex ^(data|stringData)$`), registered in the ksops generator. Round-trip verified against the live Secret. - `terraform/files/coredns/Corefile` — `rewrite name pi.riotpiao.com ingress-nginx-controller...` so the hostname resolves in-cluster. Note: the keys currently in that Secret are the ones leaked earlier and must be rotated; re-encrypt with `sops -e -i` after rotating.Reviewed-on: https://forgejo.riotpiao.com/riotpiao-poimen/homelab/pulls/54 Co-authored-by: rock --- .../secrets/pi-sandbox-secrets.enc.yaml | 24 ++++++++ k8s/argocd/secrets/secret-generator.yaml | 1 + k8s/infra/poimen/kustomization.yaml | 1 + k8s/infra/poimen/pi-agent-workspace-pv.yaml | 57 +++++++++++++++++++ terraform/files/coredns/Corefile | 1 + 5 files changed, 84 insertions(+) create mode 100644 k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml create mode 100644 k8s/infra/poimen/pi-agent-workspace-pv.yaml diff --git a/k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml b/k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml new file mode 100644 index 0000000..55b1359 --- /dev/null +++ b/k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml @@ -0,0 +1,24 @@ +apiVersion: v1 +kind: Secret +type: Opaque +metadata: + name: pi-code-agent-api-keys + namespace: poimen-sandbox +stringData: + ANTHROPIC_API_KEY: ENC[AES256_GCM,data:TBcno/dzj3qykHjeX96UKZekuA/WMQT2E7a1ziPyYo5J6lWLPydgli84KeJZhH62/8CnnFtFOhVIe4FWCGPGjvqKH6M8WbiQHaP9y7FN0qQUpFHKNVAQ2R+s0wInhw0eUmANvd7bRx8aIZ2X,iv:oex19yVtWUevoJ0A3mkMLCfWu85o1gPSNp43x1iV0LE=,tag:BW3zQnwvagPceeNird9CZg==,type:str] + DEEPSEEK_API_KEY: ENC[AES256_GCM,data:vq/5gOsMVjlVMd0oXiBtHFHFEr+IF2YR2iP2OBFWx3ZFNoQ=,iv:QhLU/BRIJb/7QKDCRyw+HWPphZifVEhIOLywYDgOb20=,tag:BZdtXw+WsuK+P4E0caTa0Q==,type:str] +sops: + age: + - enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqNm1LNE5nWktnd3c3STg3 + UlVWbWJxWElxUHMwa0JVMk81Z0c4d0JaQm1vCnpLQWJWV3hxOU00NzJIaG51Mmlr + ZWpNTFJWcmdtV0w4YndQeHpvSUx2MTAKLS0tIDVnNkNuVWhPUFZ5V2hFOEt1eitG + blVqRy8yNnYzSU1FNzZGNkxrRElvWVEKL3J77INje75+CvKXaJHC3CvONgHv7d0a + a10MITvfreFr6M4AN3Z/mno49SDMp15Q4oQRagHHxSTS1K54mWHo8g== + -----END AGE ENCRYPTED FILE----- + recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla + encrypted_regex: ^(data|stringData)$ + lastmodified: "2026-10-06T01:19:49Z" + mac: ENC[AES256_GCM,data:GpYl8IRefms36GvO3RW41c0US7SOgoOhd4rumxsCdEf5ME9kJ0eOxPx8klMgnlYrkBnwHmad8jj7C+HcA5Eim7mdUaYOZZWPFohvC6N4GLHX85aH78P09CelqG5tdm/Gm+1QfxYu0vANYGCAVjLK7YbBbZjbZmbvVfQvAbs3mOY=,iv:y3lglGu8FhTHQpO1poSIKgGYJFu5J9yjIaOjulda6fY=,tag:5Sgh4TbtNFMw7xOJjYRn0g==,type:str] + version: 3.13.2 diff --git a/k8s/argocd/secrets/secret-generator.yaml b/k8s/argocd/secrets/secret-generator.yaml index 6ade342..f034421 100644 --- a/k8s/argocd/secrets/secret-generator.yaml +++ b/k8s/argocd/secrets/secret-generator.yaml @@ -33,3 +33,4 @@ files: - gotify-smtp-secrets.enc.yaml - forgejo-smtp-secrets.enc.yaml - paperless-ai-secrets.enc.yaml + - pi-sandbox-secrets.enc.yaml diff --git a/k8s/infra/poimen/kustomization.yaml b/k8s/infra/poimen/kustomization.yaml index 32fdd35..ae6ad04 100644 --- a/k8s/infra/poimen/kustomization.yaml +++ b/k8s/infra/poimen/kustomization.yaml @@ -9,3 +9,4 @@ resources: - poimen-sandbox-namespace.yaml - poimen-sandbox-quota.yaml - pi-session-netpol.yaml + - pi-agent-workspace-pv.yaml diff --git a/k8s/infra/poimen/pi-agent-workspace-pv.yaml b/k8s/infra/poimen/pi-agent-workspace-pv.yaml new file mode 100644 index 0000000..1ba6bf0 --- /dev/null +++ b/k8s/infra/poimen/pi-agent-workspace-pv.yaml @@ -0,0 +1,57 @@ +apiVersion: v1 +kind: PersistentVolume +metadata: + finalizers: + - kubernetes.io/pv-protection + labels: + app: pi-coding-agent-sandbox + part-of: poimen-toolkit + name: pi-agent-workspace-pv + annotations: + argocd.argoproj.io/sync-wave: '-1' +spec: + accessModes: + - ReadWriteOnce + capacity: + storage: 10Gi + claimRef: + apiVersion: v1 + kind: PersistentVolumeClaim + name: pi-agent-workspace + namespace: poimen-sandbox + hostPath: + path: /var/lib/pi-agent-sandbox + type: DirectoryOrCreate + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: + - talos-cp-2 + persistentVolumeReclaimPolicy: Retain + volumeMode: Filesystem +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + annotations: + pv.kubernetes.io/bind-completed: 'yes' + argocd.argoproj.io/sync-wave: '-1' + finalizers: + - kubernetes.io/pvc-protection + labels: + app: pi-coding-agent-sandbox + part-of: poimen-toolkit + name: pi-agent-workspace + namespace: poimen-sandbox +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: '' + volumeMode: Filesystem + volumeName: pi-agent-workspace-pv diff --git a/terraform/files/coredns/Corefile b/terraform/files/coredns/Corefile index b1b0380..dbdcb7c 100644 --- a/terraform/files/coredns/Corefile +++ b/terraform/files/coredns/Corefile @@ -39,6 +39,7 @@ rewrite name comfy.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name comfyui.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name gotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local + rewrite name pi.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name igotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local