Compare commits

..
Author SHA1 Message Date
rock 71b3cef2eb fix(minio): replace the dead minio/mc image in the provision hook and backup job 2026-10-06 10:10:56 +08:00
rock b6e317b34c feat(poimen): pi sandbox workspace volume, encrypted api keys, cluster dns rewrite (#54)
Adds the pi coding-agent sandbox's cluster side:
- `k8s/infra/poimen/pi-agent-workspace-pv.yaml` — static hostPath PV + PVC for the sandbox /workspace (Longhorn cannot attach new volumes in this cluster; this gives real persistence and survived a pod delete/recreate test: marker file + 38 KB session.jsonl intact, the session answered from its own history afterwards).
- `k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml` — the sandbox's API-key Secret, SOPS-encrypted (recipient from .sops.yaml, payload only: `encrypted_regex ^(data|stringData)$`), registered in the ksops generator. Round-trip verified against the live Secret.
- `terraform/files/coredns/Corefile` — `rewrite name pi.riotpiao.com ingress-nginx-controller...` so the hostname resolves in-cluster.

Note: the keys currently in that Secret are the ones leaked earlier and must be rotated; re-encrypt with `sops -e -i` after rotating.Reviewed-on: #54
Co-authored-by: rock <[email protected]>
2026-10-06 01:49:12 +00:00
rock f3015522e3 fix(storage): pin minio tenant to a published image release (#53)
## Why

`minio/minio:RELEASE.2025-07-23T15-54-02Z` no longer exists. The Docker Hub repository
`minio/minio` returns **404 for every tag** (checked: the pinned tag and `latest`), and
MinIO has moved its images behind registry authentication (`quay.io/minio/minio` → 401,
`registry.min.io/minio/minio` → auth required, `dl.min.io` binaries → 410 Gone).

The tenant therefore sat in `ImagePullBackOff` — 44,229 attempts over 6d23h — and the chain
from there is:

```
minio tenant pod not ready (1/2)
  -> headless Service minio-cluster-hl has no endpoints
  -> CoreDNS: lookup minio-cluster-hl.storage.svc.cluster.local -> no such host
  -> Vault S3 storage init fails at startup
  -> vault-0 CrashLoopBackOff, 1943 restarts, ~7 days
```

## Control checks (so this is not a network story)

| check | result |
| --- | --- |
| `alpine:3.20` pulled on talos-cp-1 | Succeeded — docker.io works |
| `quay.io/prometheus/node-exporter:v1.8.2` on the same node | Succeeded — public quay works |
| `minio/minio` (docker.io) | HTTP 404, repo removed |
| old tag `RELEASE.2024-06-13T20-48-48Z` with `IfNotPresent` | not cached on the node |

## Change

Pin `spec.image` to `coollabsio/minio:RELEASE.2025-10-15T17-29-55Z` — a public mirror, and a
newer upstream release than the previous pin. Verified by pulling it on talos-cp-1 and
running it: the image carries `/usr/bin/docker-entrypoint.sh` (the layout the operator
expects, which runs `minio server … --console-address :9090`) and reports
`minio version RELEASE.2025-10-15T17-29-55Z`.

## Caveat

This is a community mirror, not MinIO's official image, because the official ones now
require credentials. If MinIO registry credentials are available, add an imagePullSecret
and switch back to `quay.io/minio/minio`. Supply-chain decision, flagged deliberately.

## After merge (ArgoCD, not kubectl)

The storage app tracks `k8s/infra/minio`, so the sync rolls the tenant; then:

```
kubectl get pods -n storage          # minio-cluster-az-a-0 -> 2/2
kubectl get pods -n iam | grep vault  # vault-0 -> Running (storage init succeeds)
```

Residual risk, unrelated to this fix: the tenant's 4 x 25Gi volumes are Longhorn-backed, and
Longhorn currently fails to create engines for new volume attaches cluster-wide. If the
restarted pod cannot attach, that is a second, separate fault.Reviewed-on: #53

Co-authored-by: rock <[email protected]>
2026-10-06 01:48:57 +00:00
7 changed files with 88 additions and 2 deletions
+2 -1
View File
@@ -48,7 +48,8 @@ spec:
mountPath: /backup
containers:
- name: mc-mirror
image: quay.io/minio/mc:latest
# bitnamilegacy/minio-client:2025.7.21-debian-12-r3 replaces the dead quay.io/minio/mc image
image: bitnamilegacy/minio-client:2025.7.21-debian-12-r3
env:
- name: ACCESS_KEY
valueFrom:
@@ -0,0 +1,24 @@
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: pi-code-agent-api-keys
namespace: poimen-sandbox
stringData:
ANTHROPIC_API_KEY: ENC[AES256_GCM,data:TBcno/dzj3qykHjeX96UKZekuA/WMQT2E7a1ziPyYo5J6lWLPydgli84KeJZhH62/8CnnFtFOhVIe4FWCGPGjvqKH6M8WbiQHaP9y7FN0qQUpFHKNVAQ2R+s0wInhw0eUmANvd7bRx8aIZ2X,iv:oex19yVtWUevoJ0A3mkMLCfWu85o1gPSNp43x1iV0LE=,tag:BW3zQnwvagPceeNird9CZg==,type:str]
DEEPSEEK_API_KEY: ENC[AES256_GCM,data:vq/5gOsMVjlVMd0oXiBtHFHFEr+IF2YR2iP2OBFWx3ZFNoQ=,iv:QhLU/BRIJb/7QKDCRyw+HWPphZifVEhIOLywYDgOb20=,tag:BZdtXw+WsuK+P4E0caTa0Q==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqNm1LNE5nWktnd3c3STg3
UlVWbWJxWElxUHMwa0JVMk81Z0c4d0JaQm1vCnpLQWJWV3hxOU00NzJIaG51Mmlr
ZWpNTFJWcmdtV0w4YndQeHpvSUx2MTAKLS0tIDVnNkNuVWhPUFZ5V2hFOEt1eitG
blVqRy8yNnYzSU1FNzZGNkxrRElvWVEKL3J77INje75+CvKXaJHC3CvONgHv7d0a
a10MITvfreFr6M4AN3Z/mno49SDMp15Q4oQRagHHxSTS1K54mWHo8g==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
encrypted_regex: ^(data|stringData)$
lastmodified: "2026-10-06T01:19:49Z"
mac: ENC[AES256_GCM,data:GpYl8IRefms36GvO3RW41c0US7SOgoOhd4rumxsCdEf5ME9kJ0eOxPx8klMgnlYrkBnwHmad8jj7C+HcA5Eim7mdUaYOZZWPFohvC6N4GLHX85aH78P09CelqG5tdm/Gm+1QfxYu0vANYGCAVjLK7YbBbZjbZmbvVfQvAbs3mOY=,iv:y3lglGu8FhTHQpO1poSIKgGYJFu5J9yjIaOjulda6fY=,tag:5Sgh4TbtNFMw7xOJjYRn0g==,type:str]
version: 3.13.2
+1
View File
@@ -33,3 +33,4 @@ files:
- gotify-smtp-secrets.enc.yaml
- forgejo-smtp-secrets.enc.yaml
- paperless-ai-secrets.enc.yaml
- pi-sandbox-secrets.enc.yaml
@@ -59,7 +59,8 @@ spec:
mountPath: /shared
containers:
- name: provision
image: quay.io/minio/mc:latest
# bitnamilegacy/minio-client:2025.7.21-debian-12-r3 replaces the dead quay.io/minio/mc image
image: bitnamilegacy/minio-client:2025.7.21-debian-12-r3
volumeMounts:
- name: shared
mountPath: /shared
+1
View File
@@ -9,3 +9,4 @@ resources:
- poimen-sandbox-namespace.yaml
- poimen-sandbox-quota.yaml
- pi-session-netpol.yaml
- pi-agent-workspace-pv.yaml
@@ -0,0 +1,57 @@
apiVersion: v1
kind: PersistentVolume
metadata:
finalizers:
- kubernetes.io/pv-protection
labels:
app: pi-coding-agent-sandbox
part-of: poimen-toolkit
name: pi-agent-workspace-pv
annotations:
argocd.argoproj.io/sync-wave: '-1'
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: pi-agent-workspace
namespace: poimen-sandbox
hostPath:
path: /var/lib/pi-agent-sandbox
type: DirectoryOrCreate
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- talos-cp-2
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: 'yes'
argocd.argoproj.io/sync-wave: '-1'
finalizers:
- kubernetes.io/pvc-protection
labels:
app: pi-coding-agent-sandbox
part-of: poimen-toolkit
name: pi-agent-workspace
namespace: poimen-sandbox
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: ''
volumeMode: Filesystem
volumeName: pi-agent-workspace-pv
+1
View File
@@ -39,6 +39,7 @@
rewrite name comfy.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name comfyui.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name gotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name pi.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name igotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local