homelab/k8s/storage/test/README.md

MinIO CRUD Example (Go)

A minimal Go program that exercises the cluster's object storage through the universal storage frontend — minio.storage.svc.cluster.local:9000 — the single DNS name that load-balances across both node-pinned MinIO instances (minio-az-a on talos-cp-1, minio-az-b on talos-worker-1).

It runs a full CRUD cycle with a random text file:

Step	S3 call	What it proves
Ensure bucket	BucketExists / MakeBucket	bucket crud-test exists (idempotent)
Create	PutObject	write path through the frontend
Read	GetObject + byte compare	content round-trips intact
Update	PutObject (overwrite)	S3 update semantics (objects are replaced, not edited)
List	ListObjects prefix demo/	enumeration
Delete	RemoveObject + StatObject	object gone (NoSuchKey confirmed)

Every operation emits [SERVICE_METRIC] s3.<op>.latency_ms=<n> ms; any failure emits [APP_METRIC] ERROR s3.<op> failed ... | trace=... and exits non-zero.

Run it

# 1. expose the frontend locally (leave running in another terminal)
kubectl port-forward svc/minio -n storage 9000:9000

# 2. credentials — same root creds used by both MinIO sites
source logging/.env        # exports MINIO_ROOT_USER / MINIO_ROOT_PASSWORD

# 3. run
cd storage/test
go mod tidy && go run .

In-cluster (e.g. from a Job), skip the port-forward and set MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000.

Expected output:

[SERVICE_METRIC] s3.ensure_bucket.latency_ms=145 ms
[SERVICE_METRIC] s3.put.latency_ms=19 ms
created crud-test/demo/<unix-ts>.txt (256 bytes of random text)
[SERVICE_METRIC] s3.get.latency_ms=9 ms
read back and verified content
[SERVICE_METRIC] s3.update.latency_ms=70 ms
updated (overwrote) object
  demo/<unix-ts>.txt  140 bytes  <timestamp>
[SERVICE_METRIC] s3.list.latency_ms=11 ms
[SERVICE_METRIC] s3.delete.latency_ms=68 ms
deleted and verified gone — CRUD cycle complete

Validating each state with kubectl

The program verifies itself in-process (read-back compare, post-delete stat), but every state is also independently observable from outside with kubectl. The helper below drops you into a throwaway mc shell wired to both sites — all subsequent checks use it:

source logging/.env
kubectl run mc-shell --rm -it --restart=Never --image=minio/mc -n storage \
  --env="U=$MINIO_ROOT_USER" --env="P=$MINIO_ROOT_PASSWORD" \
  --command -- /bin/sh -c '
mc alias set front http://minio.storage.svc.cluster.local:9000 "$U" "$P"
mc alias set az-a  http://minio-az-a.storage.svc.cluster.local:9000 "$U" "$P"
mc alias set az-b  http://minio-az-b.storage.svc.cluster.local:9000 "$U" "$P"
exec /bin/sh'

The demo deletes its object at the end, so to inspect the CREATE/UPDATE states at your own pace, comment out the // DELETE block in main.go and re-run (the delete is idempotent to re-apply later).

0. Frontend is healthy (before running anything)

kubectl get endpoints minio -n storage          # expect TWO pod IPs on :9000
kubectl get pods -n storage -o wide             # az-a on talos-cp-1, az-b on talos-worker-1

1. Bucket created — and replicated to BOTH sites

# inside mc-shell — the bucket must appear on each site individually
mc ls az-a | grep crud-test
mc ls az-b | grep crud-test     # proves site replication propagated it

2. Object created (CREATE) — 256 bytes, present on both nodes

mc ls az-a/crud-test/demo/      # <ts>.txt, 256 B
mc ls az-b/crud-test/demo/      # same object, replicated (allow ~seconds of lag)
mc cat front/crud-test/demo/<ts>.txt   # the random text itself

3. Object updated (UPDATE) — size changed 256 → 140 bytes, content starts with UPDATED ---

mc stat az-a/crud-test/demo/<ts>.txt   # Size: 140 B, fresh LastModified
mc cat  az-b/crud-test/demo/<ts>.txt | head -1    # "UPDATED ---" (replicated overwrite)

4. Object deleted (DELETE) — gone from both sites

mc ls az-a/crud-test/demo/      # empty
mc ls az-b/crud-test/demo/      # empty — deletes replicate too
mc stat front/crud-test/demo/<ts>.txt   # error: Object does not exist

5. Replication layer itself

# inside mc-shell
mc admin replicate status az-a   # buckets/policies/users "in sync"

6. Storage layer under it

kubectl get volumes.longhorn.io -n longhorn-system   # both volumes attached / healthy

Troubleshooting

• connection refused on localhost:9000 → the port-forward isn't running.
• [APP_METRIC] ERROR config missing → source logging/.env first.
• Object visible on az-a but not az-b → check mc admin replicate status az-a; replication is near-synchronous, not instant. Persistent divergence: mc admin replicate resync start az-a az-b.
• Frontend has one endpoint instead of two → a MinIO pod is unready; kubectl describe pod -n storage <pod>. Traffic still flows via the surviving pod (that's the failover design — see minio_migration.html).