# MinIO CRUD Example (Go)

A minimal Go program that exercises the cluster's object storage through the
**universal storage frontend** — `minio.storage.svc.cluster.local:9000` — the
single DNS name that load-balances across both node-pinned MinIO instances
(`minio-az-a` on talos-cp-1, `minio-az-b` on talos-worker-1).

It runs a full CRUD cycle with a random text file:

| Step | S3 call | What it proves |
|------|---------|----------------|
| Ensure bucket | `BucketExists` / `MakeBucket` | bucket `crud-test` exists (idempotent) |
| **C**reate | `PutObject` | write path through the frontend |
| **R**ead | `GetObject` + byte compare | content round-trips intact |
| **U**pdate | `PutObject` (overwrite) | S3 update semantics (objects are replaced, not edited) |
| List | `ListObjects` prefix `demo/` | enumeration |
| **D**elete | `RemoveObject` + `StatObject` | object gone (`NoSuchKey` confirmed) |

Every operation emits `[SERVICE_METRIC] s3.<op>.latency_ms=<n> ms`; any failure
emits `[APP_METRIC] ERROR s3.<op> failed ... | trace=...` and exits non-zero.

## Run it

```bash
# 1. expose the frontend locally (leave running in another terminal)
kubectl port-forward svc/minio -n storage 9000:9000

# 2. credentials — same root creds used by both MinIO sites
source logging/.env        # exports MINIO_ROOT_USER / MINIO_ROOT_PASSWORD

# 3. run
cd storage/test
go mod tidy && go run .
```

In-cluster (e.g. from a Job), skip the port-forward and set
`MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000`.

Expected output:

```
[SERVICE_METRIC] s3.ensure_bucket.latency_ms=145 ms
[SERVICE_METRIC] s3.put.latency_ms=19 ms
created crud-test/demo/<unix-ts>.txt (256 bytes of random text)
[SERVICE_METRIC] s3.get.latency_ms=9 ms
read back and verified content
[SERVICE_METRIC] s3.update.latency_ms=70 ms
updated (overwrote) object
  demo/<unix-ts>.txt  140 bytes  <timestamp>
[SERVICE_METRIC] s3.list.latency_ms=11 ms
[SERVICE_METRIC] s3.delete.latency_ms=68 ms
deleted and verified gone — CRUD cycle complete
```

## Validating each state with kubectl

The program verifies itself in-process (read-back compare, post-delete stat),
but every state is also independently observable from outside with `kubectl`.
The helper below drops you into a throwaway `mc` shell wired to both sites —
all subsequent checks use it:

```bash
source logging/.env
kubectl run mc-shell --rm -it --restart=Never --image=minio/mc -n storage \
  --env="U=$MINIO_ROOT_USER" --env="P=$MINIO_ROOT_PASSWORD" \
  --command -- /bin/sh -c '
mc alias set front http://minio.storage.svc.cluster.local:9000 "$U" "$P"
mc alias set az-a  http://minio-az-a.storage.svc.cluster.local:9000 "$U" "$P"
mc alias set az-b  http://minio-az-b.storage.svc.cluster.local:9000 "$U" "$P"
exec /bin/sh'
```

> The demo deletes its object at the end, so to inspect the CREATE/UPDATE
> states at your own pace, comment out the `// DELETE` block in `main.go`
> and re-run (the delete is idempotent to re-apply later).

**0. Frontend is healthy (before running anything)**

```bash
kubectl get endpoints minio -n storage          # expect TWO pod IPs on :9000
kubectl get pods -n storage -o wide             # az-a on talos-cp-1, az-b on talos-worker-1
```

**1. Bucket created** — and replicated to BOTH sites

```bash
# inside mc-shell — the bucket must appear on each site individually
mc ls az-a | grep crud-test
mc ls az-b | grep crud-test     # proves site replication propagated it
```

**2. Object created (CREATE)** — 256 bytes, present on both nodes

```bash
mc ls az-a/crud-test/demo/      # <ts>.txt, 256 B
mc ls az-b/crud-test/demo/      # same object, replicated (allow ~seconds of lag)
mc cat front/crud-test/demo/<ts>.txt   # the random text itself
```

**3. Object updated (UPDATE)** — size changed 256 → 140 bytes, content starts with `UPDATED ---`

```bash
mc stat az-a/crud-test/demo/<ts>.txt   # Size: 140 B, fresh LastModified
mc cat  az-b/crud-test/demo/<ts>.txt | head -1    # "UPDATED ---" (replicated overwrite)
```

**4. Object deleted (DELETE)** — gone from both sites

```bash
mc ls az-a/crud-test/demo/      # empty
mc ls az-b/crud-test/demo/      # empty — deletes replicate too
mc stat front/crud-test/demo/<ts>.txt   # error: Object does not exist
```

**5. Replication layer itself**

```bash
# inside mc-shell
mc admin replicate status az-a   # buckets/policies/users "in sync"
```

**6. Storage layer under it**

```bash
kubectl get volumes.longhorn.io -n longhorn-system   # both volumes attached / healthy
```

## Troubleshooting

- `connection refused` on localhost:9000 → the port-forward isn't running.
- `[APP_METRIC] ERROR config missing` → `source logging/.env` first.
- Object visible on az-a but not az-b → check `mc admin replicate status az-a`;
  replication is near-synchronous, not instant. Persistent divergence:
  `mc admin replicate resync start az-a az-b`.
- Frontend has one endpoint instead of two → a MinIO pod is unready;
  `kubectl describe pod -n storage <pod>`. Traffic still flows via the
  surviving pod (that's the failover design — see `minio_migration.html`).